Deep Belief Network for the Enhancement of Ultrasound Images with Pelvic Lesions

1.1 Our Contributions

The contributions of this paper are as follows:

1. Contribution 1: An effort is made to enhance ultrasound images with lesions of the pelvic region, as such work is not prevalent, to the best of our knowledge.

2. Contribution 2: Inspired from neural network (NN), we employ DBN for enhancing the lesion images of the ultrasound modality.

3. Contribution 3: A new paradigm for constructing the training library is proposed using the higher-order statistics of texture features and the decomposing mode of filter.

1.2 Article Outline

The paper is organized to the following sections: Section 2 reviews the literature, and Section 3 illustrates the texture analysis on the images to support image enhancement. Section 4 details the procedure for training library construction and the information about DBN. Section 5 discusses the results, and Section 6 concludes the paper.

2.1 Related Works

In 2010, Alush et al. [2] introduced novel steps for extracting and segmenting the class-specific object automatically by using class-specific boundaries, in addition to focusing on the determination of lesions in the uterine cervix images. They exploited the Markov random field to model the watershed segmentation map of the input image, and evaluated whether the region is a lesion or not. Also, they used the local pair-wise factors that depend on supervised learning for visual word distribution to indicate whether the arc is a part of the lesion or not.

In 2011, Park et al. [28] developed a framework using domain-specific automated image analysis. The proposed framework was used for detecting the cancerous as well as the precancerous lesions in the uterine cervix. Thus, for automatic diagnosis, they employed domain-specific diagnostic features using conditional random fields and a novel window-based performance assessment strategy. The new window-based performance scheme solved the problem of image misalignment.

In 2013, Lee and Won [18] developed a tactile sensation imaging system for capturing the lesions of breast cancer. They applied the principle of total internal reflection for lesion capturing, and introduced a novel method for evaluating the lesion size, depth, and elasticity. They applied two algorithms, namely neural-network-based inversion algorithm and three-dimensional finite-element-model-based forward algorithm, for evaluating the lesions and validated the results with realistic tissue phantom. Harmouche et al. [13] developed a fully automatic probabilistic method for classifying the lesions of multiple sclerosis. Based on the multimodal magnetic resonance imaging intensities, they built the regional likelihood distributions for each tissue class in the segmented brain. They applied Markov random fields for ensuring smoothness of the local class by using the neighboring voxel information. They used various metrics, such as dice overlap, positive predictive rates, and sensitivity for evaluating and comparing the voxel and the lesion classification.

In 2015, Zhan et al. [39] exploited T1 and fluid-attenuated inversion recovery image modalities for the automatic segmentation of white matter lesions. The T1 image was segmented into cerebrospinal fluid, gray matter, and white matter using the brain tissue segmentation method. Also, they defined the threshold for the preliminary identification of abnormalities in the tissues. The boundaries of white matter lesions were extracted using the level set method, which is based on the local Gaussian distribution energy.

2.2 Problem Definition

A literature review on the diagnostic methodologies of lesion presence in multimodal images reveals two primary research gaps (Table 1). First, adequate contributions have been made toward diagnostic methods, but not on improving the diagnostic precision. Improved diagnostic precision can be achieved only if the image is informative. However, the imaging techniques often produce unwanted signals and, as a result, the images easily become corrupted by noise. The corrupted image loses many of its useful information and, thus, the diagnosis will not be precise enough to meet the practical constraints. Second, lesion diagnosis has been done in many human parts. Lesions in the pelvic region have not been considered sufficiently in the literature. Despite the fact that the existing diagnostic methodologies are significant for medical applications, they suffer from serious drawbacks. In Ref. [39], the level set method was adopted for the segmentation of white matter lesions from the brain tissue. The enhancement of tissue visualization was indirectly performed using the Gaussian distribution function, as the majority of the noise follows the characteristics of the Gaussian function. Though the performance of the level set method is not affected by local intensities, it suffers due to complex geometrical structures. Hence, handling of pelvic lesions under a low-quality scenario remains challenging with the level set method-based segmentation.

Markov random field was deployed in Ref. [13] for the classification of multiple sclerosis lesions. The Markov random field was primarily employed for classification purposes, yet it removed the noise from the subjected image because of its probabilistic characteristics. Its probabilistic nature established relativity among the spatial data of the image and, hence, noisy data were skipped off from the image. As a result, the classification was done in a noise-mitigated environment. However, the Markov random field suffers due to its nature of sticking with the local optimal points to establish the relativity. Hence, noise removal cannot be achieved to a substantial level, degrading the quality of classification. Conditional random field, which is a member of the family of Markov random field, was exploited in Ref. [28] to determine the lesions in the uterine cervix. As the imaging systems often produce misalignment in the resultant image, a window-based scheme was used along with the conditional random field. The window-based processing improved the image for the betterment of diagnosis accuracy. The noise removal task was tackled through the conditional random field, yet it suffered due to the iterative scaling problem. As a result, sufficient image enhancement was not accomplished. In Ref. [2], watershed segmentation and Markov random field were exploited to learn the characteristics of the lesion portion of the image and to segment the lesions from the uterine cervix image. The Markov random field was employed to mitigate the noise, and later, watershed segmentation was applied to carry out the segmentation process. While the Markov random field suffers from the inability to distinguish the local optimal points and the global optima, the watershed segmentation often results in oversegmentation. The oversegmentation that resulted from the image provided a wide heterogeneous environment.

Under a circumstance of heterogeneous noisy characteristics, the image undergoes multimodal format, where both the Markov random field and watershed segmentation fail to perform image enhancement and segmentation, respectively. In Ref. [18], NN was deployed to understand the degree of lesions in breast cancer. The NN supported image enhancement, unless the data interpolation deviated from the actual data. Moreover, the finite element modeling technique interpolation was based on the statistical nature of the image. The lack of practical measurements possibly degraded the performance of the NN.

3.1 Texture Analysis

Assume a US image that contains a number of pixels as N_V and N_H in the vertical and horizontal directions, respectively. In every pixel, the gray level is quantized to a level of N_g. Let I_V={1, 2, …, N_y}, I_H={1, 2, …, N_H}, and G={1, 2, …, N_g} represent the vertical spatial domain, horizontal spatial domain, and set of N_g quantized gray levels, respectively. The resolution clique of an image that is ordered by its row-column designations is represented as a set of I_V×I_H. Let I be an image that is assigned as a function with G as the gray level in all the resolution cells or as a coordinate pair in I_V×I_H; I_V×I_H→G.

The texture features are taken from the four closely related measures, named as angular nearest-neighbor gray tone spatial dependence matrices, and these matrices represent the adjacent neighbor resolution cells. A resolution cell without the peripheral image and with eight nearest-neighbor resolution cells is also considered. In an image I, the information of the texture content is occupied in the average or the complete spatial relationship. Assume the information of the texture content to be indicated with a matrix of relative frequencies f_ij and that contains two neighbor resolution cells with distance d on the image, which has the gray tone i and j. This type of matrix with gray tone spatial dependence frequencies indicates two functions, namely the distance between the cells and the angular relationship between the nearby resolution cells. In a horizontal neighboring resolution cell set with distance 1 and gray tone image, the distance 1 horizontal gray tone spatial dependence matrix is estimated. In a 45° quantized angle interval, the unnormalized frequencies are given by

where Eqs. (1)–(4) are performed for all the elements that are available in the set. The matrices are symmetric in nature and, so, f(i, j; d, a)=f(j, i; d, a). Further, ρ is suggested as the distance metric and, so, ρ((k, l), (m, n))=max{|k−m|, |l−n|}.

Assume a 4×4 image with number of gray tones as 4, which ranges from 0 to 3. In a gray tone spatial dependence matrix, the (2, 1) position’s element with distance 1 horizontal f_H matrix refers to the number of times both the gray tones with values 1 and 2 appear nearby to each other horizontally. This number is estimated using the number of pairs in the resolution cells in R_H, so that the first as well as the second resolution cell consist of gray tone 2 and 1, respectively. All the four distance 1 gray tone spatial dependence matrices and the frequency normalizations are estimated. In each row, there are about 2(N_H−1) neighboring resolution cell pairs, N_V rows, and a total of 2N_V(N_H−1) adjacent horizontal neighbor pairs. This shows the nearest horizontal neighbor relationship (d=1, a=0°).

A total of 2(N_V−1)(N_H−1) adjacent right diagonal neighbor pairs are noted, when the neighboring resolution cell pair for each row, except the first row, is 2(N_H−1)45° and there are N_V rows. The existing relationship is the nearest right diagonal neighbor (d=1, a=45°). With respect to symmetry, there are 2N_H(N_V−1) and 2(N_H−1)(N_V−1) nearest vertical and nearest left diagonal neighbor pairs, respectively. The gray tone spatial dependence matrix is normalized by dividing the elements in the matrix with R, if the neighbor resolution pair R is applied on the matrix. During image processing, the number of resolution cells n is directly proportional to the number of operations in the proposed method. Specifically, if Hadamard or Fourier transform is used to extract the texture information, then the number of operations will be in the order of n log n. It is necessary to provide two lines of image data at a time to calculate the gray tone spatial dependence matrix entries.

In this paper, entropy that is mathematically depicted below and the autocorrelation of the gray level co-occurrence matrix are considered to understand the texture characteristics of the image. To demonstrate the significance of selecting the two features, we select five US images with pelvic lesions, as given in Figure 1. For the selected images, entropy and autocorrelation are determined and illustrated through the quantile-quantile plots of Figure 2A and C. As the quantile-quantile plots show the mutual distribution of the images, it is capable of distinguishing the texture means of the images. This is well visualized through Figure 2B and C, where each image has its own plot, rather than overlapping with the other plots. This shows that the selected entropy and autocorrelation represent the US images effectively.

Figure 1:

Sample US Images of the Pelvic Region with Pelvic Lesions.

Figure 2:

Texture Analysis.

(A) Skewness; (B) entropy; (C) autocorrelation for five images.

1. Entropy: In texture analysis, entropy refers to the spatial disorder measure [12, 32]:

In random distribution, entropy tends to be so high due to the chaos. However, the entropy value sets to zero for a solid tone image. This feature of entropy is helpful in predicting the type of textures that are statistically more chaotic in nature.

3.2 Alignment Analysis

In order to ensure the alignment of the US image, an alignment feature of the image, termed as skewness, is extracted. Skewness is defined in relation to the third as well as the second moments around the mean, and it is represented as

The traditional Fisher-Pearson coefficient of skewness is given as

g₁ may occur in negative values and, so, the Pearson statistic is known to be β1. For the case of g₁, when used as a test for deviation from normality, a table is introduced by Pearson and Hartley in the year 1970 [29]. This Pearson formula exists as a modification of the sample size, and an adjusted Fisher-Pearson standardized moment coefficient has been introduced, which is given as

The selection of skewness to represent the alignment of the US, with respect to its texture features, is substantiated using Figure 2A that is plotted for the five images of Figure 1. Figure 2A depicts the power spectral density of the skewness parameter, with respect to its frequency of occurrence. The skewness of those images show high variations with increase in the normalized frequency. Each image exhibits a good skewness variation and ensures that the alignment of the image is recognized.

Hence, the extracted features are subjected to train the DBN for allowing the decomposing band of filter to be determined.

4.1 Weightage-Based Quality Assessment

While the training attributes of the US images are extracted through texture analysis, the target of the image is set through weightage-based quality assessment. The weightage-based quality assessment includes both the reference-based quality assessment and the non-reference-based quality assessment. The reference-based quality assessment considers the peak signal-to-noise ratio (PSNR) measure, and the non-reference-based quality assessment considers the edge-based quality metric, termed as the ESSIM [6]. The PSNR measure considers the mean squared error (MSE) between the enhanced image and the original image (before noise contamination), as per Eq. (8).

where MAX_I represents the maximum image pixel. The MAX_I value is 255, if the pixels are indicated with 8 bits per sample. However, if the pixels are indicated with linear pulse-code modulation (PCM) and B bits per sample, then MAX_I=2^B−1.

1. ESSIM: The ESSIM [7] uses the information of the edge to compare the information between the original image block and the distorted image block at the same time, and to replace the structure comparison s(x, y) with the edge-based structure comparison e(x, y). The edge information can be obtained using local gradients, Sobel operator, or simple edge detection algorithm. Hence, the ESSIM calculation is performed in three steps sequentially, namely edge map calculation, determination of the edge direction vector, and edge comparison.

Let the pixel be p_i,j and the edge vector for each pixel be given as D→i,j={dxi,j, dyi,j}, where dy_i,j and dx_i,j represent the horizontal edge mask and the vertical edge mask, respectively. In terms of direction and amplitude, the edge vector representation is made and it is given as

The angle, corresponding to the pixel edge direction, is given as

where dyi,jdxi,j indicates the pixel’s edge direction. An image consists of pixels with edge vector that includes edge direction as well as amplitude, which together form the edge map of the image.

The comparison of edge information among the original and the distorted image blocks is done using the edge histogram. The steps involved in obtaining the edge direction histogram are (i) estimating the edge direction as well as the amplitude of each pixel using Eqs. (13) and (14); (ii) determining the direction of each pixel that is related to one of the eight discrete directions quantitatively; and (iii) adding the amplitude of the edge of a pixel with the direction of the same pixel.

Let D_y and D_x indicate the block edge direction vector of the distorted image and the original image, respectively. Using the correlation coefficient of D_y and D_x, the edge comparison e(x, y) can be estimated and it is given as

where C₃ and σ′xy represent a small constant that prevents the denominator from becoming zero and the covariance of vector D_y and D_x, respectively; σ′y and σ′x represent the standard deviation of D_y and D_x, respectively. The ESSIM is written as

The mean of all the subimages of ESSIM is used to calculate the similarity of the full image, and it is written as

1. Cumulative quality assessment: The cumulative quality assessment is the proposed weightage-based quality assessment that includes both the PSNR and ESSIM [i.e. MESSIM of Eq. (17)], as given in Eq. (18):

where Cq is the cumulative quality assessment that is determined using the enhanced image of the training database.

4.2 Construction of the Training Library

Let T_Mn be the training library, which is to be generated for carrying out learning in the DBN [1, 15, 20], where m refers to the volume of the training records and n refers to the number of features (Figure 3).

Figure 3:

Construction of the Training Library.

The construction of the training library is shown in Figure 1.

where w₂, w₁ refer to the weightage; w and F(w) refer to the network weight and the objective function, respectively. Equation (21) is the cumulative quality assessment metric that is equivalent to Eq. (18).

4.3 Deep Belief Learning

1. DBN Model: Consider the input as x and the hidden variables in layer i as gⁱ with a joint distribution of

where the conditional layers P(gⁱ|gⁱ⁺¹) represent the factorized conditional distributions, wherein probability and sampling are not difficult. If the hidden layer gⁱ has n_i as the binary random vector with elements gji, then

where sign(•) is a non-linear function, usually referred to as sign(t)=11+e−t; bji and Wⁱ represent the biases for the j^th unit of the i^th layer and the weight matrix for the i^th layer, respectively. Equation (23) is followed by the generative model of the first layer P(x|g¹), even though g⁰=x.

1. Restricted Boltzmann machine (RBM): The RBM is P(gⁱ⁻¹|g^l), which lies between two layers l and l−1. Consider v and h as the input layer activation and the hidden layer activation for a generic RBM. The joint distribution is written as

where c, b, Z, and W represent the hidden unit bias vector, visible unit bias vector, normalization constant, and weight matrix, respectively. If the argument of the exponential is notated in minus, then it is called the energy function and it is given as

In the above equation, the RBM parameters are indicated with θ=(W, b, c) and the layer-to-layer conditional distributions are denoted as P(v|h) and Q(h|v). Similar to Eq. (15), factorization occurs in the layer-to-layer conditionals that are associated with the RBM. Therefore,

1. Gradient learning in RBM: Consider the Gibbs Markov chain for the pair of variables – hidden and visible – to determine the RBM’s estimator of the log likelihood gradient. Gibbs sampling is performed, and then sampling of h given v and v given h is done.

In the Markov chain, the t^thv sample is indicated as v_t, and it starts at t=0 with v₀ as the input observation. Hence, from joint P(v, h), the sample obtained is (v_k, h_k) for k→∞. With the RBM model, the log likelihood of a value v₀ is given as

In the above equation, the gradient corresponding to θ=(W, b, c) is given as

for k→∞.

The unbiased sample is given as −∂energy(v0, h0)∂θ+Ehk[∂energy(vk, hk)∂θ|vk], where h₀ and (v_k, h_k) represent the sample from Q(h₀, v₀) and Markov chains, respectively.

4.4 DBN-Aided Enhancement

Given a test US image with pelvic lesions, along with weightage vectors and the noise variance, the DBN estimates the decomposing model of the homomorphic wavelet filter through which the enhancement is to be done. The weightage vectors are defined based on the significance to be given for the quality assessment metrics.

Though homomorphic wavelet filter enhances the image, DBN plays the primary role in selecting the decomposing mode for the wavelet filter. As a result, enhancement is obvious over the other decomposing modes.

5.1 Experiments

The experimental investigation is carried out in the MATLAB platform, and the performance of the proposed as well as the conventional methods is studied. The pelvic images of three patients – case 1 with 4 images, case 2 with 10 images, and case 3 with 2 images – are collected from the database (http://www.ultrasoundcases.info/case-list.aspx?cat=26). As the technique is related to supervised learning enhancement, the data are trained by 50% of the images. The trained data are subjected to analysis of visual quality. The results of the developed method are compared with NN-based enhancement methods. The performance assessment is done with two types of metrics, namely reference-based quality metric (PSNR) and non-reference-based quality metric [known as the second-order derivative metric (SDME)] [35]. The results that are related to the metric measures are analyzed and discussed further.

5.2 Quality of Enhancement

The quality of the selected five images has been studied with varied variance – 10%, 20%, 30%, 40%, and 50% – by comparing the noisy image and the enhanced image for the proposed method, and they are shown in Figures 4–8, respectively. To study the quality of enhancement, the noisy images are contaminated by 10%, 20%, 30%, 40%, and 50% of noises.

Figure 4:

US Images Contaminated by 10% Speckle Noise (A–E) and the Enhanced Images (F–J).

Figure 5:

US Images Contaminated by 20% Speckle Noise (A–E) and the Enhanced Images (F–J).

Figure 6:

US Images Contaminated by 30% Speckle Noise (A–E) and the Enhanced Images (F–J).

Figure 7:

US Images Contaminated by 40% Speckle Noise (A–E) and the Enhanced Images (F–J).

Figure 8:

US Images Contaminated by 50% Speckle Noise (A–E) and the Enhanced Images (F–J).

While analyzing the enhanced image, it is found that the enhanced images exhibit improved visualization for the proposed method than the existing method.

5.3 Reference-Based Quality Assessment

The performance of the developed method, with respect to the image data collected, is estimated using the PSNR metric for the reference-based quality assessment and it is tabulated in Tables 2–4, with respect to the three sets of data. From the results obtained, it is noticed that there is no significant difference in the PSNR of image 2 with noise variance of 0.01 and 0.02. Specifically, the proposed method shows better quality enhancement.

5.4 Non-Reference-Based Quality Assessment

The performance of both the proposed and the existing methods, with respect to the image data collected, is estimated with the SDME metric with varied variance for the non-reference-based quality assessment, and it is tabulated in Tables 5–7 , with respect to the three sets of data. The proposed method shows increased performance than the NN-based method in all sets of images. The non-reference-based quality assessment is more practical because there will not be an original image in real-time while investigating the quality of the enhanced image. Under such circumstance, it is essential to quantify the quality of enhancement on the acquired images. Even under non-reference-based quality assessment, the proposed algorithm performs better, which asserts the practical significance of the proposed enhancement algorithm.