Automatic Data Clustering Using Parameter Adaptive Harmony Search Algorithm and Its Application to Image Segmentation

2.1 Basics of Clustering Problem

Let a dataset X={x₁, x₂, x₃, …, x_n} be a set of n data points and X_n×d be the data matrix with n rows and d columns. Each of the data point is described by d features, where x_j=(x_j1, x_j2, …, x_jd) is a vector representing the j^th data point, and x_ji represents the i^th feature of x_j. The goal of partitional clustering algorithm is to determine a partition C={C₁, C₂, …, C_K} such that [36]

The data points that belong to the same cluster are similar to each other as possible, while data points that belong to other clusters are dissimilar as possible. For this, an appropriate fitness function is required to judge the quality of partition. The well-known fitness function is the mean-square error, which is defined as [5]:

where ||x_i–C_l|| denotes the similarity between data point x_i and cluster center C_l. The Euclidean distance is mostly a used similarity metric, which is defined as:

The Euclidean distance as similarity metric has been used in this article. The clustering algorithm is modeled as the optimization problem.

2.2 Parameter Adaptive Harmony Search Algorithm

The concept of harmony search (HS) algorithm was first presented by Geem et al. [16]. It is a metaheuristic algorithm that imitates the music improvisation process where the musician improvises their instruments’ pitch by searching the prefect state of harmony. It has been successfully applied in a wide variety of optimization problems such as timetabling [2, 3], structure design [25], vehicle routing [17], Sudoku puzzle solving [15], tour planning [17], etc. The variants of HS have been reported in the literature [4, 26, 35].

To enrich the searching behavior and to avoid being trapped in a local optimum, Kumar et al. [21] proposed a parameter adaptive harmony search (PAHS) algorithm. In PAHS algorithm, the two control parameters named as harmony memory consideration rate (HMCR) and pitch adjustment rate (PAR), were being allowed to change dynamically. They have explored different cases of linear and exponential changes. The computational procedure of PAHS algorithm can be summarized as follows [21]:

1. Initialization of the optimization problem and algorithm parameters: The optimization problem can be defined as minimize (or maximize) f(x) such that x_i∈[LB_i, UB_i], i=1, 2, …, n, where f(x) is the objective function, x=(x₁, x₂, …, x_n) is the set of decision variables, n is the number of decision variables. LB_i and UB_i are the lower and upper bounds of decision variable x_i, respectively. The parameters of the PAHS are harmony memory size (HMS), range of harmony memory consideration rate (HMCR_min, HMCR_max), range of pitch adjustment rate (PAR_min, PAR_max), range of distance bandwidth (BW_min, BW_max), and number of improvisation (NI).

2. Initialization of harmony memory (HM): The HM consists of HMS harmony vectors. It is filled with randomly generated solution vectors and sorted by the values of the objective function f(x).

3. Improvisation of new harmony: A new harmony vector x′ = (x′1, x′2, …, x′n) is generated using three rules: memory consideration, pitch adjustment, and random selection as follows:

   • For each i∈[1, n] do

     HMCR = HMCRmin + (HMCRmax − HMCRmin)NI × gnPAR = PARmax × e(ln(PARmin/PARmax)NI × gn)BW = BWmax × e(ln(BWmin/BWmax)NI × gn)

   • ifU(0, 1)≤HMCRthen /* memory consideration */

      begin

     x′i = xil where l  ∼  U(1, 2, …, HMS)

      ifU(0, 1)≤PARthen/* pitch adjustment */

      begin

     x′i = x′i ± BW × Rand, Rand ∼ U(0, 1)

      endif

     else/* random selection */

     x′i = LBi + (UBi − LBi) × Rand

     endif

     done

4. Updation in HM: If the newly generated harmony vector x′ = (x′1, x′2, …, x′n), evaluated in terms of objective function value, is better than the worst harmony vector in HM, it replaces the worst harmony vector. This is the step of algorithm where a decision should be taken whether the new harmony vector is to be included in the HM or not.

5. Checking the termination criterion: if the maximum number of improvisation steps is reached, computation is terminated, and the algorithm returns the best harmony vector. Otherwise, Steps 3 and 4 are repeated.

2.3 Related Works

Recently, researchers tried to develop automatic clustering techniques using metaheuristic algorithms. Lee and Antonsson [23] utilized an evolutionary strategy-based method for clustering the dataset dynamically. They applied variable length genomes to search for both cluster centroids and the optimal number of clusters. Bandyopadhyay and Maulik [6] developed a variable string-length genetic algorithm for automatic clustering, which did not require any assumption on the shape of the dataset. Their algorithm was called genetic clustering for unknown K (GCUK). The chromosome encoded the cluster centers of a number of clusters, whose value normally varies. They used real cluster center encoding. Modified versions of crossover and mutation operations were used. The DB cluster validity index was used to compute the fitness of the chromosomes. For each chromosome, the cluster centers were extracted, and then, partition was obtained by assigning the data points to different clusters based on the minimum distance criterion. The cluster centers encoded in the chromosome were then replaced by their respective cluster centers.

Ye and Chen [37] used the hybridization of PSO and K-means algorithm for determining the cluster centroids of geometrical structure datasets automatically. Omran et al. [29] came up with an automatic hard clustering scheme called DCPSO. DCPSO approach automatically determined the optimal number of clusters and cluster centers with minimum user interference. The algorithm started off by partitioning the dataset into a relatively large number of clusters to reduce the effects of the initialization. They used binary PSO to select the optimal number of clusters. Finally, the cluster centers of the chosen clusters were refined through the K-means algorithm. The binary PSO was applied again on the cluster centers to find a new optimal number of clusters in the dataset. This process was repeated until convergence is reached. The algorithm was applied for segmentation of natural, synthetic, and multispectral images.

Abdule-Wahab et al. [1] used a scatter search for automatic clustering. Jarboui et al. [20] used a combinatorial particle swarm optimization (CPSO) to solve partitional clustering problems. Each particle was represented as a candidate solution to the clustering problem. A swarm of particles was initiated and flied through the solution space for targeting the optimal solution. Pan and Cheng [30] proposed an evolution-based tabu search approach (ETSA) that treated the number of clusters as variable and evolved it to an optimal number of clusters. The ETSA consisted of two stages: quantizing the given dataset into a number of user-specified clusters and searching a clustering structure for the best cluster validity.

Das et al. [13] presented a new DE-based strategy, called ACDE. The conventional DE algorithm was modified to improve its convergence properties. A new representation scheme for the search variables was also developed to find the optimal number of clusters. Two different sets of experiments with two different fitness functions, CS and DB measures, were conducted. Das et al. [12] used the modified version of classical PSO. Their algorithm employed a kernel-induced similarity measure instead of the sum-of-squares distance. A new particle representation scheme was used for selecting the optimal number of clusters. The kernel function was used to cluster the data that is linearly nonseparable in the original input space.

Das and Konar [10] proposed an evolutionary-fuzzy clustering algorithm for automatically grouping the pixels of an image. They incorporated the fuzzy concept in ACDE for image segmentation. Das et al. [14] utilized the bacterial evolutionary algorithm for automatic clustering by using a population of variable-length chromosomes to encode an entire grouping of the data. They incorporated a new operation named chromosome repair and modified both mutation and gene transfer operations to handle variable-size chromosomes. Das and Sil [11] used a modified differential evolution for clustering the pixels of a grayscale image. They used the same concept of ACDE with a kernel-induced similarity measure. Quadfel et al. [31] presented a new automatic clustering algorithm based on a modified version of particle swarm optimization. Their algorithm used variable-length particles, which evolve a number of clusters dynamically using mutation operators.

Saha et al. [32, 33] proposed variable string-length-based automatic clustering techniques, which utilized the symmetry based distances. Lee and Chen [24] developed the improved DE algorithm with oscillation strategy for automatic data clustering. Cai and Gong [9] applied the differential evolution based on the point symmetry-based cluster validity index for automatic data clustering. The validity of the corresponding partitioning was measured through point symmetry-based cluster validity index. They used the Kd-tree-based nearest neighbor search to reduce the complexity of finding the closet symmetric point. Masoud et al. [27] proposed a modified version of the combinatorial PSO (CPSO-II) for automatically finding the best number of clusters and simultaneously categorizes the data points. CPSO-II used a renumbering procedure as a preprocessing step, and several extended PSO operators were used to increase the population diversity and remove redundant particles. The renumbering procedure increased the diversity of population, speed of convergence, and quality of solutions. Recently, Kumar et al. [22] proposed a novel automatic clustering technique using the gravitational search algorithm. They used the variance present in the dataset for clustering.

The hybridization of metaheuristic algorithms has also been in use in clustering problems. Niknam and Amiri [28] proposed a cluster optimization algorithm based on the combination of PSO, ant colony optimization, and K-means and so did Supratid and Kim [34] using a combination of GA, ACO, and fuzzy C-means. However, the parameter adaptive harmony search algorithm is yet to be applied to automatic data clustering of real-life datasets as well as image segmentation.

3.1 Mathematical Foundation

The ACPAHS is inspired from well-developed ACDE [13]. The issues as mentioned below made us think of some idea to develop a new algorithm for automatic data clustering. These are:

1. The threshold setting corresponding to each cluster center does not consider the variance of data points belonging to that particular cluster center.

2. It uses the fixed value of threshold cutoff despite the fact that it is dataset dependent.

3. The effect of threshold value has not been considered in the refinement of cluster center.

Here, we investigate these issues and suggest solutions for automatic data clustering approach.

1. Threshold Setting: Das et al. [13] did not consider the variation present in the dataset for threshold setting, which decides the actual number of clusters. Using this fact, the new threshold setting concept for cluster center is proposed. The threshold value for each cluster center is set to the corresponding within-cluster variation. Hence, the proposed selection threshold measures the relevance of each cluster center.

   (6)Thl = (1nl∑i = 1nl(xil − Cl)2), where l = 1, 2, …, Kmax

   Here, Th_l denotes the selection threshold corresponding to cluster l. xil represents the i^th data points belonging to the cluster C_l. n_l denotes the number of data points belong to cluster C_l.

2. Cutoff Threshold: Das et al. [13] fixed the value of cutoff threshold to 0.5. However, it cannot be fixed to a particular value as it decides the number of optimal clusters present in the given dataset. So, it should be depended on the dataset.

   Based on this fact, a novel cutoff threshold is proposed. The cutoff threshold value is to set the mean value of the sum of the threshold in each cluster center. The cutoff threshold (Th_cutoff) is mathematically formulated as:

   (7)Thcutoff = mean(Thtotal)

   where Th_total is the sum of the threshold in each cluster center and is defined as:

   (8)Thtotal = ∑l = 1KmaxThl

3. Weighted Cluster Center Computation: Das et al. [13] did not consider the effect of threshold value during the cluster center refinement. To include this, weighted Euclidean distance is used in the proposed approach that does consider the effect of threshold during computation. It is mathematically formulated as:

   (9)dw(xi, Cj) = (∑m = 1dwj2|xim − Cjm|2)

   where w_j is the threshold assigned to the cluster center C_j. The more the value of w_j is, the better is the j^th cluster center in clustering.

3.2 Proposed Automatic Data Clustering Algorithm

The steps of the automatic data clustering using PAHS (ACPAHS) are represented with a flow chart in Figure 1. The pseudo-code of ACPAHS is given in Figure 2 and the steps of algorithm are given below.

Figure 1:

FlowChart of the ACPAHS Algorithm.

Figure 2:

Pseudo-Code of the ACPAHS Algorithm.

Algorithm: ACPAHS

1. Initialize the algorithm parameters, such as HMS, HMCR, PAR, BW, maximum number of improvisation steps, and maximum number of clusters (K_max).

2. Initialize each harmony vector such that it contains K_max number of randomly selected cluster centers and corresponding values of activation thresholds (see Section 3.2.1).

3. Repeat the following steps until the maximum number of improvisation steps is reached:

   1. Find out the active cluster centers in each harmony vector with the help of the rule mentioned in Section 3.2.3.

   2. For each data point, calculate its weighted Euclidean distance from all the active cluster centers of the i^th harmony vector.

   3. Assign data points to a particular cluster center whose distance is minimum with respect to other cluster centers.

   4. In case if the number of data points pertaining to any cluster is less than two, then reinitialize the cluster centers of the agent by an average computation described in Section 3.2.4.

   5. Update the harmony vectors using the PAHS algorithm outlined in Section 2.2. The fitness of the harmony vectors is used to guide the search process mentioned in Section 3.2.5.

4. The best harmony vector will yield the optimal cluster centers and the optimal number of clusters at the final improvisation step.

These steps of ACPAHS algorithm are now described in the preceding subsections.

3.2.1 Solution Representation

In the ACPAHS, for n data points, each having d dimensions, and for K_max, a user-specified maximum number of clusters, a harmony vector consists of real numbers of dimension K_max+K_max×d. The first K_max entries indicate threshold values corresponding to the cluster centers to decide whether the cluster is to be activated or not. The remaining entries are used for K_max cluster centers, each having d dimensions. As an illustration, let us consider the following example.

Example 1. Let K_max=4 and d=3, i.e. the user-specified maximum number of clusters being considered as four and the space is three dimensional. Then, the harmony vector is shown below:

The first four entries (0.3, 0.7, 0.4, 0.8) represent selection thresholds corresponding to the clusters, and the remaining entries indicate four cluster centers, i.e. (4.9, 3.2, 1.6), (5.7, 4.4, 1.0), (6.9, 3.0, 4.9), and (7.7, 3.1, 2.4).

3.2.3 Active Cluster Center Extraction

The cluster center in the harmony vector is active or selected and is based on the selection of the corresponding threshold value. If the threshold value is greater than the cutoff threshold value, then, the corresponding cluster center in the harmony vector is activated for partitioning the associated dataset. The rule for active cluster center extraction is given below:

• IfTh_i,j>Th_cutoffthen the j^th cluster center is active

• Elsej^th cluster center is inactive

Here, Th_i,j denotes the j^th cluster center in the i^th harmony vector. Th_cutoff is the cutoff threshold, whose computation is mentioned in Section 3.1.

When harmony vectors are updated, there might be some chances that none of the thresholds is greater than the cutoff threshold value. To eliminate this problem, we randomly select two thresholds and reinitialize them to values greater than the cutoff threshold value to ensure that the minimum number of possible clusters is two.

Example 2. The selection thresholds in the harmony vector considered in Example 1 are (0.3, 0.7, 0.4, 0.8). Let the cutoff threshold be 0.6. Then, according to the rule mentioned for active threshold, only two thresholds are higher than 0.6 (i.e. 0.7 and 0.8), and the corresponding second (5.7, 4.4, 1.0) and fourth (7.7, 3.1, 2.4) cluster centers have been activated for partitioning the dataset. The activate cluster centers are shown in rectangle.

4.1 The Datasets Used

In order to validate the performance of ACPAHS, we have carried out different experiments on five real-life datasets such as Iris, Wine, Glass, Breast Cancer, and Vowel. These datasets are obtained from the UCI machine learning repository [8]. A description of the datasets is depicted in Table 1.

4.2 Parameter Setting

The experiments were run for the different values of the parameters used in the ACPAHS, i.e. harmony memory size, maximum number of improvisation steps, range of HMCR, BW, and PAR. Thereafter, these parameters are fixed as follows: The size of the harmony memory is set to 30, respectively. The range of HMCR and PAR is set to [0.7, 0.99] and [0.01, 0.99], respectively. The BW is assigned to [0.001, 0.01]. The maximum number of improvisation steps is 200. ACPAHS is run 40 times for the fair comparison.

4.3 Results and Discussions

The performance of ACPAHS has been compared with ACDE [13], DCPSO [29], GUCK [6], and Classical DE [13]. The results have been computed in terms of mean and standard deviation over 40 independent runs in each case.

Table 2 demonstrates the optimal number of clusters obtained through the above-mentioned clustering techniques. For Iris dataset, ACPAHS produces three clusters in most of the runs. For Wine dataset, ACPAHS, ACDE, and DCPSO produce three clusters. For Glass dataset, ACPAHS and ACDE provide six clusters in almost each run. For Breast Cancer, ACDE and GUCK produce two clusters in every run, although ACPAHS also produces two clusters in most of the runs. For Vowel dataset, ACPAHS generates six clusters in almost each run. DCPSO, GUCK, and classical DE do not able to produce the optimal number of clusters.

In addition to the number of clusters, the proposed approach was also compared in terms of inter-intra cluster ratio and number of fitness function evaluations. The former one measures separation and compactness among the clusters. The latter one measures the computational speed evolved in determining the number of clusters. Figure 3A shows the ratio of inter-cluster to intra-cluster distance among the above-mentioned five real-life datasets for clustering techniques. The results reveal that the ACPAHS produces compact and well-separated clusters compared to other existing clustering techniques. Figure 3B demonstrates the number of fitness function evaluations for the above-mentioned clustering. The number of fitness function evaluations for ACDE, DCPSO, GUCK, and Classical DE are quoted from [13]. The results illustrates that ACPAHS is faster than the other clustering techniques. Besides the fitness function evaluations, the execution time was also computed for all the datasets.

Figure 3:

Performance Comparison of ACPAHS Over Other Well-known Clustering Techniques in Terms of (A) Ratio of Inter- to Intra-cluster Distance; (B) Fitness Function Evaluations.

4.4 Statistical Evaluation

Here, we have done statistical tests to establish the superiority of the proposed approach, ACPAHS. The unpaired t-tests were done to determine whether the proposed approach is statistically significant or not. We have taken 40 as the sample size for the unpaired t-tests. Table 3 shows the results of the unpaired t-tests based on the number of clusters presented in Table 2. As can be seen from Table 3, ACPAHS is statistically significant compared to other techniques for all the datasets except the Glass dataset.

5.1 Image Segmentation as a Clustering Problem

Image segmentation is a process of partitioning an image space into some non-overlapping meaningful homogenous regions. The problem of image segmentation is often posed as clustering in the intensity space. The automatic clustering technique has been applied to grayscale image for segmentation. Here, each pixel corresponds to a pattern, and image region corresponds to a cluster. The image segmentation is mathematically defined as follows:

Let there be a set of all image pixels. By applying segmentation on I, different non-overlapping regions R₁, R₂, …, R_n are formed such that

Every pixel of the image must be exhibited in one and only one segmented region. The proposed automatic clustering technique has been applied to grayscale images for segmentation.

5.2 Experimentation 1: Grayscale Image Segmentation

The developed technique has been applied to five well-known grayscale images; Clouds, Peppers, Science Magazine, Mumbai City, and Robot. The size of each image is 256×256. The intensity of each pixel serves as a feature for clustering. Therefore, the data points are single dimensional, the number of data points is 65,536. The same parameter setting is used as mentioned in Section 4.2. However, K_max is set to 25 as it is commonly used. The results are compared with ACDE, DCPSO, GUCK, and Classical DE. Figure 4 shows the original images and segmented images obtained from ACPAHS. The results, in terms of mean and standard deviations of number of clusters computed over 40 runs, are tabulated in Table 4.

Figure 4:

(A) The Original Cloud Image; (B) Segmented Image Obtained from ACPAHS (K=4); (C) The Original Peppers Image; (D) Segmented Image Obtained from ACPAHS (K=6); (E) The Original Science Magazine Image; (F) Segmented Image Obtained from ACPAHS (K=4); (G) The Original Mumbai City Image; (H) Segmented Image Obtained from ACPAHS (K=6); (I) The Original Robot Image; (J) Segmented Image Obtained from ACPAHS (K=4).

For Clouds image, ACPAHS produces four clusters, and the corresponding segmented image is shown in Figure 4B. DCPSO and GUCK provide five clusters, which are not the optimal number of clusters. ACPAHS produces the number of clusters that fall in the optimal range in almost every run for Peppers, Magazine, Mumbai City, and Robot, and the corresponding segmented images are shown in Figure 4D, F, H, and J, respectively. The results reveal that ACPAHS is able to detect the number of class in the grayscale image efficiently.

Table 5 shows the results of the unpaired t-tests based on the number of clusters of Table 4 between the best two algorithms. The results reveal that the ACGSA is statistically significant compared to the other techniques for image segmentation.

5.3 Experimentation 2: Color Image Segmentation

The ACPAHS has also been applied to four well-known color images such as Lena, Mandrill, Jet, and Peppers. The size of each image is set to 256×256. Hence, the number of data points for each image is 65,536. The DCPSO is used for comparison. The parameter setting of the proposed and compared algorithm is the same as that used in the grayscale image. Figure 5 shows the original images and labeled images formed by cluster labels generated from ACPAHS.

Figure 5:

(A) The Original Lena Image; (B) Labeled Image Obtained from ACPAHS (K=6); (C) The Original Mandril Image; (D) Labeled Image Obtained from ACPAHS (K=7); (E) The Original Jet Image; (F) Labeled Image Obtained from ACPAHS (K=5); (G) The Original Peppers Image; (H) Labeled Image Obtained from ACPAHS (K=6).

The optimal number of clusters for the above-mentioned color images is mentioned in [22, 37]. The number of clusters obtained from the ACPAHS is depicted in Table 6. ACPAHS generates six and seven clusters in the case of the Lena and Mandrill images, respectively. For the Jet image, ACPAHS produces five clusters. For the Peppers image, ACPAHS carried out six clusters. The results depict that the number of clusters generated from both the ACPAHS and DCPSO are in the optimal range. For the statistical significance of ACPAHS, unpaired t-test is used. Table 7 shows the results of the unpaired t-test between ACPAHS and DCPSO for the above-mentioned color images. The results reveal that the ACGSA is statistically significant over DCPSO for image segmentation.