Abstract
This study focused on the identification problems of two-input single-output system with moving average noises based on unsupervised learning methods applied to the input signals. The input signal to the autoregressive moving average model is proposed to be arriving from a source with continuous technical and environmental changes as two separate featured input signals. These two input signals were grouped in a number of clusters using the K-means clustering algorithm. The clustered input signals were supplied to the model in an orderly fashion from cluster-1 up to cluster-K. To ensure that the output signal can be best predicted from the input signal which in turn leads to selecting good enough model for its intended use, the magnitude-squared coherence (MSC) measure is applied to the input/output signals in the cases of clustered and nonclustered inputs, which indicates best correlation coefficient when measured with clustered inputs. From collected input-output signals, we deduce a K-means clustering based recursive least squares method for estimating the parameter of autoregressive moving average system. The simulation results indicate that the suggested method is effective.
1. Introduction
Most systems that have been inferred to prove many of the assumptions proposed in the area of system identification focused on single-input single-output systems [1–4]. Recently, the focus has been on studying system identification of multivariable systems in order to deal with an appropriate modelling and estimation of dynamic systems operating in industrial applications and process control [5–7].
Several methods and techniques addressed system identification problems in multivariable models as in [8–12]; the researchers in these studies depend on hierarchical identification principle that decomposes a multi-input system into two subsystems, one containing a parameter vector and the other containing a parameter matrix [10]. The proposed method in [13] was dependent on the concept of coupling identification to avoid matrix inversion in multivariable least squares in order to reduce computational time in the algorithms. Many algorithms are used to implement these methods such as least squares recursive algorithms or iterative least squares algorithms without mentioning the impacts of the excitation signals on the proposed models.
There are identification methods focusing on the modified effects of the excitation signals on the design of system identification experiments and they yielded acceptable results in modelling and estimation process of multivariable system. These proposed methods are based mainly on applying filtering techniques to the collected input-output data in order to generate more accurate results as in [14–17]. We aim at the same objectives of these researches by considering effective preprocessing on the excitation signals of multivariable systems.
In this paper, we tried to focus on the collected input data to identify multiple-input single-output systems by introducing unsupervised learning approach such as the clustering techniques applied only to input signals by means of K-means clustering algorithm. To ensure that the system output is best predicted from the proposed excitation signals, the similarities in the frequency contents between modified input signals and generated output signal are investigated using the magnitude-squared coherence measure [18].
Most of the works that relied on the use of clustering in the system identification process were based on collected and recorded data from input and output without even assuming any slightly modified effect on the input signals. In [19], the submodels of the PieceWise AutoRegressive systems with eXogenous input (PWARX) are obtained through an algorithm inspired by competitive learning. In particular, the proposed method exploits a process of fuzzy clustering to obtain a subset of representatives from the original data set. Reference [20] assumed the number of modes of the PWARX system to be unknown and proposed a split-and-merge clustering algorithm to estimate the correct number of modes. Reference [21] discusses the use of correlation clustering algorithms for robust identification of PWARX models with reduced complexity. A PWARX model of the electronic throttle regulating the air inflow of a car engine has been identified based on clustering in [22]. The refinement algorithm proposed in [23] is repeatedly applied to the estimated clusters to improve both the data classification and the parameter estimation. Since, in complex engineering system, reliability is considered as the main issue, K-means clustering is used in [24] to estimate the reliability of the complex engineering system to obtain three or more subsystems called cluster systems and analyze the key performance parameters of constructs, which are known as family system that contains only subsystems with similar performance; then the contribution of these subsystems in the cluster system leads to estimating reliability of the whole system. Many of the above-mentioned research works were based on clustering the regressor vector to divide the system model into multiple candidate models for investigation and analysis. We pointed out here that the regressor vector contains both input and output signals.
The fundamental idea of the proposed method in this paper is to cluster only the input signals using K-means clustering algorithm without clustering the whole regressor vector. Based on this idea, we present a clustered input signals based recursive least squared algorithm. Given this recursive algorithm, we can produce more accurate parameter estimation compared to existing multivariable recursive estimation methods, for example, [14, 15]. Compared with the clustering techniques used in a PieceWise affine ARX model [19–21], the modified recursive algorithm in this paper can produce highly accurate estimation.
Furthermore, since the clustered input signals are assumed to be similar due to the similarity measure of the K-means algorithm, the same system properties will be excited and hence better identification performance will be obtained [25].
The rest of this paper is organized as follows. Section 2 gives the general model description and the formulation of the autoregressive moving average (ARMA) identification model with some important definitions. Section 3 derives a clustering input signals based algorithm for two-input single-output model. Section 4 gives the recursive extended least squares algorithm used for comparison. Section 5 gives the magnitude-squared coherence measure used to obtain the degree of similarities between input signals and the output signal of the proposed model. Section 6 provides the steps used to perform the K-means clustering algorithm. An illustrative example to demonstrate the effectiveness of the proposed algorithms is provided in Section 7. Finally, concluding remarks are given in Section 8.
2. The Model Description
Firstly, let us introduce some important notations. The superscript denotes the matrix transpose, the symbol stands for an identity matrix of order , represents an -dimensional column vector whose elements are 1, stands for the norm matrix , denotes that is defined by , the estimate of parameter at time is denoted by , represents a unit backward shift operator defined as , and is a large positive number [26].
Consider the two-input single-output system, described by the autoregressive moving average model, as shown in Figure 1, with a mathematical model aswhere are the two features of the input data points of the system; represents the output data points of the system; is a white noise sequence with zero mean and variance ; and , , , and are the polynomials with known degrees in the unit backward shift operator as in the following equation:
Assume that , and are strictly proper; that is, their values are 0 for , and the orders , , , and are known.
From two inputs and the output , the model parameters are to be identified.
Define the parameter vector and the information vector as
Based on recursive least squares in [27], the identification model of ARMA model can be expressed as
The parameter vector contains all parameters to be estimated.
3. The K-Means Clustering Algorithm
K-means clustering is an unsupervised learning approach. The unsupervised learning approach is given by grouping data with similar characteristics; this grouping process is known as clustering, which consists in separating each group with different characteristics in sets called clusters [28]. As in [29, 30], K-means clustering algorithm works with three main steps: initialization step, assignment step, and update step. Firstly, in the initialization step, the process starts with selecting data where n is the amount of data to be carried out in clustering and m is the number of variables (data dimension), and, initially, the centre of each cluster is determined arbitrarily [30]. Secondly, in the assignment step, the distance such as Euclidian distance (other types of distance measures can be applied depending on the application) of data-i to centroid k is calculated according to the following equation [30]:where is distance of object i and centroid k, represents data dimension, denotes coordinates of object i in dimension j, and denotes coordinates of object k in dimension j.
If the distance of data to specific cluster centroid k has the smallest value when compared to another cluster centroid, then the data will be a member of the cluster k. This can be examined according to the following criterion index [31]:
Finally, in the update step, after classification of data for each cluster, centroid value can be calculated by finding the average value of the data which are members of the cluster using the following equation [31]:where cluster k and denotes number of cluster k members.
As in [30], the steps of the K-means clustering algorithm to cluster input signals used in the proposed model are listed as follows:(1)Suppose given input data matrix with .(2)Determine the number of clusters (k), with centroid chosen randomly.(3)Calculate the distance of each input data point to the centroid using equation (5).(4)Classify input data into clusters with the shortest distance using equation (6).(5)Calculate the new centroid using equation (7).(6)Repeat steps 3 through 5 until there is no more input data movement to another cluster.
In our proposed method, we apply the above algorithm to the model input signals which are assumed to be represented as two input features collected from different sources.
The optimal number of clusters was chosen according to the elbow method in [32]. In elbow method, a decreasing cost function is analyzed for different values of k and it can be defined as the sum or average distance from all points in the data set to its centroid. Figure 2 shows the candidate elbow point as an optimal point to select the best number of clusters according to the best total sum of distances. As seen from Figure 2, the optimal value chosen for the illustrative example used to verify the proposed method is k = 2.
4. The Clustered Input Signals Based Recursive Least Squares (CIB-RLS) Algorithm
Our goal in this paper is the identification of the ARMA model based on clustered input signals. We start with the assumption that the two-input signals represent two different attributes supplied from a source with continuous technical and environmental changes. The first step is to apply these signals to the K-means algorithm to group them in several k-clusters to share a common statistical property.
Let the two-input data set signals and be applied to the K-means algorithm; the algorithm grouped these data sets into k-clusters, known as cluster-1 to cluster-k; see Figure 3 below. Every cluster contains part of data set belonging to and . Assume this data set to be known as , which represents a part of in the cluster k, and , which is a part of in the same cluster k; Figure 3 illustrates the assumptions. At this point, we select the parts of input signals and from each generated cluster to form new modified inputs as
The modified input signals, represented by equation (8) and equation (9), are then applied to the general system under investigation as shown in Figure 4.
The proposed CIB-RLS algorithm will be as follows:
To initialize the algorithm, we choose
The steps involved in the CIB-RLS algorithm are listed as follows:(1)Set .(2)Let ; set the initial values of the parameter estimate vectors and the covariance matrix according to (20), and give the parameter estimation precision .(3)Collect input data and .(4)Select the optimal number of clusters according to collected by using elbow method.(5)Classify input data and into clusters with the shortest distance by (6).(6)Collect and as in (8) and (9), respectively, and collect the output data .(7)Construct the information vectors by (14), by (15), and by (13).(8)Compute by (16), and update the estimate by (19).(9)Construct by (17).(10)Compute the gain vector by (11) and the covariance matrix by (12).(11)Update the parameter estimate by (10).(12)If , obtain the parameter estimate ; otherwise, increase by 1 and go to step 6.
5. The Recursive Extended Least Squares (RELS) Algorithm
The main algorithm used to estimate the model parameters in our proposed method is the recursive extended least squares (RELS) algorithm. With the aid of the algorithm in [14], the recursive least squares algorithm of the ARMA model described by equation (1) with its identification model expressed in equation (4) can be applied. Before we proceed, based on this identification model, the unknown variables in the information vector in (3) are replaced with their estimates [14]. So, the recursive extended least squares (RELS) algorithm [33] is as follows:
6. The Magnitude-Squared Coherence (MSC)
The magnitude-squared coherence (MSC) is a measure that estimates the extent to which one real- or complex-valued signal can be predicted from another real- or complex-valued signal [18]. In our work, we used these measures to investigate the similarities in the frequency content between input signals and the output signal used to identify the proposed ARMA model. Comparisons were done between the cases where the two inputs are clustered before being applied to the model and the two inputs are applied directly without passing to the clustering algorithm. If we define the power spectrum at frequency of the input signal and the output as and , respectively, and the cross power spectrum as , then the magnitude-squared coherence can be given aswhere .
The value of MSC lies in the range which is analogue of the correlation coefficient in statistics.
In our proposed model, the MSC gives the better correlation between the clustered applied input signals and the measured output signal compared to unclustered input signals with measured output.
Figures 5 and 6 represent the MSC for the illustrative example in Section 7 with noise variances of and , respectively. The dotted line represents the case in which the input signals are unclustered, whereas the solid line represents the case of clustered input signals.
As seen from these figures, it is clear that best measure is obtained in the case of clustered input signals. According to the type of signals, we can choose the best values of sampling rate, number of overlaps, and number of fast Fourier transform points (FFT) to achieve suitable results of MSC calculation.
7. Illustrative Example
In this illustrative example, we use MATLAB version R2020a and Wolfram SystemModeler software for simulation [34] to run recursive lest squares with K-means algorithms and to collect input and output data, respectively.
Consider the electric circuit model with two-input voltage and a single-output voltage measured from a capacitor C as shown in Figure 7. From the circuit of the Figure 7 and the concepts in [35–36], we can obtain the state-space model as
The circuit parameter is assumed to be according to Table 1, where represents the inductor current as the first state and represents the capacitor voltage as the second state. From the above state-space model, we can find the continuous and discrete transfer functions [35], respectively, as
with sampling time = 0.05 sec.
Consider the discrete polynomials model in (25) above, the proposed ARMA model to be identified can be written in the form
The parameter to be estimated is
The inputs are taken as a chirp signal with different ranges of frequency and different amplitudes and carry the characteristics of persistent excitement using simulation program in [34], and is taken as a white noise sequence with zero mean and variances of and , respectively.
Applying the recursive extended least squares algorithm for a comparison and clustering inputs based recursive least squares algorithm to estimate the parameter of the electric system, the parameter estimates and their errors are shown in Tables 2–5, and the estimation errors versus for and are shown in Figures 8 and 9, respectively.
Through Tables 2–5 and Figures 8 and 9, we can infer the following observations:(i)With a suitable selection of the input signals and choosing the optimal value of k in K-means clustering algorithm, the parameter estimation error decreases gradually with the data length t increasing. This, in turn, shows that the proposed algorithm gives effective results.(ii)Since, at a certain time, the model is excited with clustered input signals that share the same characteristics with a high degree of similarity due to the K-means clustering algorithm, the same system properties could be excited and the accuracy of the model identification is improved.(iii)Compared with the recursive extended least squares algorithm, the proposed clustering inputs based recursive least squares algorithm has better identification performance.(iv)Using clustering inputs based recursive least squares algorithm, the parameter estimates converge to their true values faster than using recursive extended least squares algorithm.
8. Conclusions
This paper investigates one aspect of unsupervised learning effects on the system identification field. The clustering inputs based recursive least squares algorithm for the two-input single-output system with moving average noise is proposed by introducing clustering techniques to the excitation input signals. The simulation results indicate that the proposed algorithm can yield more accurate parameter estimates under certain conditions (i.e., selection of the optimal value of K-means) compared with the recursive extended least squares algorithm. The proposed method can be extended to multiple-input multiple-output systems and stretched forth to investigate the identification of nonlinear systems and time-delay systems.
Data Availability
The data used to support the findings of this study are included within the article.
Conflicts of Interest
The author declares that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
The author gives special greeting to his wife who provided help during the research and preparation of the manuscript.