Abstract
The interactive projection systems based on deep images are usually disturbed by the mixed noise. Generally, several filtering methods are used in combination to resolve this problem. Although the hybrid filter can guarantee the accuracy of the image, but the algorithm is complex and time-consuming, which affects the real-time performance of the interactive projection system. In this paper, the switching system method is introduced into the filter for the first time, and an arbitrary switching filter algorithm is proposed and applied to the depth image filtering system based on Kinect sensor. The experimental results demonstrate and validate that the proposed switching filter algorithm not only effectively removes the noise but also ensures the real-time performance of tracking and achieves good target tracking performance, which makes it applicable in various image filtering processing systems.
1. Introduction
Large-screen interactive projection systems based on depth images are inevitably polluted by a variety of mixed noise due to various factors. These noises reduce the image accuracy, which affects the subsequent work such as image segmentation, tracking, feature extraction, and recognition, and also cause serious disturbance to target recognition and tracking. Therefore, it is essential to smooth the image in the preprocessing. The currently available digital image filtering methods are also applicable to deep images. From the view of design method, the filtering algorithm can be divided into two categories: linear filtering and nonlinear filtering. The commonly used linear filtering methods include mean filtering, Gaussian filtering, and their improved algorithms. Meanwhile, the nonlinear filtering methods include median filtering, bilateral filtering, and their improved algorithms. All of these algorithms have their own advantages and disadvantages and are proposed for specific noise. In practice, a variety of noise generation methods have demonstrated that a single filtering algorithm cannot be effective for all types of image filtering [1–6]. In the case of the filter system based on Kinect sensor depth image, light intensity and sensor temperature are the main factors that generate the noise in the image. In addition, the short stay at the time of image formation and the interference of the channel during image transmission are also the main reasons for the noise generation [3]. Among these noises, Gaussian noise and the salt-and-pepper noise are the two most important types of noise. These noises often appear at the same time. If the same image has a variety of hybrid filtering, it will affect the system real-time performance. Therefore, the maximum interference at the current time should be analyzed first and then the filtering algorithm should be selected corresponding to the main interference source.
Although the image denoising hybrid filtering method can effectively improve the signal-to-noise ratio improvement factor, it does not consider the image processing time, which is not chosen in practical systems. The filtering method with impulse noise filter is relatively accurate, but the algorithm is more complicated, the steps are numerous, and the calculation takes a long time, which is not conducive to the real-time performance of the interactive projection system, and the accuracy requirements of the actual system for the filtered image are not so strict. In addition, based on the joint filtering algorithm of the image processing system, the processed filtering image is accurate and smooth, but it also violates the real-time requirements of the system. On the other hand, the switching system is the concept of control engineering field and “switching” was first introduced into the control system by the switching servo system. It is determined by the switching rule to switch the subsystem, and only one subsystem can work at each moment [7–12]. The switching system theory was proposed and established to fulfill the needs of industrial production and development [7–9].
Inspired by the switching control scheme, the switching rules are introduced into the filter system, and an arbitrary switching filter algorithm in the depth image filtering process based on Kinect sensor is proposed. For the main noise (salt-and-pepper noise and Gaussian noise) of each frame image, the filtering algorithm is switched to the corresponding filtering algorithm by the switching rule in order to achieve balance between the accuracy and the real-time performance. The experimental verification and performance analysis of the entire switching system are carried out through three aspects: subjective evaluation, objective evaluation, and running time analysis, to obtain the optimization effect of system tracking. The main innovations in this paper are as follows: (1)The switching rule is introduced into the field of digital image processing for the first time, and a switching filter algorithm based on arbitrary switching rules is proposed(2)Adaptive median filtering and adaptive adjustment Gaussian filtering are selected as filtering subsystems to constitute an arbitrary switching filter system(3)The proposed arbitrary image filtering algorithm is applied to image noise preprocessing of interactive projection system that realizes the balance between filter accuracy and real time
The remaining parts of this paper are organized as follows: Section 2 describes the principle of the switching filter algorithm. The two switching subsystems of adaptive median filter and adaptive adjustment Gaussian filter are presented in Section 3 and Section 4. In order to evaluate the proposed switching filter performance, the simulation results are presented in Section 5 through three aspects: subjective evaluation, objective evaluation, and running time analysis. Finally, Section 6 provides conclusions and highlights the future work.
2. Principle of Switching Filter Algorithm
A switching system consists of a series of sequential or discrete differential equation subsystems and switching rules or a switching strategy, as shown in Figure 1. If the entire filtering process is regarded as a hybrid dynamic system, then each filter algorithm can be regarded as a subsystem of the system. The proposed system is constructed by coupling the switching system with arbitrary switching rules and using two subsystems composed of arbitrary switching filter system, as shown in Figure 2.
During the switching process, the switching rules determine which subsystem the switching system should be switched to at each moment and only one subsystem can be activated at each moment. That is, at each moment, the entire system only conforms to one control law of the switching systems.
The meaning of the switching signal is as follows: where denotes that the -th subsystem is switched on at and the -th subsystem is switched off at . Thus, in finite time, the switching sequence is finite, and there exists no state transition during the switching moment, where is the initial time and is the -th switching time. When , the trajectory of the switched nonlinear system is produced by -th subsystem, defining , as the minimum interval time of the -th subsystem.
For a given time, assume that the arbitrary switching occurs at the time instants . Since the time intervals between successive switching can never be less than , and for any fixed switching laws, no finite time escape phenomenon is possible.
3. Switching Subsystem of Adaptive Median Filter
Generally, the salt-and-pepper noise filtering is based on the median filtering. The principle of the median filtering is to take a sliding window with odd points to sort the gray values of the pixels in the window and take the middle value of the result as the gray value of the pixel in the center of the original window. However, the traditional median filtering algorithm does not consider the difference between the signal point and the noise point and selects the median value of the sliding window instead of the pixel, which can easily lead to the loss of image details.
Assume that a one-dimensional switching sequence is set and the window length is an odd number . The numbers are extracted from the input switching sequence, where is the center value of the window. Then, the values of the points will be sorted by numerical values, and the serial number is taken as the center of the value as the median filter output , where is the switching signal that is a continuous segment constant function. When , it means that the -th subsystem is activated and the remaining subsystems are deactivated, indicating that the -th number is extracted from the input switching sequence.
Similarly, the operation of the two-dimensional median filter is , where is the window, and is the two-dimensional data sequence of the image.
Based on the traditional median filter, several improved algorithms have been proposed such as adaptive median filtering algorithm [5], weighted median filter algorithm [6], and extreme value median filtering algorithm [13]. In this paper, the adaptive median filtering is selected as the filtering subsystem. The difference between the median filter and the adaptive median filter is that the adaptive filter can change the size of the filter window according to the current situation of the image. When it replaces the pixel, it only substitutes the point where the noise is disturbed without changing the current pixel value of its valuable signal. The adaptive median filter can better protect the details of the image.
Adaptive median filtering can be roughly divided into the following steps in Figure 3: (1)Determine the noise of each area of the image(2)Detect the size of the filter window according to the noise pollution of each region(3)Filter the detected noise point
The first layer: calculate and .
If and , go to the second layer; otherwise, increase the window size. If the window size , repeat the first layer; otherwise, output .
The second layer: calculate and .
If and , output ; otherwise, output
where and are the minimum and maximum values of the gray level in the filter window , respectively. is the gray value at the coordinates , and specifies the maximum allowed value. The adaptive process of the adaptive median filter is as follows:
4. Switching Subsystem of Adaptive Adjustment Gaussian Filter
Generally, the Gaussian filter is employed to filter the Gaussian noise. But the classic Gaussian filter often causes the image blur. Several improvements have been made on the basis of classical algorithms [13–16]. In a linear filter, the weighted sum of the pixels within a continuous window function is used to achieve smooth filtering of the image. As a typical linear smoothing filter, the weight of the Gaussian filter template is selected according to the shape of the Gaussian function. While the steepness of the peak is determined by the parameter . The larger value of implies stronger degree of smoothness. The ideal effect is that the filter only smoothens the noise signal, so the parameter corresponding to the noise signal pixel should be made as large as possible, while the parameter corresponding to the noisy region and the edge region should be as small as possible [1, 2, 10–12, 16, 17]. Therefore, the value of the parameter is adjusted to determine the selected Gaussian filter while switching the filter in the proposed system. The Gaussian filter under the switching rules is designed in the following: where is the image signal after Gaussian filtering, that is, the convolution of the image signal and the Gaussian filter, is the two-dimensional Gaussian kernel, is the steepness parameter under the original image signal, is the steepness parameter under the noise signal, and is the image signal that is composed of the original signal and the noise signal .
The current pixel is , and the size of the filter window is . is the signal after Gaussian filtering, and the gray value of the corresponding image window is as follows:
The difference between the gray values of the current pixel and the window is as follows:
When the window area is a fully smooth area, that is, when the pixels in the window are equal to each other,
Then we have:
If the current pixel is an isolated noise point in the perfectly smooth area,
From these two extremes, it can be seen that the size of is inversely proportional to the smoothness of the image. The smoother the image is, the smaller the is, while the marginal zone is between the two extremes.
Therefore, let
The smoothness of the filter window is considered as a scale parameter. The filter window within the gray mean and the current pixel gray value of the difference is calculated as the size of the scale, and the Gaussian filter is designed as follows:
The above two filtering algorithms are improved algorithms based on the median filtering and the Gaussian filtering, for the salt-and-pepper noise and the Gaussian noise, respectively [18–21]. This paper directly references these two improved filters as switching filter subsystems, coupled with arbitrary switching rules, which constitutes an arbitrary switching filter system.
5. Experimental Results and Analysis
In order to verify the filtering effect and the real-time performance of the proposed algorithm, an interactive projection system platform is designed based on Kinect depth image. Figure 4 illustrates the schematic diagram of the designed platform, while Figures 5(a) and 5(b) show the photograph of the deep image interactive projection system platform and the platform interface, respectively. The platform uses OpenCV function library and Visual C++ 6.0 programming. It has various functions including the real-time data acquisition, the filter, the demonstration, the algorithm simulation, and the system control.
(a) Laboratory-built depth image interactive projection system platform
(b) Depth image interactive projection system platform interface
The platform shoots a video sequence of six frames with 512 × 424 pixels in each frame. In order to highlight the filtering effect, the density salt-and-pepper noise is added to frames 1, 3, and 5, while the standard deviation Gaussian noise is added to frames 2, 4, and 6. The effects of the adaptive median filtering, the adaptive adjustment Gaussian filtering, and the arbitrary switching filtering algorithm are, respectively, compared. The subjective and the objective criteria are used to evaluate the filtering effect.
5.1. Subjective Evaluation
In order to verify the effectiveness of the proposed filter, the simulation results of different filtering algorithms are compared in MATLAB 6.5 environment. The first simulation results are shown in Figure 6. Frames 1, 3, and 5 are the images of added density salt-and-pepper noise, and frames 2, 4, and 6 are the images of added standard deviation Gaussian noise, as shown in Figure 6(a). Firstly, all the noise-contaminated images are filtered by median filter and the filtering effects are shown in Figure 6(b). Secondly, all the noise-contaminated images are filtered by Gaussian filter and the filtering effects are shown in Figure 6(c). Lastly, all the noise-contaminated images are filtered by the proposed filter and the resultant filtering effects are shown in Figure 6(d).
(a) Depth map with noise
(b) Adaptive median-filtered images
(c) Adaptively adjust the Gaussian-filtered images
(d) Arbitrary switching-filtered images
It can be seen that the noise caused by the salt-and-pepper noise is almost completely filtered after adaptive median filtering. However, the white noise of the images polluted by the Gaussian noise is still obvious. Similarly, after adjusting the Gaussian filter adaptively, the white noise of the images polluted by Gaussian noise is noticeably reduced and the image becomes smooth. However, the added salt-and-pepper noise in the image is still existed. Thus, for random noise, only one filter algorithm is ineffective. It can be observed from the arbitrary switching-filtered images that each frame image has better filtering effect compared to the single filter algorithms.
In order to further validate the filtering effect, the filter simulations are performed again. Similar to the first experiment, the density salt-and-pepper noise is added to frames 1, 3, and 5, and the standard deviation Gaussian noise is added to frames 2, 4, and 6. The second experimental results are shown in Figure 7. It can be seen that the second simulation results are consistent with the first. The adaptive median filtering or the adaptive adjustment Gaussian filtering only has a significant effect on a single noise, while the arbitrary switching filtering has a relatively high probability of switching to the corresponding most efficient filtering algorithm for any frame image, enabling smooth and efficient filtering.
(a) Depth map with noise
(b) Adaptive median-filtered images
(c) Adaptively adjust the Gaussian-filtered images
(d) Arbitrary switching-filtered images
5.2. Objective Evaluation
The objective evaluation uses the PSNR (peak signal-to-noise ratio) to measure the filter quality numerical index. It is defined as follows: where and are the length and the width of the image, respectively. MSE is the mean square error between the original image and the processed image. The smaller the MSE is, the closer the filtered and the original images are. The larger the PSNR is, the better the filtering effect is.
MATLAB is used to calculate the peak signal-to-noise ratios of the images subjected to three different filters, the adaptive median filter, the adaptive adjustment Gaussian filter, and the switching filter. The PSNR values of Figures 4 and 5 are shown in Tables 1 and 2, respectively.
It can be seen from the tables that the PSNR values of several frames with added salt-and-pepper noise are larger by adaptive median filtering than the PSNR values of several frames with Gaussian noise. After adjusting the Gaussian filter adaptively, the PSNR values of several frames with Gaussian noise are larger than the PSNR values of the few images with added salt-and-pepper noise. The filter proposed in this paper has small variation in the PSNR values of various noises with overall value of about 30.It is shown that the Gaussian filter is most suitable for Gaussian white noise and the median filter is more suitable for the salt-and-pepper noise. The proposed switching filter has small effect on the processing of various noises, and the processing effect is clearly better than the other two filters.
5.3. Runtime Comparison
The original intention of this paper is that the improved switching filtering algorithm is better than the single filtering algorithm and is less time-consuming than the hybrid filtering (that is, multiple filtering of one frame of image). Thus, the balance between precision and real time can be realized. In order to verify the timeliness of arbitrary switching filter, a time calculation statement in the program is added to obtain the total time for the six frame graphs with different filtering algorithms. The runtimes of two experiments are shown in Table 3.
Table 3 shows that although the runtime of the proposed arbitrary switching filter is longer than the adaptive median filter and the adaptive adjustment Gaussian filter, it is shorter than that of the hybrid filter. Therefore, the optimization of the switching filter algorithm’s runtime can be achieved.
Remark 1. In this paper, the proposed switching filtering algorithm can effectively filter the noise, improve the positioning accuracy, and enhance the quality of the human-computer interaction. By comparing with traditional filter algorithm without switching filtering method, the presented arbitrary switching filter algorithm based on Kinect sensor has the main advantages: (1) the switching filter algorithm based on arbitrary switching rules is proposed for the image noise preprocessing of interactive projection system, (2) the two switching filtering subsystems of adaptive median filtering and adaptive adjustment Gaussian filtering are employed to constitute an arbitrary switching filter system, and (3) the proposed arbitrary image filtering algorithm is applied to image noise preprocessing of interactive projection system, and the balance between the accuracy and the real time has been achieved.
Remark 2. As we have known, switching rules play an important role in improving the stability and robustness of the image noise preprocessing of interactive projection system. The arbitrary switching is random and cannot guarantee that each frame of the image is switched to the suitable adaptive filtering algorithm [22–25]. Therefore, the switching algorithm with arbitrary switching rule needs to be improved in terms of accuracy and the switching rules can be optimized in the next step. In future work, the average dwell time switching method or the event-driven switching method will be considered to improve the filtering accuracy and the robustness for the image processing filter system [24–31].
6. Conclusion
In this paper, the switching system is introduced into the Kinect depth image filtering process for the first time and an arbitrary switching filter algorithm is proposed. For each frame image, the suitable adaptive filtering algorithm is selected by arbitrary switching rule according to the main noise. The experimental results have demonstrated that the filtering effect of the proposed arbitrary switched filter is better and the running time is shorter than that of the general algorithms, which realize the balance between filter accuracy and real time. Thus, the proposed switching filter method can be applied to image processing system, pattern recognition system, gesture control system, etc.
Data Availability
The data used to support the findings of this study are included within the article.
Conflicts of Interest
The author declares no conflicts of interest.
Acknowledgments
This work was supported by the National Natural Science Foundation of China under Grant 61403268 and Grant 61873176 and the Natural Science Foundation of Jiangsu Province under Grant BK20181433.