Kinect Controlled NAO Robot for Telerehabilitation

1 Introduction

Stroke is one of the major causes of human upper limb impairment. Moreover, it is the fifth leading cause of death for Americans. Statistics reveal that in every 40 seconds someone in the United States suffers from a stroke and in every 4 minutes, someone dies [1]. Including the cost of health care services, medicines, and missed days of work, the total estimated cost is approximately $34 billion each year in the United States [1]. According to the London: Royal College of Physicians, 77% of the stroke survivors, experience altered arm function, among them 40% are left with a non-functional arm [2]. Recovery from a post-stroke disability is obtained by the rehabilitation programs. Such rehabilitation programs are often labor intensive, and therapists usually work with patients one-on-one. Rehabilitation therapy in all phases of post-stroke includes intensive, highly repetitive tasks and task-specific training [3, 4]. However, in the US, only 30% of stroke survivors can receive rehabilitation [5] due to an insufficient number of rehabilitation facilities [6]. All these post-stroke rehabilitation challenges demand to develop alternate/new rehabilitation technologies. Robot assistive rehabilitation technology is one of the emerging technologies in post-stroke rehabilitation. Such technology may improve rehabilitation quality, productivity, and reduce costs of individuals with a disability.

Extensive research has been done to develop robotic devices for rehabilitation of patients with upper limb impairment. The most commonly known robotic orthotic devices include, but are not limited to, ETH Arm rehabilitation (ARMin) [7], InMotion2 developed by Interactive Motion Technologies, Inc., Boston, MA (commercially available version of the MITManus) [8], The Mirror Image Movement Enabler (MIME) [9], ETS-MARSE [10].

Recently, researchers have been interested in developing robotic systems to work as a coach and perform a therapeutic role through human-robot interaction. Studies reveal that using robots in therapeutic roles keep patients motivated and provide promising and positive interaction responses [11]. Tele-operation of robotic devices is one of the emerging technology that enables researchers to use robotic devices in a therapeutic role. The main idea behind such an approach is to extract motion information of an actor’s movements and replicate these movements to a robot within the anatomical joints limits and constraints of the robot. The major challenge in this approach is to record joints position (such as upper limb joint) coordinates of the actor and accurately convert it to a joints angle to control a robotic device. Different motion tracking devices exist that are used by the researchers to track human motion. Thobbi and Sheng [12] used a Vicon tracker that use multiple markers to track human movements and applied this information to teleoperate an NAO robot. Cole et al. [13] used a markerless system consists of a single camera that tracks the motion of an actor to map it on a simulated HOAP-2 humanoid robot. To track the actor’s joint position, the actor needs to wear a special colored suit. Different researchers also used different methods to obtain joint information by tracking the actor’s performance.Wang et al. [14] transformed actor’s motion information into a critical frame, which is obtained by a Kinect sensor and applied inverse kinematics through an optimization process to obtain joint information of the NAO robot. Suleiman et al. [15] used Kinect sensors to track an actor’s movement information, then formulate it as an optimization problem where constraints are humanoid robot’s physical limitations. Rosado et al. [16] also considered a motion mimicking problem as an optimization problem, and teleoperated a simulated 4 DoF robot model by tracking actor movement information through Kinect. Ningjia et al. [17] calculated the joint angles of the robot using the spatial vector method from the data obtained by tracking a human actor through Kinect. Ude et al. [18] used a standard non-linear least square optimization method to obtain robot motion data from the actor’s tracked data.

In this paper,we used the Kinect v2 sensor to track human motion. In the recent decade, the Kinect sensor extensively used for human motion assessment and rehabilitation purposes. Kinect sensor integrated with virtual reality for rehabilitation gaming therapy shows promising results for patient’s motor and functional recovery [19, 20, 21, 22, 23]. Kinect sensor also used for human upper and lower extremity motion assessment for the purpose of recovery improvement tracking and providing feedback [24, 25, 26, 27]. Kinect motion tracking sensor system is a markerless system, and does not require any sophisticated calibration. The markerless system allows us to estimate motion more accurately than other motion capture systems, where there is a possibility for inaccurate marker placement. Instead of solving complex inverse kinematics to estimate joints angles, we proposed a simple complete geometric solution to find joint’s angles. This geometric solution allows quick estimation of joint angle and needs less computational power.We used the NAO robot to demonstrate upper limb exercises by mimicking the actor’s motion. The humanoid robot NAO can communicate verbally. Verbal communication provides a ground for social interaction and share emotions [28, 29]. NAO robot shows promising usability in the fields of pediatric rehabilitation, socially assistive robotics, human motion imitation and learning artificial intelligence [30, 31, 32, 33, 34].

2 Humanoid Robot NAO

For a demonstration of telerehabilitation, we have used a humanoid robot NAO due to its human-like (a) appearance; (b) acrobatic movement capability; and (c) communication capability using vision sensors, speech sensors, hearing sensors, and touch sensors. All these features make NAO one of the most promising autonomous programmable robot [35]. NAO was developed by Aldebaran Robotics. It has 25 joints, two cameras, four microphones, two loudspeakers, nine tactile sensors, and eight pressure sensors. It is 574 mm tall and 275 mm in width. NAO robot has two joints in the head (neck), five joints in each arm along with hand close and open motions, five joints in each leg and two hip joints. All these sensors and actuators make NAO capable of performing all simple human motions such as walking forward and backward; sitting down and standing up; waving hand; playing soccer, etc. In this study, we focused on upper extremity rehabilitation scheme; therefore, throughout the study, we have focused on the upper arm (right) of the NAO. NAO has its own central processing unit (CPU) to execute command and control all the sensors. Its motherboard consists of Intel ATOM 2530 1.6 GHz processor with cache memory 512KB, clock speed 1.6 GHz and FSB speed 533mHz, 1GB RAM, 2GB flash memory and 8GB Micro SDHC. Also, it provides communication through WiFi using IEEE 802.11 a/b/g/n protocol, Ethernet, and USB interface. Its USB port mainly used for updating the robot operating system. For advanced programming of NAO robot. Aldebaran provides API for language C++, Python and Java for NAO V5. Python SDK for the NAO robot requires Python 2.7-32 bits. It only works for this environment (Python 2.7-32 bits). C++ SDK requires a C++ compiler for different OS. For Windows, C++ SDK only works for Visual Studio 2010. Windows also have some limitations; there is no C++ cross compiler. As a solution to this problem, Windows users need to install qiBuild, a tool design to generate cross-platform projects using CMake along with the C++ SDK. In our study, we have used Python 2.7.13 and Python SDK to control the NAO for advanced functionality.

Figure 1

Active joints of NAO robot [36]

3 Human Motion Tracking Device

The most important thing in teleoperation is to track the human upper limb motion. For this purpose, we use the markerless vision-based tracking device Kinect. Kinect sensor developed by Microsoft, has a depth sensor, a color camera, and a four-microphone array. It can track the full body of a person. Due to its low price, Kinect is used for different applications in different fields. Microsoft Kinect is equipped with an IR depth sensor, one RGB camera, and a four-microphone array. Using depth sensors, it can produce a 3D depth image with regulation 512x424 at 30 FPS (frame per sec) and range 0.5 to 8 meters. It provides light independentinfrared and able to remove the ambient light effect. RGB camera provides full HD 1920 x 1080 regulation. It also provides either 30 or 15 FPS based on lighting. The most amazing feature of the Kinect sensor is that it can track a fully human body. It can track 25 joints of the human body with clear identification of hand states whether it is open or close. Body joint tracking means that it provides the coordinates of each joint in 3D space. It can track a total of six-persons simultaneously in front of it within a range of 0.5 to 4.5 meters. Through the microphone, it also can detect sound direction. Field of view of its depth camera is 70^∘ in horizontal and 60^∘ in the vertical direction. To access the Kinect data, we used the Kinect V2 SDK. This SDK provides the necessary API to extract Kinect data using Visual Studio 2012 or higher. It is (SDK) compatible with programing language C++, C#, Java. In this research, we have extracted Kinect data to analyze human arm kinematics and also to teleoperate the NAO (in real-time) to establish the proposed telerehabilitation scheme. As mentioned earlier Kinect can track 25 joints of the human body. However, in this study, our focus was on the human upper extremity. To serve our purpose, we tracked only 11 joints of human upper extremities that include shoulder, elbow and wrist joint of each arm along with the head, neck, spine shoulder, spine mid and spine base joint.

4 Human Upper Arm Kinematics

In the proposed telerehabilitation scheme human right arm movement was considered to actuate the NAO’s right arm. To replicate NAO’s arm movement, human upper arm was modeled as a two degrees of freedom (DoF) shoulder joint (to provide abduction/adduction, and vertical flexion/extension motion), and a one DoF elbow joint (to provide flexion/extension motion). To develop the kinematic model human right arm’s kinematics was analyzed using the modified DH convention [38]. The coordinate frame attachment for this analysis is depicted in Figure 4.

In this model, Joint 1 represents shoulder abduction/adduction, joint 2 corresponds to shoulder vertical flexion/extension, and joint 3 depicts the flexion/extension of the elbow joint. Distance between the shoulder joint and the elbow joint is d₁ and that is between the wrist and the elbow joint is d₂. The modified DH parameters for the human upper arm are summarized in Table 1. where α_i−1 is the link twist, a_i−1 corresponds to link length, d_i stands for link offset, and θ_i is the joint angle of Human Arm.

The general form of a link transformation that relates frame {i} relative to the frame {i − 1} is can be expressed as [38]:

where, ii−1 R is the rotation matrix that describes the frame {i} relative to frame {i − 1} and can be expressed as:

moreover, ii−1 P is the vector that locates the origin of the frame {i} relative to the frame {i − 1} and can be expressed as:

The homogenous transformation matrix that relates frame {4} to frame {0} can be obtained by multiplying individual transformation matrices.

The single transformation matrix thus found from equation (4) represents the positions and the orientations of the reference frame {4} attached to the wrist joint (Figure 4) with respect to the fixed reference frame {0}. The equation (4) is known as the forward kinematics equation. If the joint variables are (θ₁, θ₂, θ₃, and θ₄) known then the position of the last frame {4} and/or end-effector position with respect to base frame {0} can be determined from the forward kinematics equation.

For telerehabilitation purpose, teleoperator’s upper-limb joint variables (θ₁, θ₂, and θ₃) were computed in real-time from the Kinect coordinate data (see Figure 3b). More details on this were discussed in the next section. Note that, NAO’s upper arm range of motion (ROM) differs compared to that of the human upper arm. For instance, NAO’s elbow joint ROM is around 60^∘ less (see Table 2) compare to that of the human elbow joint (θ₃). In addition, NAO’s shoulder joint roll (abduction/adduction, θ₁) motion ROM is smaller compared to that of human shoulder joint motion. On the other hand, its shoulder pitch (vertical flexion/extension, θ₂) ROM is larger compared to that of human shoulder joint pitch motion. Human upper-arm joints motions are depicted in Figure 5.

Figure 2

Kinect Sensor (reference coordinate attached)

Figure 3

(a) Name of the tracked joints using Kinect [37]; (b). Kinect coordinate frames used in telerehabilitation

Figure 4

Coordinate frame assignment for Human upper arm

Figure 5

General motion. (a) Shoulder Flexion/Extension (shoulder pitch); (b) Shoulder Abduction/Adduction (shoulder roll) and (c) Elbow Flexion/Extension (elbow roll)

5 Control Scheme

To control NAO’s right arm movement (remotely), the Kinect sensor was used to generate the control signal. This sensor can track human motion in front of it. Note that, Kinect sensor provides Cartesian coordinate information of the human upper arm joints (shoulder, elbow, and wrist), whereas, human upper arm joint angles are required to teleoperate the NAO. Figure 6 shows the schematic of the control approach where it shows human arm joint angles were used as a reference input to the NAO’s controller. The geometric approach was used to compute human arm joint angles from Kinect data.

Figure 6

Schematic of the control approach

5.1 Geometric Calculation of Joint Angles

As stated earlier, Tele-operators upper arm joint coordinates (i.e., the coordinates of the shoulder (x_sk , y_sk , z_sk), elbow (x_ek , y_ek , z_ek), and wrist (x_wk , y_wk , z_wk) joints) in 3D space with respect to the Kinect’s reference coordinate frame (see Figure 2) can be directly obtained from Kinect. However, we need to compute the joint angles from the co-ordinates data. Note that, the orientation of the frame {0} (teleoperator’s shoulder joint frames, see Figure 4) is identical to the Kinect coordinate frame (see Figure 2). The co-ordinate of the elbow (frame {3}) and the wrist (frame {4}) joint with respect to frame shoulder frame {0}, {1} or {2} thus can be found by subtracting shoulder joint coordinate from the elbow and the wrist joint coordinates which were obtained from Kinect sensor.

Therefore, for a known shoulder joint co-ordinate (x_sk , y_sk , z_sk), the elbow and wrist joint coordinates with respect to the shoulder joint of the teleoperators can be found as follows:

• Elbow joint coordinates with respect to the shoulder frame:

x e , y e , z e = x e k − x s k , y e k − y e k − y s k , z e k − z s k ,

and

• Wrist joint coordinates with respect to the shoulder frame:

x w , y w , z w = x w k − x s k , y w k − y s k , z w k − z s k .

Let us consider a vector OA along Z₀ axis is in a negative direction with unit length (see, Figure 7). Then the point A locates at (0, 0,−1) position. Now, the angle between the d₁ vector OA is the angle θ₂. Also, consider a vector OB along the X₀ axis is in a positive direction with unit length. Then the point B locate at (1, 0, 0) position. Therefore, the angle between OB and d₁ is the angle θ₁. then,

Figure 7

Joint angle θ₁ and θ₂ calculation

cos θ1=((1−0)2+(0−0)2+(0−0)2)2+(d1)2−((xe−1)2+(ye)2+(ze)2)22d1.

and, sin θ1=±1−(cos θ1)2. θ1=tan−1(±sin θ1cos θ1).

therefore, Similarly,

cosθ2=((0−0)2+(0−0)2+(−1−0)2)2+(d1)2−((xe)2+(ye)2+(ze+1)2)22d1.

and, sin θ2=±1−(cos θ2)2.

therefore, θ2=tan−1(±sin θ2cos θ2).

Finally, from the triangle shown in Figure 8, θ₃ can be found using the cosine rule.

Figure 8

Joint angle θ₃ calculation

d2=d12+d22−2d1d2cos θ3.

then, θ3=cos−1(d12+d22−d22d1d2).

To align NAO’s upper arm’s joints axes of rotation with that of the human upper arm, it is required to orient NAO’s joints axes of rotation, which is summarized below:

θ_{1,NAO,RShoulderRoll} = θ₁ − 90^∘

θ_{2,NAO,RShoulderPitch} = θ₂

and, θ_{3,NAO,RElbowRoll} = 180 − θ₃

For the convenience of representation, three joint angles of NAO are represented as θ₁, θ₂ and θ₃ throughout this paper.

6 Performance Analysis

To analyze the performance of the developed teleoperation scheme, some basic experiments were conducted on NAO robot, where a healthy adult subject (age: 25, height: 5.4ft, mas: 65Kg) performed following single-joint movement exercises in front of the Kinect to teleoperate the NAO robot in real-time.

1. Shoulder abduction/adduction (NAO robot RShoulderRoll, θ₁),

2. Shoulder vertical flexion/extension(NAO robot RShoulderPitch, θ₂),

3. Elbow flexion/extension (NAO robot RElbowRoll, θ₃) .

The experimental results are shown below.

In our first teleoperation experiment, NAO’s shoulder abduction/adduction motion was controlled. Figure 9 shows the position of NAO’s upper arm and that of teleoperator’s upper arm at different instances while performing shoulder abduction/adduction motion. As shown in Figure 10, the motion began from −10^∘, and ended at −74^∘ (i.e., within the range of NAO’s joint ROM).

Figure 9

Arm position of the human operator and NAO robot during shoulder joint abduction/adduction teleoperation

Figure 10

Shoulder joint abduction/adduction teleoperation. (a) Joint coordinate from Kinect; (b) Comparison between the joint angles found from Kinect, and NAO; (c) NAO’s joint angles from Kinect Co-ordinates data

Figures 10(a), and 10(c) show the Kinect coordinate data of the teleoperator’s upper arm joints (i.e., shoulder, elbow, and wrist joint). While performing shoulder joint abduction/adduction motion, the Z coordinate from Kinect frame of all joint should be constant which is matched with the tracked results (Figure 10(a)). Figures 10 (b) and 10(c) display the trajectory followed by a human operator and NAO robot. Motor position feedback from the NAOwas obtained through Device Communication Manager (DCM). There are two separate data come from NAO’s DCM; one set of data that comes from motor controller output, and goes to the motor for execution which is called motor command; and the other set of data that comes from the joint position sensor MRE (Magnetic Rotary Encoders) is called sensor reading. From Figure 10(b), it is obvious that there was some difference between these two data. The reason of this difference is when NAO moves any of its (upper arm) joints, the motion is interpolated between the current value of the motor position and the targeted value, and it takes some time for the command to go to the motor cards and take some time for sending the reading back. There was also some delay between the Kinect data and the motor command because these two programs run in two different platforms. Kinect extract data in MATLAB and Python communicate with NAO robot through Wi-Fi communication. MATLAB and Python communicate through TCP/IP communication. There is some delay between sending and receiving data between MATLAB and Python. Besides, NAO’s controller needs some time to execute all its current command to take the next command. To eliminate/reduce such delay all the platforms are needed to be synchronized. However, it is a little bit difficult when there are multiple platforms are used (in this case MATLAB, Python, NAOqi). Nevertheless, from the above results, it is obvious that the operator’s upper limb trajectory pattern was nicely followed by the NAO robot.

Figure 11 shows the upper arm position of the NAO and the human operator at different instances of shoulder joint vertical flexion/extension motion. The experimental results are shown in Figure 12 where it can be found that the motion starts at 90^∘ and ends at 15^∘. Note that, the actual range for this motion was higher than the range 90^∘ to 15^∘, but we limit this motion 90^∘ to 15^∘ for experimental purpose.

Figure 11

Arm position of human operator and NAO robot during shoulder joint vertical flexion/extension teleoperation

Figure 12

Shoulder joint flexion/extension teleoperation. (a) Shoulder joint flexion/extension teleoperation, (joint coordinate from Kinect). (b) Comparison between joint angles from Kinect and NAO robot. (c) Coordinate data and joint angles from Kinect and NAO robot

Figure 13 shows the hand position of the NAO robot and human operator at different instances of elbow joint flexion/extension motion. It can be seen from the Figure 14 that the elbow motion was started at 15^∘ and ended at 85^∘. Figure 14 (a) shows the coordinate data obtained from the Kinect sensor for the shoulder, elbow, and wrist joint. It is to be noted that, while performing elbow flexion/extension motion, the joint coordinates of the elbow and the shoulder joint (measured from the Kinect frame) should remain unchanged and only the wrist joint coordinates change with elbow flexion/extension motion. In addition, the X coordinate for wrist joint should be constant as the elbow motion is in YZ plane. Thus, in Figure 14 (a), it can be seen that all the red dots (proximal red dots-indicates) and black dots (indicates shoulder joint) were clustered at the same position throughout the trial.

Figure 13

Arm position of human operator and NAO robot during elbow joint flexion/extension motion during teleoperation

Figure 14

Elbow flexion/extension teleoperation. (a) Skeleton with joint coordinate from Kinect, (b) comparison between joint angles from Kinect and NAO robot, (c) coordinate data and joint angles from Kinect and NAO robot

Finally, we conduct a few experiments on multi-joint movements. The first multi-joint experiment was shoulder joint horizontal flexion/extension. This motion is a combination of shoulder joint vertical flexion/extension and abduction/adduction. The experimental results are shown Figure 15 and Figure 16. Our second multi-joint experiment was diagonal reaching which is a combination of three joints motion- shoulder vertical flexion/extension and abduction/adduction, and elbow flexion/extension. The results of this experiment are shown in Figure 17 and Figure 18.

Figure 15

Arm position of human operator and NAO robot during Shoulder joint horizontal flexion/extension motion during teleoperation

Figure 16

Shoulder joint horizontal flexion/extension teleoperation. comparison between joint angles from Kinect and NAO robot

Figure 17

Arm position of human operator and NAO robot for diagonal reaching motion during teleoperation

Figure 18

Diagonal reaching teleoperation, comparison between joint angles from Kinect and NAO robot

From these experimental results, it is clear that there has a lagging between the human operator’s upper arm joint angle and that of the NAO robot. However, the pattern of the trajectory followed by the NAO robot was the same as that of the human operator. The objective of the developed control approach is to teleoperate the NAO robot according to the trajectory produced by a human operator. The experimental results evident that the developed control scheme can successfully be deployed to teleoperate the NAO. Therefore, we may conclude that using the developed control approach one can remotely control the NAO as long as they are

connected to the same network. Thus, a therapist can introduce/demonstrate new exercises remotely using the NAO robot.

7 Conclusion

We developed a successful telerehabilitation scheme in which a therapist can introduce new exercises to the patient (or a group of patients) remotely. By doing so, it can facilitate many patients in different places under rehabilitation as long as the NAO and Kinect sensor are connected under the same network. The motion tracking is easy and simple. In this research, a unique inverse kinematic solution was obtained to overcome the singularity problem. Studies show that patients often do not feel comfortable showing their disability to other people, or they do not feel good being in front of other people. The telerehabilitation would be a potential solution in such cases. Note that, this telerehabilitation scheme is the part of a high-level remote rehabilitation scheme in which we will utilize all the joints (5DoFs) of NAO robot and will introduce a remote communication strategy in which we will not require to connect the NAO and the Kinect in the same network.