Usability study of wearable inertial sensors for exergames (WISE) for movement assessment and exercise

Introduction

Recent years have witnessed the emergence of digital health using advanced technologies such as wearable sensors and embedded controllers to enhance access to medical diagnostics and treatments (1,2). Because of an accelerating trend in the number of stroke survivors requiring rehabilitation (3), healthcare services worldwide are considering technological solutions to enhance accessibility to assessment and treatment. For example, virtual therapists and telerehabilitation have been proposed to complement the skills of therapists (4,5). However, some of the challenges faced by these technologies are clinical acceptance, high equipment cost, accuracy, and ease of use (4).

Most tasks constituting activities of daily living have minimum range of motion (ROM) requirements at various joints (6). Restoring the ability to perform activities of daily living in individuals with impaired movement therefore requires clinicians to assess ROM and customize the exercises to each patients’ activity limitations. Commercially available devices such as goniometers (7), inclinometers (8), and videographic methods (9) are used by therapists to assess patient’s ROM in one-on-one clinical settings. Goniometers and inclinometers have limited inter-observer agreement due to variability in positioning the sensors on the patient’s body and can capture motion only for one joint at a time. Videographic methods also show low inter-observer agreement due to differences in camera positions and often require extensive post-acquisition data analysis. Furthermore, these methods are not suitable for remote assessments and individualized treatments, which are essential to enhance accessibility.

The Kinect™ V2 (Microsoft Corp., Redmond, WA, USA) is a popular markerless motion capture system used for gaming (10). The Kinect can provide high definition video output, depth information, and position information for 25 joints of the human body in 3D space (10). Several studies have reported the use of the Kinect sensor in ROM assessment and rehabilitative applications (11-19). Gait analysis and joint angle (JA) orientation measurements using the Kinect has shown varying levels of agreement for different joint segments (11,16,17,19,20). Under ideal conditions, the shoulder and elbow joint ROM measurements from the Kinect show good inter-trial repeatability and correlate with measurements taken with a goniometer (19,21). However, the placement of Kinect is a limiting factor: placing it in front of a subject yields reliable measurements in contrast to placing it on the side (22), but fails to correctly measure elbow flexion-extension accurately from a neutral position (11,22) and cannot measure forearm pronation-supination. Furthermore, a recent literature review on the use of repurposed gaming consoles, such as the Kinect, for neurorehabilitation in target populations reported an inability to provide individualized training as a major limitation (23).

An alternative approach for ROM and rehabilitation assessment applications is to use inertial sensors, which include inertial measurement units (IMU) and magnetic, angular rate, and gravity (MARG) sensors that measure the linear acceleration and angular velocity of a rigid body to which they are attached. Commercially available wearable inertial sensors for motion capture can be used for ROM assessment (24,25), but their use for rehabilitation is limited by the cost of custom data acquisition software, need for user-training, and the extensive data analysis required post-acquisition. Although (26,27) have shown that it is possible to extract absolute orientation of a rigid body from raw measurements of the IMU and MARG sensors, their methods are yet to be extended for use in rehabilitation applications requiring JA measurements.

Telerehabilitation is a branch of emerging medical innovation that permits assessment and treatment of patients remotely. Repurposed off-the-shelf media platforms (e.g., Skype, VSee, etc.) and interactive gaming consoles (e.g., Kinect, PlayStation, Wii, etc.) have been used in prior research efforts for telerehabilitation (28,29). However, most telerehabilitation applications are limited by (I) an inability to capture measurements accurately and with high inter-observer agreement easily in a remote manner and (II) provide individualized coaching asynchronously, i.e., without a live coach. To address these limitations, we designed a wearable inertial sensors for exergames (WISE) system consisting of: (I) an animated virtual coach to deliver virtual instruction for any activity and (II) a subject-model whose movements are animated by real-time sensor measurements using inertial sensors worn by a subject. In this paper, we test the WISE system’s accuracy and usability for assessment of upper limb ROM.

Methods

Design of WISE system

The WISE system consists of five wearable sensor modules affixed to the upper limb segments (Figure 1A). The sensors are labeled to correspond to placement on appropriate limb segments, i.e., sensors marked LF (i.e., left forearm) and RF (i.e., right forearm) are placed above the wrist on the left and right forearms, respectively; sensors marked LA (i.e., left arm) and RA (i.e., right arm) are placed above the elbow on the left and right arms, respectively; and the sensor marked B is placed on the back. Each WISE sensor module (30) consists of a BNO055 MARG sensor interfaced with a Gazell-protocol endowed microcontroller (µC) soldered to a printed circuit board (PCB). The PCB is connected to a lithium-ion battery housed in a 3D-printed wearable enclosure. The µC retrieves absolute orientation measurement from the MARG sensor and wirelessly streams it to a computer. A Unity3D-based exergame interface was developed to use the sensor data for animating a 3D-human model, i.e., the subject-model. The salient features of the application include (I) a sensor calibration user-interface (UI) (Figure 1B), (II) a sensor mounting UI (Figure 1C), (III) a virtual exercise environment (Figure 1D), and (IV) an instructor-programming UI (Figure 1E). Each BNO055 MARG sensor consists of in-built three-axes magnetometer, accelerometer, and gyroscope. The BNO055 sensor requires an initial calibration for streaming absolute orientation (31). A storage and calibration cube was designed and 3D-printed to house the five WISE system modules and simultaneously calibrate all the sensors prior to placement on a subject (Figure 1F). Sensor calibration UI allows intuitive visualization of each WISE module’s calibration status. The sensor mounting UI is a virtual environment that utilizes the data streamed to provide instructions to the user in neutral pose for mounting the sensors on their body to obtain accurate ROM measurements (32). The virtual exercise environment consists of an instructor model performing ROM exercises that are to be performed by the subject. The upper extremity (UE) motion of the subject-model is animated by the real-time data from the WISE modules worn by the subject. The instructor-programming environment allows an instructor to use the WISE modules to record exercises or activities in a flexible manner and replay them as “instructed exercises” performed by the virtual coach (32). We used the instructor-programming environment to record a series of ROM exercises using the WISE system for the testing protocol of this paper.

Figure 1 Salient features of WISE. (A) WISE sensor mounting arrangement shown on a human model in T-pose; (B) calibration UI allowing intuitive and automatic calibration of the WISE modules; (C) sensor mounting UI allowing accurate placement of sensors on the user body; (D) virtual environment—subject-model on left, virtual coach on right, real-time sensor data animates upper limb movement of the subject-model; (E) instructor programming UI provides a virtual coach synced with sensor data to generate individualized exercise routines; and (F) 3D-printed holder to house the WISE modules for calibration. WISE, wearable inertial sensors for exergames; UI, user-interface.

ROM measurements for WISE (30)

Each WISE module streams quaternion data of its orientation relative to the world coordinate frame (ℱ_W), which is represented by the direction of earth’s gravity and the magnetic north pole. Quaternions are four-tuple objects that provide a computationally effective way to represent orientation (33). A quaternion q = (q_w q_x q_y q_z) consists of a scalar part q_w and a vector part [q_x q_y q_z]^T. Three dimensional vectors are a subset of quaternions and a quaternion with its scalar part q_w =0 is termed a vector quaternion. Other forms of orientation representation include Euler angles, axis/angle representation, and rotation matrices (33). Consider the axis/angle representation (d, φ) where d = [d_x d_y d_z]^T is the axis of rotation and φ is the angle of rotation, then the corresponding quaternion describing this rotation is given below.

$$q_d\left(\phi \right)=\left(\begin{array}{cccc}\cos \frac{\phi }{2}& d_x\sin \frac{\phi }{2}& d_y\sin \frac{\phi }{2}& d_z\sin \frac{\phi }{2}\end{array}\right)$$^[1]

A vector $\overline{V}$ expressed in a coordinate frame $F_1\left({}^1\overline{V}\right)$ can be expressed in another coordinate frame $F_2\left({}^2\overline{V}\right)$ by using the quaternion product shown below

$$q\left({}^2\overline{V}\right)= {}^{1}q{}_2\otimes q\left({}^1\overline{V}\right)\otimes {}^{1}q{}_2^{*},q\left(\overline{V}\right)= \left(\begin{array}{cc}0& \overline{V}\end{array}\right)$$^[2]

where “⊗” and “*” denote the quaternion product (27,34) and conjugate, $q\left(\overline{V}\right)$ denotes the vector quaternion of $\overline{V}$, and ${}^{1}q{}_2$ denotes the orientation of $F_2$ relative to $F_1$.

Although quaternions provide an efficient tool for orientation computation, they are unintuitive for interpretation by rehabilitation practitioners. In contrast, Euler angles represent rotations of one coordinate frame relative to another characterized by simple rotations about their principal axes. The joint coordinate system (JCS) framework, proposed by the International Society of Biomechanics, recommends the use of Euler angles for extracting anatomical JAs for ease of use by practitioners (35). The UE JA measurements discussed in (35) utilize the proximal coordinate frame as a reference to describe the angular rotation of the distal coordinate frame, i.e., the shoulder and elbow JA computations use the back and arm WISE modules, respectively, as references. To produce a reference for the shoulder JA computation, the back WISE module’s quaternion is rotated and stored as q_LBref and q_RBref as below:

$$q_{\text{LBref}}=q_{\text{RBref}}=q_{Z_{\text{B}}}\left(-\frac{\pi }{2}\right)\otimes q_{\text{B}}$$^[3]

The sign convention for shoulder JA measurement is as follows: extension (−), flexion (+), adduction (−), abduction (+), external rotation (−), and internal rotation (+). To follow a similar sign convention, the axes of q_LA, q_RA, q_LBref, and q_RBref are flipped [see Figure 2 where (·)^† denotes the quaternions for the flipped coordinate frames]. This axis flipping operation is performed by converting the quaternions to rotation matrices (33), followed by multiplication of −1 to the row vectors corresponding to the coordinate axes that are to be flipped. Throughout the axes flipping operation, care is taken to ensure that the rotations follow the right-hand rule. The relative quaternions between the shoulder and back’s reference coordinate frame are computed as below.

$$q_{\text{LS}}=q_{{\text{LBref}}^{†}}^{*}\otimes q_{{\text{LA}}^{†}}$$^[4]

$$q_{\text{RS}}=q_{{\text{RBref}}^{†}}^{*}\otimes q_{{\text{RA}}^{†}}$$^[5]

To obtain the JA of the left and right shoulders, Wu et al. (35) suggest the use of the Y − X − Y' Euler angle convention. Since the five WISE modules were assigned sensor coordinate frames consistently (Figure 2), the appropriate placement and orientation of the LA and RA modules on arms required a slight adaptation of the framework of (35), leading to the use of Y − Z − Y' Euler angle convention (33) to obtain the JA of the left and right shoulders using the relative quaternions q_LS and q_RS. The quaternion to Euler angle conversion in the Y – Z − Y' framework produces angles θ_Y, θ_Z, and θ_Y + θ_Y' that represent the shoulder plane, shoulder elevation, and shoulder internal-external rotation angles, respectively. In the JCS framework, the shoulder elevation angle θ_Z relates to shoulder flexion-extension when θ_Y ≈90° and to shoulder abduction-adduction when θ_Y ≈0° (35).

Figure 2 Coordinate frames and WISE module axes references for computation of shoulder and elbow joint angle orientations. WISE, wearable inertial sensors for exergames.

The elbow JA computation utilizes the JCS framework to compute the ROM for flexion-extension and pronation-supination with the WISE system. Identical to the procedure of shoulder JCS ROM computation, q_LA and q_RA are used to create references q_LAref and q_RAref as below:

$$q_{\text{LAref}}=q_{Y_{\text{LA}}}\left(\frac{\pi }{2}\right)\otimes q_{\text{LA}}$$^[6]

$$q_{\text{RAref}}=q_{Y_{\text{RA}}}\left(-\frac{\pi }{2}\right)\otimes q_{\text{RA}}$$^[7]

To conserve the standard sign convention of elbow joint movements flexion (+), pronation (+), extension (−), and supination (−), the coordinate axes of q_LAref, q_RAref, q_LF, and q_RF are flipped similar to the procedure for shoulders. The relative quaternions between the elbow and shoulder’s reference coordinate frame are computed as below:

$$q_{\text{LE}}=q_{{\text{LAref}}^{†}}^{*}\otimes q_{{\text{LF}}^{†}}$$^[8]

$$q_{\text{RE}}=q_{{\text{RAref}}^{†}}^{*}\otimes q_{{\text{RF}}^{†}}$$^[9]

To obtain the elbow JA, Wu et al. (35) suggest the use of the Z − X − Y Euler angle convention. Thus, using the relative quaternions q_LE and q_RE with the quaternion to Euler angle conversion in the Z − X − Y framework produces elbow JA θ_Z, θ_X, and θ_Y that correspond to the flexion-extension, carrying, and pronation-supination angles, respectively. The carrying angle is the angle between the humerus and the ulna, which is constant depending on the gender of the person ranging between 8° to 20° (36,37). For further details on the mechanics of ROM measurements for WISE see (30).

ROM measurements for Kinect

The Kinect provides a total of 25 joint coordinates of a human in 3D workspace extracted from the image and depth information (10). However, measurements from the Kinect do not yield sufficient information to characterize movement in the JCS framework. Specifically, by using the shoulder, elbow, and wrist positions obtained from the Kinect, it is not possible to resolve the three principal axes (35) corresponding to each of the shoulder and elbow joints. Thus, instead of using the JCS framework for computing shoulder abduction-adduction and flexion-extension, we adapt the vector projection approach of (19).

To illustrate the JA computation from the Kinect, we use characters with bar accent to denote vectors, e.g., “$\overline{F}$” and hat accent “$\widehat{F}$” to denote coordinates of a point in the 3D space. We begin by constructing a subject centric body coordinate frame for the UE (Figure 3) by using the real-time joint positions obtained from the Kinect to compute the JA. The sagittal plane (S_P), coronal plane (C_P), and transverse plane (T_P) divide the human body into the left-right, frontal-rear, and top-bottom halves, respectively. The vectors ${\overline{S}}_N$, ${\overline{C}}_N$, ${\overline{T}}_N$ and represent vectors normal to the sagittal, coronal, and transverse planes, respectively. The LA and RA vectors are constructed by subtracting the proximal joint coordinate (shoulder) from the distal joint coordinate (elbow) as below.

Figure 3 Joint positions, joint vectors, and anatomical planes used to calculate joint angles using the Kinect.

$${\overline{F}}_S={\widehat{F}}_E-{\widehat{F}}_{\text{S}},\forall F\in \left\{L,R\right\}$$^[10]

The shoulder JAs are defined as follows: (I) shoulder flexion-extension (θ_FE) is the movement of the arm vector ${\overline{F}}_S$ in a plane parallel to the sagittal plane and (II) shoulder abduction-adduction (θ_BD) is the movement of the arm vector in a plane parallel to the coronal plane. The angle calculation for these movements of the left shoulder are performed as below.

$${\theta }_{\text{LFE}}=\text{atan}2\left(-{\overline{L}}_{\text{S}}\cdot {\overline{C}}_{\text{N}},{\overline{L}}_{\text{S}}\cdot {\overline{T}}_{\text{N}}\right)$$^[11]

$${\theta }_{\text{LBD}}=\text{atan}2\left(-{\overline{L}}_{\text{S}}\cdot {\overline{S}}_{\text{N}},{\overline{L}}_{\text{S}}\cdot {\overline{T}}_{\text{N}}\right)$$^[12]

The forearm vectors are constructed from the joint coordinates of elbows and wrists, similar to Eq. [10]. The left elbow flexion-extension α_LFE angle computation is performed as below.

$${\alpha }_{\text{LFE}}={\cos }^{-1}\left({\overline{L}}_{\text{S}}\cdot {\overline{L}}_{\text{E}}\right)$$^[13]

Note that the forearm pronation-supination angle cannot be computed from the Kinect data due to the lack of sufficient information.

The shoulder internal-external rotation (θ_IE) of the LA or RA cannot be computed with the information of shoulder vectors ${\overline{L}}_S$ and ${\overline{R}}_S$. Hence, we utilize the information of the elbow vectors ${\overline{L}}_E$ and ${\overline{R}}_E$ to compute this angle by projecting it on the transverse plane as below, similar to the procedure in (19).

$${\theta }_{\text{LIE}}=\text{atan}2\left({\overline{C}}_{\text{N}}\cdot {\overline{L}}_{\text{E}},{\overline{S}}_{\text{N}}\cdot {\overline{L}}_{\text{E}}\right),{\alpha }_{\text{LFE}}\ge 30°$$^[14]

Thus, the shoulder internal-external rotation calculation works only if the elbow is flexed beyond 30°. Finally, the computation for the right side of the body also utilizes similar equations as above and care is taken to ensure that extension, adduction, and external rotation are denoted as (−). Figure 3 illustrates all the UE joint coordinates and vectors used for the calculation of JA with the Kinect data.

Matching measurements from WISE and Kinect systems

As delineated above, unlike with the WISE system, measurements from the Kinect cannot use the JCS framework. Thus, the WISE system is modified to facilitate one-on-one comparison with the Kinect-based measurements of the computation of shoulder flexion-extension {Eq. [11]} and abduction-adduction {Eq. [12]}. The modified approach, adapted from (19), is outlined below.

The principal axes of the WISE module B $\left(X_B\text{-}{Y}_B\text{-}{Z}_B\right)$ are used to recreate the transverse, sagittal, and coronal planes on the human body such that the X_B, Y_B, and Z_B axes are normal to the transverse, sagittal, and coronal planes, respectively. The $\left(X_B,Y_B,Z_B\right)$ axes represented in the sensor coordinate frame ℱ_S are transformed to $\left({\tilde{X}}_B,{\tilde{Y}}_B,{\tilde{Z}}_B\right)$ in ℱ_W using Eq. [2]. The shoulder flexion-extension angle θ_FE and abduction-adduction angle θ_BD are computed for the LA as below.

$${\theta }_{\text{LFE}}=\text{atan}2\left(Y_{\text{LA}}\cdot {\tilde{Z}}_{\text{B}},Y_{\text{LA}}\cdot {\tilde{X}}_{\text{B}}\right)$$^[15]

$${\theta }_{\text{LBD}}=\text{atan}2\left(Y_{\text{LA}}\cdot {\tilde{Y}}_{\text{B}},Y_{\text{LA}}\cdot {\tilde{X}}_{\text{B}}\right)$$^[16]

The angles for the right side are computed similar to Eq. [15] and Eq. [16] so that the signs of flexion and abduction are (+).

Subject recruitment and data collection

Seventeen healthy subjects (11 male and 6 female) were recruited for the study in the following age groups: 18–24 years (n=8), 25–34 years (n=6), 35–44 years (n=1), 45–54 years (n=1), and 55–64 years (n=1). The participants signed informed consent as approved by the NYU Institutional Review Board (IRB-FY2019-3426). Data collection took place at the Mechatronics, Controls, and Robotics Lab, NYU Tandon School of Engineering. The participants wore the WISE modules without any obstruction to their active ROM and stood at a distance of six feet in front of the Kinect with a white screen on their back to mitigate any changes to the ambient lighting. Video tutorials recorded with a 3D animated human model served as the virtual coach displayed on a television screen. The virtual coach demonstrated each exercise first and then instructed the subjects to perform the demonstrated exercises along with the coach for eight trials of the ten ROM exercises. At the end of the session, subjects were presented with a system usability scale (SUS) questionnaire that consisted of ten questions (with five positive and five negative statements). The questionnaire sought respondents’ opinion about using the virtual coach for ROM exercises on a five-point Likert scale.

Using MATLAB version 2019a (MathWorks, Inc., Natick, MA, USA), a routine was created for real-time data acquisition to compare the WISE and Kinect measurements. The experimental setup is illustrated in Figure 4. Three user interfaces were created: (I) a video display for subjects to view the virtual coach (Figure 4B); (II) Kinect’s video stream superimposed with numerical values of the JAs computed from the Kinect and WISE systems (Figure 4E); and (III) animated plots for JA visualization from the Kinect and WISE systems. Each subject performed the ROM exercises in the following sequence: (I) left shoulder flexion-extension (θ_LFE), (II) left shoulder abduction-adduction (θ_LBD), (III) left elbow flexion-extension from neutral-pose $\left({\alpha }_{{\text{LFE}}_1}\right)$, (IV) left elbow flexion-extension with 90° shoulder abduction $\left({\alpha }_{{\text{LFE}}_2}\right)$, (V) left shoulder internal-external rotation with 90° elbow flexion (θ_LIE), (VI) right shoulder flexion-extension (θ_RFE), (VII) right shoulder abduction-adduction (θ_RBD), (VIII) right elbow flexion-extension from neutral-pose $\left({\alpha }_{{\text{RFE}}_1}\right)$, (IX) right elbow flexion-extension with 90° shoulder abduction $\left({\alpha }_{{\text{RFE}}_2}\right)$, and (X) right shoulder internal-external rotation with 90°elbow flexion (θ_RIE). The complete procedure lasted approximately 20 minutes for each subject. The JA data computed from the Kinect and WISE systems were saved in a text file for post-processing.

Figure 4 Schematic representation of subject testing. (A) Subject wearing WISE modules and standing in front of the Kinect; (B) ROM exercises instructed by the virtual coach displayed on a TV screen; (C) Kinect markerless motion capture system; (D) tripod stand for mounting Kinect; (E) MATLAB interface for sampling data from the Kinect and WISE systems. WISE, wearable inertial sensors for exergames; ROM, range of motion.

Data processing and statistical analysis

The data was analyzed using MATLAB with the command findpeaks to determine the peak ROM measured using the Kinect $\left(p_{{\text{ROM}}_{\text{K}}}^{\text{E}}\right)$ and WISE $\left(p_{{\text{ROM}}_{\text{W}}}^{\text{E}}\right)$ systems, where superscript “E” represents specific ROM exercises. The temporal signal for each movement was then spliced to separate the trials (see Figure 5). The ROM from the two systems were compared by determining the root mean square error (RMSE) $\left(R_{{\text{S}}_i}^{\text{E}}\right)$ for each trial, where subscripts “_S” and “_i” represent the subject number and trial number, respectively. The extracted peaks and RMSE were used to compute the mean error relative to the peak ROM (${\mu }_{\text{R}/\text{p}}=\frac{1}{n}{\displaystyle {\sum }_{i=1}^n\frac{R_{{\text{S}}_i}^{\text{E}}}{{\left(p_{{\text{ROM}}_{\text{K}}}^{\text{E}}\right)}_i}}$, where n=6_trials×17_subjects) (20), as well as the mean (μ) and standard deviation (σ) of the RMSE of trials two to seven across all subjects.

Figure 5 Kinect and WISE system data obtained from the right arm movements. WISE, wearable inertial sensors for exergames.

MATLAB environment’s data acquisition used a line-by-line program execution which caused a systematic lag/lead between the Kinect and WISE measurements. To mitigate the temporal error, we applied dynamic time warping (DTW) (38) to the Kinect and WISE measurements using the MATLAB command dtw similar to the procedure outlined in (14). Prior to applying DTW, the time series JA signals were scaled to [−1,1] by using the corresponding peak values for the Kinect and WISE measurements. Following the application of DTW, the data were rescaled by the peak values used previously for scaling. The peak and RMSE calculations described above were repeated using the JA output of DTW to compute the mean (μ_DTW) and standard deviation (σ_DTW) of the RMSE for each trial.

The Bland-Altman statistic establishes the agreement between two measurement systems by computing the limits of agreement (LOA) (39). Following the application of DTW, the error signal between the Kinect and WISE measurement systems exhibited high kurtosis and skewness indicating a non-normal distribution. Thus, Kimber’s outlier rejection technique (40) was applied to reject outlier data points beyond [Q₁ − γ(M − Q₁), Q₃ + γ(Q₃ − M)], where Q₁, Q₃, and M denote the first quartile, third quartile, and median, respectively. The multiplier γ =1.5 is a commonly used parameter for outlier rejection (40), thus it was used for rejecting the outliers in the Kinect and WISE measurements. The outputs of the outlier rejection procedure also exhibited non-normal distributions in the error signal. Thus, the Bland-Altman test for non-parametric signals, which defines the LOA as the median ±1.45 times the interquartile range (i.e., M ±1.45× IQR), was applied to the remainder of the Kinect and WISE data after the outlier rejection. The peaks obtained from the Kinect and WISE measurements after DTW were also used to compute the intra-class correlation coefficients (ICC). Two methods were used to determine (I) the test-retest consistency [ICC(C,1)] of the Kinect (ICC_K) and WISE (ICC_W) between trials and (II) the absolute agreement [ICC(A,1)] between the measurements of ROM peaks of the Kinect and WISE (ICC_K/W) (41). The SUS responses for ten questions obtained from the subjects were analyzed for reliability using the Cronbach’s alpha (42,43) and the final SUS score was computed using the method described in (44).

Results

The mean and standard deviations of the RMSE between the JA measurements obtained from the Kinect and WISE systems for 17 subjects and 10 ROM exercises, Bland-Altman LOA, ICC for each device (ICC_K, ICC_W), and the absolute agreement between the two devices (ICC_K/W) are shown in Table 1. The mean of the RMSE shows that the ROM measurements in the coronal, transverse, and sagittal planes had errors of less than 9°, 10°, and 12°, respectively. After the application of DTW, the mean RMSE errors in the coronal, transverse, and sagittal planes decreased to less than 8°, 8°, and 10°, respectively. The Bland-Altman LOA, with 95% confidence intervals, for ROM measurements in the coronal, transverse, and sagittal planes were determined to be in the range of (−12.0°, 10.0°), (−9.5°, 10°), and (−14.0°, 18.0°), respectively. The ICC for the Kinect (ICC_K) and WISE (ICC_W) systems indicated very good repeatability within each system especially for RA movements and slightly lower for left arm movements. The ICC was lowest for the left shoulder internal-external rotation. The ICC_K/W, which compared the consistency between the Kinect and the WISE systems, showed moderate to very good agreement for all movements except the left and right elbow flexion-extension from the neutral-pose.

Table 1 The mean and standard deviation of the root mean square error for each trial before and after dynamic time warping between the Kinect and WISE measurements, mean error relative to peak ROM, Bland-Altman limits of agreement, intra-class correlation coefficients for consistency within each measurement system and between the two systems
Full table

The Bland-Altman plots (i.e., mean versus difference) for the DTW-processed signals of the Kinect and WISE systems for the 10 ROM exercises are presented in Figure 6. The measurements for coronal plane ROM exercises (I) left and right elbow flexion-extension with 90° shoulder abduction and (II) right shoulder abduction-adduction produced Bland-Altman LOA within ±10°. However, for left shoulder abduction-adduction measurements the LOA was (−12.0°, 9.4°), which is slightly higher than the ±10° range. The transverse plane ROM exercises for the left and right shoulder internal-external rotation with 90° elbow flexion were also within the ±10° acceptance limits.

Figure 6 Bland-Altman plots for all 10 ROM exercises. WISE, wearable inertial sensors for exergames; ROM, range of motion.

The SUS response percentages are plotted in a bar chart as shown in Figure 7. The Cronbach’s alpha for the five-point SUS scale for the ten questions was computed to be above 0.7, indicating acceptable reliability. A majority of the users found the virtual coach-based ROM tutoring system to be easy to use (82% strongly agreed) and well-integrated (82% agreed or strongly agreed) and reported that they would be confident in using it (88% strongly agreed) and would like to use it (82% agreed or strongly agreed). Overall, the SUS score showed relatively high third and first quartile scores of 97.5 and 82.5, respectively, with the interquartile range of 15 and the minimum score of 65, suggesting that the subjects were interested in using the animated virtual coach for the guided ROM exercises.

Figure 7 Horizontal bar chart of responses (in percentage) of the system usability scale.

Discussion

Accurate assessment of ROM limitations and compliance monitoring of exercises to restore movement are necessary to assess and tailor treatments for individuals with motor deficits across care settings. In this paper, we presented the overview of the WISE system consisting of: (I) an animated virtual coach to deliver virtual instruction for any activity and (II) a subject-model whose movements are animated by the real-time sensor measurements using inertial sensors worn by a subject. We compared the accuracy and usability of the WISE system for assessment of upper limb ROM with the Kinect, which has been used extensively in prior studies as a markerless motion capture system for exergames and rehabilitation.

The results show moderate to very good within-device agreement for each of the measurement systems. The discrepancy between the two devices was within ±10° for most of the ROM exercises. The between-device agreement was moderate to very good in the coronal and transverse planes for the following ROM exercises: (I) shoulder abduction-adduction, (II) elbow flexion-extension with the shoulder abducted at 90°, and (III) shoulder internal-external rotation. Even though, there are no quantified clinical acceptance limits for the ROM assessment, prior literature (8) has suggested ±10° as acceptable LOA for the Bland-Altman statistic. Furthermore, the RMSE and Bland-Altman LOA results suggest the concurrence between the two measurement systems was best in the coronal plane. However, the RMSE for exercises in the sagittal plane, i.e., (I) elbow flexion-extension from neutral-pose and (II) shoulder flexion-extension showed greater discrepancy between the two devices. These discrepancies persisted despite the adaptation of the Kinect-based vector projection method (19) for computing JAs with the WISE system. This can be explained by the problem of joint occlusion during movements in the sagittal plane when the Kinect is placed in front of the subject (22). Alternatively, when the elbow flexion-extension exercise was performed with the shoulder abducted at 90°, the occlusion did not occur and resulted in the least discrepancy between the two systems. Next, the ROM exercises for internal-external rotation in the transverse plane were modified by the introduction of 90° elbow flexion, which enabled the use of elbow vector obtained from the Kinect measurements for the computation of shoulder internal-external rotation angle. Furthermore, the forearm pronation-supination angles could not be computed from the Kinect measurements. Although Kinect has been used extensively for exergames and rehabilitation, above results suggest that JA measurements from such markerless motion capture devices lack the ability to resolve motion in three planes of movement for each joint (i.e., shoulder and elbow) as required by the JCS framework. In contrast, the WISE system provides a robust integration of measurements from multiple wearable sensors for the shoulder and elbow joints allowing continuous real-time measurements of JAs in three planes for each joint. Finally, the SUS scores suggest that subjects were interested in using the animated virtual coach for the guided ROM exercises.

In a prior study, we developed a method to remotely assess grasping performance using real-time data (45) in patients with multiple sclerosis (46). Using the WISE system, a similar telerehabilitation intervention can be developed for patients in need of UE ROM assessment and rehabilitation exercises. In such a scenario, the therapists can use the virtual coach to provide an individualized battery of exercises, enabling patients to perform exercises at home while asynchronous measurements using the WISE platform can capture and transmit the information to the therapist. The granular ROM data in all three planes can be useful to bridge the gap between laboratory research on motion analysis and translation to clinical practice. Although the current implementation of the WISE system was restricted to a computer-based interface, in our prior work we have demonstrated the feasibility of interfacing medical devices with smartphones (47). In a similar vein, the WISE system can be interfaced to mobile devices such as tablets and smartphones to render a portable mHealth system. Future work will consider the use of the WISE virtual coach and a guided mounting interface for the sensors, as well as feedback systems that enable the virtual coach to tailor exercises based on the data acquired from the sensors.

Acknowledgments

The authors acknowledge the members of the Mechatronics, Controls, and Robotics Lab for assisting with subject recruitment and Mr. Hassam Khan Wazir for the selection of MARG sensors.

Funding: This work is supported in part by the National Science Foundation DRK-12 Grant DRL-1417769, RET Site Grant EEC-1542286, and ITEST Grant DRL-1614085; NY Space Grant Consortium Grant 48240-7887; and Translation of Rehabilitation Engineering Advances and Technology (TREAT) grant NIH P2CHD086841.

Footnote

Provenance and Peer Review: This article was commissioned by the Guest Editor (Mei R. Fu) for the series “Real-Time Detection and Management of Chronic Illnesses” published in mHealth. The article was sent for external peer review organized by the Guest Editor and the editorial office.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/mhealth-19-199). The series “Real-Time Detection and Management of Chronic Illnesses” was commissioned by the editorial office without any funding or sponsorship. PR and VK have patented technology for a Game-Based Sensorimotor Rehabilitator through NYU. PR reports consulting for Mirrored Motion Works, Inc., outside the submitted work. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The participants signed informed consent as approved by the NYU Institutional Review Board (IRB-FY2019-3426).

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Edwards J. Wireless sensors relay medical insight to patients and caregivers. IEEE Signal Processing Magazine 2012;29:8-12. [Crossref]
2. Kim J, Campbell AS, de Ávila BE, et al. Wearable biosensors for healthcare monitoring. Nat Biotechnol 2019;37:389-406. [Crossref] [PubMed]
3. Benjamin EJ, Muntner P, Alonso A, et al. Heart disease and stroke statistics—2019 update: A report from the American Heart Association. Circulation 2019;139:e56-528. [Crossref] [PubMed]
4. Burdea GC. Virtual rehabilitation—Benefits and challenges. Methods Inf Med 2003;42:519-23. [Crossref] [PubMed]
5. van Vuuren S, Cherney LR. A virtual therapist for speech and language therapy. In: Bickmore T, Marsella S, Sidner C. editors. Intelligent Virtual Agents. Springer, Cham, 2014;8637:438-48.
6. Gates DH, Walters LS, Cowley J, et al. Range of motion requirements for upper-limb activities of daily living. Am J Occup Ther 2016;70:7001350010p1-7001350010p10.
7. Gajdosik RL, Bohannon RW. Clinical measurement of range of motion. Review of goniometry emphasizing reliability and validity. Phys Ther 1987;67:1867-72. [Crossref] [PubMed]
8. de Winter AF, Heemskerk MA, Terwee CB, et al. Inter-observer reproducibility of measurements of range of motion in patients with shoulder pain using a digital inclinometer. BMC Musculoskelet Disord 2004;5:18. [Crossref] [PubMed]
9. Khadilkar L, MacDermid JC, Sinden KE, et al. An analysis of functional shoulder movements during task performance using Dartfish movement analysis software. Int J Shoulder Surg 2014;8:1-9. [Crossref] [PubMed]
10. Terven JR, Córdova-Esparza DM. Kin2. A Kinect 2 toolbox for MATLAB. Science of Computer Programming 2016;130:97-106. [Crossref]
11. Bonnechère B, Jansen BMP, Salvia P, et al. What are the current limits of the Kinect sensor? In: Proceedings International Conference on Disability, Virtual Reality, and Associated Technologies, Laval, France, 2012:287-94.
12. Sin H, Lee G. Additional virtual reality training using Xbox Kinect in stroke survivors with hemiplegia. Am J Phys Med Rehabil 2013;92:871-80. [Crossref] [PubMed]
13. Kurillo G, Han JJ, Obdržálek S, et al. Upper extremity reachable workspace evaluation with Kinect. Stud Health Technol Inform 2013;184:247-53. [PubMed]
14. Kurillo G, Chen A, Bajcsy R, et al. Evaluation of upper extremity reachable workspace using Kinect camera. Technol Health Care 2013;21:641-56. [Crossref] [PubMed]
15. Kitsunezaki N, Adachi E, Masuda T, et al. KINECT applications for the physical rehabilitation. In: 2013 IEEE International Symposium on Medical Measurements and Applications (MeMeA), Gatineau, Canada, 2013:294-9.
16. Bonnechère B, Jansen B, Salvia P, et al. Validity and reliability of the Kinect within functional assessment activities: Comparison with standard stereophotogrammetry. Gait Posture 2014;39:593-8. [Crossref] [PubMed]
17. Pfister A, West AM, Bronner S, et al. Comparative abilities of Microsoft Kinect and Vicon 3D motion capture for gait analysis. J Med Eng Technol 2014;38:274-80. [Crossref] [PubMed]
18. Choppin S, Lane B, Wheat J. The accuracy of the Microsoft Kinect in joint angle measurement. Sports Technology 2014;7:98-105. [Crossref]
19. Lee SH, Yoon C, Chung SG, et al. Measurement of shoulder range of motion in patients with adhesive capsulitis using a Kinect. PLoS One 2015;10:e0129398. [Crossref] [PubMed]
20. Fern'ndez-Baena A, Susín A, Lligadas X. Biomechanical validation of upper-body and lower-body joint movements of Kinect motion capture data for rehabilitation treatments. In: 2012 Fourth International Conference on Intelligent Networking and Collaborative Systems, Bucharest, Romania, 2012:656-61.
21. Hawi N, Liodakis E, Musolli D, et al. Range of motion assessment of the shoulder and elbow joints using a motion sensing input device: A pilot study. Technol Health Care 2014;22:289-95. [Crossref] [PubMed]
22. Huber ME. Validity and reliability of Kinect skeleton for measuring shoulder joint angles: A feasibility study. Physiotherapy 2015;101:389-93. [Crossref] [PubMed]
23. Knippenberg E, Verbrugghe J, Lamers I, et al. Markerless motion capture systems as training device in neurological rehabilitation: A systematic review of their use, application, target population and efficacy. J Neuroeng Rehabil 2017;14:61. [Crossref] [PubMed]
24. Cuesta-Vargas AI, Galán-Mercant A, Williams JM. The use of inertial sensors system for human motion analysis. Phys Ther Rev 2010;15:462-73. [Crossref] [PubMed]
25. El-Gohary M, McNames J. Shoulder and elbow joint angle tracking with inertial sensors. IEEE Trans Biomed Eng 2012;59:2635-41. [Crossref] [PubMed]
26. Bachmann ER, Duman I, Usta UY, et al. Orientation tracking for humans and robots using inertial sensors. In: Proceedings IEEE International Symposium on Computational Intelligence in Robotics and Automation. CIRA'99 (Cat. No.99EX375), Monterey, CA, USA, 1999:187-94.
27. Madgwick SOH, Harrison AJL, Vaidyanathan R. Estimation of IMU and MARG orientation using a gradient descent algorithm. In: 2011 IEEE International Conference on Rehabilitation Robotics, Zurich, Switzerland, 2011:1-7.
28. Jonsdottir J, Bertoni R, Lawo M, et al. Serious games for arm rehabilitation of persons with multiple sclerosis: A randomized controlled pilot study. Mult Scler Relat Disord 2018;19:25-9. [Crossref] [PubMed]
29. Caggianese G, Cuomo S, Esposito M, et al. Serious games and in-cloud data analytics for the virtualization and personalization of rehabilitation treatments. IEEE Transactions on Industrial Informatics 2019;15:517-26. [Crossref]
30. RajKumar A, Vulpi F, Bethi SR, et al. Wearable inertial sensors for range of motion assessment. IEEE Sensors Journal 2020;20:3777-87. [Crossref] [PubMed]
31. GmbH BS. BNO055 Intelligent 9-axis Absolute Orientation Sensor [Internet]. 2016. Available online: https://cdn-learn.adafruit.com/assets/assets/000/036/832/original/BST_BNO055_DS000_14.pd
32. Bethi SR, RajKumar A, Vulpi F, et al. Wearable inertial sensors for exergames and rehabilitation. In: IEEE Engineering in Medicine and Biology Conference. 2020. [In press].
33. Spong M, Hutchinson S, Vidyasagar M. Robot Modeling and Control. New York: Wiley, 2006:1-478.
34. Chen X. Human Motion Analysis with Wearable Inertial Sensors. PhD Dissertation. University of Tennessee, Knoxville; 2013.
35. Wu G, van der Helm FCT. ISB recommendation on definitions of joint coordinate systems of various joints for the reporting of human joint motion—Part II: shoulder, elbow, wrist and hand. J Biomech 2005;38:981-92. [Crossref] [PubMed]
36. An KN, Morrey BF, Chao EY. Carrying angle of the human elbow joint. J Orthop Res 1984;1:369-78. [Crossref] [PubMed]
37. Paraskevas G, Papadopoulos A, Papaziogas B, et al. Study of the carrying angle of the human elbow joint in full extension: A morphometric analysis. Surg Radiol Anat 2004;26:19-23. [Crossref] [PubMed]
38. Sakoe H, Chiba S. Dynamic programming algorithm optimization for spoken word recognition. In: IEEE Transactions on Acoustics, Speech, and Signal Processing, 1978;2643-9.
39. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. International Journal of Nursing Studies 2010;47:931-6. [Crossref]
40. Kimber AC. Exploratory data analysis for possibly censored data from skewed distributions. Journal of the Royal Statistical Society. Series C 1990;39:21-30. [Crossref]
41. McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychol Methods 1996;1:30-46. [Crossref]
42. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika 1951;16:297-334. [Crossref]
43. Wessa P. Cronbach Alpha (v1.0.5) Free Statistics Software (v1.2.1) [Internet]. 2017. Available online: https://www.wessa.net/rwasp_cronbach.wasp/
44. Brooke J. SUS: A “quick and dirty” usability scale. In: Jordan PW, Thomas B, Weerdmeester BA,et al. editors. Usability Evaluation in Industry. London: Taylor and Francis, 1996:189-94.
45. Raj Kumar A, Bilaloglu S, Raghavan P, et al. Grasp rehabilitator: A mechatronic approach. In: Proceedings of the Design of Medical Devices Conference, 2019:V001T03A007.
46. Feinberg C, Shaw M, Palmeri M, et al. Remotely supervised transcranial direct current stimulation (RS-tDCS) paired with a hand exercise program to improve manual dexterity in progressive multiple sclerosis: A randomized sham controlled trial. Neurology 2019;92:P5.6-009.
47. RajKumar A, Arora C, Katz B, et al. Wearable smart glasses for assessment of eye-contact behavior in children with autism. In: Proceedings of the Design of Medical Devices Conference, 2019:V001T09A006.