2024, Number 1
Methodology for muscle identification in gesture recognition using machine learning methods
Full text
10.35366/113828
Language: English
References: 33
Page: 29-41
PDF size: 425.13 Kb.
ABSTRACT
Human-Machine-Interfaces (HMIs) can use surface electromyography (sEMG) signals to control equipment that assists disabled people in their activities of daily living (ADLs). The use of sEMG signals in HMIs is currently the subject of extensive research. However, some drawbacks of previous research are that weight loads that directly impact sEMG signals, movement velocities, electrode positioning, and a criterion for selecting sEMG features are not considered for the best performance in HMI. Therefore, the article's main contribution is the presentation of a methodology that allows identifying the muscles and features that have the most significant contribution in sEMG-based gesture recognition, considering electrode positioning and avoiding compensatory movements. The article highlights how load weights affect sEMG signals and how principal component analysis determines the best sEMG features for gesture classification. We compared seventeen machine learning classifier models for classifying four upper limb movements based on decision trees, support vector machines, k-Nearest Neighbors, and ensembled methods classifier models. The results show that the signal square integral and Mean Frequency features of sEMG make it possible for classifiers to get an accuracy of above 90%.INTRODUCTION
Surface electromyography (sEMG) has been used as input for human machine interfaces (HMIs) to recognize the user's body movements and translate them into machine commands. Some applications of sEMG-based HMI in upper limb rehabilitation include bionic hands,1,2 rehabilitation devices,3 and assistive devices.4 For an sEMG signal to be used on an HMI, a classification process, better known as pattern recognition, must be done. The pattern recognition process has three fundamental stages: signal pre-processing, feature extraction, and classifier model training.5 Signal pre-processing includes hardware and software denoising, full-wave rectification, and smoothing to get the sEMG signal envelope.6 Feature extraction plays an essential role in classification accuracy and feasibility. Several time-domain, frequency-domain, and time-frequency-domain features have been used to analyze sEMG signals.7 Due to the vast number of available features, methods such as principal component analysis (PCA) have been used to reduce feature space dimensionality.8,9 On occasions, computationally simpler classification methods are preferred over more accurate but more complex ones due to their low processing time, making them more feasible to be applied in real-time.10,11 For instance, time-domain features are the most straightforward feature extraction technique since no transformation is required and are preferred over frequency-domain and time-frequency-domain features.12,13 In the classifier model training stage of the sEMG signals pattern recognition process, we can find several classifier models in the literature, such as Gaussian mixture models,14 support vector machines (SVM),15 k-Nearest Neighbours (KNN),11 decision trees (DTs),16 or ensemble methods.17
Most previous works on pattern recognition systems using sEMG signals focus on motion classification. For example, McDonald et al.18 classified the direction of eight single degrees of freedom (DoF) and multi-DoF movements of the elbow and wrist using nine time-domain features recorded from eight forearm muscles. McDonald et al.18 concluded that using linear discriminant analysis (LDA), only the RMS and MAV are enough for detecting movement direction with 100% accuracy in nine able-bodied subjects.
Sun et al.19 compared four machine learning classifier models (K-Nearest Neighbor (KNN), anfis neural network (ANN), ensembled methods, random forest, and support vector machine (SVM)) to identify four wrist movements, using only four sEMG signals from four wrist muscles. Sun et al. demonstrated that muscle selection impacts classifier model accuracy since they found that extensor carpi radialis (ECR) and flexor carpi ulnaris (FCU) muscles can obtain the best classification results (over 95%) on random forest classification models.
Said et al.8 found that the best classifier model for identifying four hand gestures (hand close, hand open, wave-in, and wave-out) using eight sEMG signals from the forearm using the MAV and standard deviation (SD) features is an SVM model with an 89.93% accuracy.
Few works on pattern recognition systems using sEMG consider motion and sEMG signals generated by different load weights. For example, in the paper by Azis S,20 the pattern recognition of elbow flexion/extension using the sEMG signal of the Biceps Brachii with load classification of three loads, 1, 3 and 7 kg, is presented. Azis S.20 determined that the best classification model is a cubic SVM with a 99% accuracy.
Current trends in sEMG-based gesture recognition focus on deep learning (DL) techniques,21 and their final prediction results are affected by the quality and quantity of sEMG signals. Even though DL techniques have proven to be an effective method to solve sEMG-based gesture recognition problems, they need many electrodes and, therefore, additional hardware and computing power to process signals, which might be inconvenient for developing some prosthetic or rehabilitation devices.
It is important to note that assistive devices such as robotic prostheses, among others, should be designed to be used in activities of daily living since daily living activities involve performing movements under external loads' influence, causing adjustments to be made in motor control.22 Motor control adjustments are reflected by altering spatiotemporal features of sEMG signals. As mentioned above, body movement classification models based on sEMG signals have been evaluated with good results. However, most reviewed studies do not consider how to properly place sEMG electrodes, avoid compensatory movements when performing motion movements that might affect sEMG-based gesture recognition, apply load weights on multiple forearm movements, or consider movement velocities.
Therefore, the article's main contribution is the presentation of a methodology that identifies the muscles and features that have the most significant contribution to the gesture recognition of sEMG signals generated by different load weights and movement velocities. The proposed methodology also considers the positioning of the electrodes and tries to avoid compensatory movements. In addition, this article aims to improve the design and development of future HMIs that are adapted to the needs of each individual.
MATERIAL AND METHODS
This methodology describes collecting and processing sEMG signals generated by various external load conditions to identify single-DoF upper limb movements. The methodology section contains three subsections: 1) The experimental configuration subsection describes the acquisition protocol and mechanical configuration used for collecting the sEMG signals. 2) The signal preprocessing subsection describes how the sEMG signal was cleaned of noise. 3) The gesture recognition subsection describes how the sEMG signal's features were computed. In addition, the subsection on gesture recognition describes how machine learning classifier models were applied and how reduction feature tests were carried out.
EXPERIMENTAL CONFIGURATION
We proposed three measurement configurations to acquire sEMG signals generated by single-DoF upper limb movements under varying external loads (Figure 1). The measurement configurations are based on Von Werder's article.22 In all measurement setups, a pulley machine (Lojer, Finland) is used to apply a constant external load to the understudy joint throughout its entire range of motion. The pulley machine has attached a 4-centimeter-diameter deflection pulley, which allows it to connect to the same axis an elbow flexion/extension, a wrist flexion/extension, or a wrist medial/lateral deviation joint pulley.
In the measurement configurations for elbow flexion/extension and wrist flexion/extension, the participant was seated on a chair parallel to the pulley machine in an upright position. The participant's joint was parallel to the deflection pulley. In the setup for measuring wrist medial/lateral deviation, the participant was seated in front of the pulley machine. The participant was then instructed to align the wrist joint with the medial/lateral deflection pulley (Figure 1).
The participant was instructed to rest their arm on a table in the setups for measuring wrist flexion/extension and medial/lateral deviation. The arm was then secured to the table with Velcro straps to prevent compensatory body movements. The misalignment of the joints between the pulley machine and the participant was prevented by adjusting the deflection pulley machine to the participant's height.
The sEMG signals were collected at a sampling rate of 1000 Hz, using 2 cm diameter bipolar Ag-AgCl electrodes with the commercial sEMG system Datalog (Biometrics, Newport, UK). Since, in this paper, elbow flexion/extension, wrist flexion/extension, and wrist medial/lateral deviation movements were under study, we evaluated the muscles of brachioradialis (BQR), biceps brachii long head (BBLH), triceps brachii long head (TBLH), triceps brachii lateral head (TBLAH), flexor carpi radialis (FCR), FCU, ECR, extensor carpi ulnaris (ECU), due to their relevance for the designated movements by their myotome map. The placement of the BBLH and TBLH electrodes followed the SENIAM guidelines.23 While the TBLAH, FCR, FCU, ECR, ECU, and BQR electrodes were placed following the Perotto book's instructions.24
A twelve-camera "Flex13" Optitrack System (Natural Point, Corvallis, Oregon, USA) was used to acquire the kinematics of the elbow and wrist joints. The kinematics of the elbow and wrist movements were sampled at 100 Hz. The Optitrack system's reflective markers were positioned per the Baseline Upper Body "25" biomechanical marker set.25
The testing procedure was done by a 30-year-old participant who was considered healthy and gave informed consent for testing. The participant performed elbow flexion/extension, wrist flexion/extension, and wrist medial/lateral deviation at the following cadences for each movement: 3 cycles in 2 seconds, 4 cycles in 1 second, 5 cycles in 0.5 seconds, and 6 cycles in 0.5 seconds (where a cycle corresponds when the participant completes a flexion/extension or medial/lateral deviation movement of a corresponding evaluated joint). The participant was instructed to rest for sixty seconds between each cycle set to prevent fatigue. This procedure was done with the following movements and external loads: elbow flexion/extension at 0.5, 2.5, 5 kg, wrist flexion/extension at 0.5, 2.5 kg, and wrist medial/lateral deviation at 0 and 0.5 kg. Loads were chosen based on the participant's capabilities to perform full maneuvers without fatigue.22 The angular velocities of the movement cycles range from 20 to 280°/s. These speeds were determined by comparing a slow movement performed by a participant with a movement disorder to that of a healthy subject.
SIGNAL PRE-PROCESSING
The joint angles were calculated by inverse kinematic analysis of the Optitrack-obtained marker-set trajectories using the OpenSim software26 and a previously described biomechanical model.27 The joint angle signals were resampled from 100 Hz to a 1 kHz frequency, and a second-order Butterworth low-pass filter with a f_3db = 5 Hz cutoff frequency was then used for smoothing the signal.
The sEMG signals were filtered with a fourth order bandpass digital Butterworth filter with a f_3db = 40 - 450 Hz cutoff frequencies. All the sEMG signals were subsequently full wave rectified and normalized to a maximum value of one. To obtain the envelope of the signals, all the sEMG signals were low-pass filtered with a second-order Butterworth filter with a f_3db = 10 Hz cutoff frequency.
By computing the correlation between the resampled joint's angle signal and the sEMG from the BBLH processed signal, synchronization was achieved. The minimum and maximum joint angle signal values were selected to segment and label the data. The used data labels were elbow flexion 0.5 kg, elbow extension 0.5 kg, elbow flexion 2.5 kg, elbow extension 2.5 kg, elbow flexion 5.0 kg, elbow extension 5.0 kg, wrist flexion 0.5 kg, wrist extension 0.5 kg, wrist flexion 2.5 kg, wrist extension 2.5 kg, wrist flexion 5.0 kg, wrist extension 5.0 kg, wrist medial deviation 0.0 kg, wrist lateral deviation 0.0 kg depicts the entirety of signal preprocessing (Figure 2).
GESTURE RECOGNITION
Once the data had been segmented and labeled, mobile windows of 256 ms were applied, and features from the time domain and frequency domain were calculated over the sEMG signals. The features were selected for their frequent use in gesture recognition articles.5,18,19,28 The selected features were the Integrated sEMG (IEMG) (equation 1),28 the MAV (equation 2),5,19,28,29 the simple square integral (SSI) (equation 3),28 the variance (VAR) (equation 4),19,28 the RMS (equation 5),5,19,28,29 the WL (equation 6),5,19,28,29 the median frequency (MDF) (equation 7),19,28 the MNF (equation 8) (See equations).19,28
Where Xn is the sEMG signal in a segment. N is the total length of the sEMG signal. Pj is the power spectrum at a frequency bin j. Finally, fj is the spectrum frequency bin i at frequency. n and j are n - sample the and j - bin, respectively.
Seventeen machine learning classifier models were evaluated using DT, KNN, SVM, and ensembled classifier models with different Kernel functions. The total number of evaluated samples was 320,649 samples. Due to the large dataset size and to compare the models based on the model accuracy on the test dataset and select the best model, the 50% hold-out validation method was used. The feature extraction was done in MATLAB (MATLAB. (2022). version 9.12.0.1927505 (R2022a). Natick, Massachusetts, USA).30 The classification models were trained with the classification learner toolbox from MATLAB. The classifiers were selected based on previous works explained in the introduction section. The measure used to compare the different classifier models was accuracy (Acc). Acc31 is defined in equation 9. Where TP stands for true positive, TN stands for true negative, FP stands for false positives, and FN stands for false negatives.
(Acc) = (∑TP + ∑TN) / (∑TP + ∑TN + ∑FP + ∑FN) (9)
To better understand how the sEMG signal features of time and frequency contribute to motion classification and identify which features contribute the most to a better classification, we proposed five experimental tests that we will explain next.
The frequency domain was ignored in the first test to understand the contribution of time domain features. This evaluation analyzed eight different muscle signals by calculating their time domain features. Forty-eight features were considered, and fourteen classes were considered.
The second experiment used frequency-domain features exclusively as the feature set. The analysis considered sixteen features and fourteen classes were considered.
On the third test, we considered both the time-domain and frequency-domain features. Sixty-four features in total were assessed. Also, in this test, fourteen different classes were considered.
Following a PCA for feature reduction, a fourth experiment was conducted. The IEMG feature of all sEMG signals was used to test the classifier models. Eight features and fourteen classes were considered in this test.
The fifth test is a baseline test. In this test, only the movements without loads are considered. Therefore, only six classes were classified. In this test, we only checked for time-domain features. A complete overview of the proposed methodology is shown in Figure 3.
RESULTS
The classifiers evaluated an average of 22,904 samples for each obtained sEMG signal. Table 1 displays the results from the five tests that compared the classification of elbow flexion/extension, wrist flexion/extension, and medial/lateral wrist deviation movements through sEMG signals produced by various external loads. We can see from test 1, where only time-based features were considered, that SVM, KNN, and ensemble-based classifier models produced the best classification accuracy, reaching 99.8% accuracy. No method could classify data more accurately than 79% in test two, which only considered frequency features. The time-frequency features were applied in test 3, but the results did not improve over those obtained in test 1. Only the IEMG feature was used for classification in test 4. This was done since the PCA analysis shown in Figure 4 indicated that this feature contributed the most to the classification models. Also, Figure 4 shows that the muscles that contribute more than 40% according to the PCA analysis to identify the classification of the evaluated muscles are FCU, FCR, EQU, ECR, and BQR. The results of test 4 indicate a classification that is comparable to that of test 1. Only the movements were classified in test five, and the time features were applied. We observe an improvement in the classification models, like DT-based models, in test five.
Some confusion matrices have been included to understand better how the outcomes in Table 1 performed. For example, the confusion matrix in Figure 5 illustrates how classification models with low classification rates (below 80%) produce a lot of false positives. The confusion maps from test 3 (Figures 6 and 7) were added to compare the behavior of time and time-frequency features.
DISCUSSION
According to the results of the five proposed tests, body movements can be detected through sEMG signals generated by different loads. Additionally, we have pinpointed the features that make it easier to categorize body movements using sEMG signals, thanks to the five proposed tests. The PCA allowed for identifying the sEMG signal feature that contributes the most to the classification of upper limb movements.
The results from the proposed test five showed that KNN and SVM-based classification models produce the best outcomes. These results are consistent with those of Sun et al.,19 where wrist flexion/extension classification scores with over 90% accuracy. These results demonstrate that the classifier is operating as intended. According to test 1 accuracy results compared to the other test (Table 1), time-domain features are the main contributor to the classification of upper limb movements. The results of tests one, two, and three show that frequency-domain features don't have a relevant effect on the classification of upper limb movements through sEMG signals generated by different loads. These previously discussed results are consistent with Sun et al. and McDonald et al.18,19 Comparing the accuracy results from the other classifier models with varying kernel functions showed that kernel functions affect the results of the different classifier models. Therefore, even though KNN and SVM have the best results, the Fine KNN and Fine Gaussian kernel functions score up to 99.8% accuracy. Comparing the positive and negative confusion matrices from test 1 of the DT with a kernel function of a Fine Tree and test 3 SVM model with a coarse Gaussian kernel function (Figure 4 and 6) against test 1, the SVM model with a Fine Gaussian kernel function (Figure 6), it can be seen that the models were scoring 70%, and 80% have a high number of false positives, even in some cases, high misclassification rates are achieved, unlike the SVM model from test 1, where high true positives are achieved. This supports the criteria of McDonald,18 where a classifier model on gesture recognition should have an accuracy of over 90%.
The worst accuracy results were obtained with DT and some Ensemble methods with kernel functions of boosted trees, subspace discriminator, and RUsBoosted Trees, scoring results less than 89%. Therefore, it is not recommended for upper limb motion classification using sEMG signals. Test 3 and PCA analysis shown in Figure 4 show that the FCR, FCU, ECR, ECU, and BQR sEMG signals have the best performance in signal classification. This might be because more movements related to the wrist are evaluated. Also, from PCA analysis, as shown in Figure 2, the best time domain feature that contributes the most to upper limb motion classification is IEMG from all the measured muscles. Comparing accuracy results from test 1, 48-time domain features of the sEMG signal were used to train the models against the accuracy results from test 4, where only 8 IEMG sEMG time domain features are used, can be seen in Table 1, that most classification learner models score higher accuracy. Still, in SVM with Fine Gaussian Kernel, KNN achieves up to a 99.8% accuracy, indicating that feature reduction is possible. The IEMG feature has information on the sEMG signal related to movement velocity and weight load conditions.
From the obtained results, for mapping unknown individuals, a high set of muscles is preferable for scoring high accuracy results; however, a reduction of muscles is possible. This is important since making faster and more comfortable HMIs is possible. Also, the results show that it is possible to classify upper limb motions by sEMG signals generated by low loads. This is important since the development of HMI based on sEMG singles for people with low sEMG activity can be implemented. Although we have only evaluated one healthy participant, we consider that the results obtained are relevant since the age and gender of the participant involved are within 80% of the statistical range of upper limb amputations,32 and furthermore that the design of HMI for upper limb amputees is usually performed on an individualized basis. This is because factors such as stump shape, length, or etiology influence obtaining sEMG signals. That is why one of the main contributions of the article is the methodology that allows identifying the muscles that have the greatest contribution in the classification of movements and the identification of the sEMG feature that allows its better classification.
Through testing of the participant-involved subject using the PCA, it has been identified that the time features have the greatest contribution to the classification of upper limb movements, indicating that it is possible to save processing time in HCI and thus improve its performance. Also, through the tests performed, it has been shown that having many sEMG signals for gesture classification is unnecessary, which can result in the reduction of signal processing time and hardware components, which are useful mainly in prosthetic devices. In the tests performed, we have demonstrated that it is possible to identify three upper limb motor gestures using only the ECR, ECU, FCR, FCU, and BQR muscles.
In test 5, where the gestures were classified without considering the unloaded labels, we can observe that the accuracy values increased compared to the other tests. This could indicate that using loads to identify gestures increases the accuracy of the classifiers compared to not using loads because using loads could increase the muscle frequency and amplitude of the sEMG signal or the values of some features. We currently do not have a direct comparative test, however, we will study this hypothesis in the future.
CONCLUSION
The proposed paper presents a methodology that identifies the muscles and features that contribute most to sEMG-based gesture recognition, considering electrode positioning and avoiding compensatory movements of upper limb movements of the elbow and the wrist. Previous research methodologies presented in the introduction commonly only focus on gesture movements, but this might affect the performance of HMI since a user might interact with Activities of Daily Living that modify load speeds of movements and that affect the sEMG signal generation. From the obtained results, we can conclude that it is possible to identify different body movements considering different load weights, which isn't done by Aziz S et al.20 and Liu Y et al.33 This may change how users interact with assistive technology since different activities require a different response. Also, according to our research, using KNN and SVM with the Fine KNN and Fine Gaussian kernel functions with the IEMG feature is the most accurate machine learning classifier that has proved efficient and successful. These results can be used directly in the design of real-time EMG classifiers for rehabilitation and assistive devices.
REFERENCES
Aziz S, Khan MU, Aamir F, Javid MA. Electromyography (EMG) data-driven load classification using empirical mode decomposition and feature analysis. undefined. 2019 International Conference on Frontiers of Information Technology (FIT), Islamabad, Pakistan, [Published online December 1], 2019, 272-277. doi: 10.1109/FIT47737.2019.00058.
Phinyomark A, Hirunviriya S, Limsakul C, Phukpattaranont P. Evaluation of EMG feature extraction for hand movement recognition based on Euclidean distance and standard deviation. In: ECTI-CON2010: The 2010 ECTI International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology, Chiang Mai, Thailand, 2010, 856-886.
AFFILIATIONS
1 Motion Analysis Lab, National Institute of Rehabilitation "Luis Guillermo Ibarra Ibarra", Mexico City, Mexico.
2 CONACYT-CINVESTAV, Mexico City, Mexico.
3 Tecnológico de Monterrey, School of Engineering and Sciences, Mexico City, Mexico.
4 LAFMIA UMI, Center for Research and Advanced Studies of National Polytechnic Institute, Mexico City, Mexico.
5 UTC-CNRS UMR, Sorbonne Universités, Compiégne, France.
Conflict of interest: the authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
CORRESPONDENCE
Ivett Quiñones-Urióstegui. E-mail: iquinonesu@gmail.comReceived: July 14, 2023. Accepted: November 6, 2023