How a digital coach helps with rehabilitation
As part of the international eHealth project “CARE”, the company Ateleris have developed a digital exercise coach that accompanies people as they perform movement exercises. The exercises are recorded with a webcam and analysed and evaluated in real time with the help of artificial intelligence (AI). People can watch themselves on the screen during the exercises and receive visual feedback.
Medical studies[1] show that heart attack patients who have undergone a rehabilitation programme suffer from fewer heart problems in the longer term and have a lower mortality rate. At the same time, however, studies also show that such offers are often not taken up or are prematurely discontinued[2]. In Europe, in some cases only up to 50% of all heart patients take advantage of these programmes and between 12 and 56% do not complete them[3].
The reasons for not taking up or dropping out of such a programme are manifold. Lack of personal support, logistical problems such as finding appointments or poor accessibility of rehabilitation facilities, but also the stress of doing the exercises in a group are the most frequently cited reasons[4],[5].
Personal counselling for positive long-term effects
In the Eurostars project “CARE”, a cooperation with the two Danish companies Tutee[6] and Wellfaster[7] and the Radboud University Hospital[8] in the Netherlands, was about developing an eHealth platform for better rehabilitation support for heart patients after a heart attack or the onset of other cardiovascular diseases.
Part of the project was to investigate which digital methods can increase rehab participation, reduce dropouts and overall achieve a more sustainable readiness to exercise. To this end, various aspects were implemented, for example digital counselling channels via video for individual support at home between patient and therapist or a rehab companion app for digital motivation and support during and after rehabilitation. The app was also used in a study by the partner Radboud University Hospital[9].
Figure 1: Rehab companion app of the partner Wellfaster, from the paper https://bmjopensem.bmj.com/content/7/3/e001159
The digital training coach
We developed the digital training coach with the idea of enabling a more individualised accompaniment of independently performed rehabilitation exercises at home in order to increase training fidelity. The application plays various training videos, which can also be combined into exercise sequences, and records the person training with a camera.
To start an exercise, the patient stands in front of the camera, the digital trainer checks the posture and if it is correct, the exercise starts automatically after a few seconds. During the exercise, the patient not only sees himself, but also a video of the coach demonstrating the exercise. In the background, the digital coach automatically counts the exercise repetitions and automatically switches to the next exercise.
Fig.2: Source https://customer-zmmtzxtiel4baype.cloudflarestream.com/6ffa709f5084e97372b1cc6996c36f2a/watch
The AI recognises the posture and counts repetitions
The application uses Google’s machine learning (ML) model “MediaPipe Pose”, which can recognise the posture of a person in the video in real time (so-called “pose detection”). The special feature of the chosen ML model is the high recognition performance even on simple hardware, which even enables execution on mobile devices.
Fig. 3: The 33 reference points of the ML model for “Pose Detection”, from https://google.github.io/mediapipe/solutions/pose.html
With the help of 33 different reference points on the body, which are mostly located along the different joints, the ML model can recognise different body postures, distinguish them from each other and follow the movements. As an orientation aid during training, the recognised body skeleton is superimposed on the real-time video image so that permanent visual posture control is possible.
On the other hand, it can analyse the movement sequence between the trainer video and the exerciser and recognise and assign the exercise. This enables the application to control the correct execution of the exercise and to count the corresponding repetitions.
Experiences and developments
In the course of the CARE project, we were able to gain various experiences with the Training Coach and develop it further. For example, it became apparent early on that a smooth playback speed is essential for a good training experience. An initial slower ML model caused the video playback to stall, interrupting the flow of the exercise and making it unusable for training.
Another challenge for the ML algorithm was exercises that did not have to be performed standing frontally, but bent, sideways or lying on the floor. Especially in such situations, the correspondence with the trainer video was insufficient. Depth information in the video image, such as provided by a Microsoft Kinect camera, would have been helpful, but could only have been realised with special hardware. Here, too, the further developed ML model “MediaPipe Pose” could help: It has learned to extract certain depth and angle information from a normal video image and can thus provide the 33 reference points as 3D positions.
Source: Example video from Google for 3D posing
Further areas of application for “Pose Detection
In principle, the digital training coach can be used in both the health and fitness sectors, although as a pure “lifestyle app” it faces fewer hurdles. For use in the health sector, e.g. in rehabilitation, the app would first have to be certified as a medical device, which was not part of the project.
There are many other possible applications for “Pose Detection” ML models. For example, to detect the activities of people in sports, in security and surveillance, in the field of ergonomics or to detect work processes, which is also known as “process mining”.
References
[1] https://doi.org/10.1007/s12471-020-01413-1
[2] https://doi.org/10.1002/14651858.CD007131.pub4
[3] https://doi.org/10.1097/hcr.0000000000000580
[4] https://doi.org/10.1186/s12872-017-0512-7
[5] https://doi.org/10.1097/HCR.0000000000000709
[6] https://tutee.dk/en/tutee-english/
[7] https://wellfaster.com/indexen.html
Leave a Reply
Want to join the discussion?Feel free to contribute!