Mixed reality can help detect neurodegenerative diseases

Mr Jerzy, for quite some time, has been experiencing difficulties in walking on uneven ground. During his walks, he has been tripping more and more often. His steps were becoming shorter and shorter and less confident. On one occasion, he tripped so badly and fell, breaking his leg. On that day, it dawned on him that his difficulties in walking do not necessarily stem only from his old age. The diagnosis was not optimistic: Mr Jerzy suffers from a type of Parkinson’s. Despite the implemented treatment, his condition was systematically degrading. Not only did walking become an arduous endeavour; getting out of bed became a troublesome task, too. He found it more and more difficult to say particular words. Hand tremors began to occur. In the final stage of the disease, he was unable to take care of his needs by himself. However, what has not left him until the very end was his spirit... .

A novel approach to the diagnose of neurodegenerative diseases

We can expect that there will be more Mr Jerzys in the future. Neurodegenerative diseases, which are accompanied by progressive and irreversible degradation of nerve cells – one of such is parkinsonism – manifest most often in old age. According to the Polish Central Statistical Office (GUS), the percentage of Poles older than 60 years in 2018 amounted to 24.75%. Taking into account the fact that the problem of population aging is a dominant trend in developed countries, the percentage of people suffering from neurodegenerative disorders will also continue to rise. Therefore, not only the development of effective treatment methods becomes a burning issue, but also methods of precise and quick diagnosis.

Two of the most fundamental methods for assessing the condition of patients with dementia are the anamnesis and physical examination. On their basis, doctors fill out medical charts which are then used to make a diagnosis on the type of disease and its stage. It is easy to guess that this process is time-consuming. Worse still, it is not error-free, since it is based on subjective feelings of the patient and observations of the doctor. It would be much more reliable to work with hard data. Biomedical engineering offers a helping hand, which, for a long time, has been developing methods for collecting and processing various signals generated by the human body.

To help you imagine it, we can compare our organisms to a TV set. When we use the remote control to set the volume to a particular level, each time the device generates equally loud sound, that is, a parameter expressed in a numerical value. The TV allows us to adjust many other parameters related to sound and image. Their combination influences our general impression of the message. This can be compared to the signals generated by the human body. We can register them and describe them using objective metrics. When we collect enough data from a considerable amount of patients, we will be able to compare it, determine norms, and search for correlations. Circling back to our TV analogy: when, despite the desired volume level, only every other word from our favourite actress’ mouth reaches us, and her actually snow-white dress seems to be covered with dark spots – it is time to take our TV to a service centre.

Goggles for detecting Parkinson’s and Huntington’s diseases

Developing methods to register signals and process them to diagnose neurodegenerative diseases, such as parkinsonism or Huntington’s disease, is the objective of the team led by Daria Hemmerling, DSc, from the Faculty of Electrical Engineering, Automatics, Computer Science, and Biomedical Engineering. Her project titled ‘The use of mixed reality to diagnose and evaluate the stage of neurodegenerative diseases’ has recently received funding (PLN 1.5 million) from the National Centre for Research and Development (NCBR) as part of the LIDER programme. To obtain the signals, the scientists want to use mixed reality (MR) goggles HoloLens 2.

How do the researchers conceive it?

Before we get into that, let us first look at what mixed reality is and what distinguishes it from virtual reality (VR). The latter is commonly associated with video games, where putting special goggles allows the player not only to see the action on the screen, but also to become directly involved in it. In the case of mixed reality, we are dealing with a hybrid of the real world with incorporated digital objects that can be manipulated. It has been used increasingly in professional settings. This allows surgeons, for example, to perform tutorial surgeries on 3D holograms of the patient’s organs and technicians to plan the installation of various elements of the physical space using their virtual models.

To make such chiselled surgeries happen, the goggles developed for this purpose are equipped with precise cameras, microphones, and sensors that track the behaviour of the user. And this is exactly what the AGH UST engineers want to use to collect diagnostic data from patients. Using mixed reality, patients would be able to see the image of their doctor who would be able to ask them to perform one or several actions at a time, for example, walk a few steps back and forth, move their arms in a specified way, or speak a few words. The system will collect the following data in real time, which can prove essential in making a diagnosis: posture, movement smoothness, articulation, and speech rate. It will also track saccades, that is, involuntary movements of the eyes, which vary between sick and healthy people. The scientists want their solution, using the data collected via MR goggles, to contribute to making a proper diagnosis with a 80% efficiency rate. If this is successful, it will be the first device in the world to facilitate simultaneous registration and analysis of multiple signals that are crucial in the process of detecting neurodegenerative diseases.

Hopes and challenges

‘We came up with a prospective idea of having one device that produces one examination record. This will make it possible to evaluate all parameters that we can measure with it (the same way for every patient, at various moments in time, etc.). The device might prove very useful for filling out medical charts, an activity that doctors have to do themselves. Now, instead of doing paperwork, they will be able to focus on observing their patients because it will be the mixed reality system that will perform the tests’, says Dr Hemmerling.

The scientist also hopes that the data collected in this way from a considerable number of patients will make it possible in the future to discover correlations between particular parameters, deepening our knowledge of parkinsonism and Huntington’s disease.

However, before that happens, the AGH UST researchers have a long way to go, as the project has only just launched. Completion of the work requires the participation of specialists from various disciplines, but especially doctors and their patients. Diagnostic data will be collected from patients from two hospitals and medical centres in Krakow. However, before putting on the goggles, the patients will undergo standard diagnostic examinations that will constitute a point of reference for further research work. To analyse the data collected using goggles, the engineers have to design artificial intelligence algorithms capable of discovering regularities in the accumulated signals and tying them with specific nosological units. The final stage, before clinical trials, will be the creation of a user-friendly application for computers and smartphones that will allow doctors to use the benefits of the new system. The completion of the project should take no longer than 3.5 years.

Science originating from personal experience

In her research work, Dr Hemmerling has dealt with speech signal analysis and its use in, e.g., the diagnosis of various nosological units. When she was writing her doctoral dissertation, she went on an internship at the Universidad de Medellin in Colombia, where she met people suffering from Parkinson’s disease. ‘Old patients are stressed when visiting their doctors. They forget most of the things that had happened during their previous appointment. Additionally, they require care – someone has to drive them to the doctor, lead them, hand in hand, to the doctor’s office, and stay with them at all times. Most of the people I have dealt with felt unwanted because of this. This was a huge motivation to start researching this field’, recalls the scientists.

The AGH UST researcher adds that it is important for patients to find a way to monitor their diseases constantly. This is significant for adjusting the pharmacotherapy to the stage and symptoms of the disease, which fluctuate over time.

‘In South America, engineers visit patients for a few hours to simultaneously register two signals, that is, definitely less than in our case. Ultimately, I would like the patient to have an independent device that requires no assistance from an engineer sitting in their house and perhaps disturbing them a bit’, says Dr Hemmerling, sharing her dreams.

Today, the necessary technology is still too expensive to find its way under the roofs of Everymen. Furthermore, in the event of its successful implementation, it would be more likely that it will be used in clinical conditions. Nevertheless, who would have thought not so long ago that the majority of us will be carrying computers with the access to the knowledge of the entire world in our pockets?