Skip to main content

Breathy, tense, tight, flat: a person’s voice can hint at depression, and AI can pick it up

RADAR-CNS is experimenting with using wearable devices and smartphones to pick up on biomarkers of central nervous system disorders

Parrot Aksioma CC BY-NC-SA 2.0
From the artwork "Uncertainty-in-the-Loop " by artist Sanela Jahić. Image by Aksioma Project Space Komenskega Ljubljana  CC BY-NC-SA 2.0


When Slovenian artist Sanela Jahić was working on a thought-provoking installation about how art and creativity can be turned into dispassionate data points, she came across a scientific paper by Dr Nick Cummins, a member of the RADAR-CNS team. Dr Cummins’ area of expertise is vocal variation as a signifier of depressed mental states, and how AI can be used to detect these variations.

The artist was designing PATAKA, a visual and auditory journey that ties speech patterns and machine learning to vocal biomarkers and mental health. The name is a reference to a vocal exercise given by speech pathologists; the way a person utters the rapidly-alternating three-syllable sequence can signal changes in muscle control that can offer clues as to their mental state. As part of the exhibition, parrots were trained to repeat the PATAKA sequence, and the cacophony was combined with data visualisations to make a point about what our own voices can tell machines about us.   

Dr Cummins’ paper claims that alterations in speech motor control in depressed individuals have been found to manifest as a reduction in ‘spectral variability’. He says, “Depression can affect fine muscle control which means it requires more effort to speak, making the speech sound flatter. Other common descriptors would be a breathy or tense or tight.” Together with Dr Judith Dineley from the University of Augsburg, he happily provided the artist with insights into the analysis of depression in speech that helped her with the machine-based-learning approach of PATAKA.

From subjective to objective diagnosis, thanks to AI

“There are currently no objective biomarkers to diagnose depression, that I know of, that are used in mainstream clinical practice. This is what makes a project like RADAR-CNS so important: digital bio-signatures could provide this breakthrough.” Speech can currently be used by clinicians in a subjective manner to aid their diagnosis, he says. “One of the earliest references in the literature is from German psychologist Emil Kraepalin in his 1921 book Manic Depressive Insanity and Paranoia: ‘patients speak in a low voice, slowly, hesitatingly, monotonously, sometimes stuttering, whispering, try several times before they bring out a word, become mute in the middle of a sentence.’ But AI and related computing advances is what will really enable us to move from the subjective to the objective.”

RADAR-CNS is using wearable tech and smartphones to detect and validate biosignatures not only for depression but also epilepsy and multiple sclerosis. The research covers other biosignatures, including heart rate, physical and social activity, facial muscle movements and EEG signals. Dr Cummins and Dr Dineley extract the meaningful information from these raw signals recorded during clinical trials before aggregating them for further analysis.

Other disorders have well known effects on speech

Links for speech changes are not as strong for the other disorders as they are for major depressive disorder, says Dr Cummins. “However, we are in the process of beginning to form a hypothesis of what to start search for, so I am very excited to see what we find. “Anxiety, fatigue, stress and bipolar disorders have well known effects on speech, as do neurodegenerative disorders such as Huntington's disease, Parkinson’s disease and Alzheimer’s disease. I have also got recently involved in a project exploring speech and language changes associated with psychosis.”

Thus far, what we know is based on relatively small datasets that were collected in controlled conditions. RARAR-CNS is taking the next, important step: do their findings hold on data collected outside so-called laboratory conditions, where noise becomes a major issue? “We also need to consider that a condition such as depression is heterogeneous, and the best options we have for aiding diagnosis is analysis of multiple signals and biomarkers.”


This is new ground, and getting regulatory approval for any innovations that stem from the research of Dr Cummins and his colleagues could be challenging. “Identifying best practices for regulating digital technologies and AI approaches is a major hurdle that needs overcoming. It will require multidisciplinary efforts from key stakeholders: patients, clinicians, academia, tech and pharma companies and governments.”


Pataka was part of Sanela Jahić’s Uncertainty-in-the-Loop exhibition which ran from 23 September to 23 October 2020 at the Institute for Contemporary Art, Ljubljana.

Read more 


Regulators need solid evidence that new technologies are reliable. IMI can provide it

Can computers learn to think like chemists?

Finally, a reliable way to track drug side effects

One step closer: digital readouts of walking as a measure of health


Related projects: 

Manage your newsletter subscriptions
Stay informed - subscribe to our newsletter.