Everyone remembers where they were when they first learned two planes had struck the World Trade Centre on September 11, 2001, but what did you have for dinner on the second Tuesday of last month?
Psychologists studying the nature of memories have for decades had to rely on major events as a known truth against which to compare people’s recollections. The problem is that such events do not create run-of-the-mill memories.
“Those are special kinds of events. They’re special because they have high emotional content. They’re also special because they’re often ones you tell other people about. So they’re not just memories, there’s an overlay of the telling too,” explains the University of Melbourne’s Professor Simon Dennis.
Alternatively, in their quest to understand memory, researchers have studied individuals’ ability to recall numbers or faces or words presented to them in a laboratory setting. But such studies are too artificial and contrived to really learn much from.
“If you think about the difference between doing a laboratory experiment versus me in the real world with real memories – that’s a big difference. There’s no guarantee the things I learned in the lab are what’s actually going on out there,” Dennis adds.
Now a new approach, leveraging smartphone monitoring and machine learning techniques, is being tested. It promises to not only provide near-term benefits for people with bipolar disorder and depression, but also change the field of psychology forever.
“This is going to be a paradigm change for psychology,” Dennis says. “This is actually the point where psychology will move out of its adolescence into being a grown up science.”
Unforgettable, that’s what? Who? Where?
Dennis is director of the University of Melbourne’s Complex Human Data Hub which launched last week.
There are already a number of studies underway within the hub that use mobile monitoring and data collection to get what Dennis calls the “ground truth” against which memories can be assessed.
In one, participants wear a mobile phone in a pouch around their necks to work as a multi-sensor device. At hourly intervals, an app – Unforgettable.Me created by Dennis – captures an image, records its location, a snippet of audio, an accelerometer reading and the temperature. The app is paired with the web service "If This Then That" (IFTTT) to record extra data like when an email is received, a phone call is made or a participant reads a news article.
The individual’s data is then combined with other data streams like the weather and moon phase, and processed.
Machine learning techniques are then applied to the audio snippets to determine if someone is talking, if the participant is in a road vehicle and so on. Those classifiers are further trained to improve their accuracy.
“After a delay we show them the images they collect and ask when they were taken, how confident they are in their answer and how emotional the event was. Using a range of data sources, our models reconstruct the experience of the participants and predict the errors they will make,” Dennis, who has a PhD in Computer Science, says.
“This is a quantum leap in the state of the art, and I believe will allow us to construct a more comprehensive, ecologically valid and translationally relevant memory science.”
Another project uses a similar methodology, but is focused on bipolar disorder sufferers. Although images aren’t captured – the app runs on the participant’s usual phone, rather than one worn round their neck – their location and communications metadata is. This data is combined with additional readings from a Fitbit device worn by participants.
The passive data gathering method is particularly useful with people with conditions like bipolar and schizophrenia, which are associated with forgetfulness, complicating the feedback they might give at clinic visits.
“We know things like sleep and movement and amount of social interaction are predictive of the states bipolar patients might be in. When they’re depressed they tend to stay at home, not talk to people and so on,” says Dennis. “If we monitor those things automatically, then we can predict when they might be in a bad state and when intervention might be necessary.”
Predictive modelling – although still some way off yet – could be used to determine medication dosages.
“At the moment the bipolar medications are prescribed to account for a worst case scenario, so you’ve got enough of the drug on board so in your worst state you’re ok. The problem is the drugs have some serious side effects, particularly with long term use,” Dennis says.
By better understanding where a patient is on a cycle between manic and depressive, means their medication could be dosed far more accurately and other treatments like therapy could be recommended.
“We’re not there yet but that’s the vision,” Dennis adds.
Promising phenotype
Passive data collection from mobiles holds great promise for the detection and treatment of range of health issues, but especially mental illnesses.
A ‘digital phenotype’ for a person can be created by determining their typical interactions with their smartphone. Deviations from the norm could indicate an episode of psychosis or the onset of an illness.
“Our use of these devices generates, as a byproduct, a surprisingly rich tapestry of social and behavioral fingerprint,” wrote Harvard's Jukka-Pekka Onnela and Scott Rauch in their summary of the emerging field in Neuropsychopharmacology.
“Given that these digital fingerprints reflect the lived experiences of people in their natural environments, with granular temporal resolution, it might be possible to leverage them to develop precise and temporally dynamic disease phenotypes and markers to diagnose and treat psychiatric and other illnesses.”