Back in 2003, Lawrence Hettinger and colleagues penned this paper on the topic of neuroadaptive interface technology. This concept described a closed-loop system where fluctuations in cognitive activity or emotional state informs the functional characteristics of an interface. The core concept sits comfortably with a host of closed-loop technologies in the domain of physiological computing.

One great insight from this 2003 paper was to describe how neuroadaptive interfaces could enhance communication between person and system. They argued that human-computer interaction currently existed in an asymmetrical form. The person can access a huge amount of information about the computer system (available RAM, number of active operations) but the system is fundamentally ‘blind’ to the intentions of the user or their level of mental workload, frustration or fatigue. Neuroadaptive interfaces would enable symmetrical forms of human-computer interaction where technology can respond to implicit changes in the human nervous system, and most significantly, interpret those covert sources of data in order to inform responses at the interface.

Allowing humans to communicate implicitly with machines in this way could enormously increase the efficiency of human-computer interaction with respects to ‘bits per second’. The keyboard, mouse and touchscreen remain the dominant modes of input control by which we translate thoughts into action in the digital realm. We communicate with computers via volitional acts of explicit perceptual-motor control – the same asymmetrical/explicit model of HCI holds true for naturalistic modes of input control, such as speech and gestures. The concept of a symmetrical HCI based on implicit signals that are generated spontaneously and automatically by the user represents a significant shift from conventional modes of input control.

This recent paper published in PNAS by Thorsten Zander and colleagues provides a demonstration of a symmetrical, neuroadaptive interface in action.

The authors constructed an implicit control task where a cursor moved once every few seconds in one of eight directions within a grid. The task ended when the cursor reached a target location at the edges of the grid. Participants wore EEG apparatus (64 electrodes) and passively watched as the cursor moved randomly through the grid. This was a training phase where implicit EEG responses to desirable cursor moves (in the direction of the target location) were collected along with those instances when the cursor moved in an undesirable direction, away from the target location. The authors applied two types of analyses to the EEG data that they collected at this stage. They created a linear classifier to distinguish EEG-based responses to desirable cursor moves from undesirable moves. They also performed a source localisation analysis on the EEG data. For those who are unfamiliar with source localisation, it is a technique used to identify the neurological source of cortical activity recorded in the EEG signal. This analysis revealed a positive peak in EEG amplitude that occurred 180ms after each cursor movement. It was especially interesting that the size of this peak had a linear relationship with angular deviance from the desirable direction; in other words, the 180ms peak was maximal at 0 degrees from desired direction and minimal at 180 degrees. The source localisation analyses identified this positive peak as originating from the medial area of the prefrontal cortex.

The second phase of the study was similar to the first phase, participants wore an EEG head cap and passively viewing the movements of the cursor within the grid. But on this occasion, EEG data was classified in real-time to indicate whether cursor movement was in a desirable or undesirable direction. This output was subsequently deployed as an input to a reinforcement learning algorithm, which basically increased the probability of cursor movements in a desirable directions occurring in future. By definition, the same algorithm would reduce the probability of undesirable cursor movements within the grid. One clever part of this approach was the use of an output from one machine learning algorithm (linear classification) as input to a second algorithm performing reinforcement learning that allowed the user to implicitly communicate a task goal to the computer without any volition or conscious control. The resulting performance of the system during phase 2 was also impressive. For a 4×4 grid, it took 27 random movements on average for the cursor to reach the target, if those movements had been perfectly reinforced, the cursor would reach the target in 10 moves; the online EEG classifier achieved an average performance of 13 moves. When the grid was extended to a more complex 6×6 grid, the online classifier reached the target in 23 moves compared to 90 in the random condition and 14 when movements were perfectly reinforced.

The study perfectly represents an implicit process of monitoring, inference and interface adaptation, which is the modus operandi of a symmetrical, neuroadaptive system. Given that participants were simply instructed to watch the movements of the cursor, the efficiency of communication between person and system is very good.  With respect to efficiency, this type of symmetrical HCI is less concerned with information transfer rates, which are very important for active BCI systems, as opposed to increasing the possible communication bandwidth between user and computer. Allowing a technological system to act on the basis of implicit neurophysiological activity has an advantage of enabling HCI with minimal conscious awareness. This approach potentially allows the person to conduct two levels of interaction with technology, a conscious act of reading/editing/writing accompanied by a series of unconscious actions, which act at the meta-level of human-computer interaction: shaping context, making ambient alterations, pacing the interaction.

The big questions for future research in this field concerns the utility of the approach: is it sufficiently flexible? Can it add genuine value to the enjoyment, safety or productivity of interacting with computers? There are a range of sticky issues in the future, from the recognition of unintentional inputs to data privacy. It is worth remembering that liberation of computers from the restrictions of asymmetrical interaction comes at the cost of reduced autonomy for the human user as I’ve argued before here.

For now, it’s baby steps for symmetrical HCI and neuroadaptive technology but this PNAS paper is a really interesting start.