Like a lot of people, I came to the area of physiological computing via affective computing. The early work I read placed enormous emphasis on how systems may distinguish different categories of emotion, e.g. frustration vs. happiness. This is important for some applications, but most of all I was interested in user states that related to task performance, specifically those states that might precede and predict a breakdown of performance. The latter can take several forms, the quality of performance can collapse because the task is too complex to figure out or you’re too tired or too drunk etc. What really interested me was how performance collapsed when people simply gave up or ‘exhibited insufficient motivation’ as the psychological textbooks would say.
People can give up for all kinds of reasons – they may be insufficiently challenged (i.e. bored), they may be frustrated because the task is too hard, they may simply have something better to do. The prediction of motivation or task engagement seems very important to me for biocybernetic adaptation applications, such as games and educational software. Several psychology research groups have looked at this issue by studying psychophysiological changes accompanying changes in motivation and responses to increased task demand. A group led by Alan Gevins performed a number of studies where they incrementally ramped up task demand; they found that theta activity in the EEG increased in line with task demands. They noted this increase was specific to the frontal-central area of the brain.
We partially replicated one of Gevins’ studies last year and found support for changes in frontal theta. We tried to make the task very difficult so people would give up but were not completely successful (when you pay people to come to your lab, they tend to try really hard). So we did a second study, this time making the ‘impossible’ version of the task really impossible. The idea was to expose people to low, high and extremely high levels of memory load. In order to make the task impossible, we also demanded participants hit a minimum level of performance, which was modest for the low demand condition and insanely high for the extremely high demand task. We also had our participants do each task on two occasions; once with the chance to win cash incentives and once without.
The results for the frontal theta are shown in the graphic below. You can clearly see the frontal-central location of the activity (nb: the more red the area, the more theta activity was present). What’s particularly interesting and especially clear in the incentive condition (top row of graphic) is that our participants reduced theta activity when they thought they didn’t have a chance. As one might suspect, task engagement includes a strong component of volition and brain activity should reflect the decision to give up and disengage from the task. We’ll be following up this work to investigate how we might use the ebb and flow of frontal theta to capture and integrate task engagement into a real-time system.
Hummm… let me be a reviewer for 5 min 🙂
From what I read in the literature, engagement and workload are often associated to the ratio of the Beta power to the Theta + Alpha power. This ratio is supposed to increase with increasing workload, which would mean that either Beta energy increases or Theta / Alpha energy decreases which is not what you obtain (even though you did not check Beta energy which might increase more than Theta). Any comments or explanation ?
My EEG knowledge is a bit hazy, so Steve will have to correct me if I’m wrong (which I probably am :)). As you may or may not know theta activity has been found to be associated with different types of mental activity (Schacter 1977) (e.g. increases with lack of attention).
Depending on the location theta is measured from the activity your inferring changes. The beta/(theta+alpha) ratio which was tested by (Pope 1995) in his engagement work does work as you describe. However their recordings are based on a monopolar recordings at CZ (I’m using his videogame work which is basically the same, sorry can’t find my 1995 paper at the moment). The work here uses a different placement and so the inferred mental activity (theta increases with lack of attention) is not applicable.
(Schacter 1977)
EEG THETA WAVES AND PSYCHOLOGICAL PHENOMENA: A REVIEW AND ANALYSIS
(Pope 1995)
Biocybernetic system evaluates indices of operator engagement in automated task
The engagement index was derived from the work of Alan Pope as Kiel points out. These kinds of EEG ratio scores where faster activity is divided by slower activity are a good way of capturing the central idea that slower (theta, alpha) activity is associated with lower levels of brain activation whilst the opposite holds true for faster activity (beta). Studies of alpha activity and fMRI activity (I don’t have the ref to hand) have supported this notion.
However, theta which is fairly slow (3-7Hz) does not seem to fall into the same category. It has been pointed out that the relationship between theta and cortical activity is very sensitive to topography (as first pointed out by Schacter in 1977). Whilst occipital theta at the back of the head has been associated with sleepiness, it seems that frontal theta is increased in the presence of cognitive demand. The guy who did the most work on this is Alan Gevins. His studies demonstrated that: (1) theta at frontal-central site (Fz) increases with working memory load, and (2) this theta activity was probably derived from the anterior cingulate cortex (ACC). Basically we followed up Gevins’ work rather than Pope’s because: (1) I’ve used the engagement index in the past and didn’t find it to be particularly sensitive, and (2) Gevins’ work has been replicated and has a neuroanatomical basis. If you’re really interested, here are some references for his work:
Gevins, A., & Smith, M. E. (2003). Neurophysiological measures of cognitive workload during human-computer interaction. Theoretical Issues in Ergonomic Science, 4(1-2), 113-121.
Gevins, A., Smith, M. E., Leong, H., McEvoy, L., Whitfield, S., Du, R., et al. (1998). Monitoring working memory load during computer-based tasks with EEG pattern recognition models. Human Factors, 40(1), 79-91.
Gevins, A., Smith, M. E., McEvoy, L., & Yu, D. (1997). High resolution EEG mapping of cortical activation related to working memory: effects of task difficulty, type of processing and practice. Cerebral Cortex, 7, 374-385.
Hey Stephen, hope you’re well. After putting “gevins” “engagement” in google (see if there was anything new!), it came up with this. Clever Google. Sounds interesting, few quick questions..What was your sample size? Is the above data averaged over all participants (i think unlikely, if not how many were excluded and for what reasons)? What was the task?
Just for some further neurophysiological evidence, check this link: http://lpp.psycho.univ-paris5.fr/pdf/2529.pdf, seems that theta recorded from frontal regions possibly reflects intentional action (read effortful engagement), and involves the frontomedian wall (specifically the preSMA and the rostal cingulate zone). The frontomesian complex also contains the ACC, so converging evidence with Gevin’s work.
Frontal theta activation (together with parietal alpha inhibition) is definitely associated with effortful engagement…in some people, some of the time. Possibly the role of intentional v automatic reponses helps explain some of the individual differences which plague this research. The same people do the same task in different ways (the terms ‘same’ and ‘different’ are entirely interchangeable), sometimes fully engaged (intentional/deep processing), sometimes fully distracted (automatic/shallow processing). Usually somewhere in between with periods of engagement and periods of distraction.
Am gonna stop rambling, have just checked the date of this thread…mothballs?
Hi Mike,
Until Steve is back online,
You can check out his recent publications on capturing user engagement here: –
http://web.mac.com/shfairclough/Stephen_Fairclough_Research/Publications_physiological_computing_mental_effort_stephen_fairclough.html
And if I recall correctly this paper has a description of the study this post was relating to.
http://web.mac.com/shfairclough/Stephen_Fairclough_Research/Publications_physiological_computing_mental_effort_stephen_fairclough_files/IJAACS_2010_Fairclough.pdf
Thanks for the links Kiel, much appreciated. I’m curious to know, with a starting sample of 34 participants, why only 18 were used for the ‘grand average’ in figure 3? Also, any problems with some individuals having very low spectral power (in frontal theta), even though task performance was high?
^^ Could be because you had to pay them for the incentive study…pesky budgets!
Hi Michael
Good to hear from you. Hope all’s well with you.
The figure in this post is taken from a study of 20 participants. Unfortunately Kiel directed you to the wrong experiment – this one has not been published yet except as an abstract at SPR. Thanks a lot for the link; I hadn’t seen that one. As you say, there is converging evidence here for a frontomesian effect taking in the ACC and the rostral cingulate.
As you might know, there is also a lot of evidence from Gendolla and Wright for systolic blood pressure reactivity being linked to effort. I mention this in passing because we found some convergence between our frontal theta results and systolic reactivity; also, ACC has been implicated in BP control.
Best,
Steve
I remember writing a lot of stuff about the link between ANS activity and frontal neural networks during cognitive processing (ended up on the cutting room floor!). Had a whole section on ECG/EEG interactions, and can remember reading about systolic BP. No suprise really, since during effortful task engagement we would expect significant (system wide) mobilisation of energetical resources. Seems likely that these frontal neuronal oscillations represent preparatory, anticipatory and regulative co-ordination of parasympathetic activity, which in turn subserves cognitive functioning. “The master timing mechanism…”?
What a shame that part of the thesis got cut. I would have liked to have read it. Interactions between autonomic and neuroscience measures remains relatively unexplored. I recommend checking out Hugo Critchley‘s work on connections between ACC and cardiovascular system if you haven’t already. As you say, we would expect to see some convergence between the groups of measures and you would expect them to pull in the same direction.
I’ve still got it somewhere, will see if i can dig it up. Have you checked this work: http://www.plosbiology.org/article/info:doi/10.1371/journal.pbio.0060159
Diffusion spectral imaging shows really interesting clustering around frontomesian regions and major cluster of interconnected cortico-cortical propogations in parietal and posterior medial regions. Looks like two control centres, one fronto-mesian, the other parietal/posterior. Not to mention the really impressive looking piccies!
Thanks Michael. I hadn’t seen that work with spectral imaging. I’m reminded of some work that is indirectly related on EEG coherence patterns during working memory load, such as this one by Sauseng et al (2005) – which links frontomesian patterns with a parietal regions.
Thanks for that, the link doesn’t work but found the paper. Seems there’s a few different groups all saying similar stuff! Old Sauseng, Doppelmayr, Klimesch & Hanslmayr are busy fella’s…