Cutting edge technology is helping researchers track eye movement and brain activity to the millisecond in a bid to understand how we process language and extract meaning from what we see, read and hear. One of the leaders in the field is psycholinguistics expert Dr Pia Knoeferle from Berlin’s Humboldt University, where she studies how humans learn, use, and understand language.
As a discipline, psycholinguistics traces its history back to the early 20th century debate about acquired and innate behaviours. That distinction between the two was developed in the 1950s and 1960s by psychologists such as BF Skinner, who argued that all language must be learned, and the philosopher and linguist Noam Chomsky, who believed that humans possess an innate language facility.
Modern psycholinguists focus on various aspects of language, for example how children and adults learn language (acquisition and language learning), how language develops from an initial idea to the spoken or written word (production), and the processes by which a wide variety of language users understand language (comprehension). Dr Knoeferle’s work focuses on comprehension and what types of information in a sentence are used to build meaning. She is particularly interested in how context – especially visual context – contributes to language processing.
Dr Knoeferle’s research makes use of technologies which study ocular and neural activity to the millisecond, to record in real time how subjects respond to language stimuli. These include eye-tracking technology, which monitors the subject’s gaze and eye movements towards visuals during language tasks, for example when looking at text and images on a computer screen. It also involves the study of brain activity in response to a similar stimulus – called ‘event-related brain potentials’.
Language processing: theory and empiricism
In a recent paper “Predicting (variability of) context effects in language comprehension”, published in the Journal of Cultural Cognitive Science, Dr Knoeferle presents an in-depth review of language processing theory. She focuses in particular on research into prediction in language comprehension; that is, how we can begin to understand and predict meaning before we actually encounter a specific word.
The development of technologies has further enabled the study of brain activity while people perform various tasks. Among the most notable findings for linguistics has been the discovery of event-related brain potentials – brain responses to words and other stimuli. Measured to around 400 / 600 milliseconds post stimulus – ‘the N400 / P600 effect’ – brain potentials show how quickly the brain reacts to expectations of a word and semantic associations (N400) as well as to structure-related processes (P600).
Our immediate environment can affect language comprehension in real time.
Language in context
Dr Knoeferle looks at what the research literature says about context-focused approaches to comprehension. Studies have looked at verification and the interaction between listening and looking, for example asking subjects to verify a sentence in the context of a picture. Research has also considered ‘semantic priming’, in which brain responses to a second stimulus are found to be faster when that second stimulus is related to a first one (such as the word ‘cat’ following ‘dog’) than when it is unrelated (e.g., the word ‘disc’ following ‘dog’).
Other lines of research have included ‘construction grammar’. Instead of the traditional view that words (‘lexicon’) and language structure (‘syntax’) are distinct, this approach to linguistics argues that meaning and form are inseparable, and that constructions within sentences form the central units of language.
Following her extensive study of the literature, Dr Knoeferle identifies three key processes and associated mechanisms in language comprehension. Her paper discusses these processes at length in light of extant academic research. The first is “building structure and assigning an interpretation”. This includes forming a “mental representation” of sentence meaning which is “informed by linguistic and world knowledge, and the immediate linguistic context”.
The second is “grounding the interpretation in a model of the world/visual context”, for example via (internal/visual) attention guided by the language content or concept and associated personal experience. Finally, the third process is “verifying the structure and interpretation against representations of the world/ non-linguistic visual context and revising as necessary” and understanding by linking it to personal experience.
Dr Knoeferle argues that research approaches to date suggest that more attention should be paid to the potential differences between individuals. She writes: “Empirical research has provided more and more evidence that we must not only consider object and event contexts (and their associated representations) for developing theories of real time language comprehension but also (representations of) the listener and his characteristics such as age, or educational background among others.”
Predicting context effects
Dr Knoeferle’s research review leads her to conclude that context is important in language comprehension, but that context effects are variable, and some are more robust than others. She draws three conclusions. These concern the link between context effects and comprehender characteristics, the situation and environment in which language arises, and the comprehension process.
Context is important in language comprehension, but context effects are variable, and some are more robust than others.
Dr Knoeferle explains: “Our immediate environment can affect language comprehension in real time. Such context effects vary by our characteristics as language users, how words relate to the environment and what comprehension process they contribute to.” For example, she argues that comprehender characteristics such as age, literacy and language skills modulate context effects in language comprehension. Age particularly affects how quickly individuals integrate context into comprehension. The research literature suggests that a visually depicted action affects comprehension more rapidly in young adults than it does with kindergarten children, or with adults aged over 60.
Regarding how words relate to the situation and environment in which they arise, Dr Knoeferle finds there is a relatively direct and strong connection between action words and images, for example between seeing an image of someone painting and understanding the word ‘paint’. However, seeing someone smile is far more ambiguous and could relate to a range of things or words. They may be happy or amused, be looking forward to something, or be trying to cheer someone up, and related words could be ‘happy’, ‘amused’, or ‘compassionate’.
As a result, Dr Knoeferle argues that context has a greater effect for language that relates to things and events: “Referential cueing of information in visual context elicits – on average and by comparison – more rapid and stronger effects than non-referential mediation of visual context.”
Considering comprehension processes, when either distinct or similar language cues or prompts contribute to the same comprehension process, Dr Knoeferle argues they affect comprehension in a similar manner: “Imagine you hear language about a musician and it’s unclear whether she is acting or acted upon. Such ambiguity could be resolved via marking in language or also showing who is acting.” However, if the comprehension processes to which marking in language versus context depictions contribute are distinct, the context effects will show up as distinct.
Dr Knoeferle’s study aimed “to derive a more principled account of context effects” in language comprehension and her scholarly review of 40 years of psycholinguistics research and the concepts it has produced more than fulfils her brief. Providing ample evidence that language does not take place in a vacuum and cannot be separated from its (social) context,
Dr Knoeferle equally finds that context effects are variable.
The use of eye-tracking and the event-related brain potential technology is particularly interesting. Are you working with colleagues in other departments to exploit the findings from these studies using other techniques?