Parallel analysis of eye movements and verbal production – evidence for a component model of perception of emotional expressions

Ekman’s neurocultural theory of emotions assumes a fully consolidated synchronous nature of the action of the components associated with the experience of emotions. The experimental results accumulated to date contradict this hypothesis. Using the example of the interaction of eye movements and the generation of a verbal description of an image, we demonstrate that the processes potentially involved in the experience of emotions can proceed asynchronously, being determined by their own logic of development and determinants. We consider the component theory of K. Scherer to be a relevant way of describing it. In this case, the task of the researcher is to identify the relevant components involved in solving a specific experimental problem and to study the interaction between them.

Introduction

For a long time, studies of the perception of emotional expressions of the face relied on Paul Ekman’s neurocultural theory of emotions. The basis for the creation of a neurocultural theory of emotions was the study of the perception of emotional expressions by the natives of Papua New Guinea (Ekman & Friesen, 1971). Experiments were also carried out in which studied: recognition of spontaneous facial expressions in Japan and the United States cultures; actual measurement of facial behavior of American and Japanese subjects; demonstration, that for happiness, sadness, anger, fear, surprise, and disgust the same facial expression is interpreted as showing the same emotion in Brazil, Argentina, Chile, the United States, and Japan (Ekman, 1971). Ekman explained, that theory is named neuro-cultural, “because it emphasizes two very different sets of determinants of facial expressions, one which is responsible for universal and the other for cultural differences. Neuro refers to the facial affect program – the relationships between particular emotions and the firing of a particular pattern of facial muscles. <…> Cultural refers to the other set of determinants – most of the events which elicit emotion, the rules about controlling the appearance of emotion, and most of the consequences of emotion” (Ekman, 1971, 212). Farther, Ekman describe appearance of face in three zones (Brows – Forehead; Eyes-Lids; Lower face) for six emotions: Surprise, Fear, Anger, Disgust, Sadness, Happiness (Ekman, 1971, 251 – 252).

The book “Emotion in the human face: Guidelines for Research and an Integration of Findings” (Ekman, Friesen, & Ellsworth, 1972) was written as a guide for future researchers, covers the main conceptual ambiguities, methodological decisions and a review of recent Research Findings. Authors notice the lack of a clear definition of emotion (Chapter 1) and select most significant aspects of definition, which should be taken as the basis for assessing relevance of the behavior studied to emotion: a special class of stimuli which usually elicit emotional behavior; physiological responses; motor responses; verbal responses; interactive consequence of certain behavior.

The next significant step was the creation of a set of reference images of emotional facial expressions Pictures of Facial Affect (Ekman & Friesen, 1976). Until now POFA remains the reference material for conducting research on the perception of emotional expressions of the face and creating new databases of photo and video images. Facial Affect Scoring Technique – FAST (Ekman, Friesen & Tomkins, 1971) and Facial Action Coding System – FACS (Ekman & Friesen, 1978) have been developed to describe facial expressions.

In 1984 Ekman again pointed to the absent of agreement about a definition of emotion and proposed “ten characteristics which can help in beginning to define what distinguishes emotion from other psychological states” (Ekman, 1984, p 319). Ekman gave the following statements (Ekman, 1984):

1. There is a Distinctive Pan-Cultural Signal for Each Emotion.
2. Distinctive Universal Facial Expressions of Emotion Can Be Traced Phylogenetically.
3. Emotional Expressions Involve Multiple Signals, Involving the Voice as Well as the Face.
4. There Are Limits on the Duration of an Emotion.
5. The Timing of an Emotional Expression Reflects the Specifics of a Particular Emotional Experience.
6. Expressions Are Graded in Intensity, Reflecting Variations in the Strength of Felt Experience.
7. Emotional Expression Can Be Totally Inhibited.
8. Emotional Expressions Can Be Convincingly Simulated.
9. There Are Pan-Human Commonalities in the Elicitors for Each Emotion.
10. There is a Pan-Human, Distinctive Pattern of Changes in the Autonomic and Central Nervous System for Each Emotion.

Different versions of these characteristics were proposed in (Ekman, 1993; Ekman, 1999). The final list is given in (Ekman & Cordaro, 2011):

• distinctive universal signals;
• distinctive physiology;
• automatic appraisal;
• distinctive universals in antecedent events;
• presence in other primates;
• capable of quick onset;
• can be of brief duration;
• unbidden occurrence;
• distinctive thoughts, memories, and images;
• distinctive subjective experience;
• refractory period filters information available to what supports the emotion;
• target of emotion unconstrained;
• the emotion can be enacted in either a constructive or destructive fashion.

Based on the early work of P. Ekman, N. Etcoff proposed an experiment that, in the case of the expected result, would confirm that emotional expressions represent as universal communication signals that carry information about the subject’s state (Etcoff & Magee, 1992).

The idea of the experiment is based on the studies of the categorical perception of phonemes conducted by A. Lieberman (Liberman et. al., 1957) and general ideas about the categorical perception (Harnad, 1987). The paradigm of the experiment assumed a comparison of the results of identification task and discrimination task on a uniform transition series between stimuli perceived as belonging to opposite categories. In forced choice identification task subject must indicate which category the presented stimulus belongs to. ABX-Discrimination task involves sequential exposure of two near stimuli A and B, and then stimulus X, exactly matching A or B. Subject must answer: X = A or X = B? It is assumed that in the case of a categorical perception, the result of solving the identification task will be an S-shaped curve, reflecting the presence of a clear boundary between opposing categories. Since objects will be distinguished only due to their different categorical affiliation, the result of solving the discrimination task will be a horizontal line at chance level with a local maximum corresponding to the boundary between categories.

Justifying this paradigm, N. Etcoff believes that the use of expressions as outward-facing communication signals (as follows from the works of Ch. Darwin and P. Ekman) requires unambiguous expression of the emotional state and its identification by the communicant. “However, even if multiple emotions can coexist, the state relevant to the perceiver may still be categorical. A person experiencing both anger and fear may flee the situation or fight, but would not do something halfway in between. If the perceiver is built to detect states in others that can motivate behavior, it might be best to see the face in terms of the single most likely underlying state, rather than some mixture” (Etcoff & Magee, 1992).

Experimental results consistent with the proposed hypothesis would be sufficient confirmation of Ekman’s neurocultural theory of emotions and would allow creating effective means of recognizing a person’s emotional state by facial expression. Etkoff’s work inspired a large number of experimental studies, the history of research is partially given in in the monograph by V.A. Barabanshchikov, A. V. Zhegallo & O.A. Korolkova “Perceptual categorization of facial expressions” (Barabanschikov, Zhegallo, & Korolkova, 2015), but the results obtained do not agree with the initial hypothesis. Disappointment in Ekman’s neurocultural theory in its extreme form was expressed in the creation of an alternative “constructivist” theory of emotions (Barrett, 2017). Emotional experiences are assumed to be dynamically constructed, based on incoming sensory information, current body state, and past experiences. Thus, the researcher needs to study the accumulation of experience in the neural network of the brain and the dynamic formation of the emotional state mediated by the accumulated experience. We consider that a significant shortcoming of this theory is the impossibility of extracting concrete, easily verifiable empirical consequences.

Further study of the perception of emotional expressions requires the use of an intermediate theory that allows one to explain the specific experimental results obtained so far. One of the significant problems is the unsatisfactory agreement between the results of solving the identification task and the discrimination task. In our studies, the categorical affiliation of distinguishable images, the subjective distance in the perceptual space of the observer, and the modality of reference expressions explain up to 20% of the dispersion of the results (Barabanschikov, Zhegallo, & Korolkova, 2015, 248 – 253).

In the initial study (Liberman, 1957), on which the experimental paradigm proposed by N. Etkoff was based, the categorical perception of transitional phonemes was studied. In this case, the sound (content) of the phoneme seems to be directly perceived in both the case of identification and discrimination, so that a comparison of the results is admissible.

Our experiment demonstrates some features of the formation of elements of a verbal description when considering images of emotional expressions. This shows the possible incorrectness of a direct comparison of the results of solving the identification task and the discrimination task.

Method

Experimental procedure and methods

Images of emotional facial expressions from the catalog of the exhibition “About Face: Human Expression on Paper”, image size 884×1024 px, were used as stimulus material. The images were displayed on a 17” LCD screen with a resolution of 1280×1024, the distance from the screen was 60 cm. The study participants were placed in the KE-1 electroencephalographic chair (Neurobotics), which ensured a comfortable position and a stable position of the subject’s head. Eye movements were recorded using an SMI REDm eye tracker, recording frequency 120 Hz. Presentation of stimulus material and registration of responses were performed using the modified PxLab software (Zhegallo, 2016). The task of the study participants was to freely describe the emotional state of the sitters. Exposure time is 30 sec. Speech products were recorded using an external microphone Fifine Technology K058. Eye movement recordings were processed in the R statistical processing environment (R Core Team, 2020) using the ETRAN package (Zhegallo & Marmalyuk, 2015). Fixation detection was performed using the I-DT (Dispersion Threshold Identification) algorithm, the minimum duration of fixation was 6 samples (50 ms), and the threshold dispersion was 40 px.

The study involved psychology students from Moscow universities, a total of 29 people, mean age m=23.4 years, sd=6.7; 13 men and 16 women.

Results

• Relationship between eye movements, image viewing patterns and generated verbal production

The main characteristics of the eye movements in this study were previously described in (Zhegallo, 2020)2]. In current paper, we consider an example of correlating eye movements and speech production. In the original album, the facial images were accompanied by descriptions of facial expressions, reflecting the author’s interpretation of the sitter’s facial expressions.

As an example, consider a Plate II image that has the following characteristic: «The effects of attention are to make the eye-brows sink and approach the sides of the nose, to turn the eye-balls towards the object that causes it, to open the mouth, and especially the upper part, to decline the head a little, and fix it, without any other remarkable alteration».

The subject gave the following description: “ahh so well, he probably has some kind of displeasure on his face, because his eyebrows are furrowed, ahh he looks askance uu(x) his mouth is open as if he doesn’t know he is going to quarrel with someone. Most likely, this is dissatisfaction with something, irritation, maybe even anger. Okay, everything about him probably”.  This description indicates that the subject noted the main features of the image (drooping (frowning) eyebrows and open mouth), but gave his own interpretation of the emotional state.

Parallel analysis of eye movements and speech production was performed by us using the original python program. The program visualizes the spatial and temporal representation of eye movements in the form of “raw” data and visual fixations; visualization of audio recording, playback of selected fragments of audio recording. The application of the text of verbalizations and the correlation of spatial and temporal markings were done manually in the Inkscape program. To increase the clarity of the presentation of the results, an experimental situation with a duration of 30 sec. divided into three consecutive intervals lasting 10 seconds. Visualization is presented in Fig. 1a, Fig. 1b, Fig. 1c. As can be seen from the visualizations, viewing the image is mainly based on the areas of the eye, mouth, and ear. At the same time, several visual fixations may correspond to a separate verbalization containing a generalized description. The completion of the description is characterized by a sequence of prolonged visual fixations in the region of the eye.

Consider separately visual fixations, corresponding to references to specific key elements of the face. The description “mouth open” corresponds to a sequence of three fixations in the mouth area, starting 300 ms before the start of verbalization. In this case, “open” corresponds to the fourth fixation, shifted back, closer to the junction of the upper and lower lips (Fig. 2a). Thus, when describing specific partial elements of facial expression, eye movements and speech production are synchronized, however, even in these cases, comparison of the type is the only fixation – the replica turns out to be irrelevant. Eye movements have their own internal structure. Descriptions of integral characteristics correspond to large-scale sequences of fixations covering the entire surface of the
face. Thus, eye movements and generated speech production can be considered as two parallel processes that have their own external manifestations and obey their own laws. Explicitly expressed synchronization of processes is observed in the generation of descriptions corresponding to individual partial elements The description of “frown eyebrows” corresponds to a cyclic sequence of six fixations with a total duration of 2.5 seconds: pupil, bridge of the nose, outer corner of the eyebrow, upper lip, nostril, pupil of the eye (Fig. 2b).

Figure 1а. Spatial and temporal representation of eye movements, time interval 0 – 10 sec. The diameters of the circles denoting visual fixations correspond to the dispersion value. The transcript is given in Russian for a visual representation of the exact volume of speech production.

Figure 1b. Spatial and temporal representation of eye movements, time interval 10 – 20 sec.

Figure 1c. Spatial and temporal representation of eye movements, time interval 20 – 30 sec.

Figure 2а. A sequence of fixations corresponding to the description “mouth open”.

Figure 2b. A sequence of fixations corresponding to the description “brows furrowed”.

The presented experience of analysis shows the promise of a parallel analysis of eye movements and speech production, but at the same time indicates the need for further improvement of methodological methods of analysis. The unit of analysis should be considered a verbal unit of description (Nosulenko, 2021). The software must ensure the imposition of the decoding of verbal units on the time base; visualization of the spatial sweep corresponding to the analyzed verbal unit, the ability to adjust the time boundaries of verbal units, the ability to calculate statistics for each verbal unit (average duration of fixations, length of the trajectory, coverage area). A separate problem is the correlation of the localization of visual fixations with the image under consideration. Until now, in our studies, markings containing 5 areas of interest seemed to be sufficient: left eye, bridge of the nose, right eye, nose, mouth (Zhegalo, 2017). The examples discussed above demonstrate the need to introduce more fractional zones of interest. A possible solution here can be the marking of areas of interest, based on the actual distribution of gaze density in the image under consideration.

The possible range of individual components that contribute to the recognition of emotional expression is not limited to eye movements based on the actual structure of the image of the sitter’s face and generated by speech production. Isolation of other components that do not have specific external manifestations can be based only on indirect experimental data. Thus, the absence of links between the accuracy of solving the discriminative ABX problem and the asymmetry of erroneous answers (Zhegalo & Korolkova, 2019) may indicate that the asymmetry of erroneous answers is associated with a specific perceptual component. The actual contribution of the semantic component to the solution of the discrimination problem requires additional study. The main difficulty here is that the possible verbal component here, if present, is in a condensed, implicit form of “inner speech”. Accordingly, it is necessary to search for methodological solutions that allow it to be recorded either in the form of residual manifestations characteristic of internal speech (Sokolov, 2007) or in the form of the electrical activity of the brain accompanying speech.

In the future, as possible candidates for the external manifestations of individual components of the process of recognition of facial expressions, we consider the dynamics of ECG, GSR, integral changes in facial expressions (Zhegallo & Basyul, 2021).

The division of the process of perception of facial expressions into separate interacting components and the transition to the study of the interaction between them corresponds to the component theory of emotions by Klaus Scherer. In Scherer’s original theory, the components of the emotional process include: Multilevel appraisals, Action tendencies, Physiological responses, Motor expression, Component-integration experienced feeling, categorization labeling (Scherer, 2022).

We believe that the emotional process in the general case may include duplicating components running in parallel, working to obtain a combined result; taking into account the context of the situation; the attitude of the subject to the current situation and the experimental problem being solved, etc. The task of the researcher when analyzing the results of a particular experiment is to take into account all the components that are actually involved in the performance of the task set by the experimenter. Thus, the solution of the ABX-discrimination task can, in principle, occur simultaneously at the semantic and perceptual levels, and at the perceptual level, it is possible to compare both integral images and their fragments, depending on the strategy used by the subject. The end result will be determined by the competition of the involved convergent processes.

The main characteristics of the eye movements in this study were previously described in (Zhegallo, 2020).

Conclusions

Funding: The work was supported by the RF State Assignments nos. 0138-2023-0006.

Competing interests: None.

References

1. Barabanschikov, V.A., Zhegallo, A.V. & Korolkova, O.A. (2015). Perceptual categorization of facial expressions. Moscow: Cogito-Center. [Barabanshchikov V.A., Zhegallo A.V., Korol’kova O.A. Pertseptivnaya kategorizatsiya vyrazheniy litsa. M.: KogitoTsentr, 2015.].
2. Barrett, L.F. (2017). How emotions are made. The Secret Life of the Brain. Boston, New York Houghton Mifflin Harcourt.
3. Ekman, P., Friesen, W.V. & Tomkins, S.S. (1971). Facial Affect Scoring Technique: A First Validity Study. Semiotica, 3(1), 37-58. DOI: 10.1515/semi.1971.3.1.37
4. Ekman, P. & Friesen, W.V. (1971). Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17(2), 124 – 129. DOI: 10.1037/h0030377
5. Ekman, P. (1971). Universals and Cultural Differences in Facial Expressions of Emotion. In Nebraska Symposium on Motivation 4. Ed. J.K. Code Lincoln, Nebraska, University of Nebraska Press, 207 – 283.
6. Ekman, P., Friesen, W.V. & Ellsworth, P. (1972). Emotion in the human face: Guidelines for Research and an Integration of Findings. New York, Cambridge University Press.
7. Ekman, P. & Friesen W.V. (1976). Pictures of Facial Affect. Consulting Psychologists Press, Palo Alto, CA.
8. Ekman, P. & Friesen, W.V. (1978). Facial Action Coding System: A Technique for the Measurement of Facial Movement. Palo Alto, CA: Consulting Psychologist Press.
9. Ekman, P. (1984). Expression and the nature of emotion/ In Approaches to emotion. Ed. Scherer K., Ekman P. Hillsdale, NJ: Erlbaum. pp. 319 – 344.
10. Ekman, P. (1993). Facial Expression and Emotion. American Psychologist, 48(4), 376 – 379.
11. Ekman, P. (1999). Basic Emotions. In Handbook of Cognition and Emotion. Ed. Dalgleish T., Power M. Sussex, UK: John Wiley & Sons, 45 – 60.
12. Ekman, P. & Cordaro, D. (2011). What is meant by calling emotions basic. Emotion Review, 3(4), 364 – 370.
13. Etcoff, N. & Magee, J. (1992). Categorical perception of facial expressions. Cognition, 44(3), 227 – 240.
14. Harnad, S. (ed.). (1987). Categorical perception. Cambridge, UK: Cambridge University Press.
15. Liberman, A.M., Harris, K.S., Hoffman, H.S., & Griffith, B.C. (1957). The discrimination of speech sounds within and across phoneme boundaries. Journal of Experimental Psychology, 54(5), 358-368.
16. Nosulenko, V.N. (2021). Questions of integration of qualitative and quantitative methods in psychological study. Experimental Psychology, 14(3), 4- 16. DOI: https://doi.org/10.17759/exppsy.2021140301 [Nosulenko V.N. Voprosy integratsii kachestvennykh i kolichestvennykh metodov v psikhologicheskom issledovanii // Eksperimental’naya psikhologiya. 2021. Tom 14. № 3. S. 4–16.].
17. R Core Team. (2020). R: A language and environment for statistical computing. In R Foundation for Statistical Computing. Vienna, Austria. URL: https://www.R-project.org/ (Available at: 10.01.2023).
18. Scherer, K. (2022). Theory convergence in emotion science is timely and realistic. Cognition and Emotion, 36(2), 154 – 170. DOI: 10.1080/02699931.2021.1973378
19. Sokolov, A.N. (2007). Inner speech and thinking. Ed. 2nd Moscow: LKI. [Sokolov A.N. Vnutrennyaya rech’ i myshleniye Izd. 2 URSS. 2007.].
20. Zhegallo, A.V. (2016). Using the PXLab software to prepare experiments involving the registration of eye movements using Sensomotoric Instruments eye trackers. Russian Journal of Cognitive Science, 3(3), 43 – 57. [Zhegallo, Alexander. (2017). Ispol’zovaniye PO PXLab dlya podgotovki eksperimentov, vklyuchayushchikh registratsiyu dvizheniy glaz s pomoshch’yu aytrekerov Sensomotoric Instruments. Rossiyskiy zhurnal kognitivnoy nauki, 3, 43-57.].
21. Zhegalo, A.V. (2017). Indicators of oculomotor activity of the observer during the perception of video and photo images of the face. In Сognitive mechanisms of non-verbal communication. Ed. V.A. Barabanschikov. Moscow: Cogito-Center, 146 – 188. [Zhegalo, A.V. (2017). Pokazateli glazodvigatel’noy aktivnosti nablyudatelya pri vospriyatii video- i fotoizobrazheniy litsa. / Kognitivnyye mekhanizmy neverbal’noy kommunikatsii. Red. V.A. Barabanshchikov. Moskva: Kogito-Tsentr, 146 – 188.].
22. Zhegalllo, A.V. (2020). Principles for comparing indicators oculomotor activity when looking different types of images of emotional expressions. In Abilities and mental resources of a person in the world of global change. Ed. Zhuravlev A.L., Kholodnaya M.A., Sabadosh P.A. Moscow: IPRAS, 1572 – 1581. [Zhegallo, A.V. (2020). Printsipy sravneniya pokazateley glazodvigatel’noy aktivnosti pri prosmotre razlichnykh tipov izobrazheniy emotsional’nykh proyavleniy / Sposobnosti i mental’nyye resursy cheloveka v mire global’nykh peremen. Red. Zhuravlev A.L., Kholodnaya M.A., Sabadosh P.A. Moskva: IPRAN, 1572 – 1581.].
23. Zhegallo, A.V. & Basyul, I.A. (2021). Indicators of oculomotor activity as indicators of the reliability / unreliability of the information reported. Cognition and experience, 2(4), 34—47. DOI: 10.51217/cogexp_2021_02_04_03 [Zhegallo, A.V., Basyul, I.A. Pokazateli okulomotornoy aktivnosti kak indikatory dostovernosti/nedostovernosti soobshchayemoy informatsii // Poznaniye i perezhivaniye, 2021. T. 2. № 4. S. 37-53.].
24. Zhegalo, A.V. & Korolkova, O.A. (2019). “Basic” Emotional Expressions: verification the Categorical Adjustment Model. In Cognitive Studies: Emergence and Complexity, Cognitive Practices: VIII Intern. interdisciplinary. conf. Issue. 8 Minsk: BSPU, 417 – 424. [Zhegalo, A.V. & Korol’kova O.A. (2019). «Osnovnyye» emotsional’nyye vyrazheniya: proverka modeli kategorial’noy adaptatsii. V kognitivnykh issledovaniyakh: vozniknoveniye i slozhnost’, kognitivnyye praktiki: VIII mezhdunar. mezhdistsiplinarnyy. konf. Problema. 8 Minsk: BGPU, 417 – 424.].
25.  Zhegallo, A.V. & Marmalyuk, P. A. (2015). ETRAN – R Extension Package for Eye Tracking Results Analysis. Perception, 8–9, 1129