Cognitive architectures have often been applied to data from individual experiments. In this paper, I develop an ACT-R reader that can model a much larger set of data, eye-tracking corpus data. It is shown that the resulting model has a good fit to the data for the considered low-level processes. Unlike previous related works, the model achieves the fit by estimating free parameters of ACT-R using Bayesian estimation and Markov-Chain Monte Carlo techniques, rather than by (...) relying on the mix of manual selection + default values. The method used in the paper is generalizable beyond this particular model and data set and could be used on other ACT-R models. (shrink)

This paper develops a novel psycholinguistic parser and tests it against experimental and corpus reading data. The parser builds on the recent research into memory structures, which argues that memory retrieval is content‐addressable and cue‐based. It is shown that the theory of cue‐based memory systems can be combined with transition‐based parsing to produce a parser that, when combined with the cognitive architecture ACT‐R, can model reading and predict online behavioral measures (reading times and regressions). The parser's modeling (...) capacities are tested against self‐paced reading experimental data (Grodner & Gibson, 2005), eye‐tracking experimental data (Staub, 2011), and a self‐paced reading corpus (Futrell et al., 2018). (shrink)

Research in experimental philosophy has increasingly been turning to corpus methods to produce evidence for empirical claims, as they open up new possibilities for testing linguistic claims or studying concepts across time and cultures. The present article reviews the quasi-experimental studies that have been done using textual data from corpora in philosophy, with an eye for the modeling and experimental design that enable statistical inference. I find that most studies forego comparisons that could control for confounds, and (...) that only a little less than half employ statistical testing methods to control for chance results. Furthermore, at least some researchers make modeling decisions that either do not take into account the nature of corpora and of the word-concept relationship, or undermine the experiment's capacity to answer research questions. I suggest that corpus methods could both provide more powerful evidence and gain more mainstream acceptance by improving their modeling practices. (shrink)

We explore the interaction between oculomotor control and language comprehension on the sentence level using two well-tested computational accounts of parsing difficulty. Previous work (Boston, Hale, Vasishth, & Kliegl, 2011) has shown that surprisal (Hale, 2001; Levy, 2008) and cue-based memory retrieval (Lewis & Vasishth, 2005) are significant and complementary predictors of reading time in an eyetracking corpus. It remains an open question how the sentence processor interacts with oculomotor control. Using a simple linking hypothesis proposed in Reichle, Warren, (...) and McConnell (2009), we integrated both measures with the eye movement model EMMA (Salvucci, 2001) inside the cognitive architecture ACT-R (Anderson et al., 2004). We built a reading model that could initiate short “Time Out regressions” (Mitchell, Shen, Green, & Hodgson, 2008) that compensate for slow postlexical processing. This simple interaction enabled the model to predict the re-reading of words based on parsing difficulty. The model was evaluated in different configurations on the prediction of frequency effects on the Potsdam Sentence Corpus. The extension of EMMA with postlexical processing improved its predictions and reproduced re-reading rates and durations with a reasonable fit to the data. This demonstration, based on simple and independently motivated assumptions, serves as a foundational step toward a precise investigation of the interaction between high-level language processing and eye movement control. (shrink)

Multimodal Critical Discourse Analysis (MCDA) as a sub-discipline of Critical Discourse Analysis (CDA) emerged from the availability of social semiotic frameworks describing multimodal meaning making. However, weaknesses of these frameworks have raised concerns and prompted recent methodological reflections in MCDA. Inspired by these reflections, this paper critically assesses MCDA research on advertising persuasion and identifies a lack of attention in studies to account for the social and ideological impact of advertising. This shortcoming is argued to be attributable to the weak (...) empirical grounding of social semiotic frameworks which in turn is rooted in a ‘discursive gap’ (Bateman, Citation2019, p. 300) in multimodality as a discipline. The authors then propose methodological solutions involving eye tracking experiments and higher order data analytics which can strengthen the empirical basis of theoretical assumptions made in MCDA research on advertising. Drawing on methodological guidance for empirical multimodal research, the authors discuss specific issues in realizing this proposal, focusing on a strategy addressing operationalization challenges in experimental design and the affordance of a data cluster algorithm that calculates the common viewing path of a group of viewers. (shrink)

This article aims to describe the construction and annotation of a multimodal and multilingual (French, English, and Mandarin Chinese) corpus for the study of second language acquisition and professional development in language teaching. The corpus was built within a research project whose main objective is to determine which techno-semio-pedagogical competencies are developed informally and which ones require formal training. In this paper, we explain the theoretical framework adopted, characterized by an ecological approach to the interactive environment. We subsequently (...) illustrate the procedures for the collection of audio, video and eye-tracking data. We also provide details of the annotation of raw data to produce a corpus of analysis drawing on different automatic and semi-automatic annotation tools. Finally, we explain how such a corpus enables us to study the development of learners’ multimodal competence and tutors’ techno-pedagogical competence and point to some other possible research questions. (shrink)

How do people evaluate causal relationships? Do they just consider what actually happened, or do they also consider what could have counterfactually happened? Using eye tracking and Gaussian process modeling, we investigated how people mentally simulated past events to judge what caused the outcomes to occur. Participants played a virtual ball‐shooting game and then—while looking at a blank screen—mentally simulated (a) what actually happened, (b) what counterfactually could have happened, or (c) what caused the outcome to happen. Our findings (...) showed that participants moved their eyes in patterns consistent with the actual or counterfactual events that they mentally simulated. When simulating what caused the outcome to occur, participants moved their eyes consistent with simulations of counterfactual possibilities. These results favor counterfactual theories of causal reasoning, demonstrate how eye movements can reflect simulation during this reasoning and provide a novel approach for investigating retrospective causal reasoning and counterfactual thinking. (shrink)

Eye-tracking technology has become a powerful tool for biomedical-related applications due to its simplicity of operation and low requirements on patient language skills. This study aims to use the machine-learning models and deep-learning networks to identify key features of eye movements in Alzheimer's Disease (AD) under specific visual tasks, thereby facilitating computer-aided diagnosis of AD. Firstly, a three-dimensional (3D) visuospatial memory task is designed to provide participants with visual stimuli while their eye-movement data are recorded and used to build (...) an eye-tracking dataset. Then, we propose a novel deep-learning-based model for identifying patients with Alzheimer's Disease (PwAD) and healthy controls (HCs) based on the collected eye-movement data. The proposed model utilizes a nested autoencoder network to extract the eye-movement features from the generated fixation heatmaps and a weight adaptive network layer for the feature fusion, which can preserve as much useful information as possible for the final binary classification. To fully verify the performance of the proposed model, we also design two types of models based on traditional machine-learning and typical deep-learning for comparison. Furthermore, we have also done ablation experiments to verify the effectiveness of each module of the proposed network. Finally, these models are evaluated by four-fold cross-validation on the built eye-tracking dataset. The proposed model shows 85% average accuracy in AD recognition, outperforming machine-learning methods and other typical deep-learning networks. (shrink)

Face perception and emotion categorization are widely investigated under laboratory conditions that are devoid of real social interaction. Using mobile eye-tracking glasses in a standardized diagnostic setting while applying the Autism Diagnostic Observation Schedule, we had the opportunity to record gaze behavior of children and adolescents with and without Autism Spectrum Conditions during social interaction. The objective was to investigate differences in eye-gaze behavior between three groups of children and adolescents either with ASC or with unconfirmed diagnosis of ASC or (...) with neurotypical development during social interaction with an adult interviewer in a diagnostic standard situation using the ADOS-2. In a case control study, we used mobile eye-tracking glasses in an ecologically valid and highly standardized diagnostic interview to investigate suspected cases of ASC. After completion of the ASC diagnostic gold standard including the ADOS-2, the participants were assigned to two groups based on their diagnosis and compared with a matched group of neurotypically developed controls. The primary outcome measure is the percentage of total dwell times assessed for different areas of interest with regard to the face and body of a diagnostic interviewer and the surrounding space. Overall, 65 children and adolescents within an age range of 8.3–17.9 years were included in the study. The data revealed significant group differences, especially in the central-face area. Previous investigations under laboratory conditions gave preferential attention to the eye region during face perception to describe differences between ASC and NTD. In this study – using an ecologically valid setting within a standard diagnostic procedure – the results indicate that neurotypically developed controls seem to process faces and facial expressions in a holistic manner originating from the central-face region. Conversely, participants on the Autism Spectrum seem to avoid the central-face region and show unsystematic gaze behavior, not using the preferred landing position in the central-face region as the Archimedean point of face perception. This study uses a new approach, and it will be important to replicate these preliminary findings in future research. (shrink)

One of the most important social cognitive skills in humans is the ability to “put oneself in someone else’s shoes,” that is, to take another person’s perspective. In socially situated communication, perspective taking enables the listener to arrive at a meaningful interpretation of what is said and what is meant by the speaker. To successfully decode the speaker’s meaning, the listener has to take into account which information he/she and the speaker share in their common ground. We here further investigated (...) competing accounts about when and how CG information affects language comprehension by means of reaction time measures, accuracy data, event-related potentials, and eye-tracking. Early integration accounts would predict that CG information is considered immediately and would hence not expect to find costs of CG integration. Late integration accounts would predict a rather late and effortful integration of CG information during the parsing process that might be reflected in integration or updating costs. Other accounts predict the simultaneous integration of privileged ground and CG perspectives. We used a computerized version of the referential communication game with object triplets of different sizes presented visually in CG or PG. In critical trials, CG information had to be integrated while privileged information had to be suppressed. Listeners mastered the integration of CG. Yet, slower RTs, and enhanced late positivities in the ERPs showed that CG integration had its costs. Moreover, eye-tracking data indicated an early anticipation of referents in CG but an inability to suppress looks to the privileged competitor, resulting in later and longer looks to targets in those trials, in which CG information had to be considered. Our data therefore support accounts that foresee an early anticipation of referents to be in CG but a rather late and effortful integration if conflicting information has to be processed. We show that both perspectives, PG and CG, contribute to socially situated language processing and discuss the data with reference to theoretical accounts and recent findings on the use of CG information for reference resolution. (shrink)

Both linguistic and extralinguistic consultations are essential in translation practice and have been commonly investigated as an integral topic in previous studies. However, since extralinguistic information is usually longer in extent and not specifically designed for a linguistic purpose, extralinguistic consultations involve different search strategies compared with linguistic consultations. Drawing on eye-tracking and screen-recording data, this study compares linguistic and extralinguistic consultations in terms of cognitive resources allocation and information processing patterns in English–Chinese translation. It also explores the differences (...) among 17 language learners, 20 student translators, and 21 professional translators, and the effect of extralinguistic consultation on their translation quality. The findings are as follows: all participants allocate more attention and lower cognitive load to extralinguistic consultations than to linguistic consultations; participants’ translation experience levels and their attention allocated to extralinguistic consultation show an inverted U-shaped relationship; and participants who consult extralinguistic information before drafting or devote more attention to extralinguistic consultation produce target texts with significantly higher scores. (shrink)

Attentional control theory suggests that high cognitive demands impair the flexible deployment of attention control in anxious adults, particularly when paired with external threats. Extending this...

At the core of Pickering & Garrod's (P&G's) theory is a monitor that uses forward models. I argue that this account is challenged by neuropsychological findings and visual world eye-tracking data and that it has two conceptual problems. I propose that conflict monitoring avoids these issues and should be considered a promising alternative to perceptual loop and forward modeling theories.

Installation art is one of the most important and provocative developments in the visual arts during the last half century and has become a key focus of artists and of contemporary museums. It is also seen as particularly challenging or even disliked by many viewers, and-due to its unique in situ, immersive setting-is equally regarded as difficult or even beyond the grasp of present methods in empirical aesthetic psychology. In this paper, we introduce an exploratory study with installation art, utilizing (...) a collection of techniques to capture the eclectic, the embodied, and often the emotionally-charged viewing experience. We present results from an investigation of two pieces, both part of Olafur Eliasson's exhibition "Baroque, Baroque" held at the Belvedere museum in Vienna. These were assessed by administered pre- and post-viewing questionnaires focusing on emotion, meaning-making, and appraisals, in tandem with mobile eye tracking to consider viewers' attention to both installed artworks and/or the museum environment. The data showed differences in participants' emotional states, appraisals, and visual exploration, which together paint a picture of the aesthetic reactions to the works. These differences also showed how viewers' appraisal strategies, meaning making, and physical actions facilitated relatively more or less deep engagement with, and enjoyment of, the art pieces. The results are discussed in terms of their implications for museum studies, art education, and theory in empirical aesthetics. (shrink)

What is philosophy of science? Numerous manuals, anthologies or essays provide carefully reconstructed vantage points on the discipline that have been gained through expert and piecemeal historical analyses. In this paper, we address the question from a complementary perspective: we target the content of one major journal of the field—Philosophy of Science—and apply unsupervised text-mining methods to its complete corpus, from its start in 1934 until 2015. By running topic-modeling algorithms over the full-text corpus, we identified 126 (...) key research topics that span across 82 years. We also tracked their evolution and fluctuating significance over time in the journal articles. Our results concur with and document known and lesser-known episodes of the philosophy of science, including the rise and fall of logic and language-related topics, the relative stability of a metaphysical and ontological questioning (space and time, causation, natural kinds, realism), the significance of epistemological issues about the nature of scientific knowledge as well as the rise of a recent philosophy of biology and other trends. These analyses exemplify how computational text-mining methods can be used to provide an empirical large-scale and data-driven perspective on the history of philosophy of science that is complementary to other current historical approaches. (shrink)

This study looked for evidence of biases in the allocation and disengagement of attention in dysphoric individuals. Participants studied images for a recognition memory test while their eye fixations were tracked and recorded. Four image types were presented (depression-related, anxiety-related, positive, neutral) in each of two study conditions. For the simultaneous study condition, four images (one of each type) were presented simultaneously for 10 seconds, and the number of fixations and the total fixation time to each image was measured, similar (...) to the procedure used by Eizenman et al. (2003) and Kellough, Beevers, Ellis, and Wells (2008). For the sequential study condition, four images (one of each type) were presented consecutively, each for 4 seconds; to measure disengagement of attention an endogenous cuing procedure was used (Posner, 1980). Dysphoric individuals spent significantly less time attending to positive images than non-dysphoric individuals, but there were no group differences in attention to depression-related images. There was also no evidence of a dysphoria-related bias in initial shifts of attention. With respect to the disengagement of attention, dysphoric individuals were slower to disengage their attention from depression-related images. The recognition memory data showed that dysphoric individuals had poorer memory for emotional images, but there was no evidence of a conventional mood-congruent memory bias. Differences in the attentional and memory biases observed in depressed and dysphoric individuals are discussed. (shrink)

Starting with the hypothesis that analogical reasoning consists of a search of semantic space, we used eye-tracking to study the time course of information integration in adults in various formats of analogies. The two main questions we asked were whether adults would follow the same search strategies for different types of analogical problems and levels of complexity and how they would adapt their search to the difficulty of the task. We compared these results to predictions from the literature. Machine learning (...) techniques, in particular support vector machines (SVMs), processed the data to find out which sets of transitions best predicted the output of a trial (error or correct) or the type of analogy (simple or complex). Results revealed common search patterns, but with local adaptations to the specifics of each type of problem, both in terms of looking-time durations and the number and types of saccades. In general, participants organized their search around source-domain relations that they generalized to the target domain. However, somewhat surprisingly, over the course of the entire trial, their search included, not only semantically related distractors, but also unrelated distractors, depending on the difficulty of the trial. An SVM analysis revealed which types of transitions are able to discriminate between analogy tasks. We discuss these results in light of existing models of analogical reasoning. (shrink)

This article argues that a theory of media selectivity needs a theory of attention, because attention to a media stimulus is the starting point of each process of reception. Attention sequences towards media stimuli – pages of newspapers and online-newspapers – were analyzed using eye-tracking patterns from three different perspectives. First, attention patterns were compared under varying task conditions. Second, different types of media were tested. Third, attention sequences towards different forms of news with different design patterns were compared. Attention (...) was seen as a prerequisite for reception: Its selective functions for these processes are especially important. Reception itself was examined within an action-theoretical framework and therefore described as a form of interaction between recipient and the media. Eye-tracking data were used as indicators of attention. Starting with a hypothesis on the impact of different media such as printed newspapers and online newspapers on the agenda-setting process of their audience, the study examined how the type of media and the form of news influences attention and selectivity. Our findings showed that visual cues such as salient photos or graphics and information hierarchies signalled by design and layout guide attention processes, not as an automatic process driven from the bottom up, but as stimuli for an active, intention-driven selection process. The results indicate that the form of news affects these patterns of interactive attention more than the medium itself. (shrink)

Online data collection with infants raises special opportunities and challenges for developmental research. One of the most prevalent methods in infancy research is eye-tracking, which has been widely applied in laboratory settings to assess cognitive development. Technological advances now allow conducting eye-tracking online with various populations, including infants. However, the accuracy and reliability of online infant eye-tracking remain to be comprehensively evaluated. No research to date has directly compared webcam-based and in-lab eye-tracking data from infants, similarly to (...) class='Hi'>data from adults. The present study provides a direct comparison of in-lab and webcam-based eye-tracking data from infants who completed an identical looking time paradigm in two different settings. We assessed 4-6-month-old infants in an eye-tracking task that measured the detection of audio-visual asynchrony. Webcam-based and in-lab eye-tracking data were compared on eye-tracking and video data quality, infants’ viewing behavior, and experimental effects. Results revealed no differences between the in-lab and online setting in the frequency of technical issues and participant attrition rates. Video data quality was comparable between settings in terms of completeness and brightness, despite lower frame rate and resolution online. Eye-tracking data quality was higher in the laboratory than online, except in case of relative sample loss. Gaze data quantity recorded by eye-tracking was significantly lower than by video in both settings. In valid trials, eye-tracking and video data captured infants’ viewing behavior uniformly, irrespective of setting. Despite the common challenges of infant eye-tracking across experimental settings, our results point toward the necessity to further improve the precision of online eye-tracking with infants. Taken together, online eye-tracking is a promising tool to assess infants’ gaze behavior but requires careful data quality control. The demographic composition of both samples differed from the generic population on caregiver education: our samples comprised caregivers with higher-than-average education levels, challenging the notion that online studies will per se reach more diverse populations. (shrink)

Recent advances in neuroscience indicate that analysis of bio-signals such as rest state electroencephalogram and eye-tracking data can provide more reliable evaluation of children autism spectrum disorder than traditional methods of behavior measurement relying on scales do. However, the effectiveness of the new approaches still lags behind the increasing requirement in clinical or educational practices as the “bio-marker” information carried by the bio-signal of a single-modality is likely insufficient or distorted. This study proposes an approach to joint analysis of (...) EEG and eye-tracking for children ASD evaluation. The approach focuses on deep fusion of the features in two modalities as no explicit correlations between the original bio-signals are available, which also limits the performance of existing methods along this direction. First, the synchronization measures, information entropy, and time-frequency features of the multi-channel EEG are derived. Then a random forest applies to the eye-tracking recordings of the same subjects to single out the most significant features. A graph convolutional network model then naturally fuses the two group of features to differentiate the children with ASD from the typically developed subjects. Experiments have been carried out on the two types of the bio-signals collected from 42 children. The results indicate that the proposed approach can achieve an accuracy of 95% in ASD detection, and strong correlations exist between the two bio-signals collected even asynchronously, in particular the EEG synchronization against the face related/joint attentions in terms of covariance. (shrink)

Behavioral studies on language processing rely on the eye-mind assumption, which states that the time spent looking at text is an index of the time spent processing it. In most cases, relatively shorter reading times are interpreted as evidence of greater processing efficiency. However, previous evidence from L2 research indicates that non-native participants who present fast reading times are not always more efficient readers, but rather shallow parsers. Because earlier studies did not identify a reliable predictor of variability in L2 (...) processing, such uncertainty around the interpretation of reading times introduces a potential confound that undermines the credibility and the conclusions of online measures of processing. The present study proposes that a recently developed modulator of online processing efficiency, namely, chunking ability, may account for the observed variability in L2 online reading performance. L1 English – L2 Spanish learners’ eye movements were analyzed during natural reading. Chunking ability was predictive of overall reading speed. Target relative clauses contained L2 Verb-Noun multiword units, which were manipulated with regards to their L1-L2 congruency. The results indicated that processing of the L1-L2 incongruent units was modulated by an interaction of L2 chunking ability and level of knowledge of multiword units. Critically, the data revealed an inverse U-shaped pattern, with faster reading times in both learners with the highest and the lowest chunking ability scores, suggesting fast integration in the former, and lack of integration in the latter. Additionally, the presence of significant differences between conditions was correlated with individual chunking ability. The findings point at chunking ability as a significant modulator of general L2 processing efficiency, and of cross-language differences in particular, and add clarity to the interpretation of variability in the online reading performance of non-native speakers. (shrink)

In this study, 44 late elementary students’ visual behavior patterns when reading mind maps were investigated, more particularly, the intuitive processing nature of their visual characteristics, reading sequence and presentation mode. Eye-tracked data were investigated by means of static early attention and dynamic educational process mining analysis and combined with learning performance and retrospective interview data. All students seem to struggle with the map’s radial structure during initial reading. Also, the picture’s position in the map diverts students from (...) consecutively reading interconnected branches. EPM analyses revealed different reading patterns in proceeding reading behavior. Students receiving a text first, seem to grasp the radial structure slightly more and show higher information integration attempts. They also attained higher free recall and coherence scores. The study concludes with instructional design principles for urgently needed explicit visual literacy instruction in elementary grades. (shrink)

The aim of this study is to assess the role of readers’ proficiency and of the base-word distributional properties on eye-movement behavior. Sixty-two typically developing children, attending 3rd, 4th, and 5th grade, were asked to read derived words in a sentence context. Target words were nouns derived from noun bases (e.g., umorista, ‘humorist’), which in Italian are shared by few derived words, and nouns derived from verb bases (e.g., punizione, ‘punishment’), which are shared by about 50 different inflected forms and (...) several derived words. Data showed that base and word frequency affected first-fixation duration for nouns derived from noun bases, but in an opposite way: Base frequency had a facilitative effect on first fixation, whereas word frequency exerted an inhibitory effect. These results were interpreted as a competition between early accessed base words (e.g., camino, chimney) and target words (e.g., caminetto, fireplace). For nouns derived from verb bases, an inhibitory base frequency effect but no word frequency effect were observed. These results suggest that syntactic context, calling for a noun in the target position, lead to an inhibitory effect when a verb base was detected, and made it difficult for readers to access the corresponding base+suffix combination (whole word) in the very early processing phases. Gaze duration was mainly affected by word frequency and length: for nouns derived from noun bases, this interaction was modulated by proficiency, as length effect was stronger for less proficient readers, while they were processing low-frequency words. For nouns derived from verb bases, though, all children, irrespective of their reading ability, showed sensitivity to the interaction within frequency of base+suffix combination (word frequency) and target length. Results of this study are consistent with those of other Italian studies that contrasted noun and verb processing, and confirm that distributional properties of morphemic constituents have a significant impact on the strategies used for processing morphologically complex words. (shrink)

Purpose This paper aims to discuss research ethics in mixed-methods research and MMR development with a focus on ethical challenges that stem from working with technical instruments such as mobile eye-trackers. Design/methodology/approach The case of an interdisciplinary mixed-methods development study that aimed at researching the impacts of emerging mobile augmented-reality technologies on the perception of public places serves as an example to discuss research-ethical challenges regarding the practical implementation of the study, data processing and management and societal implications of (...) developing instruments to track and understand human practices. Findings This study reports challenges and experiences in ethical decision-making in the practical implementation of the study regarding the relationship to research subjects, the use of mobile research instruments in public places and the interdisciplinary cooperation among research team members. Further, this paper expounds on ethical challenges and recommendations in data processing and management and with a view to societal implications of method development and the aspirations of transdisciplinarity. This study concludes that institutionalized ethics need to become more flexible, while applied ethics and reflection must make their entry into university curricula across disciplines. Originality/value Complex interdisciplinary mobile and mixed-methods projects that involve sensors and instruments such as mobile eye-trackers are on the rise. However, there is a significant lack of engagement with practical research ethical challenges, practices and requirements in both mixed-methods and method-development literature. By taking a context- and process-oriented perspective focusing on doing ethics, the paper contributes a concrete empirical case to these underdeveloped fields. (shrink)

Visual working memory is a construct hypothesized to store a small amount of accurate perceptual information that can be brought to bear on a task. Much research concerns the construct's capacity and the precision of the information stored. Two prominent theories of VWM representation have emerged: slot-based and continuous-resource mechanisms. Prior modeling work suggests that a continuous resource that varies over trials with variable capacity and a potential to make localization errors best accounts for the empirical data. Questions (...) remain regarding the variability in VWM capacity and precision. Using a novel eye-tracking paradigm, we demonstrate that VWM facilitates search and exhibits effects of fixation frequency and recency, particularly for prior targets. Whereas slot-based memory models cannot account for the human data, a novel continuous-resource model does capture the behavioral and eye tracking data, and identifies the relevant resource as item activation. (shrink)

In 1990, Vermeerbergen started the first larger-scale corpus study with spontaneous language data from adult signers on the morpho-syntactic aspects of Flemish Sign Language. After this, a number of lexicographic projects, including the collection of a 90-h corpus, led to the launch of the first online bilingual Dutch/VGT—VGT/Dutch dictionary in 2004. Since then, researchers have developed several corpora of variable sizes, with the greatest realization being the VGT Corpus. The main focus of this chapter is twofold. (...) On the one hand the run-up to, the development and the use of the VGT Corpus will be discussed, while on the other hand smaller specific research corpora will be highlighted such as the corpus on early parent-child interaction and the multifocal eye-tracking corpus. The current chapter will discuss the research and community value of the corpora and future directions. Finally, it will elaborate on the need for corpus research, the associated advantages and disadvantages, and the obstacles faced in smaller deaf communities. (shrink)

This study aimed to evaluate if eye blinks can be used to discriminate players with different performance in a session of Nintendo Entertainment System (NES) Tetris. To that end, we developed a state-of-the-art method for blink extraction from EAR measures, which is robust enough to be used with data collected by a low-grade webcam such as the ones widely available on laptop computers. Our results show a significant decrease in blink rate per minute (blinks/m) during the first minute of (...) playing Tetris. After having defined 3 groups of proficiency based on in-game performance (Novices, Intermediates, and Experts) we found out that expert players display a significantly lower decrease in blinks/m compared to novices during the first minute of gameplay, which shows that Tetris players' proficiency can be detected by looking at eye blinks/m variations during the early phase of a game session. This difference in blinks/m is observed throughout the entire game session, which supports the general conclusion that proficient Tetris players have a lower decrease in blinks/m, even when playing more difficult levels. Finally, we offer some interpretations of this effect and the relationship that our results may have with the visual cognitive workload experienced during the gameplay. (shrink)

Computational modeling and eye-tracking were used to investigate how phonological and semantic information interact to influence the time course of spoken word recognition. We extended our recent models to account for new evidence that competition among phonological neighbors influences activation of semantically related concepts during spoken word recognition . The model made a novel prediction: Semantic input modulates the effect of phonological neighbors on target word processing, producing an approximately inverted-U-shaped pattern with a high phonological density advantage at an (...) intermediate level of semantic input—in contrast to the typical disadvantage for high phonological density words in spoken word recognition. This prediction was confirmed with a new analysis of the Apfelbaum et al. data and in a visual world paradigm experiment with preview duration serving as a manipulation of strength of semantic input. These results are consistent with our previous claim that strongly active neighbors produce net inhibitory effects and weakly active neighbors produce net facilitative effects. (shrink)

This volume offers insights on experimental and empirical research in theoretical linguistic issues of negation and polarity, focusing on how negation is marked and how negative polarity is emphatic and how it interacts with double negation. Metalinguistic negation and neg-raising are also explored in the volume. Leading specialists in the field present novel ideas by employing various experimental methods in felicity judgments, eye tracking, self-paced readings, prosody and ERP. Particular attention is given to extensive crosslinguistc data from French, Catalan (...) and Korean along with analyses using semantic and pragmatic methods, corpus linguistics, diachronic perspectives, and longitudinal acquisitional studies as well as signed and gestural negation. Each contribution is situated with regards to major previous studies, thereby offering readers insights on the current state of the art in research on negation and negative polarity, highlighting how theory and data together contributes to the understanding of cognition and mind.. (shrink)

Parameters in E-Z Reader models are estimated on the basis of a simple data set consisting of 30 means. Because of heavy aggregation, the data have a severe problem of multicolinearity and are unable to adequately constrain parameter values. This could give the model more power than the empirical data warrant. Future models should exploit the richness of eye movement data and avoid excessive aggregation.

Until recently, experimental philosophy has been associated with the questionnaire-based study of intuitions; however, experimental philosophers now adapt a wide range of empirical methods for new philosophical purposes. New methods include paradigms for behavioural experiments from across the social sciences as well as computational methods from the digital humanities that can process large bodies of text and evidence. This book offers an accessible overview of these exciting innovations. The volume brings together established and emerging research leaders from several areas of (...) experimental philosophy to explore how new empirical methods can contribute to philosophical debates. Each chapter presents one or several methods new to experimental philosophy, demonstrating their application in a key area of philosophy and discussing their strengths and limitations. Methods covered include eye tracking, virtual reality technology, neuroimaging, statistical learning, and experimental economics as well as corpus linguistics, visualisation techniques and data and text mining. The volume explores their use in moral philosophy and moral psychology, epistemology, philosophy of science, metaphysics, philosophy of language, philosophy of mind and the history of ideas. Methodological Advances in Experimental Philosophy is essential reading for undergraduates, graduate students and researchers working in experimental philosophy. (shrink)

Eye tracking is used to investigate human choice procedures. We infer from eye movement patterns in choice problems where the deliberation process is clear to deliberations in problems of choice between two lotteries. The results indicate that participants tend to compare prizes and probabilities separately. The data provide little support for the hypothesis that decision makers use an expected utility type of calculation exclusively. This is particularly true when the calculations involved in comparing the lotteries are complicated.

In theatre rehearsals, actors can occasionally be seen getting stuck in the play text, which is called blanking. To overcome such textual difficulties and continue with the given text, a prompter can verbalize the line in question, thus contributing to an actor’s word search by prompting. The paper focuses on interactional practices by which prompters and actors interactionally resolve blanking situations. This study’s data comprises a case collection of 67 prompting situations, which are taken from a 200-h video (...) class='Hi'>corpus of a rehearsal process at a professional theatre. These cases demonstrate how theatre professionals organize prompting situations and how they negotiate/sanction prompting actions such as when there was no blanking but a dramatic pause or when the dramatic performance is interrupted due to a missing prompt. In addition to the audiovisual recordings, eye tracking data of the person prompting is also used to describe the coordination of the visual resources in the context of multimodal interaction analysis. The analysis suggests that prompting and blanking persons interactively resolve blankings with the help of verbal and visual markers. (shrink)

Based on the Competition Model, the current study investigated how cue availability and cue reliability as two important input factors influenced second language (L2) learners' cue learning of the English article construction. Written corpus data of university-level Chinese-L1 learners of English were sampled for a comparison of English majors and non-English majors who demonstrated two levels of L2 competence in English article usage. The path model analysis in structural equation modeling was utilized to investigate the relationship between (...) the input factors and L2 usage (frequency and accuracy of article cue production). The findings contribute novel and scarce empirical evidence that confirms a central claim of the Competition Model, i.e., the changing importance of cue availability and cue reliability in the frequency and accuracy of production. Cue availability was found to determine L2 production frequency regardless of level of L2 competence. Cue reliability was the input factor that differentiated competence levels. When learners stayed at a relatively lower L2 proficiency, cue reliability played an important role in influencing L2 frequency of usage rather than accuracy of usage. When learners developed increased exposure to and stronger competence in the target language, cue reliability played a significant role in determining learners' success of cue learning. The study is methodologically innovative and expands the empirical applicability of the Competition Model to the domain of second language production and construction learning. (shrink)

ABSTRACTFractions are defined by numerical relationships, and comparing two fractions’ magnitudes requires within-fraction and/or between-fraction relational comparisons. To better understand how individuals spontaneously reason about fractions, we collected eye-tracking data while they performed a fraction comparison task with conditions that promoted or obstructed different types of comparisons. We found evidence for both componential and holistic processing in this mixed-pairs task, consistent with the hybrid theory of fraction representation. Additionally, making within-fraction eye movements on trials that promoted a between-fraction comparison (...) strategy was associated with slower responses. Finally, participants who performed better on a non-numerical test of reasoning took longer to respond to the most difficult fraction trials, which suggests that those who had greater facility with non-numerical reasoning attended more to numerical relationships. These findings extend pri... (shrink)

We know that reading involves coordination between textual characteristics and visual attention, but research linking eye movements during reading and comprehension assessed after reading is surprisingly limited, especially for reading long connected texts. We tested two competing possibilities: (a) the weak association hypothesis: Links between eye movements and comprehension are weak and short‐lived, versus (b) the strong association hypothesis: The two are robustly linked, even after a delay. Using a predictive modeling approach, we trained regression models to predict comprehension (...) scores from global eye movement features, using participant‐level cross‐validation to ensure that the models generalize across participants. We used data from three studies in which readers (Ns = 104, 130, 147) answered multiple‐choice comprehension questions ~30 min after reading a 6,500‐word text, or after reading up to eight 1,000‐word texts. The models generated accurate predictions of participants' text comprehension scores (correlations between observed and predicted comprehension: 0.384, 0.362, 0.372, ps <.001), in line with the strong association hypothesis. We found that making more, but shorter fixations, consistently predicted comprehension across all studies. Furthermore, models trained on one study's data could successfully predict comprehension on the others, suggesting generalizability across studies. Collectively, these findings suggest that there is a robust link between eye movements and subsequent comprehension of a long connected text, thereby connecting theories of low‐level eye movements with those of higher order text processing during reading. (shrink)

Remote interpreting via video-link is increasingly being employed in investigative interviews chiefly due to its apparent increased accessibility and efficiency. However, risks of miscommunication have been shown to be magnified in remote interpreting and empirical research specifically on video-link remote interpreting is in its infancy which greatly limits the evidence base available to inform and direct evidence-based policy and best practice, particularly in the identification of the optimal mode of interpreting to be used, namely consecutive and simultaneous. Consecutive interpreting refers (...) to a process in which the interpreter transfers short segments of speech from one language into the other as each person speaks in managed turn-taking, while simultaneous interpreting refers to the transfer of natural speech from one language into another in a concurrent manner without the need for speakers to segment their speech. This study provides novel empirical evidence by using eye tracking to compare the overt visual attention of interpreters working in a remote setting in which an English-speaking Interviewer interacts with a non-English-speaking Suspect in person, for whom interpretation is provided via video-link in real time. Using a within-subject design, we analyze eye-movement data from 28 professionally accredited interpreters who interpreted via video-link an investigative interview in which consecutive and simultaneous interpreting modes were counterbalanced. Taking interpreting performance into account, our results showed that, the consecutive mode yielded significantly less gaze time and therefore significantly less on-screen overt visual attention due to off-screen notetaking, an essential component of the consecutive interpreting mode. Relative to gaze time, the consecutive mode also resulted in significantly more and longer fixations and shifts of attention. Participants also allocated significantly more overt visual attention to the Interviewer than the Suspect, particularly in the consecutive mode. Furthermore, we found informative significant correlations between eye tracking measures and interpreting performance: accuracy, verbal rapport, and management. Finally, we found no significant differences between the three language pairs tested. We conclude with a discussion of limitations and the contributions of the study and an outline for future work on this topic of growing importance. (shrink)

Based on various stimuli, the findings for the inattentional blindness paradigm suggest that many observers do not perceive an unexpected object in a dynamic setting. In a first experiment, inattentional blindness was combined with eye tracking data from children. Observers who did not notice the unexpected object in the basketball game test by Simons and Chabris spent on average as much time looking at the unexpected object as those subjects who did perceive it. As such, individual differences that are (...) responsible for the recognition of unexpected objects have to be found as further indicators. In a second experiment, the expert-novice paradigm was used to show that the probability of seeing an unexpected object can be increased with specific previous experience. The results in the same task indicate significant differences between basketball experts and basketball novices. The ages of the subjects in both experiments are discussed in connection with the inattentional blindness paradigm. (shrink)

There is increasing interest in how the pupil dynamics of the eye reflect underlying cognitive processes and brain states. Problematic, however, is that pupil changes can be due to non-cognitive factors, for example luminance changes in the environment, accommodation and movement. In this paper we consider how by modeling the response of the pupil in real-world environments we can capture the non-cognitive related changes and remove these to extract a residual signal which is a better index of cognition and (...) performance. Specifically, we utilize sequence measures such as fixation position, duration, saccades, and blink-related information as inputs to a deep recurrent neural network (RNN) model for predicting subsequent pupil diameter. We build and evaluate the model for a task where subjects are watching educational videos and subsequently asked questions based on the content. Compared to commonly-used models for this task, the RNN had the lowest errors rates in predicting subsequent pupil dilation given sequence data. Most importantly was how the model output related to subjects' cognitive performance as assessed by a post-viewing test. Consistent with our hypothesis that the model captures non-cognitive pupil dynamics, we found (1) the model's root-mean square error was less for lower performing subjects than for those having better performance on the post-viewing test, (2) the residuals of the RNN (LSTM) model had the highest correlation with subject post-viewing test scores and (3) the residuals had the highest discriminability (assessed via area under the ROC curve, AUC) for classifying high and low test performers, compared to the true pupil size or the RNN model predictions. This suggests that deep learning sequence models may be good for separating components of pupil responses that are linked to luminance and accommodation from those that are linked to cognition and arousal. (shrink)

Subject relative clauses (SRCs) are typically processed more easily than object relative clauses (ORCs), but this difference is diminished by an inanimate head‐noun in semantically non‐reversible ORCs (“The book that the boy is reading”). In two eye‐tracking experiments, we investigated the influence of animacy on online processing of semantically reversible SRCs and ORCs using lexically inanimate items that were perceptually animate due to motion (e.g., “Where is the tractor that the cow is chasing”). In Experiment 1, 48 children (aged (...) 4;5–6;4) and 32 adults listened to sentences that varied in the lexical animacy of the NP1 head‐noun (Animate/Inanimate) and relative clause (RC) type (SRC/ORC) with an animate NP2 while viewing two images depicting opposite actions. As expected, inanimate head‐nouns facilitated the correct interpretation of ORCs in children; however, online data revealed children were more likely to anticipate an SRC as the RC unfolded when an inanimate head‐noun was used, suggesting processing was sensitive to perceptual animacy. In Experiment 2, we repeated our design with inanimate (rather than animate) NP2s (e.g., “where is the tractor that the car is following”) to investigate whether our online findings were due to increased visual surprisal at an inanimate as agent, or to similarity‐based interference. We again found greater anticipation for an SRC in the inanimate condition, supporting our surprisal hypothesis. Across the experiments, offline measures show that lexical animacy influenced children's interpretation of ORCs, whereas online measures reveal that as RCs unfolded, children were sensitive to the perceptual animacy of lexically inanimate NPs, which was not reflected in the offline data. Overall measures of syntactic comprehension, inhibitory control, and verbal short‐term memory and working memory were not predictive of children's accuracy in RC interpretation, with the exception of a positive correlation with a standardized measure of syntactic comprehension in Experiment 1. (shrink)

The Joint Battle Management Language (JBML) is an XML-based language designed to allow Command and Control (C2) systems to interface easily with Modeling and Simulation (M&S) systems. While some of the XML-tags defined in this language correspond to types of entities that exist in reality, others are mere syntactic artifacts used to structure the messages themselves. Because these two kinds of tags are not formally distinguishable, JBML messages in effect confuse data with what the data represent. In (...) this paper we show how a realism-based ontology combined with a rule language can be used to make these distinctions explicit. The approach allows storage of the contents of JBML messages in a Referent Tracking System in a format that mimics the structure of reality thereby providing an aid to message validation. (shrink)

Eye gaze plays an important role in communication but understanding of its actual function or functions and the methods used to elucidate this have varied considerably. This systematized review was undertaken to summarize both the proposed functions of eye gaze in conversations of healthy adults and the methodological approaches employed. The eligibility criteria were restricted to a healthy adult population and excluded studies that manipulated eye gaze behavior. A total of 29 articles—quantitative, qualitative and mixed methods were returned, with a (...) wide range of methodological designs. The main areas of variability related to number of conversants, their familiarity and status, conversation topic, data collection tools—video and eye tracking—and definitions of eye gaze. The findings confirm that eye gaze facilitates turn yielding, plays a role in speech monitoring, prevents and repairs conversation breakdowns and facilitates intentional and unintentional speech interruptions. These findings were remarkably consistent given the variability in methods across the 29 articles. However, in relation to turn initiation, the results were less consistent, requiring further investigation. This review provides a starting point for future studies to make informed decisions about study methods for examining eye gaze and selecting variables of interest. (shrink)

This article describes the Vaccine/Injury Project Corpus, a collection of legal decisions awarding or denying compensation for health injuries allegedly due to vaccinations, together with models of the logical structure of the reasoning of the factfinders in those cases. This unique corpus provides useful data for formal and informal logic theory, for natural-language research in linguistics, and for artificial intelligence research. More importantly, the article discusses lessons learned from developing protocols for manually extracting the logical structure and (...) generating the logic models. It identifies sub-tasks in the extraction process, discusses challenges to automation, and provides insights into possible solutions for automation. In particular, the framework and strategies developed here, together with the corpus data, should allow “top–down” and contextual approaches to automation, which can supplement “bottom-up” linguistic approaches. Illustrations throughout the article use examples drawn from the Corpus. (shrink)

(This version was submitted to Behavioral and Brain Science. A revised version was published by Biological Theory) Estimates of autism’s incidence increased 5-10 fold in ten years, an increase which cannot be genetic. Though many mutations are associated with autism, no mutation seems directly to cause autism. We need to find the direct cause. Complexity science provides a new paradigm - confirmed in biology by extensive hard data. Both the body and the personality are complex dynamic systems which spontaneously (...) self-organize from simple dynamic systems. Autism may therefore be caused by the failure of a simple dynamic system. We know that infants who cannot track their mother’s face often become autistic, that eye-contact initiates intersubjectivity which is blocked in autism, and that the infant-mother pair seems designed to promote eye-contact, as does the eye’s appearance. This author earlier proposed that failure of eye-contact might directly cause autism and that early non-maternal childcare, including television/video, would therefore be statistically linked to autism. Waldman et al. (2008; 2006) recently proved that autism is strongly linked to precipitation (indoor activity) and to the introduction of cable. The most plausible explanation? Early exposure to television/video is linked to autism. Furthermore a normal developmental cascade (blocked in autism) has been deciphered: (a) Infant-mother eye-contact triggers increased maternal attention. (b) Early maternal attention permanently increases not only baseline vasopressin but also that oxytocin release which is triggered by subsequent maternal attention. (c) Vasopressin and oxytocin promote face recognition, gazing-at-the-eyes, emotion recognition, and social bonding. The eye-contact hypothesis suggests a clinical study addressing prevention: recruit prospective parents who agree to curtail television/video/computer/wi-fi in their families; measure autism’s incidence in their children. (shrink)

Recent work on the application of neural networks to language modeling has shown that models based on certain neural architectures can capture syntactic information from utterances and sentences even when not given an explicitly syntactic objective. We examine whether a fully data-driven model of language development that uses a recurrent neural network encoder for utterances can track how child language utterances change over the course of language development in a way that is comparable to what is achieved using (...) established language assessment metrics that use language-specific information carefully designed by experts. Given only transcripts of child language utterances from the CHILDES Database and no pre-specified information about language, our model captures not just the structural characteristics of child language utterances, but how these structures reflect language development over time. We establish an evaluation methodology with which we can examine how well our model tracks language development compared to three known approaches: Mean Length of Utterance, the Developmental Sentence Score, and the Index of Productive Syntax. We discuss the applicability of our model to data-driven assessment of child language development, including how a fully data-driven approach supports the possibility of increased research in multilingual and cross-lingual issues. (shrink)