Scan Patterns Predict Sentence Production in the Cross-Modal Processing of Visual Scenes

Cognitive Science 36 (7):1204-1223 (2012)
Most everyday tasks involve multiple modalities, which raises the question of how the processing of these modalities is coordinated by the cognitive system. In this paper, we focus on the coordination of visual attention and linguistic processing during speaking. Previous research has shown that objects in a visual scene are fixated before they are mentioned, leading us to hypothesize that the scan pattern of a participant can be used to predict what he or she will say. We test this hypothesis using a data set of cued scene descriptions of photo-realistic scenes. We demonstrate that similar scan patterns are correlated with similar sentences, within and between visual scenes; and that this correlation holds for three phases of the language production process (target identification, sentence planning, and speaking). We also present a simple algorithm that uses scan patterns to accurately predict associated sentences by utilizing similarity-based retrieval.
Keywords Scan patterns  Scene understanding  Cross‐model processing  Similarity measures  Language production  Eye‐movements
Categories (categorize this paper)
DOI 10.1111/j.1551-6709.2012.01246.x
 Save to my reading list
Follow the author(s)
My bibliography
Export citation
Find it on Scholar
Edit this record
Mark as duplicate
Revision history Request removal from index
Download options
PhilPapers Archive

Upload a copy of this paper     Check publisher's policy on self-archival     Papers currently archived: 16,658
External links
Setup an account with your affiliations in order to access resources via your University's proxy server
Configure custom proxy (use this if your affiliation does not provide a proxy)
Through your library
References found in this work BETA

Add more references

Citations of this work BETA

Add more citations

Similar books and articles
Michael Kubovy & Michael Schutz (2010). Audio-Visual Objects. Review of Philosophy and Psychology 1 (1):41-61.

Monthly downloads

Added to index


Total downloads

16 ( #165,587 of 1,725,959 )

Recent downloads (6 months)

3 ( #210,870 of 1,725,959 )

How can I increase my downloads?

My notes
Sign in to use this feature

Start a new thread
There  are no threads in this forum
Nothing in this forum yet.