Scan Patterns Predict Sentence Production in the Cross-Modal Processing of Visual Scenes

Cognitive Science 36 (7):1204-1223 (2012)
Abstract
Most everyday tasks involve multiple modalities, which raises the question of how the processing of these modalities is coordinated by the cognitive system. In this paper, we focus on the coordination of visual attention and linguistic processing during speaking. Previous research has shown that objects in a visual scene are fixated before they are mentioned, leading us to hypothesize that the scan pattern of a participant can be used to predict what he or she will say. We test this hypothesis using a data set of cued scene descriptions of photo-realistic scenes. We demonstrate that similar scan patterns are correlated with similar sentences, within and between visual scenes; and that this correlation holds for three phases of the language production process (target identification, sentence planning, and speaking). We also present a simple algorithm that uses scan patterns to accurately predict associated sentences by utilizing similarity-based retrieval
Keywords Scan patterns  Scene understanding  Cross‐model processing  Similarity measures  Language production  Eye‐movements
Categories (categorize this paper)
Options
 Save to my reading list
Follow the author(s)
My bibliography
Export citation
Find it on Scholar
Edit this record
Mark as duplicate
Revision history Request removal from index
 
Download options
PhilPapers Archive


Upload a copy of this paper     Check publisher's policy on self-archival     Papers currently archived: 11,085
External links
Setup an account with your affiliations in order to access resources via your University's proxy server
Configure custom proxy (use this if your affiliation does not provide a proxy)
Through your library
References found in this work BETA
Citations of this work BETA

No citations found.

Similar books and articles
Michael Kubovy & Michael Schutz (2010). Audio-Visual Objects. Review of Philosophy and Psychology 1 (1):41-61.
Analytics

Monthly downloads

Added to index

2012-04-10

Total downloads

11 ( #137,061 of 1,101,585 )

Recent downloads (6 months)

4 ( #81,941 of 1,101,585 )

How can I increase my downloads?

My notes
Sign in to use this feature


Discussion
Start a new thread
Order:
There  are no threads in this forum
Nothing in this forum yet.