For humans, the ability to communicate and use language is instantiated not only in the vocal modality but also in the visual modality. The main examples of this are sign languages and gestures. Sign languages, the natural languages of Deaf communities, use systematic and conventionalized movements of the hands, face, and body for linguistic expression. Co-speech gestures, though non-linguistic, are produced in tight semantic and temporal integration with speech and constitute an integral part of language together with (...) speech. The articles in this issue explore and document how gestures and sign languages are similar or different and how communicative expression in the visual modality can change from being gestural to grammatical in nature through processes of conventionalization. As such, this issue contributes to our understanding of how the visual modality shapes language and the emergence of linguistic structure in newly developing systems. Studying the relationship between signs and gestures provides a new window onto the human ability to recruit multiple levels of representation in the service of using or creating conventionalized communicative systems. (shrink)

The aim of the present study is to evaluate Sonja Foss’s Rhetorical Schema for the Evaluation of Visual Imagery as well as reflect upon several points for further consideration; and finally suggest a renewed triadic approach as a method for analyzing art-relevant visual imagery. The triadic approach to be discussed assumes three correlative layers: the intellectual knowledge, function of the artistic content as the visual representational component, and aesthetics as modality. This study will include the analysis (...) of a print advertisement that used an artwork as its content of visual rhetoric, and this will inform further discussions on the proposed approach. The contribution of this renewed triadic approached to the field of visual rhetoric has the advantage of expanding and possibly improving rhetorical analysis methods of visual imagery. (shrink)

Three priming experiments were conducted to determine how information about the self from different sensory modalities/cognitive domains affects self-face recognition. Being exposed to your body odor, seeing your name, and hearing your name all facilitated self-face recognition in a reaction time task. No similar cross-modal facilitation was found among stimuli from familiar or novel individuals. The finding of a left-hand advantage for self-face recognition was replicated when no primes were presented. These data, along with other recent results suggest the brain (...) processes/represents information about the self in highly integrated ways. (shrink)

Although Pessoa et al. imply that many aspects of the filling-in debate may be displaced by a regard for active vision, they remain loyal to naive neural reductionist explanations of certain pieces of psychophysical evidence. Alternative interpretations are provided for two specific examples and a new category of filling-in (of visual detail) is proposed.

Modality specificity in priming is taken as evidence for independent perceptual systems. However, Easton, Greene, and Srinivas (1997) showed that visual and haptic cross-modal priming is comparable in magnitude to within-modal priming. Where appropriate, perceptual systems might share like information. To test this, we assessed priming and recognition for visual and auditory events, within- and across- modalities. On the visual test, auditory study resulted in no priming. On the auditory priming test, visual study resulted in (...) priming that was only marginally less than within-modal priming. The priming results show that visual study facilitates identification on both visual and auditory tests, but auditory study only facilitates performance on the auditory test. For both recognition tests, within-modal recognition exceeded cross-modal recognition. The results have two novel implications for the understanding of perceptual priming: First, we introduce visual and auditory priming for spatio-temporal events as a new priming paradigm chosen for its ecological validity and potential for information exchange. Second, we propose that the asymmetry of the cross-modal priming observed here may reflect the capacity of these perceptual modalities to provide cross-modal constraints on ambiguity. We argue that visual perception might inform and constrain auditory processing, while auditory perception corresponds to too many potential visual events to usefully inform and constrain visual perception. (shrink)

Training can influence behavioral performance and lead to brain reorganization. In particular, training in one modality, for example, auditory, can improve performance in another modality, for example, visual. Previous research suggests that one of the mechanisms behind this phenomenon could be the cross-modal recruitment of the sensory areas, for example, the auditory cortex. Studying expert musicians offers a chance to explore this process. Rhythm is an aspect of music that can be presented in various modalities. We designed (...) an fMRI experiment in which professional pianists and non-musicians discriminated between two sequences of rhythms presented auditorily or visually. Behavioral results showed that musicians performed in both visual and auditory rhythmic tasks better than non-musicians. We found no significant between-group differences in fMRI activations within the auditory cortex. However, we observed that musicians had increased activation in the right Inferior Parietal Lobe when compared to non-musicians. We conclude that the musicians’ superior visual rhythm discrimination is not related to cross-modal recruitment of the auditory cortex; instead, it could be related to activation in higher-level, multimodal areas in the cortex. (shrink)

ABSTRACTBoth auditory and visual emotional memories can be made less emotional by loading working memory during memory recall. Taxing WM during recall can be modality specific (giving an audit...

In a recent paper, Brogaard presents counter-arguments to the conclusions of an experiment with blindsight subject GR. She argues that contrary to the apparent findings that GR’s preserved visual abilities relate to degraded visual experiences, she is in fact fully unconscious of the stimuli she correctly identifies. In this paper, we present arguments and evidence why Brogaard’s argument does not succeed in its purpose. We suggest that not only is relevant empirical evidence in opposition to Brogaard’s argument, her (...) argument misconstrues necessary criteria to decide whether a conscious experience is visual or not visual. (shrink)

The distinction between true belief and knowledge is one of the most fundamental in philosophy, and a remarkable effort has been dedicated to formulating the conditions on which true belief constitutes knowledge. For decades, much of this epistemological undertaking has been dominated by a single strategy, referred to here as the modal approach. Shared by many of the most widely influential constraints on knowledge, including the sensitivity, safety, and anti-luck/risk conditions, this approach rests on a key underlying assumption — the (...) modal profiles available to known and unknown beliefs are in some way asymmetrical. The first aim of this paper is to deconstruct this assumption, identifying its plausibility with the way in which epistemologists frequently conceptualize human perceptual systems as excluding certain varieties of close error under conditions conducive to knowledge acquisition. The second aim of this paper is to then argue that a neural phase phenomenon indicates that this conceptualization is quite likely mistaken. This argument builds on the previous introduction of this neural phase to the context of epistemology, expanding the use of neural phase cases beyond relatively narrow questions about epistemic luck to a much more expansive critique of the modal approach as a whole. (shrink)

This article presents a fresh perspective on the interplay among visual, verbal, and auditory modalities, positing that these modalities, as semogenic resources, compete to express dynamic meanings. The theoretical paradigm emphasizes that whether a modality or an element within a modality gets or loses semantic status, it will elicit an additional layer of social meaning to depict a comprehensive picture of a story together with an explicit semiotic meaning. The article adopts a qualitative method to analyze the (...) data, which are drawn from The Good Wife and My Roommate is a Gumiho and annotated in ELAN 6.3. It was found that modal competition can shed light on the dynamic meaning-making processes in semiotic and societal orientations. Modal competition may distort space and time of different stories, and reconstruct a different discursive spatio-temporal dimension in the TV world. It can diversify the dynamic orientations from New to Given in visual, verbal, and auditory texts of multimodal discourses to tell stories. Modal competition provides a lens to understand the multidimensional reality and to appreciate the aesthetics of a modern TV series. (shrink)

Most everyday tasks involve multiple modalities, which raises the question of how the processing of these modalities is coordinated by the cognitive system. In this paper, we focus on the coordination of visual attention and linguistic processing during speaking. Previous research has shown that objects in a visual scene are fixated before they are mentioned, leading us to hypothesize that the scan pattern of a participant can be used to predict what he or she will say. We test (...) this hypothesis using a data set of cued scene descriptions of photo-realistic scenes. We demonstrate that similar scan patterns are correlated with similar sentences, within and between visual scenes; and that this correlation holds for three phases of the language production process (target identification, sentence planning, and speaking). We also present a simple algorithm that uses scan patterns to accurately predict associated sentences by utilizing similarity-based retrieval. (shrink)

Clinical and laboratory studies have demonstrated that executing a demanding dual-task while recollecting emotional memories weakens the emotional intensity and vividness of these memories. While t...

This research delves into the potential of implementing artificial intelligence in architecture. It specifically provides a critical assessment of AI-enabled workflows, encompassing creative ideation, representation, materiality, and critical thinking, facilitated by prompt-based generative processes. In this context, the paper provides an examination of the concept of hybrid human–machine intelligence. In an era characterized by pervasive data bias and engineered injustices, the concept of hybrid intelligence emerges as a critical tool, enabling the transcendence of preconceived stereotypes, clichés, and linguistic prejudices. This (...) paper not only explores the applied and generative capacities of AI-enabled workflows but also suggests fundamental approaches that can enhance the creative process and confront the embedded biases and injustices within data-driven systems. (shrink)

Restrictivists hold that visual experience only represents low-level properties such as shape, spatial location, motion, color, etc. Expansionists contend that visual experience also represents high-level properties such as being a pine tree. I outline a new approach to support expansionism called the conflict cross-modal argument. What I call the conflict cross-modal effects occur when at least two perceptual systems disagree about some property belonging to a common stimulus, and this disagreement causes a change in the representational and phenomenal (...) content of the perceptual experience associated with one, or both, modalities. The conflict cross-modal argument works by accepting that if a property figures in a conflict cross-modal effect, then prima facie that property is a strong common sensible between the two modalities, and vision and another modality disagree about a property that is high-level for vision. After outlining this argument, and showing how it overcomes two obstacles that face traditional methods employed to defend expansionism, I turn to the well known Mcgurk effect as a case where vision and audition disagree about phoneme properties in order to employ my argument for the conclusion that phoneme properties are represented in visual experience. The upshot is that since phoneme properties are high-level for vision, we have an argument supporting expansionism. (shrink)

Theories of embodied cognition hold that the conceptual system uses perceptual simulations for the purposes of representation. A strong prediction is that perceptual phenomena should emerge in conceptual processing, and, in support, previous research has shown that switching modalities from one trial to the next incurs a processing cost during conceptual tasks. However, to date, such research has been limited by its reliance on the retrieval of familiar concepts. We therefore examined concept creation by asking participants to interpret modality-specific (...) compound phrases (i.e., conceptual combinations). Results show that modality switching costs emerge during the creation of new conceptual entities: People are slower to simulate a novel concept (e.g., auditory jingling onion) when their attention has already been engaged by a different modality in simulating a familiar concept (e.g., visual shiny penny). Furthermore, these costs cannot be accounted for by linguistic factors alone. Rather, our findings support the embodied view that concept creation, as well as retrieval, requires situated perceptual simulation. (shrink)

The nature of attention is one of the oldest and most central problems in psychology. A huge amount of research has been produced on this subject in the last half century, especially on attention in the visual modality, but a general explanation has remained elusive. Many still view attention research as a field that is fundamentally fragmented. This book takes a different perspective and presents a unified theory of visual attention: the TVA model. The TVA model explains (...) the many aspects of visual attention by just two mechanisms for selection of information: filtering and pigeonholing. These mechanisms are described in a set of simple equations, which allow TVA to mathematically model a large number of classical results in the attention literature. The theory explains psychological and neuroscientific findings by the same equations; TVA is a complete theory of visual attention, linking mind and brain. Aimed at advanced students and professional researchers, Principles of Visual Attention contains a detailed review of the most important research done on attention in vision, spanning cognitive psychology, brain imaging, patient studies, and recordings from single cells in the visual cortex. The book explains the TVA model and shows how it accounts for attentional effects observed across all the research areas described. Principles of Visual Attention offers a uniquely integrated view on a central topic in cognitive neuroscience. (shrink)

Early modern humans developed mental capabilities that were immeasurably greater than those of nonhuman primates. We see this in the rapid innovation in tool making, the development of complex language, and the creation of sophisticated art forms, none of which we find in our closest relatives. While we can readily observe the results of this high-order cognitive capacity, it is difficult to see how it could have developed. We take up the topic of cave art and archeoacoustics, particularly the discovery (...) that cave art is often closely connected to the acoustic properties of the cave chambers in which it is found. Apparently, early modern humans were able to detect the way sound reverberated in these chambers, and they painted artwork on surfaces that were acoustic “hot spots,” i.e., suitable for generating echoes. We argue that cave art is a form of cross-modality information transfer, in which acoustic signals are transformed into symbolic visual representations. This form of information transfer across modalities is an instance of how the symbolic mind of early modern humans was taking shape into concrete, externalized language. We also suggest that the earliest rock art found in Africa may constitute one of the first fossilized proxies for the expression of full-fledged human linguistic behavior. (shrink)

We experience our own body through both touch and vision. We further see that others’ bodies are similar to our own body, but we have no direct experience of touch on others’ bodies. Therefore, relations between vision and touch are important for the sense of self and for mental representation of one’s own body. For example, seeing the hand improves tactile acuity on the hand, compared to seeing a non-hand object. While several studies have demonstrated this visual enhancement of (...) touch effect, its relation to the ‘bodily self’, or mental representation of one’s own body remains unclear. We examined whether VET is an effect of seeing a hand, or of seeing my hand, using the rubber hand illusion. In this illusion, a prosthetic hand which is brushed synchronously—but not asynchronously—with one’s own hand is felt to actually be one’s hand. Thus, we manipulated whether or not participants felt like they were looking directly at their hand, while holding the actual stimulus they viewed constant. Tactile acuity was measured by having participants judge the orientation of square-wave gratings. Two characteristic effects of VET were observed: cross-modal enhancement from seeing the hand was inversely related to overall tactile acuity, and participants near sensory threshold showed significant improvement following synchronous stroking, compared to asynchronous stroking or no stroking at all. These results demonstrate a clear functional relation between the bodily self and basic tactile perception. (shrink)

Maintaining and coordinating multiple task-sets is difficult and leads to costs, however task-switching training can reduce these deficits. A recent study in young adults demonstrated that this training effect occurs at an amodal processing level. Old age is associated with reduced cognitive plasticity and further increases the performance costs when mixing multiple tasks. Thus, cognitive aging might be a limiting factor for inducing cross-modal training effects in a task-switching environment. We trained participants, aged 62–83 years, with an auditory task-switching paradigm (...) over four sessions, to investigate whether training-related reductions in task-switching costs would also manifest in an untrained visual modality version of the task. Two control groups trained with single tasks or not trained allowed us to identify improvements specific to task-switching training. To make statistical evaluations of any age differences in training and cross-modal transfer, the data from the Kattner cohort were incorporated into the present analysis. Despite the tendency for older adults to respond more cautiously, task-switching training specifically led to a mixing cost reduction in both trained and untrained modalities, the magnitude of which was statistically similar regardless of age. In line with a growing body of research, we failed to observe any far transfer effects in measures of inhibition, working memory or fluid intelligence. Overall, we conclude that any apparent cognitive limitations associated with aging do not prevent cognitive control processes which support set-shifting from improving at an amodal level. (shrink)

It is being increasingly recognized that the Saussurean dictum of “the arbitrariness of the linguistic sign” is in conflict with the pervasiveness of the phenomenon commonly known as “sound symbolism”. After first presenting a historical overview of the debate, however, we conclude that both positions have been exaggerated, and that an adequate explanation of sound symbolism is still lacking. How can there, for example, be (perceived) similarity between expressionsand contents across different sensory modalities? We offer an answer, based on the (...) Peircian notion of iconic ground, and G. Sonesson’s distinction betweenprimary and secondary iconicity. Furthermore, we describe an experimental study, in a paradigm first pioneered by W. Kohler, and recently popularized by V.Ramachandran, in which we varied vowels and consonants in fictive word-forms, and conclude that both types of sounds play a role in perceiving an iconic ground between the word-forms and visual figures. The combination of historical conceptual analysis, semiotic explication and psychological experimentationpresented in this article is characteristic of the emerging paradigm of cognitive semiotics. (shrink)

Neuroimaging and psychophysiological techniques have proved to be useful in comprehending the extent to which the visual modality is pervasive in mental imagery, and in comprehending the specificity of images generated through other sensory modalities. Although further research is needed to understand the nature of mental images, data attained by means of these techniques suggest that mental imagery requires at least two distinct processing components.