The purpose of this study is to discuss the effects of audiovisual input on second language acquisition and the factors that influence the difficulty of audiovisual learning through a systematic literature review. Prior to this systematic review, in this paper, we searched papers on related topics for the past 10 years from 2012 to 2022, and found 46 journal papers that met the research criteria. They can basically represent the scholarly work in this field. The 46 studies were published (...) in journals indexed in Google Scholar, Eric, Scopus, and Wiley Library. Databases were selected according to a set of inclusion and exclusion criteria. The following conclusions are drawn from the literature review: Audiovisual input can provide more authentic language input and more adequate and richer multimodal cultural and situational contexts, which can better promote learners' understanding of the content and stimulate learners' interest in participating in listening comprehension tasks. The influencing factors of multimodal input on listening difficulty include subtitles, video type, and the relationship between the audio and visual input. (shrink)

Lip reading is the ability to partially understand speech by looking at the speaker's lips. It improves the intelligibility of speech in noise when audio-visual perception is compared with audio-only perception. A recent set of experiments showed that seeing the speaker's lips also enhances sensitivity to acoustic information, decreasing the auditory detection threshold of speech embedded in noise [J. Acoust. Soc. Am. 109 (2001) 2272; J. Acoust. Soc. Am. 108 (2000) 1197]. However, detection is different from comprehension, and it remains (...) to be seen whether improved sensitivity also results in an intelligibility gain in audio-visual speech perception. In this work, we use an original paradigm to show that seeing the speaker's lips enables the listener to hear better and hence to understand better. The audio-visual stimuli used here could not be differentiated by lip reading per se since they contained exactly the same lip gesture matched with different compatible speech sounds. Nevertheless, the noise-masked stimuli were more intelligible in the audio-visual condition than in the audio-only condition due to the contribution of visual information to the extraction of acoustic cues. Replacing the lip gesture by a non-speech visual input with exactly the same time course, providing the same temporal cues for extraction, removed the intelligibility benefit. This early contribution to audio-visual speech identification is discussed in relationships with recent neurophysiological data on audio-visual perception. (shrink)

Infant and adult learners are able to identify word boundaries in fluent speech using statistical information. Similarly, learners are able to use statistical information to identify word–object associations. Successful language learning requires both feats. In this series of experiments, we presented adults and infants with audio–visual input from which it was possible to identify both word boundaries and word–object relations. Adult learners were able to identify both kinds of statistical relations from the same input. Moreover, their learning (...) was actually facilitated by the presence of two simultaneously present relations. Eight-month-old infants, however, do not appear to benefit from the presence of regular relations between words and objects. Adults, like 8-month-olds, did not benefit from regular audio–visual correspondences when they were tested with tones, rather than linguistic input. These differences in learning outcomes across age and input suggest that both developmental and stimulus-based constraints affect statistical learning. (shrink)

Recent work asked if visual cues facilitate word segmentation in adults and infants. While adults showed better word segmentation when presented with a regular visual cue, infants did not. This difference was attributed to infants' lack of understanding that objects have labels. Alternatively, infants’ performance could reflect their difficulty with tracking and integrating multiple multimodal cues. We contrasted these two accounts by looking at the effect of visual cues on word segmentation in adults and across childhood. We found that older (...) children benefitted from the regular visual cues, but younger children, who already knew that objects have labels, did not. Knowing that objects have labels was not enough to use visual cues as an aid for segmentation. These findings show that the ability to integrate multimodal cues develops during childhood, and it is not yet adult-like in children. (shrink)

Cerebral Visual Impairment is an umbrella term which includes abnormalities in visual acuity, or contrast sensitivity or colour; ocular motility; visual field and the conscious and unconscious filtering or processing of visual input. Children with CVI have specific needs and problems relating to their development from infancy to adulthood which can impact on their wellbeing. Recent research indicates the complexities of living with CVI but there remains limited information of the full impact of CVI on families’ everyday lives. The (...) qualitative interviews reported here explored families’ experiences to discover the impact of CVI on all aspects of everyday life. Parents and children were invited to participate in semi-structured interviews, either face to face, by phone or video call between January 2018 and February 2019. Topics covered everyday practicalities of living with CVI, focusing on challenges and what worked well at school and home. Interviews were audio-recorded and subject to thematic analysis to look for patterns across the data. Twenty families took part in interviews, with eight children/young people within those families contributing interviews of their own. Four themes were developed from the interviews: Assessment and understanding implications of CVI, Education, Family life, Psychological wellbeing and quality of life. The interviews provide valuable insights into the impact of living with CVI and highlight the need for more awareness of the condition among professionals in both health and education settings. (shrink)

We examined whether facial expressions of performers influence the emotional connotations of sung materials, and whether attention is implicated in audio-visual integration of affective cues. In Experiment 1, participants judged the emotional valence of audio-visual presentations of sung intervals. Performances were edited such that auditory and visual information conveyed congruent or incongruent affective connotations. In the single-task condition, participants judged the emotional connotation of sung intervals. In the dual-task condition, participants judged the emotional connotation of intervals while performing a secondary (...) task. Judgements were influenced by melodic cues and facial expressions and the effects were undiminished by the secondary task. Experiment 2 involved identical conditions but participants were instructed to base judgements on auditory information alone. Again, facial expressions influenced judgements and the effect was undiminished by the secondary task. The results suggest that visual aspects of music performance are automatically and preattentively registered and integrated with auditory cues. (shrink)

People can discriminate the synchrony between audio-visual scenes. However, the sensitivity of audio-visual synchrony perception can be affected by many factors. Using a simultaneity judgment task, the present study investigated whether the synchrony perception of complex audio-visual stimuli was affected by audio-visual causality and stimulus reliability. In Experiment 1, the results showed that audio-visual causality could increase one's sensitivity to audio-visual onset asynchrony (AVOA) of both action stimuli and speech stimuli. Moreover, participants were more tolerant of AVOA of speech stimuli (...) than that of action stimuli in the high causality condition, whereas no significant difference between these two kinds of stimuli was found in the low causality condition. In Experiment 2, the speech stimuli were manipulated with either high or low stimulus reliability. The results revealed a significant interaction between audio-visual causality and stimulus reliability. Under the low causality condition, the percentage of “synchronous” responses of audio-visual intact stimuli was significantly higher than that of visual_intact/auditory_blurred stimuli and audio-visual blurred stimuli. In contrast, no significant difference among all levels of stimulus reliability was observed under the high causality condition. Our study supported the synergistic effect of top-down processing and bottom-up processing in audio-visual synchrony perception. (shrink)

In this paper we offer a theory of cross-modal objects. To begin, we discuss two kinds of linkages between vision and audition. The first is a duality. The the visual system detects and identifies surfaces ; the auditory system detects and identifies sources . Surfaces are illuminated by sources of light; sound is reflected off surfaces. However, the visual system discounts sources and the auditory system discounts surfaces. These and similar considerations lead to the Theory of Indispensable Attributes that states (...) the conditions for the formation of gestalts in the two modalities. The second linkage involves the formation of audiovisual objects, integrated cross-modal experiences. We describe research that reveals the role of cross-modal causality in the formation of such objects. These experiments use the canonical example of a causal link between vision and audition: a visible impact that causes a percussive sound. (shrink)

In a century where humans and machines—powered by artificial intelligence or not—increasingly work together, it is of interest to understand human processing of multi-sensory stimuli in relation to attention and working memory. This paper explores whether and when supporting visual information with rhythmic auditory stimuli can optimize multi-sensory information processing. In turn, this can make the interaction between humans or between machines and humans more engaging, rewarding and activating. For this purpose a novel working memory paradigm was developed where participants (...) are presented with a series of five target digits randomly interchanged with five distractor digits. Their goal is to remember the target digits and recall them orally. Depending on the condition support is provided by audio and/or rhythm. It is expected that the sound will lead to a better performance. It is also expected that this effect of sound is different in case of rhythmic and non-rhythmic sound. Last but not least, some variability is expected across participants. To make correct conclusions, the data of the experiment was statistically analyzed in a classic way, but also predictive models were developed in order to predict outcomes based on a range of input variables related to the experiment and the participant. The effect of auditory support could be confirmed, but no difference was observed between rhythmic and non-rhythmic sounds. Overall performance was indeed affected by individual differences, such as visual dominance or perceived task difficulty. Surprisingly a music education did not significantly affect the performance and even tended toward a negative effect. To better understand the underlying processes of attention, also brain activation data, e.g., by means of electroencephalography, should be recorded. This approach can be subject to a future work. (shrink)

This paper has already been published in Sociological Research Online, 21/3, August 2016. All rights reserved. © SOCIOLOGICAL RESEARCH ONLINE. It is available from Kent Academic Repository. We thank Dawn Lyon for the permission to republish it here.: This article documents, shows and analyses the everyday rhythms of Billingsgate, London's wholesale fish market. It takes the form of a short film based an audiovisual montage of time-lapse photography and sound recordings, and a - Sociologie – Nouvel article.

Utilizing a pre-experimental pre-test post-test design, this study investigated the effect of an authentic audio-visual American sitcom on the intercultural sensitivity and intercultural effectiveness of a sample of male and female upper-intermediate English students. To this aim, 34 Iranian EFL students were selected through convenient non-random sampling. In order to assure the participants' homogeneity in English proficiency, the selected students were given the Oxford Quick Placement Test prior to the intervention. Over a 10-week period, the participants were presented with nearly (...) 30-min-long episodes of Friends in a flipped context, two episodes every week uploaded to the accompanying website. An online 10-item quiz on the day of watching and a next-day meeting were held. During the online discussion meetings, the participants discussed cultural questions tailored to the aims of the study. The required data were collected through the administration of the intercultural sensitivity scale and intercultural effectiveness scale in pre- and post-test assessments. The analyses of the data indicated that both ICS and ICE improved significantly during the intervention. However, ICS indicated more improvement. In addition, a semi-structured interview was administered to the participants to examine their perception of the flipped classroom experience. Having analyzed the data using MAXQDA 2020, some significant themes emerged which are reported. On the whole, the participants portrayed positive attitudes toward using technology in flipped classes. The educational and practical implications are discussed. (shrink)

Experimental research has shown that pairs of stimuli which are congruent and assumed to 'go together' are recalled more effectively than an item presented in isolation. Will this multisensory memory benefit occur when stimuli are richer and longer, in an ecological setting? In the present study, we focused on an everyday situation of audio-visual learning and manipulated the relationship between audio guide tracks and viewed portraits in the galleries of the Tate Britain. By varying the gender and narrative style of (...) the voice-over, we examined how the perceived congruency and assumed unity of the audio guide track with painted portraits affected subsequent recall. We show that tracks perceived as best matching the viewed portraits led to greater recall of both sensory and linguistic content. We provide the first evidence that manipulating crossmodal congruence and unity assumptions can effectively impact memory in a multisensory ecological setting, even in the absence of precise temporal alignment between sensory cues. (shrink)

It is commonly believed that human perceptual experiences can be, and usually are, multimodal. What is more, a stronger thesis is often proposed that some perceptual multimodal characters cannot be described simply as a conjunction of unimodal phenomenal elements. If it is the case, then a question arises: what is the additional mode of combination that is required to adequately describe the phenomenal structure of multimodal experiences? The paper investigates what types of audio–visual experiences have phenomenal character that cannot (...) be analysed as a mere conjunction of visual and auditory elements; and how can we properly characterise the required, additional mode of perceptual combination. Three main modes of combination are considered: instantiation, parthood, and grouping. It is argued that some phenomena involving intermodal relations, like spatial and temporal ventriloquism, can be analysed in terms of audio–visual, perceptual grouping. On the other hand, cases of intermodal binding need a different treatment. Experiences involving audio–visual binding should be analysed as experiences presenting objects or events which instantiate, or which have a proper part instantiating, both visually and auditorily determined properties. (shrink)

This article investigates the hypothesis of creating non-linear audio-visual narratives, through an unanticipated use of traditional print-based games, enriched with videos, via augmented reality (AR) possibilities. A ludic system has been created and presented. Based on a traditional card game, a non-linear cinematic narrative occurs. We attempt to examine the following questions: in which way can we bring together different forms of visual communication, such as graphic design and video? Can the above forms create a complex narrative whole and what (...) kind of rules will be needed for this? How can we enrich traditional forms of gaming with the potentials of AR? Gaming itself demands a set of rules. Can these rules play the role of algorithms in the combined universe that we have designed and created? In which way can the designer on the one hand and the user on the other influence the overall output of the system? What will the user experience be like? The printed card system chosen for this is Tarot and more precisely the Great Arcana, which makes use of the 22 fundamental Tarot figures. (shrink)

Previous research with speech and non-speech stimuli suggested that in audiovisual perception, visual information starting prior to the onset of corresponding sound can provide visual cues, and form a prediction about the upcoming auditory sound. This prediction leads to audiovisual interaction. Auditory and visual perception interact and induce suppression and speeding up of the early auditory event-related potentials such as N1 and P2. To investigate AV interaction, previous research examined N1 and P2 amplitudes and latencies in response to audio only, (...) video only, audiovisual, and control stimuli, and compared AV with auditory perception based on four AV interaction models. The current study addresses how different models of AV interaction express N1 and P2 suppression in music perception. Furthermore, the current study took one step further and examined whether previous musical experience, which can potentially lead to higher N1 and P2 amplitudes in auditory perception, influenced AV interaction in different models. Musicians and non-musicians were presented the recordings of a keyboard /C4/ key being played, as well as CO stimuli. Results showed that AV interaction models differ in their expression of N1 and P2 amplitude and latency suppression. The calculation of model and has consequences for the resulting N1 and P2 difference waves. Furthermore, while musicians, compared to non-musicians, showed higher N1 amplitude in auditory perception, suppression of amplitudes and latencies for N1 and P2 was similar for the two groups across the AV models. Collectively, these results suggest that when visual cues from finger and hand movements predict the upcoming sound in AV music perception, suppression of early ERPs is similar for musicians and non-musicians. Notably, the calculation differences across models do not lead to the same pattern of results for N1 and P2, demonstrating that the four models are not interchangeable and are not directly comparable. (shrink)

We investigate whether smile mimicry and emotional contagion are evident in non-text-based computer-mediated communication (CMC). Via an ostensibly real-time audio-visual CMC platform, participants interacted with a confederate who either smiled radiantly or displayed a neutral expression throughout the interaction. Automatic analyses of expressions displayed by participants indicated that smile mimicry was at play: A higher level of activation of the facial muscle that characterises genuine smiles was observed among participants who interacted with the smiling confederate than among participants who interacted (...) with the unexpressive confederate. However, there was no difference in the self-reported level of joviality between participants in the two conditions. Our findings demonstrate that people mimic smiles in audio-visual CMC, but that even though the diffusion of emotions has been documented in text-based CMC in previous studies, we find no convincing support for the phenomenon of emotional contagion in non-text-based CMC. (shrink)

Jerry Fodor has consistently cited the persistence of illusions--especially the M.

During the COVID-19 pandemic, questions have been raised about the impact of face masks on communication in classroom settings. However, it is unclear to what extent visual obstruction of the speaker’s mouth or changes to the acoustic signal lead to speech processing difficulties, and whether these effects can be mitigated by semantic predictability, i.e., the availability of contextual information. The present study investigated the acoustic and visual effects of face masks on speech intelligibility and processing speed under varying semantic predictability. (...) Twenty-six children and twenty-six adults performed an internet-based cued shadowing task, in which they had to repeat aloud the last word of sentences presented in audio-visual format. The results showed that children and adults made more mistakes and responded more slowly when listening to face mask speech compared to speech produced without a face mask. Adults were only significantly affected by face mask speech when both the acoustic and the visual signal were degraded. While acoustic mask effects were similar for children, removal of visual speech cues through the face mask affected children to a lesser degree. However, high semantic predictability reduced audio-visual mask effects, leading to full compensation of the acoustically degraded mask speech in the adult group. Even though children did not fully compensate for face mask speech with high semantic predictability, overall, they still profited from semantic cues in all conditions. Therefore, in classroom settings, strategies that increase contextual information such as building on students’ prior knowledge, using keywords, and providing visual aids, are likely to help overcome any adverse face mask effects. (shrink)

Human development takes place in a social context. Two pervasive sources of social information are faces and hands. Here, we provide the first report of the visual frequency of faces and hands in the everyday scenes available to infants. These scenes were collected by having infants wear head cameras during unconstrained everyday activities. Our corpus of 143 hours of infant-perspective scenes, collected from 34 infants aged 1 month to 2 years, was sampled for analysis at 1/5 Hz. The major finding (...) from this corpus is that the faces and hands of social partners are not equally available throughout the first two years of life. Instead, there is an earlier period of dense face input and a later period of dense hand input. At all ages, hands in these scenes were primarily in contact with objects and the spatio-temporal co-occurrence of hands and faces was greater than expected by chance. The orderliness of the shift from faces to hands suggests a principled transition in the contents of visual experiences and is discussed in terms of the role of developmental gates on the timing and statistics of visual experiences. (shrink)

Companies increasingly seek to use gay protagonists in audio-visual commercials to attract a new affluent target group. There is also growing demand for the diversity present in society to be reflected in media formats such as advertising. Studies have shown, however, that heterosexual consumers, who may be part of the company's loyal consumer base, tend to react negatively to gay-themed advertising campaigns. Searching for an instrument to mitigate this unwanted effect, the present study investigated whether carefully selected background music can (...) shape the perceived gender of gay male advertising protagonists. In a 2 × 2 between-subjects online experiment, 218 heterosexual participants watched a commercial promoting engagement rings that featured gay male protagonists, scored with feminine- or masculine-connoted background music. As expected, women generally reacted more positively than men to the advertising. Men exposed to the masculine-connoted background music rated the promoted brand more positively, and masculine music also enhanced these men's acceptance of gay men in general more than was the case for feminine background music. Carefully selected background music affecting the perceived gender of gay male advertising protagonists may prevent negative reactions from heterosexual audiences and, therefore, motivate companies to use gay protagonists in television commercials on a more regular basis. (shrink)

Online data collection with infants raises special opportunities and challenges for developmental research. One of the most prevalent methods in infancy research is eye-tracking, which has been widely applied in laboratory settings to assess cognitive development. Technological advances now allow conducting eye-tracking online with various populations, including infants. However, the accuracy and reliability of online infant eye-tracking remain to be comprehensively evaluated. No research to date has directly compared webcam-based and in-lab eye-tracking data from infants, similarly to data from adults. (...) The present study provides a direct comparison of in-lab and webcam-based eye-tracking data from infants who completed an identical looking time paradigm in two different settings. We assessed 4-6-month-old infants in an eye-tracking task that measured the detection of audio-visual asynchrony. Webcam-based and in-lab eye-tracking data were compared on eye-tracking and video data quality, infants’ viewing behavior, and experimental effects. Results revealed no differences between the in-lab and online setting in the frequency of technical issues and participant attrition rates. Video data quality was comparable between settings in terms of completeness and brightness, despite lower frame rate and resolution online. Eye-tracking data quality was higher in the laboratory than online, except in case of relative sample loss. Gaze data quantity recorded by eye-tracking was significantly lower than by video in both settings. In valid trials, eye-tracking and video data captured infants’ viewing behavior uniformly, irrespective of setting. Despite the common challenges of infant eye-tracking across experimental settings, our results point toward the necessity to further improve the precision of online eye-tracking with infants. Taken together, online eye-tracking is a promising tool to assess infants’ gaze behavior but requires careful data quality control. The demographic composition of both samples differed from the generic population on caregiver education: our samples comprised caregivers with higher-than-average education levels, challenging the notion that online studies will per se reach more diverse populations. (shrink)

In 2015 the first mass migration and the first geopolitical and transnational crisis of the digital age began. This brought along the awakening of a new phase in slavery trade and trafficking of human beings (THB) at the European borders. In this article I introduce an ongoing field study I began in January 2016, in collaboration with experts from Spanish and European institutions, that so far has taken me to hotspots and borders, and to the accumulation of a vast multimedia (...) archive. This led to the creation of zonaautonomamilitarizada.eu, an immersive/360°, interactive and reactive audio-visual performative system conceptually rooted in military technologies, second-order cybernetics and mass media developments. (shrink)

This article provides an ethical analysis of the moral conflict over access to premium audio-visual online content, which is currently regulated by geo-blocking in the domain of digital film distri...

This paper revisits some early applications of audio-visual imaging technologies used in physiology in a dialogue with reflections on Henri Bergson’s philosophy. It focuses on the aspects of time and memory in relation to spatial representations of movement measurements and critically discusses them from the perspective of the observing participant and the public exhibitions of scientific films. Departing from an audio-visual example, this paper is informed by a thick description of the philosophical implications and contemporary discourses surrounding the scientific inventions, (...) technologies and theories of moving image technologies, representations of time, and the measurement of bodies in motion such as those by the French physiologist Étienne-Jules Marey. The application of Bergson’s thinking will show how the scientific graphing and representations of the body through technologies inevitably stand in tension with the limits and potentials of the ‘human apparatus’ and its abilities, filters and internalised processes of perception, memory and consciousness. It proposes an ontological approach to scientific imaging technologies and a critical engagement with the ‘realism’ debate in the discourse of audio-visual media. (shrink)