Jennifer L. Campos1,2*
Michel Bédard3
Sherrilene Classen4
Jude J. Delparte5Deborah A. Hebert5,6Nellemarie Hyde7Geoff Law5
Gary Naglie1,8,9
Stephanie Yung1,2- 1iDAPT Research, Toronto Rehabilitation Institute, University Health Network, Toronto, ON, Canada
- 2Department of Psychology, University of Toronto, Toronto, ON, Canada
- 3Centre for Research on Safe Driving, Lakehead University, Thunder Bay, ON, Canada
- 4Department of Occupational Therapy, University of Florida, Gainesville, FL, United States
- 5Brain and Spinal Cord Rehabilitation Program, Toronto Rehabilitation Institute, University Health Network, Toronto, ON, Canada
- 6Department of Occupational Science and Occupational Therapy, University of Toronto, Toronto ON, Canada
- 7Saint Elizabeth Health Care, Toronto, ON, Canada
- 8Baycrest Health Sciences, Toronto, ON, Canada
- 9Department of Medicine and Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada
Driving simulators are powerful tools for use in research and applications concerned with the evaluation and improvement of driving performance. The value of this technology is contingent upon carefully considering the technical features of the simulator itself (e.g., type of visual display, vehicle control model), the development of appropriate hypothesis-motivated driving scenarios, and the selection of meaningful outcome measures with respect to the questions being addressed and the populations of interest. The Toronto Rehabilitation Institute's iDAPT Centre for Rehabilitation Research recently developed DriverLab, a 7 degrees-of-freedom, motion-based simulator, containing a passenger vehicle, 360 degree visual projection screen, and unique rain and glare simulators (www.idapt.com). To maximize the efficiency and effectiveness of the research conducted within this unique facility, a workshop was organized during which experts across several fields (academia, clinical, industry, government) met in four separate groups to discuss four targeted themes including: (1) use of simulators for driving assessment; (2) effects of drugs on driving safety; (3) effects of automated vehicle technologies (AVTs; e.g., adaptive cruise control) on driving safety; and (4) techniques for mitigating simulator sickness. This paper describes the consensus achieved by the driving assessment group.
The driving assessment group was specifically tasked with characterizing the role of driving simulators in assessing driving performance across a range of applications and populations including individuals with sensory, motor, or cognitive impairments, psychiatric disorders and neurological disorders. The group includes experts in driving evaluations of medically-at-risk drivers (Hyde); clinical assessment of fitness-to-drive (Hebert, Naglie, Law, Classen); and researchers studying the perceptual, cognitive, and emotional factors associated with driving performance and/or simulation design (Bédard, Classen, Campos, Hebert, Yung). The team also includes representation by government organizations that employ a large workforce of occupational drivers (Canada Post) and those involved in road safety initiatives.
Role for Simulators in Augmenting Traditional Office-based and On-road Assessments
Driving simulators hold key advantages for identifying risks to driving safety among different driver populations and across driving conditions when compared to office-based tools and on-road testing (Allen et al., 2011; Classen and Brooks, 2014). Common office-based tests of sensory and cognitive function thought to affect driving safety include, for instance: the Useful-Field-of-View test that assesses visual attention (Ball and Owsley, 1993), the Rapid Pace Walk test that measures general physical functioning (Marottoli et al., 1994), and the Montreal Cognitive Assessment that screens for mild cognitive impairment (Nasreddine et al., 2005). Note that the appropriateness of these tools varies as a function of the driver population and outcomes of interest. See, Vrkljan et al. (2011), Classen et al. (2012), and Bennet et al. (2016) for comprehensive reviews and evaluations of these tools. Although performance on these types of office-based tests have been shown to correlate with aspects of on-road and simulated driving performance, simulators offer greater face validity by recreating the multisensory, multidimensional challenges associated with this complex task in more realistic ways.
Compared to on-road testing, simulators are safer, more easily controlled and standardized, and allow for reproducible and easily modifiable conditions and scenarios. Simulators can also provide objective methods of capturing driver-response data, be used with high-risk populations, introduce more challenging environmental conditions, and create more demanding task-based conditions such as multi-tasking.
While even state-of-the-art simulator technologies are currently insufficient to fully determine fitness-to-drive for the purpose of licensing decisions, they have remarkable potential to address a broad range of research questions and to facilitate the screening of at-risk drivers in practice. For instance, driving simulator assessments may serve as a valuable intermediate stage between office-based measures and on-road assessments. By implementing a trichotomization approach of “pass,” “fail,” “indeterminate,” at the simulator stage, this may help to more precisely categorize a driver's risk status (Molnar et al., 2009; Gibbons et al., 2017). When drivers are appropriately categorized during the simulator stage of assessment, the reliance on resource-heavy on-road assessments will be reduced. For a trichotomization approach to be successful with driving simulators, sensitivity and specificity criteria related to pass or fail must be determined. At each stage of this process (office-based → simulator → on-road), customized clinical interventions, training protocols, or driving cessation plans can be initiated as appropriate.
Simulators can also serve as a tool for identifying specific driving skills that could be targeted for intervention and as a means of administering such interventions. Simulators can also be employed to compare performance metrics across different driver populations and to detect changes in the performance of the same individual over time. Repeated driving simulator assessments might be particularly important for monitoring individuals with declining health conditions such as dementia or macular degeneration, individuals with variable symptoms such as bipolar disorder, and individuals taking medications of different types or dosages.
Necessary Requirements in the Design and Implementation of Driving Simulator Protocols when Used for Assessment
With driving simulators becoming increasingly accessible due to wider availability, decreased costs, and increasingly specialized expertise, a growing appeal exists to use them widely in both clinical and research contexts. Irrespective of research domains or application, the following important conditions should be considered when using simulators for assessment.
Maintain a Consistent Language
Maintaining consistency among terminologies and taxonomies used in protocols, reports, and publications is of great importance. A recent publication in the Transportation Research Board Circular (2016) provides a helpful “Taxonomy and Terms for Stakeholders in Senior Mobility.” A lack of uniformity leads to misinterpretation of research findings, inconsistencies in research design implementation, and ineffective translation into practice. For example, authors may indicate that they are assessing “driving safety” when they are actually assessing “driving skills” (i.e., proper vehicle control choices and knowledge of the rules of the road), or “driving abilities” (i.e., sensory, cognitive, motor, functions used during vehicle control).
Establish Fidelity, Validity, Reliability
To ensure that the simulated driving experience is as comparable as possible to real-world driving experiences, establishing high fidelity, including: physical (e.g., motion capabilities), sensory (e.g., image resolution, field of view), and emotional (e.g., fear of adverse events) fidelity is important (Roza, 2005). For driving simulator outcomes to be reflective of what would occur during on-road driving, validity must also be established (ecological, content, construct, criterion; Kaptein et al., 1996; Lee et al., 2003; Bédard et al., 2010; Shechtman, 2010). While absolute validity may be difficult to obtain, relative validity is sufficient for many applications (Mullen et al., 2011). Moreover, simulator assessments must demonstrate reproducible results, and therefore reliability, including inter-rater and test-retest reliability must be evidenced (Bédard et al., 2010). Classen and Akinwuntan (2017) provide explicit operational definitions of these criteria and how they apply to driving simulation for use in assessment. While there is some limited evidence indicating that driving simulation measurements have moderate-high reliability, less is known about validity (Contardi et al., 2004; Lew et al., 2005; Bédard et al., 2010; Classen and Akinwuntan, 2017). Therefore, more evidence-based research to establish these criteria is warranted. With the advent of more sophisticated driving simulators, this is becoming increasingly possible to achieve.
Develop Hypothesis-Driven Assessment Protocols
Before developing a driving simulation assessment protocol, core objectives must be precisely defined using established conceptual frameworks, theories, and empirical evidence. The design of each study should be hypothesis-driven and the scenarios and the outcome measures chosen must be rigorously selected and inherently meaningful (see below). Investigators may be inclined to create scenarios that broadly sample from real-world driving experiences to be inclusive; however this approach lacks a clear motivation with respect to the hypotheses being evaluated. Similarly, it is tempting to analyze and report every outcome metric that the simulator software is able to extract. However, the inclusion and interpretation of specific outcome measures should be conceptualized a priori rather than discovered post-hoc. Exploratory analyses can have great utility in new and developing fields to reveal unanticipated effects, or model multivariate associations that may otherwise be difficult to conceptualize. However, caution should be taken in using results from these approaches to make recommendations regarding driving safety or to establish best practices prior to evaluating them in a prospective, hypothesis-driven manner.
Scenarios Should Be Customized to the Driving Behaviors and Populations of Interest
Driving scenarios have been categorized as targeting “operational level” (e.g., braking, steering), “tactical level” (e.g., maneuvering around obstacles, merging into traffic), and “strategic level” (e.g., mapping out goals and routes) performance based on Michon's hierarchical model of driving behaviors (Michon, 1985; Lindstrom-Forneri et al., 2010; Transportation Research Board Circular, 2016). The design of every simulated drive should deliberately target specific scenario levels based on predictions regarding the tasks for which a given population of drivers would be expected to be most at risk. For example, different tasks would challenge the risk associated with a motor impairment such as using a prosthetic device, vs. a cognitive impairment such as dementia. Driving assessments of those with motor impairments should include scenarios related to operational and tactical level performance given that mechanical or physical behaviors are most likely to be affected. In contrast, driving assessments of those with cognitive impairments might also include scenarios related to strategic level performance given that cognitive skills such as executive functioning and memory are likely to affect these types of driving behaviors.
In order to assess drivers under more challenging situations, layers of increased sensory, motor, or cognitive demands can be introduced. For example, consider the differences in sensory and cognitive load when an older driver negotiates daytime driving (optimal, low load), vs. nighttime driving in the rain (adverse, high load). Importantly, driving researchers should provide detailed descriptions of and justifications for the scenarios used in order to allow for conceptual clarity, reproducibility, and comparisons across studies.
Outcome Measures Should be Hypothesis-Driven
Outcome measures take various forms that uniquely address specific objectives (see Caird and Horrey, 2011 for a summary of common outcome measures). For instance simulator software typically collects kinematic data such as measures of longitudinal/lateral accelerations and turning rate, as well as summary statistics such as number of collisions. Notably, kinematic data provides more detailed information about the spatial and temporal dynamics of behavior compared to summary statistics.
Other outcome measures include assessments performed by trained evaluators such as licensing evaluators, occupational therapists, or driver rehabilitation specialists. These evaluators may observe drivers' behaviors during the simulated driving session or through video recordings. These might include, for instance, standardized demerit-point inventories used by licensing authorities, or other techniques used by driver rehabilitation specialists, such as structured observations of driving behaviors.
Other adjunctive measures that can be used to inform the interpretation of driving simulator performance measures include questionnaires (e.g., driving history and habits; Owsley et al., 1999), subjective rating scales (e.g., driving confidence, anxiety), and physiological response measures (e.g., heart rate, eye movements, brain activity; Johnson et al., 2011; Schweizer et al., 2013). It is also critical to monitor, measure and report common symptoms of simulator sickness (e.g., dizziness, nausea; see the Simulator Sickness Questionnaire Kennedy et al., 1993; Classen et al., 2011). Ultimately, one can evaluate how these adjunctive measures are associated with simulated driving related outcomes. For instance, one must determine whether measures of simulator sickness are associated with measures of driving performance to ensure that the interpretations of the real outcomes of interest are not confounded (Mullen et al., 2010).
Existing Gaps and Priorities for Future Research
In order to fully exploit the exciting potential of driving simulators in driving assessment there is a need to:
• Employ common and consistent operational definitions.
• Establish the validity and reliability of simulator-based assessments.
• Understand how the technological features of the simulator (e.g., field of view, motion capabilities) are associated with fidelity, validity, and reliability measures.
• Use a structured, targeted, hypothesis-driven approach when designing driving assessment scenarios and when interpreting outcome measures.
• Be transparent in reporting null results and negative outcomes (e.g., rates of simulator sickness).
While our team plans to tackle many of these issues using DriverLab by exploiting its unique, high fidelity features, these priorities apply broadly to any research or clinically-based driving simulator that is used for assessment.
Conclusions
Simulators are emerging as valuable tools for both research and driver assessment applications. The most significant current gaps that have been identified include the need to more firmly establishing criterion validity, ecological validity, and absolute validity across different configurations, applications and populations. The most significant opportunities for utilizing driving simulators occur in the context of assessing at-risk populations, modeling the effects of challenging environmental conditions in more realistic ways (night driving, rain), and evaluating the influence of emerging automated vehicle technologies on driving performance.
Author Contributions
All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.
Funding
This work was supported by a Canadian Institutes of Health Research (CIHR) Planning Grant [Awarded to Hebert (PI), Campos (Co-I) and others, # 354895]. GN is partially supported by the George, Margaret, and Gary Hunt Family Chair in Geriatric Medicine, University of Toronto.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
The reviewer LB and handling Editor declared their shared affiliation, and the handling Editor states that the process nevertheless met the standards of a fair and objective review.
Acknowledgments
We would like to thank the DriverLab Workshop planning committee for organizing the event.
References
Allen, W. R., Classen, S., and Cook, M. (2011). Driving simulator applications in research and clinical. Adv. Trans. Stud. Internat. J. 2010, 3–6. doi: 10.4399/97888548416591
Ball, K., and Owsley, C. (1993). The useful field of view test: a new technique for evaluating age-related declines in visual function. J. Am. Optom. Assoc. 64, 71–79.
Bédard, M., Parkkari, M., Weaver, B., Riendeau, J., and Dahlquist, M. (2010). Assessment of driving performance using a simulator protocol: validity and reproducibility. Am. J. Occup. Ther. 64, 336–340. doi: 10.5014/ajot.64.2.336
Bennet, J. M., Chekaluk, E., and Batchelor, J. (2016). Cognitive tests and determining fitness to drive in dementia: a systematic review. J. Am. Geriatr. Soc. 64, 1904–1917. doi: 10.1111/jgs.14180
Caird, J. K., and Horrey, W. J. (2011). “Twelve practical and useful questions about driving simulation,” in Handbook of Driving Simulation for Engineering, Medicine, and Psychology, ed J. D. Lee (Boca Raton, FL: CRC Press), 57–74.
Classen, S., and Akinwuntan, A. (2017). “Validity and Reliability in Driving Simulation,” in Best Evidence and Best Practices in Driving Simulation: A Guide for Health Care Professionals, ed S. Classen (Bethesda, MD: AOTA Press), 89–97.
Classen, S., Bewernitz, M., and Shechtman, O. (2011). Driving simulator sickness: an evidence-based review of the literature. Am. J. Occup. Ther. 65, 179–188. doi: 10.5014/ajot.2011.000802
Classen, S., and Brooks, J. (2014). Driving simulators for occupational therapy screening, assessment and intervention. Occup. Ther. Health Care 28, 154–162. doi: 10.3109/07380577.2014.901590
Classen, S., Dickerson, A. E., and Justiss, M. D. (2012). “Occupational Therapy Driving Evaluation: Using Evidence-Based Screening and Assessment Tools,” in Driving and Community Mobility: Occupational Therapy Across the Lifespan, eds M. J. McGuire and E. Schold-Davis (Bathesda, MD: AOTA Press), 221–277.
Contardi, S., Pizza, F., Sancisi, E., Mondini, S., and Cirignotta, F. (2004). Reliability of a driving simulation task for evaluation of sleepiness. Brain Res. Bull. 63, 427–431. doi: 10.1016/j.brainresbull.2003.12.016
Gibbons, C., Smith, N., Middleton, R., Clack, J., Weaver, B., Dubois, S., et al. (2017). Using serial trichotomization with common cognitive tests to screen for fitness to drive. Am. J. Occup. Ther. 71, 1–8. doi: 10.5014/ajot.2017.019695
Johnson, M. J., Chahal, T., Stinchcombe, A., Mullen, N., Weaver, B., and Bédard, M. (2011). Physiological responses to simulated and on-road driving. Int. J. Psychophysiol. 81, 203–208. doi: 10.1016/j.ijpsycho.2011.06.012
Kaptein, N. A., Theeuwes, J., and van der Horst, R. (1996). Driving simulator validity: Some considerations. Trans. Res. Rec. J. Trans. Res. Board 1550, 30–36. doi: 10.3141/1550-05
Kennedy, R. S., Lane, N. E., Berbaum, K. S., and Lilienthal, M. G. (1993). Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int. J. Aviat. Psychol. 3, 203–220. doi: 10.1207/s15327108ijap0303_3
Lee, H. C., Cameron, D., and Lee, A. H. (2003). Assessing the driving performance of older adult drivers: on-road versus simulated driving. Accid. Anal. Prev. 35, 797–803. doi: 10.1016/S0001-4575(02)00083-0
Lew, H., Poole, J., Lee, E., Jaffe, D., Huang, H., and Brodd, E. (2005). Predictive validity of driving-simulator assessments following traumatic brain injury: a preliminary study. Brain Injury 19, 177–188. doi: 10.1080/02699050400017171
Lindstrom-Forneri, W., Tuokko, H. A., Garrett, D., and Molnar, F. (2010). Driving as an everyday competence: a model of driving competence and behavior. Clin. Gerontol. 33, 283–297. doi: 10.1080/07317115.2010.502106
Marottoli, R. A., Cooney, L. M., Wagner, R., Doucette, J., and Tinetti, M. E. (1994). Predictors of automobile crashes and moving violations among elderly drivers. Ann. Intern. Med. 121, 842–846. doi: 10.7326/0003-4819-121-11-199412010-00003
Michon, J. A. (1985). “A critical review of driver behavior models: what do we know, what should we do?,” in Human Behavior and Traffic Safety. eds R. Schwing and L. Evans (New York, NY: Plenum Press), 487–525.
Molnar, F. J., Byszewski, A. M., Rapoport, M., and Dalziel, W. B. (2009). Practical experience–based approaches to assessing fitness in dementia. Geriatr. Aging 12, 83–92. Available online at: https://www.healthplexus.net/article/practical-experience-based-approaches-assessing-fitness-drive-dementia-0
Mullen, N., Charlton, J., Devlin, A., and Bédard, M. (2011). “Simulator validity: behaviors observed on the simulator and on the road,” in Handbook of Driving Simulation for Engineering, Medicine, and Psychology, eds D. L. Fisher, M. Rizzo, J. Caird, and J. D. Lee (Boca Raton, FL: CRC Press), 13–18.
Mullen, N. W., Weaver, B., Riendeau, J. A., Morrison, L. E., and Bédard, M. (2010). Driving performance and susceptibility to simulator sickness: are they related? Am. J. Occup. Ther. 64, 288–295. doi: 10.5014/ajot.64.2.288
Nasreddine, Z. S., Phillips, N. A., Bédirian, V., Charbonneau, S., Whitehead, V., Collin, I., et al. (2005). The montreal cognitive assessment (moca): a brief screening tool for mild cognitive impairments. J. Am. Geriatr. Soc. 53, 695–699. doi: 10.1111/j.1532-5415.2005.53221.x
Owsley, C., Stalvey, B., Wells, J., and Sloane, M. E. (1999). Older drivers and cataract: driving habits and crash risk. J. Gerontol. A Biol. Sci. Med. Sci. 54, M203–M211. doi: 10.1093/gerona/54.4.M203
Roza, Z. C. (2005). Simulation Fidelity Theory and Practice: A Unified Approach to Defining, Specifying and Measuring the Realism of Simulations. Delft: Delft University Press.
Schweizer, T. A., Kan, K., Hung, Y., Tam, F., Naglie, G., and Graham, S. J. (2013). Brain activity during driving with distraction: an immersive fMRI study. Front. Hum. Neurosci. 7:53. doi: 10.3389/fnhum.2013.00053
Shechtman, O. (2010). Validation of driving simulators. Adv. Trans. Stud. Int. J. 53–62. doi: 10.4399/97888548416596
Transportation Research Board Circular (2016). Taxonomy and terms for stakeholders in senior mobility. Trans. Res. Board. E-C211, 1–21. doi: 10.17226/23666
Keywords: driving simulator, scenarios, outcomes, assessment, evaluation, driving skills, risk assessment, psychomotor performance
Citation: Campos JL, Bédard M, Classen S, Delparte JJ, Hebert DA, Hyde N, Law G, Naglie G and Yung S (2017) Guiding Framework for Driver Assessment Using Driving Simulators. Front. Psychol. 8:1428. doi: 10.3389/fpsyg.2017.01428
Received: 26 May 2017; Accepted: 07 August 2017;
Published: 22 August 2017.
Edited by:
Markus Raab, German Sport University Cologne, GermanyReviewed by:
Laura Broeker, German Sport University Cologne, GermanyCopyright © 2017 Campos, Bédard, Classen, Delparte, Hebert, Hyde, Law, Naglie and Yung. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Jennifer L. Campos, Jennifer.Campos@uhn.ca