Anna Stróż Mar 06, 2023 - Interior sensing

Naturalistic vs. simulated driving – a bargain between natural feel and safety in DMS testing

Authors:
Olga Zdzienicka, DMS Researcher @ Robotec.ai
Anna Stróż, Senior Scientist @ Robotec.ai
***

In the previous blog posts concerning Driver Monitoring Systems (DMS) field, we introduced you to the domain of vision-based aspects in DMS development (here), as well as biosignal data application for psychophysiological studies among drivers (here). In today’s post, we would like to take a step back and look at the whole subject from a broader perspective. Today we want to engage you in the discussion about a methodologically important part of DMS testing, which is a careful choice of the testing environment.

Most of car accidents are caused or influenced by driver errors (see Treat et al. 1979 for a detailed report on influencing factors; Wierwille et al., 2002; Singh, 2015). Aside from legal actions and the modernization of road infrastructure, technological developments are on their way to address the need for the reduction of driver errors. Ongoing efforts in the domains of Advanced Driver Assistance Systems (ADAS) and Driver Monitoring Systems (DMS) development are a promising step forward.

Yet, before such systems will “hit the road”, there are numerous steps of iterative prototyping, testing, and subsequent development of their features. Some of the stages may be performed in a laboratory setting, such as tests of low-level features of a system, e.g. its performance in face detection in precisely defined illumination conditions, but it is also possible to test high-level features—like severe drowsiness detection—which due to safety reasons should be performed in a well-controlled environment. However, as the system reaches its maturity and higher levels of technological readiness, the testing environment should resemble the working environment of a system—natural driving conditions. Below, we will dive into specific details on the features of both laboratory settings, i.e. driving simulators, and natural setting, known as Naturalistic Driving Studies (NDS).

Naturalistic Driving Studies

As the name suggests, Naturalistic Driving Studies are conducted in an environment as natural for drivers as possible, which are streets, and—depending on a topic under investigation—they can be urban, rural, highway, or of mixed conditions. Studies of this kind are usually designed and undertaken in order to provide insight into driver behavior under natural road conditions. On-road studies can provide as many distractions, changes, and stressful situations as ordinary driving in daily tasks and everyday trips. This type of research makes it possible to observe the behavior, distraction, or fatigue of drivers in normal, undirected, and unexpected driving situations (see e.g. Bagdadi, 2013; Eenink et al., 2014; Wijayaratna et al., 2019), as during an on-road study presence of environmental stimuli cannot be controlled. Studies of this sort let the researchers not only gather data about driver’s behavior (such as distraction, fatigue, eyes, head, and hands movements), but also road and weather conditions, and therefore, observe and analyze relationships between the driver, other traffic participants and external conditions. With additional safety precautions (e.g. safety instructor and a set of double pedals), such studies may allow us to observe the driver’s engagement in secondary tasks, such as phone use, eating, or drinking (see e.g. Hanowski, Perez & Dingus, 2005 and Hickman & Hanowski, 2012; Wijayaratna et al., 2019). Because naturalistic driving studies enable the observation of so many factors in real-world driving situations, researchers can gauge the likely safety risks, limitations, and performance of tested DMS solutions.

However, despite the possibility of collecting various types of data and gaining information relating directly to drivers’ actions and behavior in a variety of traffic situations, this type of research can be more stressful and dangerous than other available research methods. In studies involving young drivers (Mayhew et al., 2011) with little driving experience, or the elderly (Lee, Cameron & Lee, 2003), the stress caused by the research procedure can lead to dangerous driving situations, as well as distort the results of the study itself while undermining its validity. Our experiences in the Human Factors team led us to the conclusion that careful preparation of participants, a comfortable study atmosphere and, last but not least, a meticulous approach to safety, may help to lower levels of stress perceived by participants and lead to a successful data collection.

To sum up, our considerations regarding NDS, we prepared a graphical summary of benefits, as well as limitations, associated with driving studies in a naturalistic context—you can find them in Figure 1.

Figure 1. Comparison of benefits and limitations of Naturalistic Driving Studies. Source: Robotec.ai. Background image source: Phillip Gorbachev from Pixabay. Licensed under Pixabay License.

Research with driving simulators

As we discussed some of the most relevant factors of the naturalistic driving studies, let’s take a look at an alternative solution, driving simulator studies, and talk about its potential benefits in the domain of human factors research.

Driving simulators vary in their complexity—with simpler simulators having a fixed base, set of screens, and steering accessories, and those much more detailed, known as high-fidelity driving simulators, which are typically built on a moving platform with a full-scale, equipped vehicle, which furthermore can also be enclosed in a chamber enabling 360° visual simulation (see e.g. Mohn, 2021). Simulators used in such studies are supposed to imitate cars as much as possible—usually, they are built on actual running car’s cabins positioned on the moving platform with the scene displayed on the monitor or flat surface in front of the car, all necessary vehicle controls, and a sound system (Carsten & Jamson, 2011). Our Human Factors team has great expertise in conducting studies with moving-base driving simulators also combining experimental protocols with psychophysiological measures, such as electroencephalography (EEG) or electrocardiography (ECG), and subjective assessment (e.g. Karolinska Sleepiness Scale, KSS). We are happy to collaborate with the simulation experts from the Motor Transport Institute in Warsaw, and to develop fascinating driving-related projects together.

Figure 2. One of the full-scale driving simulators we use in our studies, located at the Motor Transport Institute in Warsaw. Source: Robotec.ai.

Driving simulators usually try their best to reflect and pretend natural road conditions by adding elements such as urban infrastructure, pedestrians, and animals to the simulation. To further unnaturalize the driving experience, some simulators also implement various weather conditions. All of these additions help bring the simulation closer to the natural experience of drivers, but they take place in a safe environment and do not expose drivers to the dangers of, for example, a child trespassing on the street or bad weather conditions such as fog or rain.

Although simulation might feel close to real it is devoid of the dangers that exist on real roads. And while it is possible to cause an accident in the simulator it is harmless, and eventual exposure to stressors is therefore limited.

Despite the primary upside of simulator testing, which is driver safety, this solution also has some disadvantages. One of these is the possible occurrence of simulator sickness in test participants. Simulator sickness is a form of motion sickness, and it is caused by a mismatch between visual perception and the perception of acceleration or deceleration and the vestibular sensation of the same movement. The occurrence of simulation sickness is rare, but it does happen and can affect psychomotor abilities at any given time (Carsten & Jamson, 2011; Lee et al., 2003), therefore it is crucial to monitor the well-being of study participants, and exclude those who may be vulnerable to the consequences of simulation sickness.

Additionally, while the graphics performance is getting better and is under constant development, driving simulators can still give a pretty game-like feel and because of that they are not absolute replicates of the real-world experience. Assuming that we discuss the aspects of high-fidelity driving simulators, we provided you with a summary of the pros and cons in Figure 3.

Figure 3. Benefits and limitations in the usage of driving simulators in studies. Source: Robotec.ai.

NDS vs. simulator—summary. Are there any trade-offs?

To sum up our considerations, the two environments—NDS and driving simulators—may be helpful to satisfy different research needs. While choosing the proper setup for the testing process, it is worthwhile to take into consideration several aspects, such as:

What are the primary research goals, hypotheses, and eventual constraints (e.g. whether one wants to record facial images in a predefined illumination setting or to test the setup with accordance to existing regulations);

  • Technological readiness of the system and maturity of the project;
  • Assumptions regarding the study sample and project’s budget;
  • Need for preparation of experimental protocols including stimuli (e.g. dual-task paradigm) or additional psychophysiological data acquisition;
  • Availability of experts in safety and safe-oriented setup preparation.

However, as the processes in the development of ADAS and DMS solutions are multilevel and iterative, both driving simulators and naturalistic driving studies are useful, but at different stages of the project’s maturity. Driving simulators may greatly help with quick prototyping and testing of low-level performance features, or to collect input data for developed algorithms. On the other hand, naturalistic driving studies may be useful in subsequent, iterative testing of a complex solution, and for data collection in more challenging environments, testing the system’s stability and sensitivity to dynamically changing conditions. We can notice then that both environments are needed to be considered in the R&D process of DMS and ADAS development.

There is also an intermediate solution, which can be seen as a tradeoff for the discussed environments. These are test tracks, which enable us to grasp a piece of natural driving experience while maintaining a high level of safety. As a downside, it is still a mock-up of the real world with limited exposure to its true dynamics (e.g. presence of other road users), yet some aspects, such as vehicle physics and the system’s performance on different road conditions, can be successfully tested.

One is certainly for sure: these three worlds—simulators, NDS, and test tracks—will jointly satisfy all the research needs in the process of DMS and ADAS development.

References:

Bagdadi, O. (2013). Assessing safety critical braking events in naturalistic driving studies. Transportation research part F: traffic psychology and behaviour, 16, 117-126. DOI: https://doi.org/10.1016/j.trf.2012.08.006.

Carsten, O., & Jamson, A. H. (2011). Driving simulators as research tools in traffic psychology. In Handbook of traffic psychology (pp. 87-96). Academic Press. DOI: https://doi.org/10.1016/B978-0-12-381984-0.10007-4.

Eenink, R., Barnard, Y., Baumann, M., Augros, X., & Utesch, F. (2014). UDRIVE: the European naturalistic driving study. In Proceedings of Transport Research Arena. IFSTTAR. URL: https://eprints.whiterose.ac.uk/93078/.

Guo, F., Klauer, S. G., Hankey, J. M., & Dingus, T. A. (2010). Near crashes as crash surrogate for naturalistic driving studies. Transportation Research Record, 2147(1), 66-74. DOI: https://doi.org/10.3141/2147-09.

Hanowski, R. J., Perez, M. A., & Dingus, T. A. (2005). Driver distraction in long-haul truck drivers. Transportation Research Part F: Traffic Psychology and Behaviour, 8(6), 441-458. DOI: https://doi.org/10.1016/j.trf.2005.08.001.

Hickman, J. S., & Hanowski, R. J. (2012). An assessment of commercial motor vehicle driver distraction using naturalistic driving data. Traffic injury prevention, 13(6), 612-619. DOI: https://doi.org/10.1080/15389588.2012.683841.

Lee, H. C., Cameron, D., & Lee, A. H. (2003). Assessing the driving performance of older adult drivers: on-road versus simulated driving. Accident Analysis & Prevention, 35(5), 797-803. DOI: https://doi.org/10.1016/S0001-4575(02)00083-0.

Mayhew, D. R., Simpson, H. M., Wood, K. M., Lonero, L., Clinton, K. M., & Johnson, A. G. (2011). On-road and simulated driving: Concurrent and discriminant validation. Journal of safety research, 42(4), 267-275. DOI: https://doi.org/10.1016/j.jsr.2011.06.004.

Mohn, H. (2021, May). Moving-base driving simulator: Virtual reality on tracks. Magazine for Mobility and Society. Mercedes-Benz. URL: https://group.mercedes-benz.com/company/magazine/technology-innovation/100-things-driving-simulator.html. Accessed: 22.02.2023.

Singh, S. (2015, February). Critical reasons for crashes investigated in the National Motor Vehicle Crash Causation Survey. (Traffic Safety Facts Crash•Stats. Report No. DOT HS 812 115). Washington, DC: National Highway Traffic Safety Administration. DOI: http://www-nrd.nhtsa.dot.gov/Pubs/812115.pdf.

Treat, J. R., Tumbas, N. S., McDonald, S. T., Shinar, D., Hume, R. D., Mayer, R. E., … & Castellan, N. J. (1979). Tri-level study of the causes of traffic accidents: final report. Executive summary. Indiana University, Bloomington, Institute for Research in Public Safety. URL: https://deepblue.lib.umich.edu/handle/2027.42/64993.

Wierwille, W. W., Hanowski, R. J., Hankey, J. M., Kieliszewski, C. A., Lee, S. E., Medina, A., … & Dingus, T. A. (2002). Identification and evaluation of driver errors: Overview and recommendations. URL: https://rosap.ntl.bts.gov/view/dot/922.

Wijayaratna, K. P., Cunningham, M. L., Regan, M. A., Jian, S., Chand, S., & Dixit, V. V. (2019). Mobile phone conversation distraction: Understanding differences in impact between simulator and naturalistic driving studies. Accident Analysis & Prevention, 129, 108-118. DOI: https://doi.org/10.1016/j.aap.2019.04.017.

World Health Organization. (2018). Global status report on road safety 2018, Geneva. Licence: CC BY-NC-SA 3.0 IGO. URL: https://www.who.int/publications/i/item/9789241565684.

Share post: