Sylwia Mac Jan 05, 2023 - Interior sensing

Arousal level in driver state benchmarking – application of ECG in automotive 

Authors: Magdalena Kotynia, Anna Stróż

In one of our previous posts, we had the opportunity to introduce you to the domain of vision-based aspects of Driver Monitoring Systems development. If you haven’t read the post yet, you can take a look at it here.

In today’s blog post, we would like to provide a brief selection of biosignal data applications in the context of driver monitoring and safety, as available in open-sourced scientific publications. As the topic is a very broad one, today we’ll limit the scope only to the application of electrocardiography (ECG).

ECG is broadly used in medical diagnostics, mainly to detect heart rhythm anomalies. Along with this progress in technology that is sneaking into our lives more and more, applications of ECG can be found in other areas such as: sport, e.g., in wearable devices monitoring our activity during exercises; home telemedical care, or… Driver Monitoring Systems research, which from our perspective, seems to be a fascinating application of that measure. In this post, we will focus on what we can learn from ECG in the context of drivers’ physiology and behavior, as a measurement of heart activity, it can be valuable in detection of drowsiness, fatigue [3, 9, 13], activity level and involvement in driving [3], distraction [3], or even driving under the influence [7, 12].

“Listen to your heart, there’s nothing else you can do”

Roxette in their “Listen to Your Heart” gave clear instructions on how to deal with the heart—we need to listen very carefully. But instead of listening to acoustic waves (e.g., as in echocardiography), we focus on “listening” to voltage changes, with electrodes as our “microphones”. Electrocardiography shows the electrical (“electro-“) activity of our heart (“-cardio-“ from Greek kardia meaning “heart”) in a graph (“-graphy”). As a heart repetitively fills with blood and then pumps it out to vessels, its muscles relax and contract. To do so, muscle cells receive electrical impulses, and their electrical properties change for a moment. On a macro scale, these changes in voltage can tell us which phase of a so-called cardiac cycle a heart is in. Is it now contracting or is it now relaxing? ECG can tell us that and much more, as you can see on the diagram below. It has some characteristic waves and segments, and each of them has a specific meaning. Let us introduce the three main components of ECG – P wave, QRS complex, and T wave – in the table below (Table 1).

Table 1. A summary of the most relevant ECG components with a description of ongoing events, based on [1]. Background image source: Pixabay. Licensed under Pixabay License.

Let’s dive deeper

To obtain useful information from the ECG signal, we usually need to do some computations. A correct ECG signal is not perfectly regular, as it is coupled with the nervous system, respiratory system, and other factors which are not constant. Thanks to that our hearts can adapt to different conditions. For example, when we inhale, our heart beats faster, and when we exhale our heart beats slower. Such changes are difficult to see directly, but we can capture them in other ways, for example by computing heart rate variability (HRV) parameters. Heart rate variability is the physiological variance in the time interval between subsequent heartbeats [2]. It is very useful in the clinical diagnosis of arrhythmias, but also in the detection of such states as drowsiness, stress, or even drunkenness. The most commonly used methods of HRV assessment are time-domain analysis and frequency-domain analysis, which is also known as spectral analysis. Time-domain analysis consists of the statistical processing of time intervals between components occurring in the ECG signal, e.g., RR intervals, for which it could be generalized to the two steps: 1. detection of all R peaks in the ECG signal (see Figure 1), 2. the calculation of time differences between two subsequent R peaks. There also exist so-called NN intervals, meaning normal-to-normal, as they are calculated after the exclusion of abnormal peaks in the ECG signal [10].

Figure 1. Relationship between the Cardiac Cycle and ECG. Source: [1], licensed under CC-BY 4.0.

Several examples of HRV time-domain measures are presented below, which are being calculated on top of prior identification of R peaks in the signal:

• SDNN – standard deviation of NN intervals [10, 14],

• SDANN – standard deviation of the average of the NN intervals in subsequent 5-minute periods of the recording lasting 24 hours [10],

• RMSSD – root mean square of the difference of subsequent RR intervals [4, 10, 14],

• NN50 – the number of pairs of subsequent RR intervals in which the difference is more than 50 ms within a given length of measurement time [4]. Can be also defined via a percentage of such events, pNN50 [10].

As we finish the first steps and obtain lengths of RR (or NN) intervals in the ECG signals, we can create a time series, with time points at heartbeats’ occurrence time and data points given as interval’s length. If we plot them against each other, we can create a tachogram, on which we can visually assess when heart was beating faster or slower, or if such changes were rapid. Having a time series with RR/NN intervals, we can continue our analysis in the frequency domain, which is a very common practice in ECG signal processing. Frequency domain can tell us how much of a signal power is in a predefined frequency band. Two key frequency bands recognized in HRV frequency-domain processing are:

• LF – Power of the low-frequency band (0.04 – 0.15 Hz)

• HF – Power of the high-frequency band (0.15 – 0.4 Hz)

Low-frequency HRV power is to some extent being interpreted as a way to assess activity level of human sympathetic nervous system [11]—as the system’s involvement is related to increased readiness for action, increased activity in this part of the autonomic nervous system is popularly known as fight, flight or freeze response.

High-frequency HRV power is, on the other hand, to some extent interpreted as a measure of another division of the human autonomic nervous system—the parasympathetic nervous system [11]. On the contrary, increased activity of this system may indicate some relaxing processes or increasing drowsiness, and that’s why it is usually seen as a rest and digest response.

Let’s wrap it up—on the one hand there is a rest and digest response, and on the other fight, flight or freeze response—but please note that this is just a simplification and it’s not exactly a binary reaction of our organisms, as there are more complex, non-linear interactions observed [11]. Would it be possible then to use such indicators to assess driver’s increased arousal or tendency to fall asleep behind the wheel?

In [6] the authors summarized several findings about the relationship between LF/HF ratio and the driver’s fatigue level, and noticed that the reported results seem to be contradictory—with some works linking an increase in the ratio (as e.g., in [14]), and other linking decrease in the ratio with the occurrence of drowsiness. There is a similar case with another metric, RMSSD, as in some works a decrease of RMSSD was linked to stressful driving tasks, and its increase with lack of engagement [3], yet in another work [14], while comparing RMSSD for normal and drowsy states, it was decreased with a growing tendency to fall asleep. Therefore, as there is a limited consistency in utilizing single factors, there are approaches involving multifactor HRV analysis [3, 4] or Machine Learning approaches [7], with an attempt to grasp this fascinating complexity of human psychophysiology.

Researchers also combine ECG-derived indicators with other measures to improve driver monitoring systems.  In the works of Fujiwara et al. [4] and Wang et al. [9], authors validate HRV measures with the measurement of brain activity through EEG (sidenote: take a look at our blogpost about EEG in automotive here). But not only strictly physiological signals contain meaningful information. We can learn a lot from behavioural indicators, such as eye-blink frequency, and combine them together with HRV [3], or from driving parameters (e.g., speed, lane position, or steering wheel rotations), for which there were also attempts of utilization in pair with HRV measures [7].

Limitations and how to deal with them

Although the ECG can be a good indicator of a driver’s state, it is kind of challenge to apply ECG to real driving conditions. It would be indeed too much if the driver had to mount electrodes or even a wearable device for every drive. Just imagine a drunk driver wanting to drive and patiently placing ECG sensors and thinking “Oh, I need to wear my ECG sensor to improve safety on the road!”— quite an unrealistic scenario. Lohani, Payne and Strayer [3] tell us about some solutions to this problem. ECG sensors can be applied on the steering wheel [3, 9, 14] or the driving seat [3, 14]. To estimate heart rate and HRV measures, we can use another technique—photoplethysmography (PPG). This technique takes advantage of the fact that blood absorbs light proportionally to its amount in the blood vessels. With every heartbeat, blood is released into our vessels, and the pressure increases. PPG sensors with light sources can be also mounted on the steering wheel or we can use RGB cameras for heart rate estimation [7, 8]. All three techniques—ECG, PPG and camera-based estimation—have some pros and cons regarding reliability, precision and, last but not least, driver comfort.

Summary

Our goal was to introduce you to some of the basic properties of ECG signal and its potential applications in the domain of driver safety. We hope that it was a satisfying journey! If you have any questions in the matter of psychophysiology and human factors in driver safety research, you can reach us via the contact form on our website.

Sources

[1] Betts, G., Young, K. A., Wise, J. A., Johnson, E., Poe, B., Kruse, D., Korol, O., Johnson, J., Womble, M., & DeSaix, P. (2022). Cardiac Cycle. In Anatomy and Physiology 2e. OpenStax. https://openstax.org/books/anatomy-and-physiology-2e/pages/19-3-cardiac-cycle. License: CC-BY 4.0.

[2] Gu, Z., Zarubin, V. C., Mickley Steinmetz, K. R., & Martsberger, C. (2022). Heart Rate Variability in Healthy Subjects During Monitored, Short-Term Stress Followed by 24-hour Cardiac Monitoring. Frontiers in Physiology, 13. https://doi.org/10.3389/fphys.2022.897284. License: CC-BY 4.0.

[3] Lohani, M., Payne, B. R., & Strayer D.L. (2019) A Review of Psychophysiological Measures to Assess Cognitive States in Real-World Driving. Front. Hum. Neurosci. 13:57. https://doi.org/10.3389/fnhum.2019.00057. License: CC-BY 4.0.

[4] Fujiwara, K., Abe, E., Kamata, K., Nakayama, C., Suzuki, Y., Yamakawa, T., Hiraoka, T., Kano, M., Sumi, Y., Masuda, F., Matsuo, M., & Kadotani, H. (2019). Heart Rate Variability-Based Driver Drowsiness Detection and Its Validation With EEG. IEEE transactions on bio-medical engineering, 66(6), 1769–1778.  https://doi.org/10.1109/TBME.2018.2879346. License: Open Access.

[5] Tobaldini, E., Nobili, L., Strada, S., Casali K. R., Braghiroli. A. & Montano N. (2013). Heart rate variability in normal and pathological sleep. Front. Physiol. 4:294. https://doi.org/10.3389/fphys.2013.00294. License: CC-BY 3.0.

[6] Li, G., & Chung, W. Y. (2013). Detection of driver drowsiness using wavelet analysis of heart rate variability and a support vector machine classifier. Sensors, 13(12), 16494-16511.  https://doi.org/10.3390/s131216494. License: Open Access.

[7] Chen, H., & Chen, L. (2017). Support Vector Machine Classification of Drunk Driving Behaviour. International journal of environmental research and public health, 14(1), 108. https://doi.org/10.3390/ijerph14010108. License: Open Access.

[8] Chen, Q., Wang, Y., Liu, X., Long, X., Yin, B., Chen, C., & Chen, W. (2021). Camera-based heart rate estimation for hospitalized newborns in the presence of motion artifacts. Biomedical engineering online, 20(1), 122. https://doi.org/10.1186/s12938-021-00958-5. License: Open Access.

[9] Wang, F., Wang, H., & Fu, R. (2018). Real-Time ECG-Based Detection of Fatigue Driving Using Sample Entropy. Entropy, 20(3), 196. https://doi.org/10.3390/e20030196. License: Open Access.

[10] Shaffer, F., & Ginsberg, J. P. (2017). An overview of heart rate variability metrics and norms. Frontiers in public health, 258. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5624990/. License: CC-BY 4.0.

[11] Billman, G. E. (2013). The LF/HF ratio does not accurately measure cardiac sympatho-vagal balance. Frontiers in physiology, 4, 26. https://www.frontiersin.org/articles/10.3389/fphys.2013.00026/full. License: CC-BY 3.0.

[12] Wu, C. K., Tsang, K. F., Chi, H. R., & Hung, F. H. (2016). A precise drunk driving detection using weighted kernel based on electrocardiogram. Sensors, 16(5), 659. https://doi.org/10.3390/s16050659. License: Open Access.

[13] Nguyen, T., Ahn, S., Jang, H., Jun, S. C., & Kim, J. G. (2017). Utilization of a combined EEG/NIRS system to predict driver drowsiness. Scientific reports, 7(1), 1-10. ﷟ https://www.nature.com/articles/srep43933. License: Open Access.

[14] Jung, S. J., Shin, H. S., & Chung, W. Y. (2014). Driver fatigue and drowsiness monitoring system with embedded electrocardiogram sensor on steering wheel. IET Intelligent Transport Systems, 8(1), 43-50. https://ietresearch.onlinelibrary.wiley.com/doi/full/10.1049/iet-its.2012.0032. License: Free Access.