Sylwia Mac May 16, 2024 - Interior sensing

DMS Benchmarking – regulatory perspective

Author: Sylwia Mac, DMS Dataset Collection Coordinator @ Robotec.ai

The information provided in this blog is for general informational purposes only and does not constitute legal advice. The content and materials available here may not reflect the most current legal developments or other information.

If you are an OEM or Tier I, you probably already feel the pressure of forthcoming regulations on Driver Monitoring Systems in the European market. As of July 2024, the Driver Drowsiness and Attention Warning (DDAW; European Commission, 2021) systems will be mandatory, and two years later Advanced Driver Distraction Warning (ADDW; European Commission, 2023) regulation will take effect. A separate, but not less important from a safety (and marketing) perspective is the Euro NCAP rating (Euro NCAP, 2024), which also considers Driver State Monitoring while assessing new cars – both for drowsiness and distraction detection. So, a properly working algorithm detecting driver state is a crucial thing to include in a new vehicle. But how to choose a DMS supplier from so many options available on the market? In this blog post, we will discuss the regulatory perspective of driver state monitoring system validation focusing on test procedures.

What is important in DMS benchmarking?

Two main features need to be included in your Driver State Monitoring device – drowsiness and distraction detection. Regulatory-wise, both need to be tested with an appropriate group of drivers (e.g., holding a specific type of driving license). It is important to have a diversified testing sample to ensure that the system works for different-looking drivers (in terms of age, gender, or skin color, among others) with different occlusions (such as clothing and accessories partially covering the face or eyes). The data collected from them shall be processed in accordance with the European Union data protection law.

The testing environment is another very important issue here – from safe and relatively cheap simulator studies to complex naturalistic driving studies that give the most lifelike results but are expensive and least safe. More detailed information about testing environment selection can be found in one of our previous blog posts. Since the system is expected to operate effectively during the day and night (under a variety of illumination intensities), the environmental conditions must be implemented and measured during the study.

Comparing two or more DUTs could be difficult to do simultaneously, as most cameras have algorithms for effective illumination that may interfere with other infrared-based devices. On the other hand, it would be most reliable if the cameras could be verified on the exact same example of driver behavior. This would only succeed if the recordings were made at the same time since it is impossible to replicate gazes and movements exactly when testing with humans. Recommended mounting locations for different cameras should also be considered, which can be a tricky issue especially when testing the devices simultaneously (unless the camera position has been imposed by the OEM).

Last but not least, the actual warning. Only the system issuing the alert can be assessed properly, as it is the time of occurrence of this feature that is evaluated. This may be a tricky part for the DMS suppliers while testing in the simulated environment, as many systems retrieve data from the vehicle and use it as an additional logic layer when assessing the driver’s state (such indicators may come from e.g., lane position, temporal metrics, or steering input). Please note that during the validation testing, the warnings should be muted, but the time at which the warning from the system was provided should be recorded to establish if it was a true positive event.

DROWSINESS

The recommended measure to use as a ground truth for drowsiness detection is the Karolinska Sleepiness Scale (KSS). Other means of measuring drowsiness can also be used, but they are required to be validated against KSS. The most troubling issue is that the KSS is subjective – the driver himself needs to assess his alertness or drowsiness on a 9-level scale every 5 minutes. It is very important (and also required by regulations) to provide proper KSS training for all the participants taking part in DMS testing. Based on our experience, a proprietary three-step training is administered to the drivers to ensure that they can recognize internal latent states, have a deeper introspection on their alertness or sleepiness levels, and thus, assign them to the specific labels of the scale, resulting in more fine-grained outcomes. We also use a dedicated keypad to collect the answers to interact with the driver as little as possible and not affect the process of increasing sleepiness.

The drowsiness alert can be issued when the driver reaches KSS 7 and should be issued when the driver’s drowsiness self-assessment is equal to 8 or 9 on the scale. The alert present if the participant’s previous or next rating is at the KSS level of 7 or above, shall be treated as true positive – once a true positive event occurred, the data points after this event shall be considered irrelevant for this specific test. Several other rules are indicated in the regulations to validate the course of the KSS, which should be considered when analyzing and reporting the results. For drowsiness detection, alternative measurements can be used, such as electroencephalography (EEG) or PERCLOS (percentage of eyelid closure), though the evidence should be provided that the chosen measurement is a valid and accurate means to assess driver drowsiness and that the drowsiness threshold used in the validation testing is equivalent to a KSS level being the drowsiness threshold.

DISTRACTION

For distraction detection, a specific time of looking off the road is indicated within the regulation. The European Commission (2023) specifies that the alert should be issued when the driver is looking at any area below a plane, 30° downward from the driver’s ocular reference point for 6 seconds at a lower speed (between 20 and 50 km/h), or 3.5 seconds while maintaining a higher speed (above 50 km/h). These times can be extended by 1.5 seconds if there is a so-called non-nominal situation present – a system limitation that may cause the ADDW system to have its performance degraded. Non-nominal situation descriptions shall be provided by the manufacturer. There is also an indication to have at least 15 seconds of non-distracted driving between testing the subsequent locations indicated in the regulation. Spot testing for ADDW allows for repetition of the procedure (for a maximum of two times for each fixation point).

Euro NCAP (2024) provides more detailed requirements for distraction detection. There are types of movements specified – Lizard or Owl. Lizard Type is a “movement in which the driver focuses on a task by moving primarily their eye line away from the road with their head/face remaining in the forward-facing position” (Euro NCAP, 2024, p. 5), while Owl Type is “a shifting of visual attention away from the road and forward-facing position that is primarily achieved by a head rotation followed by the eyes” (Euro NCAP, 2024, p. 5). According to the Euro NCAP, an alert should be issued for different types of inattention – long, and short. Long Distraction is considered as “a single long-duration driver gaze away from the forward road to one consistent location of ≥ 3 seconds” (Euro NCAP, 2024, p. 16). Short Distraction (or VATS –Visual Attention Time Sharing) is defined as “repeated glances away from the forward road view either repeated towards one location or to multiple different locations.

A short distraction event is a build-up of multiple glances away from the forward road view and is considered to end when the driver’s attention returns to the forward road view for a period long enough for the driver to fully interpret the road situation” (Euro NCAP, 2024, p. 16). An example of suitable requirements for a driver to be classified as distracted is when a driver glances away from the forward road view for a cumulative 10 seconds within a 30-second time period, where the time period is reset if the driver’s glance returns to the forward road view for a period of ≥ 2 seconds. There is also a Phone Use distraction that is considered by Euro NCAP to be “a specific type of short distraction (visual attention time sharing) event where the driver’s repeated gaze is towards their mobile phone” (Euro NCAP, 2024, p. 17). The gaze locations are indicated for each type of distraction (Euro NCAP, 2024, p. 15).

Not only DMS, not just Europe

The Driver Monitoring System is not only required as a feature per se but is also crucial as part of Driver Availability Monitoring Systems (DAMS) required in SAE level 3 and 4 automation implementations. When the vehicle is driving in automation mode, but the driver’s supervision is still required to some extent, such systems monitor whether the driver is alert and focused enough to regain situational awareness quickly and safely take over control of the vehicle. You can find more information about the driver’s role in the automated vehicle in our blog post.

There are also other camera-based features in a vehicle related to occupants’ experience rather than safety, which are becoming increasingly popular among stakeholders. Such features include adjusting the driver’s seating position or in-cabin environment control, such as adjusting the temperature inside the vehicle.

Since our company is based in Europe and focuses mainly on satisfying European safety requirements, please note that other parts of the world may have different regulations with slightly varying indications. Just to name a few, there are:

• Chinese national standard GB/T 41797-2022 (National Technical Committee of Auto Standardization, 2022) titled “Driver Attention Monitoring System (DAMS) Performance Requirements and Test Methods” – this regulation mandates monitoring and alert in case of the driver’s eye closing, abnormal head pose, and hand-held phone answering (with the possible alert in case of the driver’s yawning or smoking behavior).

• Finalized draft of Indian Automotive Industry Standard AIS-184 (Automotive Research Association of India, 2022) “Driver Drowsiness and Attention Warning Systems for M, N2, and N3 Category Vehicles” requirements resembling those of DDAW.

• The US 2021 Infrastructure Investment and Jobs Act (from P1045; United States Congress, 2021) where can be read that: “Not later than 3 years after the date of enactment of this Act, the Secretary shall conduct research regarding the installation and use on motor vehicles of driver monitoring systems to minimize or eliminate: driver distraction, driver disengagement, automation complacency by drivers, and foreseeable misuse of advanced driver-assist systems”, so this one should be kept on the lookout for in the near future.

• UNECE (UNECE, 2019) states that including DAMS is one of their top priorities in the development of regulatory guidelines for automated vehicles: “Automated/ autonomous vehicle should include driver engagement monitoring in cases where drivers could be involved (e.g. take over requests) in the driving task to assess driver awareness and readiness to perform the full driving task. The vehicle should request the driver to hand over the driving tasks in case that the driver needs to regain proper control of the vehicle.”

  
  

At Robotec.ai, driver monitoring including DMS benchmarking is one of our areas of expertise, so if you want to discuss some related issues or explore cooperation possibilities, let us know at humanfactors@robotec.ai. We’re looking forward to hearing from you! You can also follow our LinkedIn page for the latest company updates.

Sources:

Automotive Research Association of India. (2022, November). Automotive Industry Standard AIS-184/DF. Driver Drowsiness and Attention Warning Systems for M, N2, and N3 Category Vehicles.

European Commission. (2021). Commission Delegated Regulation (EU) 2021/1341 of 23 April 2021 supplementing Regulation (EU) 2019/2144 of the European Parliament and of the Council by laying down detailed rules concerning the specific test procedures and technical requirements for the type-approval of motor vehicles with regard to their driver drowsiness and attention warning systems and amending Annex II to that Regulation. Official Journal of the European Communities, L292, 4-19. Notified under document number C/2021/2639. http://data.europa.eu/eli/reg_del/2021/1341/oj

European Commission. (2023). Commission Delegated Regulation (EU) 2023/2590 of 13 July 2023 supplementing Regulation (EU) 2019/2144 of the European Parliament and of the Council by laying down detailed rules concerning the specific test procedures and technical requirements for the type-approval of certain motor vehicles with regard to their advanced driver distraction warning systems and amending that Regulation. Official Journal of the European Communities, OJ L, 2023/2590, 22.11.2023. Notified under document number C/2023/4523. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32023R2590

European New Car Assessment Programme (Euro NCAP). (2024, February). Euro NCAP Assessment Protocol – Safety Assist Safe Driving. Version 10.4.

National Technical Committee of Auto Standardization. (2022). GB/T 41797-2022. Driver Attention Monitoring System (DAMS) Performance Requirements and Test Methods.

United Nations. Economic Commission for Europe, & Inland Transport Committee. (2019, July). World Forum for Harmonization of Vehicle Regulations (WP.29), 178th session. Framework document on automated/autonomous vehicles. ECE/TRANS/WP.29/1147

United States. Congress. (2021). Infrastructure Investment and Jobs Act, Pub. L. No. 117-58.