The automotive industry is changing rapidly. The first semi-autonomous cars bring many advantages in terms of safety, congestion reduction and environmental compatibility, but also problems. The Ad Vitam research project with the BFH Wirtschaft and the University of Fribourg, among others, is investigating where the interaction between man and machine comes up short.

The problems with (partially) autonomous driving can be of a technical but also of a human nature, whereby researchers in the field of cognitive ergonomics are particularly interested in the human aspects of automation. International research in this area has so far concentrated mainly on the intelligent processing of data from sensors that are directed outwards from the vehicle and record and analyse the surrounding situation. Our interdisciplinary consortium of scientists from the fields of computer science, cognitive neuroscience and cognitive ergonomics, on the other hand, are also interested in the situation inside the vehicle and direct our sensors towards the person driving. With the Ad Vitam research project, we aim to design novel, cooperative human-vehicle interactions to support human monitoring and intervention tasks and to enable a pleasant and safe driving experience.

Levels of vehicle automation

Automated driving changes the role of the human from actively operating the vehicle to mainly monitoring the system, with only occasional withdrawal of control in specific emergency situations. In this respect, the level of automation can be defined using the SAE standard J3016 (SAE International, 2018; see Figure 1), which can be divided into six levels ranging from level 0: no automation, for example in the VW Beetle, to level 5: full automation as in Waymo (the Google car). In the Ad Vitam project, we are dealing with vehicles at automation level 3 (conditional automation).

Figure 1. six-level classification of autonomous driving according to SAE International (source: https://www.nhtsa.gov/technology-innovation/automated-vehicles-safety).

Special challenge are emergency situations

The transfer of control in emergency situations poses a great challenge for the interaction between humans and technology. If a person is not actively involved in driving, their knowledge of the driving environment is limited – in this context we speak of reduced situational awareness. As a result, the person driving may delay or be unable to adequately take control. In addition, conditional automation allows the person to pursue other activities while driving, such as talking on the phone, reading or watching TV. These activities can have an impact on the emotional and cognitive state of the person driving, which in turn affects their ability to take control of the vehicle in an emergency situation. This is where the Ad Vitam research project comes in: We are trying to use psychophysiological data to infer the driver’s abilities and possibilities to take control and, based on this information, to adapt the interaction between vehicle and person. However, before we could dedicate ourselves to this important task, we first had to find out which critical situations can occur that require the human to take control.

Limitations of semi-autonomous vehicles

The first step was an analysis of the limitations described by the manufacturers in the user manuals of level 2 autonomous driving vehicles. Based on this analysis, a taxonomy was created containing six main categories of hazards for level 2 automation vehicles (see Figure 2, Capellara et al. ,2019a):

1. the vehicle environment (bad weather, brightness),
2. external human factors (other road users),
3. the type of road (intersections, rough road, narrow road conditions or hilly road),
4. the condition of the road (road markings, work zone),
5. temporary obstacles and
6. the modification of the vehicle (blocked or defective sensor, trailer).

Figure 2. limitations of autonomous driving vehicles of level 2 according to an analysis of user manuals. The left figure shows the overview of the six categories of the taxonomy of constraints.

Improved awareness of situations

Since the driving person might perform a secondary task during autonomous driving, there is a risk that this will have an impact on their situational awareness. Another focus of the project is therefore the development of human-computer interaction concepts that increase the driving person’s awareness of their driving environment. For this purpose, a mobile application was developed that delivers contextual information about the driving environment to the screen used to perform the secondary task (e.g. a smartphone or tablet, see Figure 3). This application enables the person driving to better identify potential hazards on the road and prepares them to regain control in the event of hazardous situations. In addition, a system of vibrations integrated in the seat has been developed, which directionally indicates potential hazards in the environment and thus increases the attention of the person driving. The effectiveness of these tools has been proven in initial studies and is currently being tested in further user trials (Meteier et al., 2021, Capellara et al., 2019b).

Figure 3. presentation of contextual information about the driving environment to alert to potential hazards when performing secondary tasks.

Simulator studies

In our stationary driving simulator in the Human Factors Lab at the University of Fribourg (see Figure 4), we have so far tested over 350 people in a conditionally automated driving environment. In each case, we have simulated specific driving situations (e.g. mental stress, distraction, fatigue, monotony, type of hazard, etc.) and recorded the behaviour of the person driving as well as psychophysiological indicators that can be derived from the measurement of heart rate (by means of electrocardiograms), skin conductance and respiration. Based on this extensive data set, we try to predict specific human states relevant for taking control using artificial intelligence, which we have succeeded in doing well for most driving situations (e.g. Meteier at al., in press, Meteier, 2021, de Salis et al., 2021; Meteier et al., in review).

Figure 4. stationary driving simulator of the Human Factors Lab at the University of Fribourg.

In one of the simulator studies, drivers had to regain control of the vehicle in five different situations, each of which involved one of the hazards described above due to limitations of autonomous driving vehicles (Meteier et al., 2021). The results showed that people who were informed about the limitations of this type of vehicle before driving had a better awareness of the environment in critical situations. Furthermore, the results show that the usefulness of receiving contextual information via the mobile application described above depends on what type of secondary task the person driving is performing (e.g. visual or auditory).

Next steps and implications

In a next step, we are now investigating how we can optimise the interaction of the vehicle with the driving person (e.g. auditory or haptic design of the alarm signal) in real time based on the knowledge of their psychophysiological data. Based on the findings from this research project, industry-relevant guidelines can be created on how to use multisensory interaction in cars and how it can be operated at different levels of attention. In addition, the physiological model for shared-control driving resulting from this project enables the definition of guidelines for increasing the safety of semi-autonomous vehicles that could be adopted by car manufacturers (and vehicle manufacturers in general). The results of this project may also help to define, in the context of future legislation, which types of secondary tasks the driving person may and may not perform during autonomous operation of the vehicle.

References

1. Capallera, M., Meteier, Q., de Salis, E., Angelini, L., Carrino, S., Khaled, O. A., & Mugellini, E. (2019a). Owner Manuals Review and Taxonomy of ADAS Limitations in Partially Automated Vehicles. Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, 156-164. https://doi.org/10.1145/3342197.3344530
2. Capallera, M., Meteier, Q., de Salis, E., Angelini, L., Carrino, S., Khaled, O. A., & Mugellini, E. (2019b). Secondary task and situation awareness, a mobile application for semi-autonomous vehicle. Proceedings of the 31st Conference on l’Interaction Homme-Machine, 1-10. https://doi.org/10.1145/3366550.3372258
3. de Salis, E., Meteier, Q., Capallera, M., Angelini, L., Sonderegger, A., Abou Khaled, O., Mugellini, E., Widmer, M., & Carrino, S. (2021). Predicting takeover quality in conditionally automated vehicles using machine learning and genetic algorithms. In: Proceedings of the 4th International Conference on Intelligent Human Systems Integration (IHSI 2021), February 22-24, 2021, Virtual Event, Palermo PA, Italy.
4. Meteier, Q., Capallera, M., Ruffieux, S., Angelini, L., Abou Khaled, O., Mugellini, E., Widemer, M., & Sonderegger, A., (in press). Classification of Drivers’ Workload Using Physiological Signals in Conditional Automation. Frontiers in Psychology, section Performance Science.
5. Meteier, Q., Capallera, M., Sonderegger, A., de Salis, E., Angelini, L., Carrino, S., Abou Khaled, O. & Mugellini, E. (2021). Physiological Response, Situation Awareness and Takeover Quality of Drivers in Critical Situations of Conditionally Automated Driving. In: Proceedings of the AutomotiveUI ’21, September 13-14, 2021, Virtual Event, DC, USA.
6. Meteier, Q., Capallera, M., de Salis, E., Sonderegger, A., Angelini, L., Carrino, S., Abou Khaled, O. & Mugellini, E. (2020). The Effect of Instructions and Context-Related Information about Limitations of Conditionally Automated Vehicles on Situation Awareness. In: Proceedings of the AutomotiveUI ’20, September 21-22, 2020, Virtual Event, DC, USA.
7. Meteier, Q., Capallera, M., de Salis, E., Widmer, M., Abou Khaled, O., Mugellini & E. Sonderegger, A. (in review). Carrying a Passenger and Relaxation Before Driving: Classification of Young Drivers’ Physiological Activation.
8. SAE International’s standard (n.d.). J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, online: http://www.sae.org/misc/pdfs/automated_driving.pdf, accessed November 2016

Acknowledgement

The Ad Vitam research project is funded by the Hasler Foundation as part of the ‘Cyber-Human Systems’ funding programme.