Driver Recognition and Fatigue Detection Systems with Devices Powered by Intel

Overview

This article discusses fatigue detection systems. These systems are designed to alert a driver when fatigue behavior is detected. When deployed in automobiles, the system can help to reduce the number of accidents caused by driver fatigue. An additional use of the system is driver identification. Only an authorized user can drive the vehicle, which helps in reducing accidents due to unauthorized drivers. The suggested idea could be the next version of the concept proposed in the video here: http://www.intel.in/content/www/in/en/internet-of-things/videos/realsense-range-rover-features-video.html

Introduction

According to the British Royal Society for the Prevention of Accidents, driver fatigue contributes up to 20% of road accidents and up to a quarter of fatal and serious accidents. The statistics are that an estimated 1,550 deaths, 71,000 injuries, and $12.5 billion in monetary losses each year due to driver fatigue. Mercedes-Benz* did its own research into car crash causes in Germany and found that 25% of all fatal crashes are a direct result of fatigue. Fatigue increases the risk of an accident. When fatigued, drivers have a reduced awareness and quick response time to potentially hazardous situations are also compromised. Another major cause of road accidents are unauthorized persons who drive the vehicle, in place of the authorized drivers. This happens mainly on long drives when the authorized driver hands over the vehicle to unauthorized persons.

Intel® RealSense™ Technology

Intel came up with a smart camera which has the capability to get depth data (z-axis) apart from normal RGB feed. This camera has the capability for 3D tracking of 78 facial landmark points supporting avatar creation, emotion recognition and facial animation. Intel® RealSense™ camera detects landmarks on the depth image of a face automatically using the Intel® RealSense™ SDK. A geometric feature based approach is used for feature extraction. The distance between facial landmarks is used for the features. And for selecting an optimal set of features, a brute force method is used. Using this technology the facial expressions characteristic of fatigue can be identified.. Facial behavior indicating that the driver is talking on a mobile device can also be captured and analyzed. Regarding unauthorized drivers, the fleet management team can monitor who is driving the truck and can take action immediately should an unauthorized driver be detected.

3-Dimensional Recognition

Addition of depth info along with RGB image. Advantage of 3D facial recognition is that it is not affected by changes in lighting like other techniques. 3D data points from a face vastly improve the precision of facial recognition.

Technical feasibility of the solution

The system captures the face of the vehicle driver using a 3d camera with Intel® RealSense™ technology and the Intel® RealSense™ SDK which has capability for 3D tracking of 78 facial landmark points which will in turn identify the closure of eyelids. If the eyelids are closed for more than a while a song can be played which alerts and wakes up the driver. The frames captured can be processed at the edge and can also be sent to cloud for remote surveillance and remote operation. The project should consist of algorithms for face recognition and emotion analytics. RealSense API’s are already exposed on RSSDK emotion analytics and face recognition.

Architecture Diagram

Intel® IoT Gateway is a device which has the capability of processing RGB and depth streams from an Intel® RealSense™ camera and also to send raw frames to cloud, which will help remote management and monitoring team to monitor authentication of driver with face recognition technology and the fatigue state of a driver. The key advantage of connecting to the cloud is remote access of the application and any updates in the application can be managed OTA (over the air). The industrial products which can be used as gateway are the Intel® NUC and USB 3.0 compute stick which does not have a CAN bus protocol. Plug in a CAN bus to USB converter to the Intel NUC or USB 3.0 compute stick which serves the purpose. SR300 AND F200 are the camera modules can be used for this case study.

Project Explanation

Includes two modules:

• Facial Recognition
• Emotion Analytics

Facial Recognition

The key reason for accidents is an authenticated driver that will not drive the entire long haul journey. The authorized person will take a nap and will ask the unauthorized person (cleaner in most of the cases) to drive the truck or he will outsource it to unauthorized drivers who are inefficient in driving it. This can be overcome by using 3D sensor technology which is highly resistant and tamper proof. Initially the database is trained with the 3d data of users who are going to use the truck or car. Face detection algorithm can be programmed to control Engine Control Unit (Car or Truck). Intel RealSense Technology captures the face of a person driving the vehicle using 1080p RGB and IR components at regular intervals and will verify the authentication. Whenever a person tries to start the engine, it will authenticate the person with the preloaded secured data. The remote monitoring will take live video streams randomly and can make sure that authorized person is using the car. OTP (One time passed) should also be used to authenticate the driver initially in case of biometric failure and poor internet connectivity. By using this technology the remote monitoring team should be able to track the person at several instances during journey. In case of network bandwidth hungry time the remote team can take a live stream and saving it at the edge (not sending to cloud) and see the complete video after the journey is completed and can make sure that authorized person drove the entire distance.

Emotion Analytics

Driver fatigue is a primary cause of of road accidents. Two main scenarios thatcanlead to fatigue state are:

1. Driving while under the influence which can lead to coordination loss and impaired judgement
2. Impaired vision due to a continuous long haul journey.

This problem can be eliminated with the help of 3D Image processing and can continuously track the emotions of a driver. The RealSense camera (with emotion tracking and landmark detection) has the capability to identify the distance between eyelids and can also check the angle with which a person is facing the camera. With the above mentioned two features, the system will be able to capture the fatigue state of a person driving the vehicle. Once the person driving the vehicle is detected in a fatigue state, music will automatically play to alert the driver and the remote monitoring team is notified of the status of the driver. Also analytics programs can be used on the captured data which can help the remote management team to take preventive measures by understanding the reasons for a driver’s fatigue.

Features

• Accuracy: > 96% Accuracy and Continuous monitoring and Authentication for 5-7 Drivers. Achieved good recognition rate on our internally collected data with varying poses
• Precision: Robustness through Face data Interpolation. Reduce false rejection rate due to aging, glasses, change in facial hair.
• Security: Database is in the format of bin.
• Gesture detection ability
• Non-frontal faces with Yaw less than 30 degrees are allowed
• Works under low light conditions
• Works with webcam quality images
• Increase in verification rates with increase in number of training images.

Challenges

• Video feed stabilization in high vibration environment like a Heavy commercial vehicle.
• Intel RealSense camera is not auto grade but there are pilot implementations on passenger vehicle
• Various influences on Intel RealSense technology
• Ultra-Low light conditions
• Accuracy reduction with larger databases. Maximum database size of less than 100 users.