Challenges for validation of automated driving for GNSS

Automotive industry will deploy vehicles equipped for automated driving. One main challenge at the time being is the validation of the intended functionality with required level of safety. The absolute position as a prerequisite today only is provided via GNSS for which no trace back to SI units exists. Therefore, today everybody is looking for a reliable way to assure integrity of provided position.

Martin Grzebellus Managing Director NavCert GmbH

Automated driving

The automation in driving as defined by SAE is categorised by the split of responsibility between driver and vehicle and the level of support functions provided by the vehicle. into six different categories as illustrated in Figure 1.

SAE defined 5 different levels of automation starting from level 0 with no command of automation to level 5, which represents fully autonomous driving. With increasing levels of automation, the system is entrusted to substitute the human driver incrementally e.g. vehicle steering, driving environment perception and vehicle fallback operation whereas in the highest level all driving functions are performed by the system fully automatic.

The absolute position of a vehicle can only be determined by GNSS. A delta to the absolute position especially in socalled GNSS denied environments may be determined by auxiliary sensors i.e. INS, odometer, etc. supporting GNSS.

For autonomous driving absolute positioning capability with, at least, lane accuracy and with high integrity in all driving environments is required. The required lane accuracy and performance integrity in AD are 10 cmlevel and meter-level, respectively (for more details see https://inlane.eu).

Validation Position Engine

All OEMs and tier 1 are developing solution / have developed solutions which shall be used in safety critical applications. However worldwide there is no way to assure the proper working of the implemented solutions. One example is Tesla creating the impression via PR that their smart vehicles drive on their own. Due to this misunderstanding driver share videos how their Tesla vehicle is driving although they are distracted by reading or even worse other activities. In case of an incident they refer to their Ts&Cs where in the contrary it is specified that the driver is of course all the time responsible for the activities of the vehicle in phases of automated driving and shall have every time the possibility to control steering wheel and speed of vehicle.

A position engine as depicted in Figure 2 is used for the determination of the position combining information form GNSS supported by other sensors like IMU and additional information as correction data.

For the development of safety critical components, the standard ISO 26262:2015 is used in the automotive industry and quite recently complemented by the ISO PAS 21448:2019. Looking to the modules of the position engine, all might be developed from scratch according to the specific requirements starting with the definition of safety goals except the GNSS correction service as this exists since quite some time.

Therefore special attention has to be given how to integrate correction data into the development and validation process according to ISO 26262:2015.

Determination of ground truth

For the evaluation of the functionality of the sensors, one main challenge is to determine the ground truth of the position of the vehicle in dynamic scenarios. Here typically expensive GNSS equipment with IMU and other sensors with a higher accuracy than the position engine under test is used as a reference system. The position determined by the reference system is regarded as ground truth and the position determined by the position engine is assessed and qualified in respect to the reference position. According to ISO 26262:2015 tools used in the development process have to be qualified if the tool may cause an error in the final product. As the only reason for using the reference system is the validation of the position engine in according to the ISO 26262:2015, the reference system obviously has a strong impact on the final product. The standard defines 4 levels of tool confidence, for the lowest level (TCL1), no confidence is needed so a tool qualification is not necessary, and all other levels up to TCL4 require qualification. The following four qualification methods are suggested with a fitting to the intended ASIL level from A to D:

1. Increased confidence from use

2. Evaluation of the development process

3. Validation of the software tool

4. Development in compliance with a safety standard

The available reference equipment is a commercial offthe- shelf product typically introduced to the market recently. Therefore method 1, 2 and 4 will not be applicable and the only alternative left is the validation.

Since quite some years various methods for assessment have been experienced, however so far nobody really was successful. Looking to the task, it’s obvious why this approach is very complex. In principle the reference system is also a position engine according to Figure 2 but with expected better performance like increased accuracy. In the past one approach was to construct tracks for which the trajectory could be determined with high accuracy. The device under test was fixed to a wagon driving on the track. By repeating the tests, one could statistically assess accuracy of the tested position engine or reference system. However, there was a sever disadvantage preventing the commercialization of this approach. The trajectory was reflecting the capabilities required for rail not for automotive and as such the potential limits of the sensors could not be assessed nor critical scenarios for road approximated.

Qualification of tools

In aviation since quite some years the assessment of GNSS receiver is done with laser trackers. The position engine is installed in an airplane to which a mirror is attached. On ground there are laser stations automatically following the mirror attached to the airplane. By this in a limited space but under real world conditions the airplane is traceable, and the trajectory can be determined as ground truth to which the calculated position of the position engine in the airplane is compared.

A similar approach is nowadays feasible with robot stations for vehicles. The robot stations may be installed in testbed suitable for required driving maneuvers with respect to speed and curve radiuses. Also, this tool requires prior use validation. However, as not any technology of the position engine is used, the assessment can be done in a different way as depicted in the following.

A laboratory accredited according to ISO 17025:2017 shall use only measurement equipment traceable back to SI units calibrated when used. In cases this is not feasible as in the GNSS environment, the laboratory shall implement its own method for validation. The laboratory has to provide evidence during accreditation and in regular internal and external audits that it is experienced in selection, verification and validation of methodologies assuring traceability and validity of results. Here special attention is required in the determination of the measurement uncertainty with an in-depth analysis of impacting factors, quantification and respective mathematical analysis.

The validation of measurement equipment is valid only for a dedicated period same as the calibration. Thereafter the validation has to be repeated by the accredited laboratory.

Validation of correction services

The usage of correction service implies same impact to the position engine as the determination of the ground truth. If the provided information as correction service is incorrect or misleading the intended accuracy of the position engine cannot be achieved and will result into an error. For that reason, the correction service itself has to be validated as well. For SBAS a certification for aviation has been done some years ago for EGNOS assessing the respective service provider ESSP in France with the result to be a certified air navigation service provider. Therefore, the correction service provided by ESSP may be used in aviation in safety critical applications. There are ideas to certify as well the new services of Galileo the High Accuracy Service (HAS) and/or the Commercial Authentication Service (CAS) in the context of automotive.

For service provider offering correction service today there is already an assessment in place resulting into a certificate by TÜV SÜD. The real time accuracy is validated and the process of offering the service assessed. Due to this approach one can derive that the offered service will work as specified for the lifetime of the certificate. The certificate is issued always for a period of one year with a recertification in year 2 and year 3. The respective certification mark is depicted in Figure 3. Latest in year 4 a complete new certification is required even if nothing changed.

Validation of position engine

The validation of the position engine can be done twofold, first driving in real world for a huge amount of time providing evidence on error behavior and second, in a simulation environment focusing on behavior in challenging situations. As a first step, an analysis has to be done identifying critical scenarios. Then the critical scenarios may be mapped to real world scenarios but mainly implemented in a simulation.

For the simulation the same challenge applies as for the validation of the reference system. All tools have to be qualified according IS26262:2015 prior usage. This applies for the used simulation environment including hard- and software. As the environment cannot be calibrated, a validation has to be developed according to a standard like ISO17025:2018. Based on the specified KPIs of the position engine the respective error impact will be analysed. Then the measurement uncertainty has to be calculated to determine if the intended methodology fulfills all specific requirements and may be used.

Conclusion

All components developed for safety critical applications like automated driving require an intensive testing according to ISO 26262. The main challenge so far was the qualification of the required tools. Here now a feasible way forward exists for qualification of reference systems used for determination of ground truth, for the correction service intended to be used as an improvement to the GNSS determined position in the position engine and finally for the simulation environment by accredited laboratories for GNSS.