Safety-Critical Oracles for Metamorphic Testing of Deep Learning LiDAR Point Cloud Object Detectors
Robustness testing is crucial for verifying autonomous vehicles, especially for safety-critical deep learning components like light detection and ranging (LiDAR) object detectors. Metamorphic testing (MT) assesses the robustness by automatically generating test cases based on abstract system specifi...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Open Journal of Intelligent Transportation Systems |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10849578/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1825207009880834048 |
|---|---|
| author | Simon Speth Maximilian Trien Dominik Kufer Alexander Pretschner |
| author_facet | Simon Speth Maximilian Trien Dominik Kufer Alexander Pretschner |
| author_sort | Simon Speth |
| collection | DOAJ |
| description | Robustness testing is crucial for verifying autonomous vehicles, especially for safety-critical deep learning components like light detection and ranging (LiDAR) object detectors. Metamorphic testing (MT) assesses the robustness by automatically generating test cases based on abstract system specifications known as metamorphic relations (MRs). However, a key challenge is ensuring a traceable safety argumentation for MRs that is in line with industry standards. To ensure this traceability, we derive seven traceable metamorphic transformations from defects identified through interviews with industry experts. Another challenge is prioritizing failures by safety criticality, as not all failing test cases, as evaluated by current intersection over union (IoU)-based metamorphic oracles, pose the same safety risk. We address this by introducing novel egocentric test oracles based on traffic participants’ bounding boxes shifted into or out of the ego vehicle’s expected lane. Testing five LiDAR object detection systems working on two datasets by executing half a million metamorphic test cases (MTCs) shows that the number of failures decreases from 48k using IoU metrics to 342 safety-critical failures with our novel test oracle “shift out of ego lane.” This reduction enables testers to stay within the test analysis budget and, hence, manually analyze each failed MTC by prioritizing safety-critical test failures. |
| format | Article |
| id | doaj-art-366038e655164978b8a2014ad470fcfa |
| institution | Kabale University |
| issn | 2687-7813 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Open Journal of Intelligent Transportation Systems |
| spelling | doaj-art-366038e655164978b8a2014ad470fcfa2025-02-07T00:02:01ZengIEEEIEEE Open Journal of Intelligent Transportation Systems2687-78132025-01-0169510810.1109/OJITS.2025.353277710849578Safety-Critical Oracles for Metamorphic Testing of Deep Learning LiDAR Point Cloud Object DetectorsSimon Speth0https://orcid.org/0000-0002-8525-1823Maximilian Trien1Dominik Kufer2Alexander Pretschner3https://orcid.org/0000-0002-5573-1201Chair of Software and Systems Engineering, TUM School of Computation, Information and Technology, Technical University of Munich, Munich, GermanyChair of Software and Systems Engineering, TUM School of Computation, Information and Technology, Technical University of Munich, Munich, GermanyAD Sensorsets & Vehicle Validation Department, IAV GmbH, Munich, GermanyChair of Software and Systems Engineering, TUM School of Computation, Information and Technology, Technical University of Munich, Munich, GermanyRobustness testing is crucial for verifying autonomous vehicles, especially for safety-critical deep learning components like light detection and ranging (LiDAR) object detectors. Metamorphic testing (MT) assesses the robustness by automatically generating test cases based on abstract system specifications known as metamorphic relations (MRs). However, a key challenge is ensuring a traceable safety argumentation for MRs that is in line with industry standards. To ensure this traceability, we derive seven traceable metamorphic transformations from defects identified through interviews with industry experts. Another challenge is prioritizing failures by safety criticality, as not all failing test cases, as evaluated by current intersection over union (IoU)-based metamorphic oracles, pose the same safety risk. We address this by introducing novel egocentric test oracles based on traffic participants’ bounding boxes shifted into or out of the ego vehicle’s expected lane. Testing five LiDAR object detection systems working on two datasets by executing half a million metamorphic test cases (MTCs) shows that the number of failures decreases from 48k using IoU metrics to 342 safety-critical failures with our novel test oracle “shift out of ego lane.” This reduction enables testers to stay within the test analysis budget and, hence, manually analyze each failed MTC by prioritizing safety-critical test failures.https://ieeexplore.ieee.org/document/10849578/Metamorphic testingLiDAR point cloudsobject detectiondeep learningautonomous vehicles |
| spellingShingle | Simon Speth Maximilian Trien Dominik Kufer Alexander Pretschner Safety-Critical Oracles for Metamorphic Testing of Deep Learning LiDAR Point Cloud Object Detectors IEEE Open Journal of Intelligent Transportation Systems Metamorphic testing LiDAR point clouds object detection deep learning autonomous vehicles |
| title | Safety-Critical Oracles for Metamorphic Testing of Deep Learning LiDAR Point Cloud Object Detectors |
| title_full | Safety-Critical Oracles for Metamorphic Testing of Deep Learning LiDAR Point Cloud Object Detectors |
| title_fullStr | Safety-Critical Oracles for Metamorphic Testing of Deep Learning LiDAR Point Cloud Object Detectors |
| title_full_unstemmed | Safety-Critical Oracles for Metamorphic Testing of Deep Learning LiDAR Point Cloud Object Detectors |
| title_short | Safety-Critical Oracles for Metamorphic Testing of Deep Learning LiDAR Point Cloud Object Detectors |
| title_sort | safety critical oracles for metamorphic testing of deep learning lidar point cloud object detectors |
| topic | Metamorphic testing LiDAR point clouds object detection deep learning autonomous vehicles |
| url | https://ieeexplore.ieee.org/document/10849578/ |
| work_keys_str_mv | AT simonspeth safetycriticaloraclesformetamorphictestingofdeeplearninglidarpointcloudobjectdetectors AT maximiliantrien safetycriticaloraclesformetamorphictestingofdeeplearninglidarpointcloudobjectdetectors AT dominikkufer safetycriticaloraclesformetamorphictestingofdeeplearninglidarpointcloudobjectdetectors AT alexanderpretschner safetycriticaloraclesformetamorphictestingofdeeplearninglidarpointcloudobjectdetectors |