STARS - Scenario-Based Testing of Automated Robotic Systems

Researchers

Till Schallau

Till Schallau received the M.Sc. degree in Computer Science from TU University, Germany, in 2019. Since then, he has been a research associate (Ph.D.) at the Chair for Software Engineering in Prof. Dr. Falk Howar's group, TU Dortmund University, Germany. His current research interests include the formal specification of scenario-based testing of autonomous systems as well as utilizing domain-specific languages.

Dominik Schmid

Dominik Schmid received the M.Sc. degree in Computer Science from TU University, Germany, in 2023. During his Master’s thesis, he worked as a research associate at the Institute for Transport Logistics in Prof. Dr. Uwe Clausen's group for mathematical optimization, TU Dortmund University, Germany. Since then, he has been a research associate (Ph.D.) at the Chair for Software Engineering in Prof. Dr. Falk Howar's group, TU Dortmund University, Germany. His current research focuses on the formal specification of traffic and the verification of autonomated driving systems using temporal logic to classify driving scenarios in recorded and live data traces from the automotive domain.

Nick Pawlinorz

Nick Pawlinorz received his B.Sc. degree in Computer Science from TU Dortmund University, Germany, in 2024. His Bachelor's thesis, titled 'Extraktion von formal Analysierbaren Fahrdaten aus Computerspielen am Beispiel von GTA V', was related to the STARS project. He began working for the Automated Quality Assurance group under Prof. Dr. Falk Howar in 2023. He is currently pursuing his M.Sc. degree at TU Dortmund University and is involved in several research projects within the STARS context.

Dr. Stefan Naujokat

Stefan Naujokat received the Dipl.-Inf. and Dr.-Ing. degrees from the Department of Computer Science at TU Dortmund University, Germany, in 2009 and 2017, respectively. He was a Ph.D. student and postdoc at the Chair for Programming Systems until 2019 and currently is senior researcher at the Chair for Software Engineering in Prof. Dr. Falk Howar's group, TU Dortmund University, Germany. His dissertation focused on simplifying language workbenches to make the development of visual domain-specific languages more accessible to a wider audience of language engineers. His research interests furthermore include metamodeling, code generation, and bridging the gap between formal specifications and domain-specific languages.

Prof. Dr. Falk Howar

Falk Howar is a professor for Software Engineering in the Department of Computer Science at TU Dortmund University and Coordinator of Software Engineering Research at Fraunhofer ISST. His research focuses on the safety and security of intelligent and autonomous software systems. He is particularly interested in the use of formal methods to analyze the behavior of such systems. After studying computer science and earning his doctorate at TU Dortmund University, Falk Howar first worked in the USA at Carnegie Mellon University (Silicon Valley) and at NASA Ames Research Center, where he developed methods for testing an autonomous air traffic control system. Subsequently, he was on the management board of the Institute for Applied Software Systems Engineering at TU Clausthal. There, he conducted research on safe autonomous driving functions together with partners from the automotive industry.

Publications

Post-hoc Scenario-based Testing of Automated Driving Systems: Classification of Driving Scenarios and Checking of Functional Requirements in Recorded Data

Till Schallau, Dominik Schmid, Nick Pawlinorz, Harun Teper, Stefan Naujokat, Jian-Jia Chen, Falk Howar

In: Intelligent Vehicles (IV 2025)

2025

to be published

Extended Abstract of Poster: STARS: Tree-Based Classification and Testing of Feature Combinations in the Automated Robotic Domain

Till Schallau, Dominik Schmid, Nick Pawlinorz, Stefan Naujokat, Falk Howar

In: ICST Workshops (IWCT 2025)

2025

DOI: https://doi.org/10.1109/ICSTW64639.2025.10962523

Preprint: Download

Tree-Based Scenario Classification

Till Schallau, Stefan Naujokat, Fiona Kullmann, Falk Howar

In: NASA Formal Methods (NFM 2024)

2024

DOI: https://doi.org/10.1007/978-3-031-60698-4_15

Preprint: Download

STARS: A Tool for Measuring Scenario Coverage When Testing Autonomous Robotic Systems

Till Schallau, Dominik Mäckel, Stefan Naujokat, Falk Howar

In: Dependable Computing - EDCC 2024 Workshops (EDCC 2024)

2024

DOI: https://doi.org/10.1007/978-3-031-56776-6_6

Preprint: Download

Validating Behavioral Requirements, Conditions, and Rules of Autonomous Systems with Scenario-Based Testing

Till Schallau, Stefan Naujokat

In: Electronic Communications of the EASST (ECEASST 2023)

2023

DOI: https://doi.org/10.14279/tuj.eceasst.82.1222

Preprint: Download

Scenario-Based Testing of Automated Robotic Systems

Researchers

Till Schallau

Dominik Schmid

Nick Pawlinorz

Dr. Stefan Naujokat

Prof. Dr. Falk Howar

Publications

Post-hoc Scenario-based Testing of Automated Driving Systems: Classification of Driving Scenarios and Checking of Functional Requirements in Recorded Data

Extended Abstract of Poster: STARS: Tree-Based Classification and Testing of Feature Combinations in the Automated Robotic Domain

Tree-Based Scenario Classification

STARS: A Tool for Measuring Scenario Coverage When Testing Autonomous Robotic Systems

Validating Behavioral Requirements, Conditions, and Rules of Autonomous Systems with Scenario-Based Testing

Theses

Current Theses

Model Checking von CMFTBL-Formeln mit SMT-Solvern am Beispiel von automatisierten Fahrsystemen

A Proof-Logic-Based Explainable Model Checker for the Temporal Logic CMFTBL

Scenario Generation based on missing feature combinations using SCENIC

Analyse der Diversität des INTERACTION-Datensatzes

Erstellung eines Autopiloten in CARLA mit einstellbarer Fahrqualität

Risikobewertung von Szenarien in STARS mittels unfallrelevanter Fahrmetriken

Implementation of a Traffic Data Acquisition Prototype for Scenario-Based Testing of Automated Driving Systems

Supervised Theses

Strategien zur Segmentierung von Simulationsfahrten für formale Analysen mit STARS

Continuous TSC evaluation and visualization for driving in CARLA

Aufbau und Evaluierung einer quellenunabhängigen Schnittstelle für das STARS-Framework durch Nutzung einer ROS2-Umgebung am Beispiel des CARLA-Simulators

Analyse der Szenario-Diversität des KITTI-Datensatzes mit dem STARS Framework

Testfallerstellung für STARS Scenario Features durch effizientes Labeln von Fahrdaten in GTA V

Strategies for extracting static map data from GTA for formal analysis with STARS

Design und Implementierung einer DSL zur vereinfachten Beschreibung von Szenarien zum simulationsbasierten Testen von autonomen Fahrzeugen

Szenario-basierte Auswertung von formalisierten Verkehrsregeln für Autobahnverkehr mit dem STARS Framework

Extraktion von formal Analysierbaren Fahrdaten aus Computerspielen am Beispiel von GTA V

Scenario-Based Testing of Automated Robotic Systems

Researchers

Till Schallau

Dominik Schmid

Nick Pawlinorz

Dr. Stefan Naujokat

Prof. Dr. Falk Howar

Publications

Post-hoc Scenario-based Testing of Automated Driving Systems: Classification of Driving Scenarios and Checking of Functional Requirements in Recorded Data

Extended Abstract of Poster: STARS: Tree-Based Classification and Testing of Feature Combinations in the Automated Robotic Domain

Tree-Based Scenario Classification

STARS: A Tool for Measuring Scenario Coverage When Testing Autonomous Robotic Systems

Validating Behavioral Requirements, Conditions, and Rules of Autonomous Systems with Scenario-Based Testing

Theses

Current Theses

Model Checking von CMFTBL-Formeln mit SMT-Solvern am Beispiel von automatisierten Fahrsystemen

A Proof-Logic-Based Explainable Model Checker for the Temporal Logic CMFTBL

Scenario Generation based on missing feature combinations using SCENIC

Analyse der Diversität des INTERACTION-Datensatzes

Erstellung eines Autopiloten in CARLA mit einstellbarer Fahrqualität

Risikobewertung von Szenarien in STARS mittels unfallrelevanter Fahrmetriken

Implementation of a Traffic Data Acquisition Prototype for Scenario-Based Testing of Automated Driving Systems​

Supervised Theses

Strategien zur Segmentierung von Simulationsfahrten für formale Analysen mit STARS

Continuous TSC evaluation and visualization for driving in CARLA

Aufbau und Evaluierung einer quellenunabhängigen Schnittstelle für das STARS-Framework durch Nutzung einer ROS2-Umgebung am Beispiel des CARLA-Simulators

Analyse der Szenario-Diversität des KITTI-Datensatzes mit dem STARS Framework

Testfallerstellung für STARS Scenario Features durch effizientes Labeln von Fahrdaten in GTA V

Strategies for extracting static map data from GTA for formal analysis with STARS

Design und Implementierung einer DSL zur vereinfachten Beschreibung von Szenarien zum simulationsbasierten Testen von autonomen Fahrzeugen

Szenario-basierte Auswertung von formalisierten Verkehrsregeln für Autobahnverkehr mit dem STARS Framework

Extraktion von formal Analysierbaren Fahrdaten aus Computerspielen am Beispiel von GTA V

Implementation of a Traffic Data Acquisition Prototype for Scenario-Based Testing of Automated Driving Systems