Challenge Evaluation Items 100% Description Fair 70% - 80% Very Good 81% - 95% Excellent 96%-100%
Simulation Test Scenario Generation 20% Simulation Test Scenario Generation Ad-hoc Manual Approach Semi-Automatic Approach Automatic Approach
No. of Generated Scenarios Specified scenarios for a given route Generated scenarios with good diversity Generated scenarios with excellent diversity
Simulation Test Script Generation 20% Simulation Test Script Generation Ad-hoc Manual Approach Semi-Automatic Approach Automatic Approach
No. of Generated Scripts Specified scenario test scripts for a given route Generated scenario scripts with good diversity Generated scenario scripts with excellent diversity
Simulation Test Modeling 10% AV Simulation Test Modeling Methodology Ad-Hoc or No Models Well-defined Simulation Test Models Automatic Test Modeling
Simulation Scenario Test Coverage 10% Test Coverage Analysis Manual (Ad-Hoc) Approach Systematic Approach Automatic Approach
Scenario Diversity 20%
5%
5%
5%
5%
Challenge Demo Evaluation Items 100% Description Fair 70% - 80% Very Good 81% - 95% Excellent 96%-100%
Simulation Test Demo 50% Successful Executed Test Scenario using Scripts Specified scenarios in a given trip route Specified scenarios for multiple trip routes Specified and self-defined scenarios for multiple trip routes
20% Test Automation Semi-automati c tests are executed Most tests are automatic executed and validated All tests are automatically executed and validated
20% Detected Problems in test simulation Found at least one Found more than 5 issues More than K issues
Project Presentation 10% Presentation Contents and delivery Skills Fair presentation Very good presentation Excellent presentation