Skip to content

Latest commit

 

History

History
115 lines (92 loc) · 4.33 KB

README.md

File metadata and controls

115 lines (92 loc) · 4.33 KB
Raw

DeepTest: Automated testing of deep-neural-network-driven autonomous cars

DeepTest is a systematic testing tool for automatically detecting erroneous behaviors of DNN-driven vehicles that can potentially lead to fatal crashes.

Install Required Packages

OS: Ubuntu 16.04
Read through and run ./install.sh

Code Directories

models/

  • Reproducing Udacity self-driving car Challenge2[1] results for Rambo, Chauffeur and Epoch models.
  • Models and weights files are required to run these scripts.
  • For Rambo model, Keras backend should be changed to theano.

testgen/

  • Generate synthetic images, calculate cumulative coverage and record predicted outputs.
  • ncoverage.py: define NCoverage class for computing neuron coverage.

guided/

  • Combine different transformations and leverage neuron coverage to guide the search.
  • Re-runing the script will continue the search instead of starting from the beginning except deleting all pkl files.

metamorphictesting/

Models and Saved Weights

  • Rambo [2]
  • Chauffeur [3]
  • Epoch [4]

Datasets

Generate hmb3 dataset from HMB_3.bag

mkdir hmb3
python generate_hmb3.py --bagfile HMB_3.bag

The following commands are to install python-rosbag api.

sudo apt-get install python-rosbag
sudo apt-get install python-genmsg
sudo apt-get install python-genpy
sudo apt-get install python-rosgraph-msgs

Reproducing experiments

Dataset directory structure:

./Dataset/
├── hmb3/
        └── hmb3_steering.csv
        └── images
└── Ch2_001/
        └── center/
                 └── images
        └── CH2_final_evaluation.csv

Evaluating models' accuracy on existing test data

cd models/
python epoch_reproduce.py --dataset DATA_PATH/Dataset/
python chauffeur_reproduce.py --dataset DATA_PATH/Dataset/
python rambo_reproduce.py --dataset DATA_PATH/Dataset/

Generate synthetic images and compute cumulative neuron coverage

cd testgen/
./epoch_testgen_driver.sh 'DATA_PATH/Dataset/'
./chauffeur_testgen_driver.sh 'DATA_PATH/Dataset/'
python rambo_testgen_coverage.py --dataset DATA_PATH/Dataset/

Combine transformations and synthesize images by guided search

cd guided/
mkdir new/
rm -rf *.pkl
python epoch_guided.py --dataset DATA_PATH/Dataset/
python chauffeur_guided.py --dataset DATA_PATH/Dataset/
python rambo_guided.py --dataset DATA_PATH/Dataset/

Detected erroneous behaviors

https://deeplearningtest.github.io/deepTest/

Citation

If you find DeepTest useful for your research, please cite the following paper:

@article{tian2017deeptest,
  title={DeepTest: Automated testing of deep-neural-network-driven autonomous cars},
  author={Tian, Yuchi and Pei, Kexin and Jana, Suman and Ray, Baishakhi},
  journal={arXiv preprint arXiv:1708.08559},
  year={2017}
}

References

  1. Udacity self driving car challenge 2.
    https://github.com/udacity/self-driving-car/tree/master/challenges/challenge-2. (2016).
  2. Rambo model.
    https://github.com/udacity/self-driving-car/tree/master/steering-models/community-models/rambo. (2016).
  3. Chauffeur model.
    https://github.com/udacity/self-driving-car/tree/master/steering-models/community-models/chauffeur. (2016).
  4. Epoch model.
    https://github.com/udacity/self-driving-car/tree/master/steering-models/community-models/cg23. (2016).