Skip to content

Latest commit

 

History

History
713 lines (614 loc) · 24 KB

Action-spotting-usage.md

File metadata and controls

713 lines (614 loc) · 24 KB
Raw

After you have finished the basic setup steps, you can train and test action spotting models using the commands below. In particular, you must have created a particular set of folders and set the environment variables that point to them, as explained within the README.

The commands below are meant to give you a general idea of how to use the package. If you are specifically interested in reproducing results from our experiments, please see Reproducing-results-from-the-SoccerNet-action-spotting-challenge-2022.md. Those instructions also explain how to download and use our pretrained models.

Training models

Currently, we prefer to use a two-step approach to action spotting, which consists of training two models. The first one predicts confidence scores, and the second one predicts temporal displacements.

Train a model that predicts confidence scores using the ResNet_TF2_PCA512 features:

CONFIDENCE_MODEL_NAME="RESNET_PCA_CONFIDENCE_LR1e-3_DWD2e-4"
./bin/train.py \
  -dt soccernet_v2 \
  -cd ./configs/soccernet_confidence/ \
  -ld ./data/labels/ \
  -fd ./data/features/resnet/ \
  -sd ./data/splits/ \
  -fn ResNET_TF2_PCA512 \
  -dc dense \
  -cw 1.0 \
  -lr 1e-3 \
  -dwd 2e-4 \
  -m $MODELS_DIR/$CONFIDENCE_MODEL_NAME

To monitor the training progress, start Tensorboard, pointing to the MODELS_DIR folder:

tensorboard --logdir="$MODELS_DIR"

The training script will save the best model as it runs, based on the average-mAP metric. If you notice that the model is overfitting, you can kill the run before it finishes and then use the best saved model.

After training the above model, you will be able to train a temporal displacement regression model. In order to do that, first run inference on the validation set using the confidence model. This will produce confidence scores that will be used when training the temporal displacement model, inside the validation step.

./bin/test.py \
  -dt soccernet_v2 \
  -cd ./configs/soccernet_confidence/ \
  -ld ./data/labels/ \
  -fd ./data/features/resnet/ \
  -sd ./data/splits/ \
  -fn ResNET_TF2_PCA512 \
  -dc dense \
  -cw 1.0 \
  --test_split validation \
  --evaluate 0 \
  -m  $MODELS_DIR/$CONFIDENCE_MODEL_NAME/best_model \
  -rd $RESULTS_DIR/$CONFIDENCE_MODEL_NAME

We can now train the temporal displacement regression model, using the above results in the validation step. Note that the training step below uses a different configuration folder: ./configs/soccernet_delta/, and a different detector setting: dense_delta. Within our code, delta refers to the temporal displacements, which were denoted as d in our paper.

DELTA_MODEL_NAME="RESNET_PCA_DELTA_LR2e-3_DWD5e-4"
./bin/train.py \
  -dt soccernet_v2 \
  -cd ./configs/soccernet_delta/ \
  -ld ./data/labels/ \
  -fd ./data/features/resnet/ \
  -sd ./data/splits/ \
  -fn ResNET_TF2_PCA512 \
  -dc dense_delta \
  -dw 1.0 \
  -lr 2e-3 \
  -dwd 5e-4 \
  -rd $RESULTS_DIR/$CONFIDENCE_MODEL_NAME \
  -m  $MODELS_DIR/$DELTA_MODEL_NAME

Testing models

Inference is currently done in two steps, each using its respective model: first the confidence scores are predicted, and then the temporal displacements. The results are then saved and evaluated according to different metrics.

./bin/test.py \
  -dt soccernet_v2 \
  -cd ./configs/soccernet_confidence/ \
  -ld ./data/labels/ \
  -fd ./data/features/resnet/ \
  -sd ./data/splits/ \
  -fn ResNET_TF2_PCA512 \
  -dc dense \
  -cw 1.0 \
  --throw_out_delta 1 \
  --evaluate 0 \
  -m  $MODELS_DIR/$CONFIDENCE_MODEL_NAME/best_model \
  -rd $RESULTS_DIR/$DELTA_MODEL_NAME
./bin/test.py \
  -dt soccernet_v2 \
  -cd ./configs/soccernet_delta/ \
  -ld ./data/labels/ \
  -fd ./data/features/resnet/ \
  -sd ./data/splits/ \
  -fn ResNET_TF2_PCA512 \
  -dc dense_delta \
  -dw 1.0 \
  --throw_out_delta 0 \
  --evaluate 1 \
  -m  $MODELS_DIR/$DELTA_MODEL_NAME/best_model \
  -rd $RESULTS_DIR/$DELTA_MODEL_NAME

We could also just evaluate the confidence scores model directly, as below. In this case, we will not benefit from the predicted temporal displacements, so we should expect worse metrics at low evaluation tolerances. In most cases there is a large improvement in the tight average-mAP when adding the temporal displacements, whereas the loose average-mAP usually does not change much.

./bin/test.py \
  -dt soccernet_v2 \
  -cd ./configs/soccernet_confidence/ \
  -ld ./data/labels/ \
  -fd ./data/features/resnet/ \
  -sd ./data/splits/ \
  -fn ResNET_TF2_PCA512 \
  -dc dense \
  -cw 1.0 \
  --throw_out_delta 1 \
  --evaluate 1 \
  -m  $MODELS_DIR/$CONFIDENCE_MODEL_NAME/best_model \
  -rd $RESULTS_DIR/$CONFIDENCE_MODEL_NAME

Creating reports with metrics across different methods

After we run evaluation with ./bin/test.py, a pickle file is created, containing a bunch of metrics. We can then read that pickle file and create a report that summarizes those metrics. The create_comparison_report.py script can create a report containing the metrics from one or more methods, as long as each method has a respective pickle file. Here is an example of how to create a report from the results of a single method. It assumes you have already ran the previous steps that create the results.

./bin/create_comparison_report.py \
  --pickles $RESULTS_DIR/$DELTA_MODEL_NAME/Evaluation/evaluation_run.pkl \
  --names $DELTA_MODEL_NAME \
  --out_dir $REPORTS_DIR/$DELTA_MODEL_NAME

Here is a run that compares the results with and without the temporal displacements .

./bin/create_comparison_report.py \
  --pickles \
    $RESULTS_DIR/$DELTA_MODEL_NAME/Evaluation/evaluation_run.pkl \
    $RESULTS_DIR/$CONFIDENCE_MODEL_NAME/Evaluation/evaluation_run.pkl \
  --names with_delta without_delta \
  --out_dir $REPORTS_DIR/add_delta/

Comparing against baseline methods

We can also create reports comparing against other published methods, as long as they can generate results in standard SoccerNet-v2 JSON format. To do so, we'll need to run our evaluation code on top of the JSON files, in order to generate the required pickle files. Below are some steps you can follow in order to try it out.

In order to generate JSON result files for baseline methods, consult the documentation for each specific method. For convenience, here are some commands that run prediction using the NetVLAD++ and CALF models, creating the desired output JSON files.

DATA_DIR="$BASE_CODE_DIR/spivak/data/"  # This should point to wherever you have your data.
cd $DATA_DIR
cp -r features/resnet/ features/merged/
# The rsync below does not access the network, it just merges the contents of the two directories.
rsync -r labels/ features/merged/
# Run testing for two methods.
cd $BASE_CODE_DIR
git clone https://github.com/SoccerNet/sn-spotting
pip install SoccerNet
# First, run the temporally aware pooling code (NetVLAD++)
cd $BASE_CODE_DIR/sn-spotting/Benchmarks/TemporallyAwarePooling/
# This will take a while as it will train a new model that uses the ResNET_TF2_PCA512 features.
# (It took around 40 minutes on a V100 for us.) After training, the script runs testing and
# saves the results in the models/NetVLAD++_PCA512/outputs_test/ folder.
python src/main.py \
  --SoccerNet_path=$DATA_DIR/merged/ \
  --features=ResNET_TF2_PCA512.npy \
  --model_name=NetVLAD++_PCA512
# Second, run the context-aware loss function code (CALF)
cd $BASE_CODE_DIR/sn-spotting/Benchmarks/CALF/
# This will use a pre-trained model and save results in the outputs/ folder.
python src/main.py \
  --SoccerNet_path=$DATA_DIR/merged/ \
  --features=ResNET_TF2_PCA512.npy \
  --num_features=512 \
  --model_name=CALF_benchmark \
  --test_only

As mentioned above, the necessary pickle files are generated when running bin/test.py. In order to generate similar pickle files for the other methods, we first generate their outputs in the standard SoccerNet-v2 JSON format as above. Then, we use those JSON files to generate the evaluation pickles. Here are the examples of how to generate the evaluation pickle files directly from the JSON files.

./bin/evaluate_spotting_jsons.py \
  --results_dir $BASE_CODE_DIR/sn-spotting/Benchmarks/TemporallyAwarePooling/models/NetVLAD++_PCA512/outputs_test/ \
  --features_dir ./data/features/resnet/ \
  --labels_dir ./data/labels/ \
  --splits_dir ./data/splits/ \
  --config_dir ./configs/soccernet_confidence/ \
  --output_dir $RESULTS_DIR/NetVLAD++_PCA512/
./bin/evaluate_spotting_jsons.py \
  --results_dir $BASE_CODE_DIR/sn-spotting/Benchmarks/CALF/outputs/ \
  --features_dir ./data/features/resnet/ \
  --labels_dir ./data/labels/ \
  --splits_dir ./data/splits/ \
  --config_dir ./configs/soccernet_confidence/ \
  --output_dir $RESULTS_DIR/CALF_PCA512/

The commands above assume that the JSON files are available inside the --results_dir folders.

Finally, we can generate a report comparing all the above methods.

./bin/create_comparison_report.py \
  --pickles \
    $RESULTS_DIR/CALF_PCA512/Evaluation/evaluation_aggregate.pkl \
    $RESULTS_DIR/NetVLAD++_PCA512/Evaluation/evaluation_aggregate.pkl \
    $RESULTS_DIR/$DELTA_MODEL_NAME/Evaluation/evaluation_run.pkl \
    $RESULTS_DIR/$CONFIDENCE_MODEL_NAME/Evaluation/evaluation_run.pkl \
  --names  CALF  NetVLAD++  DU_With_Delta  DU_Without_Delta \
  --out_dir $REPORTS_DIR/multiple

Visualizing results for each video

We can also create visualizations of the specific predictions for each video. Here is an example that creates visualizations for a single video.

GAME_DIR="italy_serie-a/2014-2015/2015-04-29 - 21-45 Juventus 3 - 2 Fiorentina/"
./bin/create_visualizations.py \
  --input_dir "./data/videos_224p/$GAME_DIR" \
  --output_dir "$VISUALIZATIONS_DIR/$DELTA_MODEL_NAME/$GAME_DIR" \
  --results_dir "$RESULTS_DIR/$DELTA_MODEL_NAME/$GAME_DIR" \
  --label_map ./configs/soccernet_confidence/spotting_labels.csv \
  --no_create_videos

In order to create the visualizations for all the videos in the test set, you can use the command below. In order to speed it up, we use --no_debug_graphs below, though the resulting visualizations will be less detailed.

./bin/create_visualizations.py \
  --input_dir ./data/videos_224p/ \
  --output_dir $VISUALIZATIONS_DIR/$DELTA_MODEL_NAME/ \
  --results_dir $RESULTS_DIR/$DELTA_MODEL_NAME/ \
  --label_map ./configs/soccernet_confidence/spotting_labels.csv \
  --no_create_videos \
  --no_debug_graphs