Revisiting Adversarial Training at Scale

This is the official repository for our paper: "Revisiting Adversarial Training at Scale."

Abstract:

The machine learning community has witnessed a drastic change in the training pipeline, pivoted by those ``foundation models'' with unprecedented scales. However, the field of adversarial training is lagging behind, predominantly centered around small model sizes like ResNet-50, and tiny and low-resolution datasets like CIFAR-10. To bridge this transformation gap, this paper provides a modern re-examination with adversarial training, investigating its potential benefits when applied at scale. Additionally, we introduce an efficient and effective training strategy to enable adversarial training with giant models and web-scale data at an affordable computing cost. We denote this newly introduced framework as AdvXL.

Empirical results demonstrate that AdvXL establishes new state-of-the-art robust accuracy records under AutoAttack on ImageNet-1K. For example, by training on DataComp-1B dataset, our AdvXL empowers a vanilla ViT-g model to substantially surpass the previous records of $l_{\infty}$-, $l_{2}$-, and $l_{1}$-robust accuracy by margins of 11.4%, 14.2% and 12.9%, respectively. This achievement posits AdvXL as a pioneering approach, charting a new trajectory for the efficient training of robust visual representations at significantly larger scales.

Installation

Installation and preparation follow the TIMM Repo. Additionally, RobustBench is needed to evaluate model robustness. We also provide a sample conda environment yml file here, that we used to reproduce the eval results.

Usage

Testing Instructions

For robustness under PGD attack, use validate.py. For robustness under AutoAttack, use eval_autoattack.py.

We have also provided some example eval scripts under scripts/eval. Put the proper weights under ${output_dir}/${checkpoint}, and they should be able to readily reproduce the results reported in our paper.

Model Weights

Model	Dataset	Sample@Resolution	Adv. Steps	Clean	Linf	L2	L1	Weights
ViT-H/14	DataComp-1B + ImageNet-1K	5.12B@84 + 38.4M@224 + 6.4M@336	2/3	83.9	69.8	69.8	46.0	download
ViT-g/14	DataComp-1B + ImageNet-1K	5.12B@84 + 38.4M@224 + 6.4M@336	2/3	83.9	71.0	70.4	46.7	download

License

This project is under the Apache 2.0 License.

Acknowledgement

This repo is based on TIMM and AutoAttack. Many thanks to the awesome works from the open-source community!

This work is partially supported by a gift from Open Philanthropy. We thank Center for AI Safety, TPU Research Cloud (TRC) program, and Google Cloud Research Credits program for supporting our computing needs.

Citation

@inproceedings{wang2024advxl,
  title     = {Revisiting Adversarial Training at Scale},
  author    = {Wang, Zeyu and Li, Xianhang and Zhu, Hongru and Xie, Cihang},
  booktitle = {CVPR},
  year      = {2024}
}