Skip to content

Latest commit

 

History

History
68 lines (53 loc) · 2.31 KB

README.md

File metadata and controls

68 lines (53 loc) · 2.31 KB
Raw

Repository for the paper "Incorporating Attribution Importance for Improving Faithfulness Metrics" to appear at ACL 2023.

We promise the repo will be tided up shortly.

Prerequisites

Install necessary packages by using the files conda_reqs.txt and pip_reqs.txt

conda create --name flexi --file  conda_reqs.txt
conda activate flexi
pip install -r pip_reqs.txt
python -m spacy download en_core_web_sm

Downloading Task Data

You can run the jupyter notebooks found under tasks/task_name/*ipynb to generate a filtered, processed csv file and a pickle file used for trainining the models.

Training the models

dataset="evinf"
data_dir="datasets/"
model_dir="trained_models/"

for seed in 5 10 15 
do
python finetune_on_ful.py --dataset $dataset \
                          --model_dir $model_dir \
                          --data_dir $data_dir \
                          --seed $seed 
done
python finetune_on_ful.py --dataset $dataset \
                          --model_dir $model_dir \
                          --data_dir $data_dir \
                          --seed $seed  \
                          --evaluate_models

Extracting Rationales (Fixed or Instance-Specific - For all parameters)

divergence="jsd"
extracted_rationale_dir="extracted_rationales/"

python extract_rationales.py --dataset $dataset  \
                             --model_dir $model_dir \
                             --data_dir $data_dir \
                             --extracted_rationale_dir $extracted_rationale_dir \
                             --extract_double \
                             --divergence $divergence

--extract_double is optional and is used to double $N$.

Evaluating Soft Faithfulness

extracted_rationale_dir="extracted_rationales/"
evaluation_dir="faithfulness_metrics/"

python evaluate_posthoc_zeroout.py --dataset $dataset \
                          --model_dir $model_dir \
                          --extracted_rationale_dir $extracted_rationale_dir \
                          --data_dir $data_dir \
                          --evaluation_dir $evaluation_dir\
                          --thresholder $thresh