Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

functionality to use pretrained pie model #168

Merged
merged 3 commits into from
Jul 19, 2024
Merged

Conversation

Bhuvanesh-Verma
Copy link
Collaborator

@Bhuvanesh-Verma Bhuvanesh-Verma commented Jul 18, 2024

This PR adds functionality to use a pretrained PyTorch-IE model that allows updating weights of base transformer model. Additionally, it provides an optional parameter, pretrained_pie_model_prefix_mapping, to filter and specify which model layers should be updated. Furthermore, this adds the optional parameter pretrained_pie_model_prefix_mapping that allows to restrict the loading to a subset of parameters via their prefixes.

For instance, to load the weights from a PIE token classification model for further training on conll2003:

python src/train.py experiment=conll2003 pretrained_pie_model_path=path/to/pretrained/pie/model "+pretrained_pie_model_prefix_mapping={model.model:model.model}"

Note that we use model.model:model.model as prefix mapping which will result in only loading the weights from the base transformer model, but not the classification head. This is useful to fine-tune on data with a different set of labels.

In addition, this also adds imports for all models and task modules from the pie_modules package to the train script because these are required when loading a such a model vie pytorch_ie.AutoModel.

Copy link
Owner

@ArneBinder ArneBinder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minors, see below. But the tests are failing, this needs to be addressed.

src/train.py Outdated Show resolved Hide resolved
src/train.py Outdated Show resolved Hide resolved
src/train.py Outdated Show resolved Hide resolved
@Bhuvanesh-Verma
Copy link
Collaborator Author

Implemented the suggested changes, I think that solved the test failure issue.

@ArneBinder ArneBinder merged commit c9ac6f4 into main Jul 19, 2024
2 checks passed
@ArneBinder ArneBinder deleted the load_pretrained_pie_model branch July 19, 2024 12:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants