Allennlp is hard to reproduce?

[Boltzmachine]

I use allennlp to train a model and I found the metrics differ by up to 4 percent between two runs with seed unchanged. I think it is a general issue in allennlp. How to avoid it?

[epwalsh]

Hard to say without knowing what you're training setup looks like.

The first I would do is make sure my data is ordered in the exact same way across different runs with the same seed. To do that, you could add an ID field to your data instances, and write out the ID during training so you can check the order afterwards.

If it turns out that your data is ordered in the exact same way for every run given the same seed, the next thing I would check is weight initialization. Save your model's state dict as soon as it is initialized to make sure you're starting off with the same weights for each run.

[Boltzmachine]

Thanks. I think allennlp's prepare_environment will ensure that, but I will try what you said.

[Boltzmachine]

Hard to say without knowing what you're training setup looks like.

The first I would do is make sure my data is ordered in the exact same way across different runs with the same seed. To do that, you could add an ID field to your data instances, and write out the ID during training so you can check the order afterwards.

If it turns out that your data is ordered in the exact same way for every run given the same seed, the next thing I would check is weight initialization. Save your model's state dict as soon as it is initialized to make sure you're starting off with the same weights for each run.

Sorry to bother you again. I found when I use CPU I can get exactly identical results between two runs. But when I use GPU, I cannot.
I also add torch.backends.cudnn.benchmark = False and torch.backends.cudnn.deterministic = True in allennlp's prepare_environments function but it still fail to get the same result. Do you know why?

My allennlp version is 0.8.4.