Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NaNs during NLE training #1347

Open
janfb opened this issue Jan 2, 2025 · 0 comments
Open

NaNs during NLE training #1347

janfb opened this issue Jan 2, 2025 · 0 comments
Labels
documentation Improvements or additions to documentation hackathon

Comments

@janfb
Copy link
Contributor

janfb commented Jan 2, 2025

I noticed that NaNs occuring during NLE training, e.g., during evaluation of the loss, are not detected or reported: the training finishes and just returns NaNs all the way. For NPE, we have checks on the loss, e.g., here:

https://github.com/sbi-dev/sbi/blob/9152e933871eee48a40be6f67b9f802a37c41b69/sbi/inference/trainers/npe/npe_c.py#L357C56-L358C1

and we should have a similar check in NLE as well. Or start a discussion whether to move such checks to the ConditionalDensityEstimator level.

@janfb janfb added documentation Improvements or additions to documentation hackathon labels Jan 2, 2025
@janfb janfb added this to the curate issues for hackathon 2025 milestone Jan 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation hackathon
Projects
None yet
Development

No branches or pull requests

1 participant