Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pem loss nan? #23

Open
leemengxing opened this issue Mar 10, 2020 · 8 comments
Open

pem loss nan? #23

leemengxing opened this issue Mar 10, 2020 · 8 comments

Comments

@leemengxing
Copy link

thks for your work. but I found that sometimes the batchsize is 1, pem = loss occurs. why?

@leemengxing
Copy link
Author

There is a statistical ratio in the loss function. It may happen that the number of iou> 0.7 or iou> 0.9 is 0 for small batch sizes, such as 1, which leads to nan. Is this correct?

@vhvkhoa
Copy link

vhvkhoa commented Mar 12, 2020

I have also met this problem and have to replace the loss by Focal Loss, I am still on the experiment to make sure it works

@leemengxing
Copy link
Author

Have you tried changing the two classifications to multiple classifications for detection tasks instead of proposal tasks? @vhvkhoa

@JJBOY
Copy link
Owner

JJBOY commented Jul 1, 2020

There is a statistical ratio in the loss function. It may happen that the number of iou> 0.7 or iou> 0.9 is 0 for small batch sizes, such as 1, which leads to nan. Is this correct?

Yes, you are right.

@lyx190
Copy link

lyx190 commented Jul 20, 2020

I have met this problem when batchsize was set to 16, anyone of you have some ideas to deal with it?

@longchao1
Copy link

I have met this problem when batchsize was set to 4, anyone of you have some ideas to deal with it?

@JJBOY
Copy link
Owner

JJBOY commented Jul 20, 2020

@lyx190 @longchao1
The reason why NAN loss is raised is explained by @leemengxing. There are two simple ways to solve this problem.
1、Set larger batch size
2、When the batch does not has any positive sample because of the threshold in loss function, just jump this batch

@lyx190
Copy link

lyx190 commented Jul 20, 2020

@lyx190 @longchao1
The reason why NAN loss is raised is explained by @leemengxing. There are two simple ways to solve this problem.
1、Set larger batch size
2、When the batch does not has any positive sample because of the threshold in loss function, just jump this batch

Thanks a lot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants