Requesting for the pretrained model (.pth)

[phanindrasp]

Hello,

I have been trying to replicate the results of DGCNN for the drivaernet dataset. However, as the earlier issues have mentioned, even though the loss is reducing, R2 is consistently negative/close to zero (never goes above zero). I took the exact same hyperparameters as in the drivaernet paper except for batch size, which was chosen as 8. In this regard, I request you to share a pretrained model (.pth file) which would help me validate and possibly debug my model.

Thanks,
Phani.

[phanindrasp]

Hey, actually not yet..Still waiting for the authors to respond :(

…

On Mon, Nov 18, 2024 at 6:53 PM JJ-interesting ***@***.***> wrote: Have you resolved the issue with the R2 score? If so, could you share the solution? — Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APRIBFU272HZXXGJGGFPYND2BHS4JAVCNFSM6AAAAABRKSDFSCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBTGAZTENZYGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[JJ-interesting]

Did you manage to reproduce the results from the paper? If so, could you share them? These are the results I obtained from my training. The R² score has turned positive but hasn't reached the 0.901 reported in the paper.
Total training time: 11301.56s
Model saved to models/cdPrediction DrivAerNet r2 100epochs 5k final model.pth
Testing the final model:
Test MSE: 0.000292, TeSt MAE: 0.013641, MaX MAE: 0.022632, Test R2: 0.5032
Total inference time: 1.21s for 592 samples
Testing the best model:
Test MSE: 0.000229, TeSt MAE: 0.011788, MaX MAE: 0.022385, Test R2: 0,6315
Total inference time: 1,10s for 592 samples

[phanindrasp]

I couldn't manage to bring R^2 above zero at all! What changes did you make that made your R^2 positive? Could you share a few other details from your simulation:

1. Batch size
2. Number of epochs
3. Learning rate
   Could you also share your email ID so we can discuss this further? Mine is [email protected]. Please feel free to mail.

Thanks,
Phanindra.

[LilaKen]

Did you manage to reproduce the results from the paper? If so, could you share them? These are the results I obtained from my training. The R² score has turned positive but hasn't reached the 0.901 reported in the paper. Total training time: 11301.56s Model saved to models/cdPrediction DrivAerNet r2 100epochs 5k final model.pth Testing the final model: Test MSE: 0.000292, TeSt MAE: 0.013641, MaX MAE: 0.022632, Test R2: 0.5032 Total inference time: 1.21s for 592 samples Testing the best model: Test MSE: 0.000229, TeSt MAE: 0.011788, MaX MAE: 0.022385, Test R2: 0,6315 Total inference time: 1,10s for 592 samples

As you mentioned, the R² score we calculated is quite similar to your results. We also found that it's difficult to fully reproduce the performance reported in the original paper for the DrivAerNet dataset. We believe this is mainly due to significant differences between the predicted and true values for some samples, and common metrics such as R² do not capture these discrepancies well.

We later validated our hypothesis by comparing the predicted values to the true values, and we found the relative error to be 0.0748, which is much better than the 13% reported in the original paper. In summary, the differences in the results are mainly due to the characteristics of the data and the specific metrics used.

If you have any further questions, I would be happy to discuss and help out!