unable to replicate ETTM1 results

[gdevos010]

Hello,

I find the results of this paper to be very impressive. I tried replicating the ETTM1 out 48 results running the command found in the readme. Do you have any suggestions on how to get the mse of 0.126 and mae of 0.229 found in the paper? My results are mse of 0.3808 and mae of0.3899.

https://github.com/cure-lab/SCINet#for-ettm1-dataset

here are my results:

Epoch: 7 cost time: 714.845440864563
--------start to validate-----------
mid --> normed mse:0.4309, mae:0.4268, rmse:0.6564, mape:1.5992, mspe:116.5968, corr:0.8446
mid --> denormed mse:7.6822, mae:1.4870, rmse:2.7717, mape:inf, mspe:inf, corr:0.8446
final --> normed mse:0.4345, mae:0.4250, rmse:0.6591, mape:1.5075, mspe:102.4135, corr:0.8480
final --> denormed mse:7.5653, mae:1.4714, rmse:2.7505, mape:inf, mspe:inf, corr:0.8480
--------start to test-----------
mid --> normed mse:0.4958, mae:0.4530, rmse:0.7041, mape:1.9340, mspe:219.9087, corr:0.6378
mid --> denormed mse:8.9583, mae:1.5916, rmse:2.9930, mape:inf, mspe:inf, corr:0.6378
final --> normed mse:0.4729, mae:0.4426, rmse:0.6877, mape:1.8262, mspe:176.8408, corr:0.6674
final --> denormed mse:9.1656, mae:1.5982, rmse:3.0275, mape:inf, mspe:inf, corr:0.6674
Epoch: 7, Steps: 2151 | Train Loss: 0.4837916 valid Loss: 0.8518541 Test Loss: 0.8955729
EarlyStopping counter: 5 out of 5
Early stopping
save model in  exp/ETT_checkpoints/SCINet_ETTm1_ftM_sl96_ll48_pl48_lr0.001_bs16_hid4.0_s2_l4_dp0.5_invFalse_itr0/ETTm148.bin
>>>>>>>testing : SCINet_ETTm1_ftM_sl96_ll48_pl48_lr0.001_bs16_hid4.0_s2_l4_dp0.5_invFalse_itr0<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
test 11473
Mid --> normed mse:0.3980, mae:0.3977, rmse:0.6309, mape:1.8941, mspe:239.4249, corr:0.6997
TTTT Final --> denormed mse:0.3808, mae:0.3899, rmse:0.6171, mape:1.8406, mspe:221.7707, corr:0.7211
Final mean normed mse:0.3808,mae:0.3899,denormed mse:6.8933,mae:1.3593

thank you for the help

[VEWOXIC]

Hi,

Can you please provide us with the version of your pytorch? Some experiments show that different pytorch versions can cause a large result gap. And can you please provide us with the full running log? It seems that your program only ran for 7 epochs which are not sufficient to make the model converge.

Thanks

[gdevos010]

Thank you for the help. I created a conda env using the requirements.txt so I am using pytorch1.8 env.yml.txt

I am not sure why it stops after 7 epochs. This log file was created using the following parameters
ettm1_48.txt

python run_ETTh.py --data ETTm1 --features M  --seq_len 96 --label_len 48 --pred_len 48 --hidden-size 4 --stacks 2 --levels 4 --lr 0.001 --batch_size 16 --dropout 0.5 --model_name ettm1_M_I96_O48_lr1e-3_bs16_dp0.5_h4_s2l4

[gdevos010]

@VEWOXIC I was wondering if you had a chance to look at the environment and the logs?

[lhyvip]

[lhyvip]

--------start to test-----------
normed mse:0.3907, mae:0.4096, rmse:0.6251, mape:7.6669, mspe:22488.8456, corr:0.7071
denormed mse:6.9905, mae:1.4484, rmse:2.6440, mape:inf, mspe:inf, corr:0.7071
Epoch: 7, Steps: 1071 | Train Loss: 0.3127497 valid Loss: 0.4202899 Test Loss: 0.4095928
EarlyStopping counter: 5 out of 5
Early stopping
save model in exp/ETT_checkpoints/SCINet_ETTh1_ftM_sl48_ll24_pl24_lr0.003_bs8_hid4_s1_l3_dp0.5_invFalse_itr0\ETTh124.bin

testing : SCINet_ETTh1_ftM_sl48_ll24_pl24_lr0.003_bs8_hid4_s1_l3_dp0.5_invFalse_itr0<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
test 2857
normed mse:0.3505, mae:0.3797, rmse:0.5920, mape:7.3974, mspe:23355.7293, corr:0.7344
TTTT denormed mse:6.4630, mae:1.3316, rmse:2.5422, mape:inf, mspe:inf, corr:0.7344
Final mean normed mse:0.3505,mae:0.3797,denormed mse:6.4630,mae:1.3316

[lhyvip]

很奇怪为什么会在epoch7就停止运行了呢？

[lhyvip]

哦哦好像是设置了一个早停止