Different results with the "Repeat" method

[etidav]

Hi, I tried to reproduce the result of the Repeat method and I did not find the same result as those displayed in your article for the ILI dataset. Your code seems to provide the same result as mine, is that an error in the paper?

Repeat:
illness
horizon 24 {'mae': 1.621, 'mse': 6.204}
horizon 36 {'mae': 1.904, 'mse': 7.702}
horizon 48 {'mae': 1.951, 'mse': 7.84}
horizon 60 {'mae': 1.787, 'mse': 6.875}

[AlexMV12]

@etidav Hi! Sorry to bother you, but I'm trying to reproduce the results in this paper, but I have some problems to even obtain the results you provided with the Repeat model.

Can I ask, pricesely, how you use the model? There are more ways to compute forecasts, and I cannot understand which one is used in this paper.

For instance, I can think of these two methods:

1. start from the first point in the test dataset, use the last known point to predict the next 24, and move ahead of 24:
   

2. start from the first point in the test dataset - 24, and predict the point at 24 steps ahead; then, move ahead of 1, and repeat:
   

Can I ask which method did you use, and how did you obtain your results? Thanks!

[etidav]

@AlexMV12 Hi, If I'm right, the Naive method used in this paper is the method 1) of your comment. (repeat the last known point x times).
However, you do not have to move ahead of x, just move ahead of 1.
In the case where horizon=24:

• Start at the beginning of the test set and compute the naive prediction (repeat 24 times the last accessible value)
• Compute the MAE/MSE of the prediction compared to the ground truth (the 24 first points of the test set)
• move ahead of 1
• repeat the process until you reach the end of the test set.

If you want, here is the function that I used for one of my paper to reproduce the naive results (except on the ILI dataset ^^')

[AlexMV12]

Hello @etidav ,

First, sorry for the late response and thanks for your answer! I found your code usefully and, basically, I am able to reproduce the same results as yours using Darts.

Let me ask a question: I found that in your code, for the ETTx datasets you do this:

dataset = dataset.loc[:"2018-02-20"]

Doing so, the results are the same reported in this paper. But, can I ask you where you found this particular timestamp to cut the time-series? Did the authors of this paper say this somewhere, or in the Autoformer paper?
Thanks a lot and good luck at ICLR!