Why are the test results *always* better?

[deklanw]

I see on the docs site,

Q2: Why are the test results usually better than the best valid results?

A2: For more rigorous evaluation, those user-item interaction records in validation sets will not be ranked while testing. Thus the distribution of validation & test sets may be inconsistent.
However, this doesn’t affect the comparison between models.

I don't understand this explanation. Since we're using the validation set to determine early stopping, shouldn't the results on the validation set be, on average, slightly better than the test results?

In my observations I don't think I've ever seen the validation results be better. The test results appear to be systematically better.

[hyp1231]

The key reason lies in user-item interaction records in validation sets will not be ranked while testing.

For example, for one certain user, the model may rank the items as following:

a1, b1,  ..., a5, b5, ...

, where a* represents validation ground truth items, and b* represents test ground truth items. (For a well-trained model, it will rank both a* and b* higher)

As we'll not rank validation ground truth items when calculating test results, the top-5 rank list for validation and test are as follows.

Validation: a1, b1, a2, b2, a3
Test: b1, b2, b3, b4, b5

(The items in bold are ground truth items)
Thus in RecBole, the test results appear to be systematically better.

[deklanw]

I see. Why aren't the test set items similarly excluded from the validation ranking?