Question about cdf-xy.pt

[jiazhi5]

Hello,

Thank you for this nice work. I have a few question. What is the cdf-xy.pt for? I am wondering if I can directly use the output from dreamsim_l2.measure(images) as w1kp score. Thank you.

[daemon]

Thanks for your interest. It is strongly advised to use cdf-xy.pt to normalize the scores, but it isn't strictly necessary.

[jiazhi5]

Thank you for your explanation. I am curious about how those values are derived. Additionally, to obtain a reasonable k-expected maximum, how many images (N) should be used? For instance, when k = 5 and N = 5, the score appears quite noisy. I remember that in Figure 5 of the paper, the k-expected maximum was calculated for k ranging from 2 to 300 with N = 300. Is that correct? If so, is there any guideline or relationship suggesting that N should be, for example, at least ten times k to achieve a reasonable score?

[daemon]

These were derived as the ECDF of a bunch of real values on DiffusionDB prompts, e.g., we generated many image sets and sorted them.

to obtain a reasonable k-expected maximum, how many images (N) should be used?

Good question, if N=5 and k=5, then it is effectively taking the max pairwise score, which is technically unbiased but as you've noticed very high variance. The underlying theory is subsampling bootstrap, so it would be something along the lines of small-ish k and a much larger N. In the paper, we fixed N to 300 and varied k to compute that estimate. For 5, it's probably okay for N to be 50-100 (ten to twenty times).

[jiazhi5]

Thank you so much for the explanation!