Add command line option to randomize test execution order

[vkushwaha-nv]

While working on an extension implementation, I noticed that ordering of tests help isolate some bugs related to driver incorrectly saving some state. I did not see an option to randomize test order from a case list. Would that be something that can be added?

[AlexGalazin-IMG]

why is sort -r not sufficient for this?

sort -r mustpass.txt > mustpass-randomized.txt
--deqp-caselist-file=mustpass-randomized.txt

[vkushwaha-nv]

--deqp-caselist-file does not honor the order in the file. I have tried finding how it orders them by just eyeballing the runs and it's not sorted by test name.

[AlexGalazin-IMG]

What makes you think that the order in the file is not respected?

[vkushwaha-nv]

compute.txt:

dEQP-VK.compute.pipeline.basic.branch_past_barrier
dEQP-VK.compute.pipeline.basic.concurrent_compute
dEQP-VK.compute.pipeline.basic.copy_image_to_ssbo_large

deqp-vk-64.exe --deqp-caselist-file=compute.txt
Writing test log into TestResults.qpa
dEQP Core vulkan-cts-1.3.9.0-67-gea4b0dc51098a2580708d39c571decd5c5980db1 (0xea4b0dc5) starting..
target implementation = 'Default'

Test case 'dEQP-VK.compute.pipeline.basic.concurrent_compute'..
Pass (Test passed)

Test case 'dEQP-VK.compute.pipeline.basic.copy_image_to_ssbo_large'..
Pass (Compute succeeded)

Test case 'dEQP-VK.compute.pipeline.basic.branch_past_barrier'..
Pass (Pass)

DONE!

Test run totals:
Passed: 3/3 (100.0%)
Failed: 0/3 (0.0%)
Not supported: 0/3 (0.0%)
Warnings: 0/3 (0.0%)
Waived: 0/3 (0.0%)

[vkushwaha-nv]

compute.txt

dEQP-VK.compute.pipeline.basic.copy_image_to_ssbo_large
dEQP-VK.compute.pipeline.basic.branch_past_barrier
dEQP-VK.compute.pipeline.basic.concurrent_compute

deqp-vk-64.exe --deqp-caselist-file=compute.txt
Writing test log into TestResults.qpa
dEQP Core vulkan-cts-1.3.9.0-67-gea4b0dc51098a2580708d39c571decd5c5980db1 (0xea4b0dc5) starting..
target implementation = 'Default'

Test case 'dEQP-VK.compute.pipeline.basic.concurrent_compute'..
Pass (Test passed)

Test case 'dEQP-VK.compute.pipeline.basic.copy_image_to_ssbo_large'..
Pass (Compute succeeded)

Test case 'dEQP-VK.compute.pipeline.basic.branch_past_barrier'..
Pass (Pass)

DONE!

Test run totals:
Passed: 3/3 (100.0%)
Failed: 0/3 (0.0%)
Not supported: 0/3 (0.0%)
Warnings: 0/3 (0.0%)
Waived: 0/3 (0.0%)

[vkushwaha-nv]

Ping.