dirac-dms-remove-catalog-replicas doesn't work?

[zhangxiaomei]

One of our site has disk corruption. I tried to remove the replicas of their SE from DFC and got the following errors:

dirac-dms-remove-catalog-replicas -ddd ./t_production JINR-EOS
2023-09-29 00:06:02 UTC dirac-dms-remove-catalog-replicas/DataManager/removeReplicaFromCatalog [139809215772480] DEBUG: Completely failed to get replicas for lfns. FileCatalog error ( 1604 : Failed to perform getReplicas from any catalog)
FileCatalog error ( 1604 : Failed to perform getReplicas from any catalog)

I used dirac_admin with "FileCatalogManagement" property to run it. The "t_prodution" contains a list of files located in JINR_EOS.

[fstagni]

Do you have possibility to look for an error printed server side?

Which DIRAC version are you using?

[zhangxiaomei]

I found the reason. If the number of files exceeds 1 million or so in the list, the problem would happen. It works fine with 10 thousand files when I split the file lists into small ones. It looks like the command can't support removing replicas beyond 1 million files.

[zhangxiaomei]

But I found another problem. I finished the cleaning and found no files with the command (dirac-dms-find-lfns -ddd --SE JINR-EOS Path=/juno/production vo=juno). But when I use "size -l /juno/production", I can still see the replicas exists in JINR_EOS:
FC:/juno/production>size -l /juno/production
directory: /juno/production
Logical Size: 711,830,180,716,466 Files: 12182671 Directories: 4179

StorageElement Size Replicas

1 IHEP-STORM 96,822,387,786,900 1054954
2 CNAF-STORM 211,000,710,105,332 2039280
3 IN2P3-DCACHE 74,010,986,797,466 1636986
4 JINR-EOS 36,070,974,891,660 248953
5 IHEP-JUNOEOS 472,528,225,183,696 11046428
6 CNAF-HTTPSTORM 541,634,931,701 29042
Total 890,974,919,696,755 16055643

It looks like the command "size" doesn't give correct number. I have no idea what happened.

[andresailer]

Maybe this #2382 happened.

That ticket also mentions that you can "rebuild" the catalog size information in the catalog CLI.

[zhangxiaomei]

"rebuild" didn't work. After rebuilding, "size -l" failed.
FC:/> rebuild
Error: Error in _read: timed out SSLTimeoutError('timed out')
FC:/> cd /juno/production
FC:/juno/production>size -l
directory: /juno/production
Logical Size: 0 Files: 0 Directories: 0
Size failed: 'TotalSize'

[zhangxiaomei]

It seems "rebuild" continues to run in the background after the command finished. After a few hours, I tried again and "size -l" is working.
FC:/juno/production>size -l
directory: /juno/production
Logical Size: 662,923,692,122,459 Files: 11845125 Directories: 4179

StorageElement Size Replicas
1 IHEP-STORM 96,763,764,248,387 1054556
2 CNAF-STORM 205,132,074,696,720 1998821
3 IN2P3-DCACHE 67,102,684,531,503 1589248
4 IHEP-JUNOEOS 423,771,484,004,803 10709945
5 CNAF-HTTPSTORM 541,634,931,701 29042
Total 793,311,642,413,114 15381612