No valid query error after 5 epochs

[edrdos101]

Training a PAP_S_PS model on market1501 data I encounter the following error:

raise RuntimeError("No valid query")
RuntimeError: No valid query 

This happens at the end of the 5th epoch where the training goes into evaluation. I wonder what could be the issue? Have you encountered it? It looks like in metric.py num_valid_queries = 0 but I have not idea why or how

I should also say that I was trying to train it on market1501 data after applying SPGAN but I was very careful with file names so I'm pretty sure they are consistent.

Below is the full log

=> Eval Statistics: dic.keys(): dict_keys(['q_feat', 'q_label', 'q_cam', 'g_feat', 'g_label', 'g_cam', 'q_visible', 'g_visible']) dic['q_feat'].shape: (3368, 2304) dic['q_label'].shape: (3368,) dic['q_cam'].shape: (3368,) dic['g_feat'].shape: (9997, 2304) dic['g_label'].shape: (9997,) dic['g_cam'].shape: (9997,) /home/ed/EANet/package/eval/metric.py:179: RuntimeWarning: invalid value encountered in double_scalars return float(np.sum(aps)) / np.sum(is_valid_query) Traceback (most recent call last): File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/ed/EANet/package/optim/eanet_trainer.py", line 134, in <module> trainer.train_phases() File "/home/ed/EANet/package/optim/eanet_trainer.py", line 125, in train_phases self.train() File "/home/ed/EANet/package/optim/reid_trainer.py", line 343, in train score_str = self.may_test() File "/home/ed/EANet/package/optim/reid_trainer.py", line 329, in may_test score_str = self.test() File "/home/ed/EANet/package/optim/reid_trainer.py", line 184, in test score_dict = eval_dataloader(self.model_for_eval, loader_dict['query'], loader_dict['gallery'], deepcopy(cfg.eval)) File "/home/ed/EANet/package/eval/eval_dataloader.py", line 33, in eval_dataloader return eval_feat(dic, cfg) File "/home/ed/EANet/package/eval/eval_feat.py", line 47, in eval_feat **input_kwargs File "/home/ed/EANet/package/eval/metric.py", line 100, in cmc raise RuntimeError("No valid query") RuntimeError: No valid query