Split by APID runtime

[WilliamParsonsV]

I'm wondering what the rough/standard time for the split_by_apid function is? I have a full decode process working when I manually enter the fixedlength pktObjects for one packet and it runs fast because split_by_apid is only looking for one packet. However, I now created a script to generate pktObject fixedlength entries for each packet using a .JSON input of our operational database. So now I am testing on an actual (12MB .PKT backorbit telemetry file) and splitting by all apids present in the file and the split_by_apid step takes 20+ minutes now, is that expected?

[ddasilva]

Thanks for reaching out. I don't think this should be happening. Do you think you could email me a google drive link to the data, so I can look into this for you? Is it all 12 MB files, or just a few?

[WilliamParsonsV]

That would be great! I can send it you first thing tomorrow when I'm in office. It's a single 12MB backorbit file. I tested with only defining one fixedLength entry packet and its quite fast, but then when I try to run with my full list of packets its very very slow

[ddasilva]

Great! In the worst case, I can rewrite this function using a C extension and it should be very fast

[WilliamParsonsV]

Sent a .zip via email!

[ddasilva]

We spoke over email.We found that the slowdown doesn't occur when the command line interface was used that does the split and writes output files. The command for this is:
$ python -m ccsdspy split file_name.tlm

@WilliamParsonsV decided to go with this solution and move forward with their project. They sent me their code/data and I wasn't able to reproduce the slowdown.

If anyone else encounters this problem in the future, feel free to reopen this thread.