Garbage data causes full file to fail decode

[Hydravine]

Me again!

As part of my testing I'm backfilling a lot of old telemetry files with our new extraction tool that uses CCSDSPy. On rare occasions some files completely fail at the split_by_apid step because of (what I believe to be) additional garbage data that made it into the file somehow during data dump from the spacecraft.

I made a quick and dirty alteration in the iter_packet_bytes function of utils.py adding a try/except so that the code could continue if it happens across garbage like this:

line 66 in utils.py

while offset < len(file_bytes):
        try:
            packet_nbytes = get_packet_total_bytes(
                file_bytes[offset : offset + PRIMARY_HEADER_NUM_BYTES].tobytes()
            )
            packet_bytes = file_bytes[offset + delta_idx : offset + packet_nbytes].tobytes()
    
            yield packet_bytes
    
            offset += packet_nbytes
        except:
            print(f"Error in iter_packet_bytes at offset {offset}, Primary header byte sequence of {len(file_bytes[offset : offset + 
                       PRIMARY_HEADER_NUM_BYTES].tobytes())} must be {PRIMARY_HEADER_NUM_BYTES} bytes long")
            offset += len(file_bytes[offset : offset + PRIMARY_HEADER_NUM_BYTES].tobytes())

I tested this version and it works in the sense that the CCSDSPy code can continue and decode the rest of the file instead of failing. I get a single error:

"Error in iter_packet_bytes at offset 35034720, Primary header byte sequence of 2 must be 6 bytes long"

and then the code continues on decoding the rest of the file because it skips over those 2 bytes and continues, vs the current release fails. Would it be possible for you to add something like this to an official release? I think it's very beneficial.

Thanks again for all your help,

Bill

[Hydravine]

When I say the current release fails this is what I mean:

 File "C:\Users\wparsons\.conda\envs\GMOC_SCHM\Lib\site-packages\ccsdspy\decode.py", line 39, in _get_packet_total_bytes
    raise ValueError(
ValueError: Primary header byte sequence of 2 must be 6 bytes long

Note: I added the "of 2" part for my debugging, this is from line 39/40 in decode.py

[Hydravine]

Gotcha, yeah I only quickly did the fix for my specific scenario so it definitely needs to be more flexible. I assume the use_previous will be used in the offset += line? The main functionality I think it needs is to just have some sort of try/except catch so that it can continue on with the rest of the file when it runs into an issue, instead of just failing the whole process.

[ddasilva]

Yeah, that's the idea. Basically that keyword argument would produce the same behavior as the snippet you put above.

I think this sounds good-- I can plan to write a patch in the next couple weeks.

[Hydravine]

Awesome, thanks!

[Hydravine]

Hi @ddasilva just circling back to this. We're very close to delivering our software package (that uses CCSDSPy) so I'm just curious if you've had a chance to look at implementing this into a new version release? I just need to know what version to put into my requirements.txt file essentially. Until then I'll continue using my local slightly modded v1.2.1

[ddasilva]

@Hydravine Sorry this slipped past me, I've been flying all around the country for meetings in December/January and haven't gotten the time. I can look at writing a patch this week. Can you send me a file with the garbage data and if possible expected output? That'll help me write at test I can implement around.