Buffered fields in packets

[tloubrieu-jpl]

We have packets with this structure:

	field_A	complete	data	field_C
1	1	yes	ertergdfg	3
2	2	no	aaaaaa	4
3	4	no	bbbbb	4
4	5	yes	cccccc	2

The complete field tells if the data field is complete yet or not.

So if complete==no that means the data value to be considered needs to be concatenated with the next packet data field value, as long as the complete value is still no.

So in this case, the data value we want to packet 2 is aaaaaabbbbbcccccc.

It is important to get the values from multiple packets together as specified here because we want to apply a decompression algorithm on top of this value (we were thinking of the CCSDSpy converters) and the algorithm only works on the full data stream, in this example the value aaaaaabbbbbcccccc.

In our case, the values for field_A and field_C of packets 3 or 4 do not matter and can be ignored, but I don't know if this would be true in similar use cases.

Do you know how this could be implemented in CCSDSpy ? Maybe the converter could be an object instance which is persisted for the full CCSDS packet streams ? So that we can buffer the data field value until the field is complete and decompression can be applied to it ? From the documentation there https://docs.ccsdspy.org/en/latest/user-guide/converters.html is sounds to be the case.

[ddasilva]

Hi @tloubrieu-jpl,

You can write a converter that loops through the array and sets the accumulated element on the complete==1 index. When you use the array, just skip over the elements where complete==0 (or just check the element is not None if you use the code I wrote below). Right now I believe this is necessary because converters must retain the same number of elements along the first dimension.

So your solution would be something like this:

from ccsdspy import FixedLength, converters

class AccumulatedBufferConverter(converters.Converter):
    def __init__(self):
        pass
    def convert(self, complete_array, data_array):
        converted = np.zeros(data_array.shape, dtype=object)
        converted[:] = None
        buff = b''

        for i in range(data_array.shape[0]):
            buff += data_array[i].tobytes()

            if completed_array[i]:
                converted[i] = buff
                buff = b''

       return converted

...

pkt.add_converted_field(
    ("complete", "data"),
    "accumulated_data",
    AccumulatedBufferConverter()
)

[tloubrieu-jpl]

Thanks very much @ddasilva , and then I think we will implement our decompression in the same converter (anytime the complete field values is yes) or do you think we can implement the decompression in a different converter which would get accumulated data as input ?
Anyway, no worries, the first option, having the decompression in the AccumulatedBufferConverter will work.

[tloubrieu-jpl]

And sorry, I missed that you had the discussions for that type of exchange, that is perfect.

[ddasilva]

I recommend implementing it as one converter. Having converters accept input from other converters would require a whole DAG model with dependency resolution and that wasn't thought of!

[tloubrieu-jpl]

Good, thanks Daniel !

[ddasilva]

If you can @tloubrieu-jpl , mark the reply as an answer so the thread shows up as "answered" in the discussion list for future users.