How can I stream a very large file (maybe using multiple flatbuffers?)

[ghost]

Hey, I am new to data streaming, and I stumbled upon FlatBuffers, however, due to the nature of Flatbuffers streaming does not seem to be possible out-of-the-box. In Flatsharp, would it be a good idea to serialize multiple Flatbuffers into a file in chunks. I am basically storing frames that can be played back in the Unity application.

[ghost]

I assume this would require length-prefixed buffer also?

[jamescourtney]

Proper streaming is a little tricky in FlatBuffers because they are read and written nonlinearly, so there are lots of jumps back and forth when writing. Some sort of contiguous segment of memory is needed (mmap'ed files, arrays, etc). FlatBuffers also has some 32 bit size limitations due to int32 offsets.

If you have a good way of chunking your data into multiple smaller Flatbuffers, then this can be a very effective option. I can think of a few approaches:

• Maintain a "main" flatbuffer that just stores a map of offset -> (index, length) tuples
• Store your "chunks" in smaller arrays that are parts of files, etc.

Length-prefixed buffers can also work if you want a way of iterating left-to-right. Keep in mind that FlatSharp does not add this length for you, so you'd need to do something along the lines of:

int maxSize = YourType.Serializer.GetMaxSize(yourObject);
byte[] buffer = new byte[sizeof(int) + maxSize]; // allow extra space for the prefix
int bytesWritten = YourType.Serializer.Write(buffer.AsSpan().Slice(sizeof(int), yourObject);
BinaryPrimitives.WrintInt32LittleEndian(buffer.AsSpan(), bytesWritten);

Writing this out is making me think that FlatSharp might want to provide LengthPrefixed parse/serialize methods as well.

One thing to note here is that FlatBuffers is an alignment-sensitive format. This is largely because it was designed back when some ARM CPUs didn't support loads of unaligned data. Since this is not really a modern concern, FlatSharp simply assumes that every buffer you hand it to serialize is 8-byte aligned. If you add 4 bytes for the prefix you will break that. However, FlatSharp does not validate alignment when parsing (and modern CPUs do not care about unaligned data much anyway). If you do want to preserve the alignment guarantees, then consider using 8 bytes of padding instead of 4.

[ghost]

I see, that makes a lot of sense. A few follow-ups questions and thoughts. First of all, I am not experienced at all with data serialization, but I have to deal with it since we don't have an engineer on the team.

For some context, I am creating a timeline that needs to stream in chunks, allowing you to scrub through the data. We are dealing with a lot of data, which is why I am looking into streaming.

(1) I guess length-prefixing wouldn't work for the timeline, as it only allows me to read from left-to-right. I also need to read right-to-left (unless I suffix as well, maybe it will be possible)
(2) I am able to chunk the data as part of a file inside an array. Every chunk could have a fixed number of frames. As you say, I would then store a map of offsets. This is in fact what we are currently doing, but using BinaryWriter and BinaryReader. I was looking for something easier to maintain than writing all the boilerplate.

[jamescourtney]

Can you separate each frame (or chunk of frames) into a separate file/array? If so, it seems pretty straightfoward to iterate in any direction you want that way, and each file could be a collection of frames.

If that doesn't work, could you share a little more about your schema? I may have some better suggestions if the problem is a little less abstract. Thanks!

[ghost]

Thanks for the advice. I managed to split it into chunks (separate files). It also solved another problem when I was trying to save the entire file at once which caused an overflow. I will continue to explore this direction :)

[jamescourtney]

I see from my emails that you had a few other comments here. One was about loading the entire array into memory. I'd like to address that so you don't stumble.

FlatBuffers requires a contiguous chunk of memory because it does jump around forwards and backwards. However, just because that chunk of memory is loaded, doesn't mean the buffer is actually parsed (in FlatSharp this is Lazy or Progressive mode -- see the wiki). If you wanted to load a chunk of data but not necessarily keep it in memory, you should look at Memory Mapped Files. FlatSharp does not provide native support for this, but it is quite easy to wrap up a mmap'ed file into a Memory<byte>, which FlatSharp does support.

The other thing you can do with mmap'ed files and FlatSharp is Write Through of some data (which is to modify the buffer in place). When your flatbuffer is actually a mmap'ed file, that update is persisted directly to disk.