State of ParquetWASMLoader as an alternative for ParquetLoader

[shaunakv1]

I have been experimenting with visualizing some AIS data using parquet files. I encountered an issue where workers don't work with the current ParquetLoaders. This issue is already identified in the following and a few other issues:

#3041

I am ofcourse able to get it to work by setting worker: false. The performance is quite fine for resolution 7 but performance at resolution of 8 h3 grid is terrible. Like I mention in the issue comments above, I suspect it's because workers are disabled?

There are also some other issues I am running into with the ParquetLoader that I have started a ticket for.

@ibgreen mentioned that using ParquetWASMLoader is a better and more performant alternative, especially becuase that loader is getting most attention development wise ( which makes sense!)

That said I am unable to find much documentation on how to use it. It doesn't seem to be listed on main docs at https://loaders.gl/docs yet.

I did find some work and PR in the repo towards the ParquetWASMLoader here: #3121 But doesn't look like that is merged yet.

What's the current state of the WASM loader and is there anywhere I can find some docs on how to use it give it go?

[ibgreen]

@shaunakv1 Thanks for engaging.

First, on the topic of performance and workers

• Moving parsing to a worker doesn't make it faster. It does however prevent the main thread from blocking, so your UI is still responsive, which can make it "feel" faster.
• If the returned data is not binary (which is the case here), workers are actually slower, since the parsed data needs to be serialized and deserialized when sent back to the main thread, which sometimes takes more time than parsing took in the first place...

[shaunakv1]

@ibgreen Right, apart from the UI getting stuck parsing about 58MB of res 8 h3 index, I was assuming that if workers are on, then some data will start drawing as it's getting worked on in parallel. Is that assumption wrong?

[ibgreen]

Many loaders.gl loaders support streaming loads (via parseInBatches()), however batched parsing is not yet implemented in workers (requires more elaborate message passing which tends to be time consuming to get right).

That said, you could use batched parsing on the main thread, and your main thread should regain control after every batch.

Not sure about the state of batched parsing with the ParquetLoader but it could be worth a try.

[ibgreen]

A bit of history on the `ParquetLoader:

• Parquet is by a large margin one of the most complex formats we have chosen support in loaders.gl. It support a range of encodings, compressions, dictionaries, and special packing and unpacking algorithms for heavily nested columns that all need to be implemented
• Parquet support in JavaScript has been weak. Most open source JS libraries out there were abandon-ware, only worked for node and didn't have great performance.
• We started by forking such a project (converting it to typescript, removing Node dependencies, etc) which became the current ParquetLoader
• We also made a pass to get a bunch of Geospatial (geoparquet) examples working (examples on the loaders.gl website)

However,

• there is still a list of small but important issues that would need work.
• And as you see the performance is not good on large data. Part of the reason is that the loader doesn't use binary columns when possible, instead being very general supporting heavily nested columns.

Fixing these things are not impossible but would require a week or two of effort, but current maintainers are spread to thin, so unlikely to happen unless someone helps out.

[ibgreen]

About the ParquetWasmLoader

• in the meantime, a WASM port of a rust Parquet loader has matured and is working really well and is over 10x faster than the current inefficient JS implementation. So the current goal is to double down on that loader instead.
• This is the experimental ParquetWasmLoader you have found.
• It is not documented as it is still experimental.

The complications of the WASM loader are that

• bundling WASM is still a hard problem
• it outputs Apache Arrow format, which is very efficient but can be hard to work with.

So there you have the current state of things...

[shaunakv1]

@ibgreen Thanks a lot for taking the time to walk through Parquet and WASM! For my current use case, the ParquetLoader performance is very acceptable if I drop the h3 resolution to 7 instead of 8. So in the short run while WASM matures we might consider that.

I also realized Arrow is supported, and I think It might be simple enough for me to switch and try that format with DuckDB (which is what generates the Parquet anyway). Only thing I need to check is compression vs performance , which is where parquet seems to strike a great balance.

Again appreciate your prompt response and this great project!

[ibgreen]

Only thing I need to check is compression vs performance , which is where parquet seems to strike a great balance.

Parquet is a binary columnar format optimized for disk storage. Arrow is a largely equivalent format optimized for in-memory use.

For your compressed parquet data to be used by the application you will most likely need to decompress/unpack it in memory anyway.

If you are passing Arrow data over the network, it can be worth compressing to parquet and then decompressing it on the other side, as network speed can be the bottleneck.