GeoParquet output with control over row groups?

[cholmes]

I've been doing a number of experiments with Overture building data, see https://beta.source.coop/cholmes/overture

The main thing I've been working on is being able to have a client where you can supply it with a GeoJSON and it'll query a global set of parquet files that have been partitioned and transform it into the GIS format that user wants. I've had some good success using DuckDB (could be gdal/ogr, etc), but recently realized that to get the performance acceptable was to format the data with more row groups - it'd take it from like a couple minutes to ~20 seconds to extract a small number of buildings.

But I couldn't find a GeoParquet tool that fully worked for me to create more row groups:

• GDAL/OGR has the option, but right now it skips the deeper nested structures of the Overture Parquet format, so it's not an option till it fixes that, as I want to faithfully represent what they have.
• GPQ doesn't have the option, and in my (brief) testing of having it do parquet to geoparquet transformation it seemed to reset the row groups to be less. It does look like the underlying Go library can give control, so likely it's not hard to add?
• GeoPandas to_parquet doesn't have the option, and I don't see it in pandas either.

DuckDB does have the option in their native parquet format, but it doesn't (yet) support GeoParquet output. So I ended up just making https://beta.source.coop/cholmes/overture/geoparquet-3 with 'compatible' Parquet (WKB in geometry column) so I could experiment. But half the point of the admin partitioning is to enable people to just download a file and load it up in QGIS, so I'd really like it to be GeoParquet.

Any other tools that might be an option? And curious from @tschaub / @rouault / @jorisvandenbossche on what's needed to get the option. I can try to do more extensive testing to compare it, but the difference felt drastic. Shoot, just realized I took the version that had bigger row groups offline - I'll try to upload it again so everyone can compare.

[kylebarron]

• GeoPandas to_parquet doesn't have the option, and I don't see it in pandas either.

It does, but it's hidden because it's actually under the kwargs...

Additional keyword arguments passed to pyarrow.parquet.write_table().

In there you see

row_group_size: Maximum number of rows in each written row group. If None, the row group size will be the minimum of the Table size and 1024 * 1024.

So you can pass GeoDataFrame.to_parquet(..., row_group_size=2000) to have a maximum of 2000 rows per row group

Edit: row_group_size=2000 is for descriptive purposes; row groups should probably have more than this number of rows 🙂

[cholmes]

Oh sweet! Thanks @kylebarron. I figured I was just missing something. Time to rip through all my files with geopandas :)

[rouault]

• GDAL/OGR has the option, but right now it skips the deeper nested structures of the Overture Parquet format

fixed per OSGeo/gdal#8262 . cf https://lists.osgeo.org/pipermail/gdal-dev/2023-August/057558.html
Note however that it maps the nested structures (I don't understand by the way what's the point of having a list of map of string->string, and not just a map of string->string) to a OGR String(JSON) field type.
So if you use ogr2ogr to output that to (Geo)Parquet, it will write a String to Parquet, not the original data type (OGR typing system doesn't keep the details of the original data type).
That said, I've in a corner of my mind the potential extension of the OGR ArrowArray capability to the writing side, so potentially a ogr2ogr Parquet->Parquet could use that to preserve things more faithfully

[kylebarron]

I've in a corner of my mind the potential extension of the OGR ArrowArray capability to the writing side, so potentially a ogr2ogr Parquet->Parquet could use that to preserve things more faithfully

🚀 The arrow read side has made pyogrio bindings super fast! If I can help on the write side, let me know! (I just don't know C++ 😅)