take

Also, a proposed followup work (to handle the general case):

Consider whether we have a testing gap for ParquetSink:

• do we need to have more e2es which ensure that the parquet encoders all encode uniformly?
• e.g. parquet encoded by either ArrowWriter (under defaults) or ParquetSink (under defaults) should be identical.

BTW I think you can write a test for the embedded schema in a parquet file using the describe command

DataFusion CLI v40.0.0
> copy (values (1)) to '/tmp/foo.parquet';
+-------+
| count |
+-------+
| 1     |
+-------+
1 row(s) fetched.
Elapsed 0.051 seconds.

> describe '/tmp/foo.parquet';
+-------------+-----------+-------------+
| column_name | data_type | is_nullable |
+-------------+-----------+-------------+
| column1     | Int64     | YES         |
+-------------+-----------+-------------+
1 row(s) fetched.

Perhaps we could extend some of the tests in https://github.com/apache/datafusion/blob/main/datafusion/sqllogictest/test_files/copy.slt with describe as a way to verify your fix

This was hit in datafusion-python here: apache/datafusion-python#957

I'd argue this is an important bug, because there are a bunch of Arrow types that aren't representable as a Parquet logical type. Additionally it's not the case of a wrong default, there's no option today to force the Parquet writer to include the Arrow metadata.

@wiedld did you get anywhere with this issue?

Since we did our own hack around it, I wasn't able to justify prioritizing the fix.

Seems like that has changed. I'll queue it up (fyi to @alamb ).

take