High latency loading parquet file on GCS.

[gwik]

Hi,

First, let me thank you for this amazing project.

I was experimenting with reading parquet from GCS and the performance looks very poor compared to downloading the file and loading from it disk.

In the example below loading my 10MB parquet file takes around 10s, although downloading the file takes less than a second. Then it is much faster to download then load the file from disk (1s + ~70ms).

I added tracing to the object store and I see more that 506 get_opts calls.

I was wondering why there was so many range requests, and what determine the chunk size ?
Is it controlled by some parameters? or is it the consequence of the how the file was written? (Record batches).

Thanks for your help.

    let store = Arc::new({
        GoogleCloudStorageBuilder::new()
            .with_bucket_name(BUCKET_NAME)
            .build()?
    });
    let url = format!("gs://{BUCKET_NAME}/").parse()?;
    let registry = DefaultObjectStoreRegistry::new();
    registry.register_store(&url, store);
    let runtime_config = RuntimeConfig::default().with_object_store_registry(Arc::new(registry));
    let runtime_env = RuntimeEnv::new(runtime_config)?;

    let ctx = SessionContext::new_with_config_rt(SessionConfig::default(), Arc::new(runtime_env));
    let df = timeit!(
        "read parquet",
        ctx.read_parquet(
            args.input,
            ParquetReadOptions {
                // file_sort_order: vec![vec![col("time").sort(true, true)]],
                parquet_pruning: true.into(),
                ..Default::default()
            },
        )
        .await?
    );

    println!("{schema}", schema = df.schema());

    let df = timeit!(
        "projection",
        df.select(vec![
            cast(
                col("time"),
                DataType::Timestamp(TimeUnit::Millisecond, None),
            )
            .alias("time"),
            col("asset_id"),
            col("asset_type"),
            col("value"),
        ])?
        .cache()
        .await?
    );

[tustvold]

Perhaps you could run the file through parquet-layout. It sounds like the file might have been written with very small row groups

[gwik]

Thank you so much @tustvold that was exactly what I needed.

parquet-layout showed a lot of row groups: 454, and the row count within each group was very uneven 4k to just 4.

After rewriting the parquet file with datafusion (default write batch size and max row group size).
There is now only one row group with helped compression (zstd) bring down the file size from 11MB to 683K, and load time from GCS from 11s to 500ms.

Awesome!