SNOW-944062: Implementation and functionality of pivot differs from PySpark and is not user-friendly

[FlorianWilhelm]

What is the current behavior?

Currently the pivot method is a method of a dataframe as shown in the docs. You thus have to pass the pivot column and even the distinct values for pivoting. It is not possible use pivot in combination with a GroupedDataframe.

This is not compatible to how PySpark implements pivot. Here pivot is a function on a GroupedDataframe. And the specific values don't need to be specified.

What is the desired behavior?

Implement the same behavior as in PySpark.

How would this improve snowflake-snowpark-python?

It will be easier to migrate from PySpark to Snowpark and Pyspark's pivot behaviour is just more user-friendly as in most cases you group before you apply it.

References, Other Background

• https://spark.apache.org/docs/3.2.0/api/python/reference/api/pyspark.sql.GroupedData.pivot.html
• https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/latest/api/snowflake.snowpark.DataFrame.pivot

[FlorianWilhelm]

Thanks @sfc-gh-aalam!

[sfc-gh-aalam]

@FlorianWilhelm please note that right now, adding values is still required. I haven't designed a good approach to evaluate all the distinct columns lazily. For now you can use Dataframe.distinct().collect() to collect all distinct values.

[FlorianWilhelm]

@sfc-gh-aalam Okay, would it then make sense to keep this issue open until this is resolved? I guess API-parity with Spark is an important goal for the acceptance of Snowpark.