Initcap behaves differently in Spark and in DataFusion (also Comet)

[Blizzara]

Describe the bug

DataFusion's initcap behaves differently than Spark's. While both do "upper-case the first letter of each word and lowercase others", Spark considers as words anything separated by whitespace (' '), while DataFusion considers anything separated by non-ascii-alphanumeric as words. (DF's code would also fail to uppercase or lowercase non-ascii chars, but that doesn't materialize as a separate issue as it considers them separators already in the first place.)

#1051 shows the problem by adding two cases to the test, one using a dash and one using non-ascii letters (from Finnish).

== Results ==
!== Correct Answer - 7 ==       == Spark Answer - 7 ==
 struct<initcap(name):string>   struct<initcap(name):string>
 [James Smith]                  [James Smith]
 [James Smith]                  [James Smith]
![James Ähtäri]                 [James äHtäRi]
 [Michael Rose]                 [Michael Rose]
 [Rames Rose]                   [Rames Rose]
![Robert Rose-smith]            [Robert Rose-Smith]
 [Robert Williams]              [Robert Williams]

Steps to reproduce

Call initcap with an input containing non-ascii-alphanumeric non-whitespace characters

Expected behavior

Match Spark

Additional context

No response

[viirya]

Thanks for reporting this bug.

[comphead]

DataFusion InitCap doesn't support nonASCII
Tracked in apache/datafusion#13711

[comphead]

![Robert Rose-smith]            [Robert Rose-Smith]

already fixed in latest DF

> select initcap('robert rose-smith');
+------------------------------------+
| initcap(Utf8("robert rose-smith")) |
+------------------------------------+
| Robert Rose-Smith                  |
+------------------------------------+
1 row(s) fetched. 
Elapsed 0.005 seconds.

UPD: Spark returns

scala> spark.sql("select initcap('robert rose-smith')").show(false)
+--------------------------+
|initcap(robert rose-smith)|
+--------------------------+
|Robert Rose-smith         |
+--------------------------+

But reg to DF SQL policy apache/datafusion#13706 DF supports a builtin functions to be in sync with PG which returns Robert Rose-Smith and Comet likely need to implement own UDF function to cover this case

[Blizzara]

already fixed in latest DF

How is that fixed? :) The diff I showed is exactly the same as the diff you meantion below, no?

[comphead]

already fixed in latest DF

How is that fixed? :) The diff I showed is exactly the same as the diff you meantion below, no?

Please refer to message update above

[kazuyukitanimura]

Looks like unicode init cap is getting fixed in DF apache/datafusion#13752