Is it possible to expand DataSetContentMismatch?

[miikargh]

Is there a way to expand the diff so that I can see every column of the dataset? Otherwise it is impossible to see where the difference lies exactly.

[info]   com.github.mrpowers.spark.fast.tests.DatasetContentMismatch: Diffs
[info] +--------------------+--------------------+
[info] |      Actual Content|    Expected Content|
[info] +--------------------+--------------------+
[info] |Event(1,161097120...|Event(1,161097120...|
[info] +--------------------+--------------------+

The above snippet illustrates the problem. The two datasets look exactly the same since they are truncated before the difference is shown.

[MrPowers]

@miikargh - I pushed up a fixed and released a new version. Can you please try libraryDependencies += "com.github.mrpowers" %% "spark-fast-tests" % "0.22.0" % "test" and let me know if that fixes the issue? Thanks!

[miikargh]

@MrPowers Thanks for the quick fix! It is definitely better now as all the keys and values are visible. The formatting for case classes with many keys is still quite messy tho:

[info] - Lab pipeline *** FAILED ***
[info]   com.github.mrpowers.spark.fast.tests.DatasetContentMismatch: Diffs
[info] +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[info] |                                                                                                                                                                        Actual Content|                                                                                                                                                                 Expected Content|
[info] +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[info] |TestEvent(123456789,1610971200000,south pole,penguing & co,2021-01-18 14:00:00.0,Lorem: Ipsum
[info]                         |Dolor: Sit
[info]                         |Amet,2.789012,success,kcal)|TestEvent(123456789,1610971200000,south pole,penguing & co,2021-01-18 14:00:00.0,Lorem: Ipsum
[info]                       |Dolor: Sit
[info]                       |Amet,2.789012,success,cal)|
[info] +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

[miikargh]

Oops. Some of the messiness was from my bad string formatting. Here is the output with better formats:

[info]   com.github.mrpowers.spark.fast.tests.DatasetContentMismatch: Diffs
[info] +------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
[info] |                                                                                                                      Actual Content|                                                                                                                   Expected Content|
[info] +------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
[info] |TestEvent(123456789,1610971200000,south pole,penguing & co,2021-01-18 14:00:00.0,Lorem: Ipsum
[info] Dolor: Sit
[info] Amet,2.789012,success,kcal)|TestEvent(123456789,1610971200000,south pole,penguing & co,2021-01-18 14:00:00.0,Lorem: Ipsum
[info] Dolor: Sit
[info] Amet,2.789012,success,cal)|
[info] +------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------+

Still messy but better.