Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support micro seconds timestamp precision with copy unload #491

Open
arthurli1126 opened this issue Mar 8, 2023 · 3 comments
Open

Support micro seconds timestamp precision with copy unload #491

arthurli1126 opened this issue Mar 8, 2023 · 3 comments

Comments

@arthurli1126
Copy link

Hi folks, due to high NAT gateway cost, we have to use copy load when reading from snowflake, but currently copy unload doesn't support micro second level precision(only at mills). I can work on the PR to add it. But wondering if you have any concerns about this.

@sfc-gh-mrui
Copy link
Contributor

@arthurli1126 Could you please try SC 2.6.0 or newer version? COPY UNLOAD is mainly used from SC 2.5.x and prior versions.

@arthurli1126
Copy link
Author

@sfc-gh-mrui thanks for your reply, the problem is with the simpleDateTime(https://github.com/snowflakedb/spark-snowflake/blob/master/src/main/scala/net/snowflake/spark/snowflake/Conversions.scala#L66) used for parse timestamp during copy unload. The format only support millisecond precision and would give wrong timestamp if the timestamp carries micro seconds. For instance, for string 2023-03-01 07:54:56.191173 it would consider it carries 191173 milliseconds so it will add
191000 / 1000 / 60 = 3 mins 11s and put 173 microseconds to milliseconds filed:
2023-03-01 07:58:07.173000.

@arthurli1126
Copy link
Author

Created a draft PR for this: #492

Please let me know if I misunderstand anything or if you have any concerns about the approach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants