Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

start_date implementation #118

Open
visch opened this issue Apr 24, 2022 · 0 comments
Open

start_date implementation #118

visch opened this issue Apr 24, 2022 · 0 comments
Labels
enhancement New feature or request

Comments

@visch
Copy link
Contributor

visch commented Apr 24, 2022

Currently

    16     def _write_starting_replication_value(self, context: Optional[dict]) -> None:
    15         """Write the starting replication value, if available.
    14
    13         Args:
    12             context: Stream partition or context dictionary.
    11         """
    10         value = None
     9         state = self.get_context_state(context)
     8
     7         if self.replication_key:
     6             replication_key_value = state.get("replication_key_value")
     5             if replication_key_value and self.replication_key == state.get(
     4                 "replication_key"
     3             ):
     2                 value = replication_key_value
     1
  321              elif "start_date" in self.config:
     1                 value = self.config["start_date"]
     2
     3         write_starting_replication_value(state, value)

From the SDK brings in start_date, which means techincally if you pass in a unix timstamp for start_date it will work. We don't advertise this as the "standard" for start_date is to use an RFC3399 datetime type

Options

  1. We could just override start_date's value with the timestamp value. right at the initialization of the tap.
  2. We could add special handling of get_starting_timestamp like https://github.com/AutoIDM/tap-clickup/pull/117/files#diff-0b811eae9421519f5a8fc873c70bd3e821c57d9e584c88de748d635e4808d1c5R279 , but we'd have to figure out when the initial state is coming from start_date and when it's coming from state as start_date is RFC3399 and state is storing the value as a unix timestamp
  3. Convert state unix time stamps to use RFC3399 values, and convert these values to timestamps when needed like here https://github.com/AutoIDM/tap-clickup/pull/117/files#diff-0b811eae9421519f5a8fc873c70bd3e821c57d9e584c88de748d635e4808d1c5R252
  4. Probably others?

Instead of figuring this out right now we're just not going to officially support start_date.

@visch visch added the enhancement New feature or request label Mar 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant