Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TDL-21378 increase page_size for all streams #123

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

TDL-21378 increase page_size for all streams #123

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f03efab
Changes:
somethingmorerelevant Dec 1, 2022
0d41895
added forcetypecast for page_size
somethingmorerelevant Dec 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions config.sample.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"api_token":"<token_here>",
"start_date":"2017-01-01",
"page_size":500
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please at a newline at the end of this file

Copy link
Member Author

@somethingmorerelevant somethingmorerelevant Jan 9, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi @luandy64 , i have tested this code. self.streams will refer to the class attribute streams declared on the class itself, it might not be a good practice to modify class attributes via instances as it can result in an inconsistent state but we are only creating one instance of this class.

does this need to be changed?

class PipedriveTap(object):
    streams = [
        CurrenciesStream(),
        ActivityTypesStream(),
        StagesStream(),
        ... 
]

    def __init__(self, config, state):
        self.config = self.get_default_config()
        ...

        page_size = int(self.config.get('page_size',500))
        for stream in self.streams:
            stream.limit = stream.next_start = page_size

5 changes: 5 additions & 0 deletions tap_pipedrive/tap.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,11 @@ def __init__(self, config, state):
self.config['start_date'] = pendulum.parse(self.config['start_date'])
self.state = state

# page size configured from config, overrides the default value set in stream.py
page_size = int(self.config.get('page_size',500))
for stream in self.streams:
stream.limit = stream.next_start = page_size
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we setting the stream.next_start while initializing the object?
I think we should capture this value from the API endpoint response within the pagination function.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i agree, and that is what its doing here(https://github.com/singer-io/tap-pipedrive/blob/master/tap_pipedrive/stream.py#L71-L82)

    def paginate(self, response):
        payload = response.json()

        if 'additional_data' in payload and 'pagination' in payload['additional_data']:
            logger.debug('Paginate: valid response')
            pagination = payload['additional_data']['pagination']
            if 'more_items_in_collection' in pagination:
                self.more_items_in_collection = pagination['more_items_in_collection']

                if 'next_start' in pagination:
                    self.start = pagination['next_start']

i don't think the attribute next_start is necessary, but i am not too confident of removing it in the context of this P.R

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you run this code?

To me it looks like self.streams doesn't exist, so I'm not sure how you are looping over it.

stream.limit = stream.next_start = page_size

This is also hard to read in my opinion.

stream.limit = page_size
stream.next_start = page_size

This makes more sense to me.


def do_discover(self):
logger.info('Starting discover')

Expand Down