make HTTP retry when the GoPro is overwhelmed #128
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
My HERO12 did not like being asked for more than about 5-7 files simultaneously, but dropped some of the /gp/gpMediaMetadata with a timeout. So after the files where the call to gpMediaMetadata was successfull were downloaded, I got presented with a bunch of errors where it didn't work, and of cause the files were not downloaded. Switching to a retrying approach means that we still hammer the GoPro for the first couple seconds, but allow the requests to proceed later. Some of this is still WIP - the config file has a timeout option, that I dropped since I do not understand enough go to incorporate it, yet.
KonradIT
force-pushed
the
retryablehttp
branch
from
de7cc01 to
d35a5bb
Compare
klaernie
force-pushed
the
retryablehttp
branch
from
d35a5bb to
f95de84
Compare
|
Now the linter is happy - seems like I did not understand what it was failing on at the time I wrote this. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
replace net.http with go-retryablehttp
My HERO12 did not like being asked for more than about 5-7 files simultaneously. Instead it dropped some of the /gp/gpMediaMetadata requests with a timeout. So after the files where the call to gpMediaMetadata was successful were downloaded, I got presented with a bunch of errors where it didn't work, and of cause those files were not downloaded.
Switching to a retrying approach means that we still hammer the GoPro for the first couple seconds, but allow the requests to proceed later.
Since go-retryablehttp allows it, I set it up to retry with a lot of jitter, so the camera should not get hit with multiple requests at the same time anymore. I also went for the approach that does not require touching all the points where the pkg/utils/client is used, but instead slipped retryablehttp underneath the
Clientthat already was used in all other places.Please note, as with my other PRs - I'm a beginner at working with golang, and fixed this mainly for me, since it cost me 3 days to import 35GB of footage due to having to retry again and again. So if anything of this jumps out as nonsensical, please tell me.
Type:
Camera:
Component:
Checklist before approval: