You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Fetch a list of proxies and save working ones to a text file.
Use the working proxies with concurrency to scrape course data.
Add retry logic after getting a 429 error to try other proxies before waiting for 60 seconds.
Why would this feature be useful?
This will significantly reduce the scraping time and allow automated daily scraping to happen on GitHub actions, thus allowing support for how full a class currently is.
Additional Notes
No response
The text was updated successfully, but these errors were encountered:
Overview
Add concurrency with proxies for the scraper.
Details
Why would this feature be useful?
This will significantly reduce the scraping time and allow automated daily scraping to happen on GitHub actions, thus allowing support for how full a class currently is.
Additional Notes
No response
The text was updated successfully, but these errors were encountered: