Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make storing a run not wait until storage is complete #1306

Open
whisperity opened this issue Jan 18, 2018 · 2 comments · May be fixed by #4326
Open

Make storing a run not wait until storage is complete #1306

whisperity opened this issue Jan 18, 2018 · 2 comments · May be fixed by #4326
Labels
API change 📄 Content of patch changes API! database 🗄️ Issues related to the database schema. enhancement 🌟 server 🖥️

Comments

@whisperity
Copy link
Contributor

With the added statelessness and the complexity of parsing a report ZIP it should be done so that this parsing does not hang one thread of the web server pool, and neither does keep a socket open. The user could time out while the server is generating the return value of massStoreRun, which could have unforeseen consequences.

Ideas:

  1. Make the Webserver process pooled instead of thread pool. This could be a big help in alleviating for the GIL.
  2. massStoreRun should start a direct, detached subprocess which is internal to the server host. This subprocess could independently handle the parsing and storing of the result ZIP. This way the user's connection is closed once the ZIP is dispatched, and instead of waiting for the reply in the same connection, it could be made so that the client actively pings the server to tell them when the build is successfully stored, or failed. This would require a change on the API and handling, and also to introduce a little bit of interprocess communication, or saving intermittent state data to the database (presumably into a new table which is recommended to be in the MEMORY backend).
@whisperity whisperity added this to the Future features milestone Jan 18, 2018
@whisperity whisperity added enhancement 🌟 API change 📄 Content of patch changes API! database 🗄️ Issues related to the database schema. server 🖥️ labels Jan 18, 2018
@gyorb
Copy link
Contributor

gyorb commented Jan 19, 2018

I was thinking about this setup for some time. I think we should check the features by Celery.

@whisperity
Copy link
Contributor Author

whisperity commented Aug 11, 2024

>mfw this has been an issue since 2018 and not just 2022 like originally thought???

Okay, while not exactly matching the design that was proposed here, #3672 is a better thought-out version of this issue without the need for GIL changes, detached subprocesses, etc.

From the user's PoV, we will simulate the blocking wait internally (or the user can just go --detach in order not to wait for anything).

@whisperity whisperity linked a pull request Aug 25, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API change 📄 Content of patch changes API! database 🗄️ Issues related to the database schema. enhancement 🌟 server 🖥️
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants