Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Implement pre-initialized Docker container pool to improve /eval #66

Open
wants to merge 6 commits into
base: develop
Choose a base branch
from

Conversation

EmmanuelStan12
Copy link

@EmmanuelStan12 EmmanuelStan12 commented Oct 17, 2024

This pull request introduces significant performance improvements for the /eval request process by implementing a pool of pre-initialized Docker containers. Previously, each request required a new container to be created and initialized, which added latency to the response time. These changes optimize that process, ensuring faster response times and better resource management.

Key changes include:

  • Container Pooling: Create one container during startup so that subsequent /eval requests can be performed quickly.
  • Pre-configured Environments: Execute startup scripts on pool containers.

@EmmanuelStan12 EmmanuelStan12 requested review from a team as code owners October 17, 2024 11:50
…l request performance

- Added a pool of pre-initialized Docker containers, each assigned a unique session ID and ready for immediate use.
- Pre-configured each container with the necessary startup scripts to ensure environments are ready for /eval requests.
- On receiving a new /eval request, the system now allocates a container from the pool, reducing the need to create and initialize a container on demand.
- Improved request latency by significantly reducing the time taken to start and initialize Docker containers during each session.
- Ran code formatting and styling checks using Spotless to ensure consistent code style.
- Performed code analysis and linting with SonarQube to identify and address potential issues.
- Fixed minor issues flagged during the linting process to enhance code quality.
@EmmanuelStan12 EmmanuelStan12 force-pushed the feature/docker-container-pool-for-eval branch from 8adf7b0 to 94ffa88 Compare October 17, 2024 12:30
- Added functionality to persist input and output streams of newly created or cached Docker containers using a ConcurrentHashMap.
- Ensured that container streams are properly managed and retrievable for future requests, improving container reuse.
- Fixed test case issues related to stream management and container lifecycle, ensuring tests pass consistently.
@EmmanuelStan12 EmmanuelStan12 force-pushed the feature/docker-container-pool-for-eval branch from fd7b998 to 168a26c Compare October 22, 2024 08:33
@EmmanuelStan12 EmmanuelStan12 force-pushed the feature/docker-container-pool-for-eval branch from 168a26c to e40b62b Compare October 22, 2024 19:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants