Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]: support LocalAI backend #54

Open
karanveersingh5623 opened this issue Aug 2, 2024 · 2 comments
Open

[Feature]: support LocalAI backend #54

karanveersingh5623 opened this issue Aug 2, 2024 · 2 comments
Assignees

Comments

@karanveersingh5623
Copy link
Contributor

Is this feature request related to a problem?
None

Problem Description
No response

Solution Description
support localai.

Benefits
The LocalAI can make it easier for users to interact with KRS.

@karanveersingh5623
Copy link
Contributor Author

@OluchukwuON

@abhimazu
Copy link
Contributor

abhimazu commented Aug 3, 2024

@OluchukwuON --> Find target open sourced models that can perform the task of log analysis and respond back. Search huggingface for target models.

Criteria:

  1. They should be under 4 GB in size
  2. They should be able to understand Kubernetes context to respond with resolutions
  3. They should be chat models that can respond

Tasks:

  1. List all candidate models in a csv file with their size, and urls from huggingface
  2. Create a list of 10 sample questions with faulty Kubernetes Pod logs that you will use to benchmark these models
  3. Test all these models using the questions by creating end points on cloud instances provided by @karanveersingh5623 one by one using either langchain, TGI or tensorflow serving.
  4. Record the individual model responses for all the 10 questions, and record the execution time for each question inside a spreadsheet, the target is to find the WPS speed of each model (word per second) when deployed locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants