diff --git a/README.md b/README.md index f7dd10d..4a4ad2e 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,10 @@ -# LLMFlowOptimizer (WIP) +# LLMFlowOptimizer -![Python 3.9](https://img.shields.io/badge/python-3.9-blue.svg) -![Python 3.10](https://img.shields.io/badge/python-3.10-blue.svg) +[![python](https://img.shields.io/badge/-Python_3.9_%7C_3.10-blue?logo=python&logoColor=white)](https://github.com/pre-commit/pre-commit) +[![hydra](https://img.shields.io/badge/Config-Hydra_1.3-89b8cd)](https://hydra.cc/) +[![Optuna](https://img.shields.io/badge/Optimize-Optuna-blue)](https://github.com/optuna/optuna) +[![black](https://img.shields.io/badge/Code%20Style-Black-black.svg?labelColor=gray)](https://black.readthedocs.io/en/stable/) +[![isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/) ## ๐Ÿ“Œ Introduction @@ -9,11 +12,13 @@ In recent years, various LLMs, embedding models, and LLM flows utilizing them ha This repository aims to treat LLMs and Embeddings as a hyperparameter, with the goal of automatically searching for the optimal hyperparameter of the LLM flow. -**Below image is the concept image of this repository (image from [Flowise](https://github.com/FlowiseAI/Flowise)). Component of LangChain like a LLM or Embedding can be treated as hyperparameter. You can find optimal component from various candidate.** -![concept_image](documents/concept.png) +**Below image is the concept image of this repository (image is took from [Flowise](https://github.com/FlowiseAI/Flowise) and slightly modified). Component of LangChain like a LLM or Embedding can be treated as hyperparameter. You will find component from various candidate that can optimize score.** +![concept_image](documents/image/concept.png) This repository is strongly inspired by [lightning-hydra-template](https://github.com/ashleve/lightning-hydra-template)๐ŸŽ‰ +Any feedback, bug reports, and suggestions are appreciated! + ## ๐Ÿ”ง Main Technologies - [LangChain](https://github.com/langchain-ai/langchain) : LangChain is a framework for developing applications powered by large language models. It can be divided conceptually into components (like llm component, embedding component), which are well-abstracted and easy to switch. This is the reason we can treat each component as hyperparameter. @@ -26,29 +31,37 @@ This repository is strongly inspired by [lightning-hydra-template](https://githu - [ragas](https://github.com/explodinggradients/ragas) : Ragas is an evaluation framework for Retrieval Augmented Generation (RAG) pipelines that provides tools based on the latest research for evaluating LLM-generated text to give insights about the RAG pipeline. -# How to use +# ๐Ÿš€ How to use + +I will explain how to use this repository by using question answering as an example. In example, we use + +- **LangChain** : for model +- **Ragas** : for evaluation +- **LangSmith** : for evaluation and monitoring + +## Step 0 : Environment Setup Please click [Use this template](https://github.com/Yongtae723/LLMFlowOptimizer/generate) to use this repository as template. -Clone your repository and install dependencies by following command. +- Clone your repository and install dependencies by following command. -```bash -git clone git@github.com:/.git -cd -``` + ```bash + git clone git@github.com:/.git + cd + ``` -## setup +- We use poetry for package management. you can install poetry and python packages by following command. -We use poetry for package management. you can install poetry and python packages by following command. + ```bash + pip install poetry + poetry install + ``` -```bash -pip install poetry -poetry install -``` +- Rename `.env.example` to `.env` and set your environment variables to use external API like OpenAI ChatGPT. -## define model architect and config. +## Step 1 : Define model architect and config. -1. Define model architect like [llmflowoptimizer/modelsample_qa.py](llmflowoptimizer/model/sample_qa.py). +1. Define model architect like [llmflowoptimizer/component/model/sample_qa.py](llmflowoptimizer/component/model/sample_qa.py). The arguments in `__init__()` can be used as hyperparameter and will be able to be optimized. @@ -57,38 +70,72 @@ poetry install Example model config: ```yaml - defaults: # you can use yaml file on this directly. - - _self_ - - embedding: OpenAI - - text_splitter: RecursiveCharacter - - llm_for_answer: ChatOpenAI + defaults: + - _self_ + - embedding: OpenAI + - text_splitter: RecursiveCharacter + - llm: OpenAI + _target_: llmflowoptimizer.component.model.sample_qa.SampleQA # what we defined on llmflowoptimizer/component/model/sample_qa.py + data_path: ${paths.reference_data_dir}/nyc_wikipedia.txt + ``` - _target_: llmflowoptimizer.models.sample_qa.SampleQA # what we defined on llmflowoptimizer/models/sample_qa.py +3. Then you can check your model and config by following command. - data_path: ${paths.reference_data_dir}/nyc_wikipedia.txt + ```bash + poetry run python llmflowoptimizer/run.py extras.evaluation=false ``` -3. then you can check your model and config by following command. +## Step 2 : Prepare dataset and define evaluation system -```bash -poetry run python llmflowoptimizer/run.py extras.evaluation=false extras.print_config=true -``` +- register data on LangSmith (If you want to use LangSmith) -## define evaluation system + In our example, we use [LangSmith](https://docs.smith.langchain.com/) for evaluation and monitoring. You can register data on LangSmith by following [example notebook](notebooks/register_data_langsmith.ipynb) -1. Define evaluation system like [llmflowoptimizer/model/evaluation.py](llmflowoptimizer/model/evaluation.py), and set argument on [configs/evaluation](configs/model). +- Define evaluation system like [llmflowoptimizer/model/evaluation.py](llmflowoptimizer/model/evaluation.py), and set argument on [configs/evaluation](configs/model). - Optuna will optimize component based on the return value of this evaluation system. + Optuna will optimize component based on the return value of this evaluation system. -2. You can check your evaluation system by following command. +- You can check your evaluation system by following command. -```bash -poetry run python llmflowoptimizer/run.py -``` + ```bash + poetry run python llmflowoptimizer/run.py + ``` + +- Since we use Hydra, you can change each LLM flow component by command line argument. For more detail, please check [Override](#override) section. + +## Step 3 : Hyperparameter search + +- Define search requirement on [configs/hparams_search](configs/hparams_search) like example. + + ```yaml + model/text_splitter: choice(RecursiveCharacter, CharacterTextSplitter) + model.text_splitter.chunk_size: range(500, 1500, 100) + model/llm: choice(OpenAI, GPTTurbo, GPT4) + ``` + + This example if a part of [configs/hparams_search/optuna.yaml](configs/hparams_search/optuna.yaml), and it means this system will search best hyperparameter from `RecursiveCharacter` or `CharacterTextSplitter` for `model.text_splitter` component, chunk_size is between 500 and 1500, and `OpenAI`, `GPTTurbo`, `GPT4` for `model.llm` component. + + Also complicated search range can be defined by python like [configs/hparams_search/custom-search-space-objective.py](configs/hparams_search/custom-search-space-objective.py) + +- You can start hyperparameter search by following command. + + ```bash + poetry run python llmflowoptimizer/run.py hparams_search=optuna + ``` + +Then you can see the best parameter on `logs/{task_name}/multirruns/{timestamp}/optimization_results.yaml`. -## dataset +And you can see the detail of each trial on LangSmith like below. -you can update sample data by following [this notebook](notebooks/register_data_langsmith.ipynb) +![langsmith_each_trial](documents/image/each_trial.png) + +This figure shows the result of each runs. + +![langsmith_each_input_output](documents/image/each_input_output.png) + +This figure shows the input and output of each questions. + +# ๐Ÿ“š Appendix ## override @@ -100,7 +147,7 @@ When you want to change single parameter, you have to use `.` between parameter Example: ```bash -poetry run python llmflowoptimizer/run.py model.llm_for_answer.model_name="gpt-4" +poetry run python llmflowoptimizer/run.py model.text_splitter.chunk_size=1000 ``` By doing this, you change `model_name` parameter of `llm_for_answer` to gpt-4. @@ -113,7 +160,7 @@ when you want to change parameter in component scale, you have to define compone poetry run python llmflowoptimizer/run.py model/llm_for_answer=OpenAI ``` -By doing this, LLM flow use [`OpenAI.yaml`](configs/model/llm_for_answer/OpenAI.yaml) model instead of [`ChatOpenAI.yaml`](configs/model/llm_for_answer/ChatOpenAI.yaml) model. +By doing this, LLM flow use [`OpenAI.yaml`](configs/model/llm_for_answer/OpenAI.yaml) model instead of [`ChatOpenAI.yaml`](configs/model/llm_for_answer/GPT4.yaml) model. ### Experiment config @@ -125,18 +172,7 @@ after you save config [configs/experiment](configs/experiment), you can override poetry run python llmflowoptimizer/run.py experiment=example ``` -## Hyperparameter search - -WIP - -## managing experimental results - -WIP -maybe use langflow? - -## Appendix - -### Test +## Test Before you make PR, you have to test your code by following command. In sample test, we only check model and evaluation class can be initialized. @@ -145,7 +181,7 @@ In sample test, we only check model and evaluation class can be initialized. make test ``` -### Auto code formatting +## Auto code formatting If you want auto code formatting, you can install by @@ -161,18 +197,36 @@ Also you can format code manually by make fix-lint ``` -# TODO - -- \[ \]ใ€€.envใฎ่ชฌๆ˜Ž -- [ ] Hyperparameter search by optuna -- [ ] manage experiment config (langsmith? WandB?) -- [ ] actual experiment result (ask specialist) - - [ ] write blog -- [ ] add test function - - [ ] reduce data and execute e2e test -- [ ] git action - - \[ \]lint-check - - [ ] test -- [ ] model can be build from GUI langchain builder - - [ ] langflow - - [ ] Flowise +
+
+
+
+ +**DELETE EVERYTHING ABOVE FOR YOUR PROJECT** + +______________________________________________________________________ + +
+ +# Your Project Name + +[![python](https://img.shields.io/badge/-Python_3.9_%7C_3.10-blue?logo=python&logoColor=white)](https://github.com/pre-commit/pre-commit) +[![hydra](https://img.shields.io/badge/Config-Hydra_1.3-89b8cd)](https://hydra.cc/) +[![Optuna](https://img.shields.io/badge/Optimize-Optuna-blue)](https://github.com/optuna/optuna) +[![black](https://img.shields.io/badge/Code%20Style-Black-black.svg?labelColor=gray)](https://black.readthedocs.io/en/stable/) +[![isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/) +Template
+ +
+ +## Description + +What it does and what the purpose of the project is. + +## Installation + +How to install the project. + +## How to run + +How to run the project.