[Roadmap] RAG

[thinkall]

Why RAG

Retrieval-Augmented Generation (RAG) is a technique that enhances the capabilities of LLMs by incorporating a retrieval mechanism into the generative process. This approach allows the model to leverage a vast amount of relevant information from a pre-existing knowledge base, which can significantly improve the quality and accuracy of its generated responses. Thus, for agents chat, incorporating a RAG agent offers several compelling advantages that can significantly enhance the performance and utility of your agent system.

RAG in AutoGen

AutoGen has provided RetrieveUserProxyAgent and RetrieveAssistantAgent for performing RetrieveChat in Aug, 2023 and announced it in blog in Oct, 2023. Given a set of documents, the Retrieval-augmented User Proxy first automatically processes documents—splits, chunks, and stores them in a vector database. Then for a given user input, it retrieves relevant chunks as context and sends it to the Retrieval-augmented Assistant, which uses LLM to generate code or text to answer questions. Agents converse until they find a satisfactory answer.

As both AutoGen and RAG are evolving very fast, we find that many users are asking for supports on customized vector databases, incremental document ingesting, customized retrieve/re-ranking algorithms, customized RAG pattern/workflow, etc. We've adjusted some of the issues and feature requests, such as we've added QdrantRetrieveUserProxyAgent for using qdrant as the vector db; we've integrated UNSTRUCTURED to support many unstructed documents. However, there are many more to do.

Our Plan

In order to better support RAG in AutoGen, we plan to refactor the existing RetrieveChat agents. The goals includes:

Primary goals

• Support launching RAG with one agent instead of two
• Support customizing vector databases with a parameter instead of extending agent class
• Support RAG in AutoGen Studio
• Support leveraging 3rd-party OSS tools
• Make RAG a capability for any conversable agent
• Support RAG as a tool like in OpenAI Assistant
• Make vector db dependency optional
• the chat interface of the RAG agent is the same as any other conversable agent

Optional goals

• Support async functions
• Support benchmarks
• Support evaluation

Tasks

Preview Give feedback

1. [Bug]: Retrieve Agents not working with function calls #1469
   bug rag +2
2. [Feature Request]: agent RAG #1387
   feature rag +2
3. [Bug]: Function calling in groupchat does not work #1440
   bug group chat/teams tool-usage +3
4. Fix issue 1440 by applying new function registration decorator #1661
   rag tool-usage +2
5. Refactor RAG agents with core functionalities #1726
   4 of 4
   rag +1
   
6. [Feature Request]: Support for different retrieval algorithms for RAG agents #1047
   0.2 feature +2
   
7. https://github.com/microsoft/autogen/discussions/484
8. Automatically decide whether RAG is needed
9. Add a score threshold for retriever
10. Add a vectordb module #2263
    +0
11. Add html parser for RAG and some improvements #2271
    rag +1
12. Add source to the answer for default prompt #2289
    +0
13. [Bug]: overlap parameter in the split_text_to_chunks not used. #1844
    bug rag +2
14. ChromaDB required even if you use QdrantRetrieveUserProxyAgent and not Chroma #531
    0.2 rag +2
    
15. How to add a new vector database? #725
    0.2 rag +2
    
16. Autogen Retrieval User Proxy generate_init_message bug #859
    0.2 rag +2
    
17. add RetrieveUserProxyAgent and RetrieveAssistantAgent from autogen studio #1723
    0.2 proj-studio rag +3
    
18. [Issue]: QdrantRetrieveUserProxyAgent is missing support for text-embedding-ada-002 embedding model #1282
    feature rag +2
19. [Issue][RAG] Reuse an existing vector database for RetrieveUserProxyAgent #1261
    rag +1
20. Support setting vector_db as a param #2313
    +0
21. Add documentation to user guide
22. Delete files from a collection?
23. Add better code parser
24. MongoDB + new abstraction of vectordb #2942
    rag +1
    
25. Add RAG Agent to AutoGen Studio #2881
    proj-studio rag vectordb +3
    
26. Improve update context condition checking rule #2883
    rag +1
27. Bugfix: PGVector/RAG - Calculate the Vector Size based on Model Dimensions #2865
    rag vectordb +2
    
28. PGVector Support for Custom Connection Object #2566
    rag vectordb +2
    
29. 2447 fix pgvector tests and notebook #2455
    bug rag vectordb +3
    
30. RetrieveUserProxyAgent, use context_max_tokens from retrieve_config if provided #2259
    rag +1
31. Fix unstructured deps installation error #2248
    rag +1
32. [Bug]: LlamaIndex checks failing #3046
    0.2 needs-triage +2
33. feat: Qdrant support for the VectorDB interface #3035
    +0
34. +mdb atlas vectordb [clean_final] #3000
    rag +1
35. Update qdrant notebook for new qdrant vectordb #3140
    rag +1
36. Update RAG blog, the custom vectordb section
37. Remove dependency on RetrieveAssistantAgent for RetrieveChat #3320
    rag +1
38. [Graph RAG] Init Commit with GraphRag interfaces #3388
    rag +1
    
39. [Roadmap]: Enhanced RAG Support #3119
    3 of 4
    roadmap +1

[Knucklessg1]

Hello, I was wondering if there was support coming for pgvector, if not I would be happy to contribute.

[julianakiseleva]

@thinkall

[thinkall]

Hello, I was wondering if there was support coming for pgvector, if not I would be happy to contribute.

Hi @Knucklessg1 , contribution is welcome, thank you for your interest!

[WaelKarkoub]

Hi @thinkall, would this PR, #2046, help out with Automatically decide whether RAG is needed?

I was thinking if the agent adds a tag like <rag context="some context"> in the message, we can intercept that by one of the hooks or even a reply, and then perform some rag operations

[thinkall]

Hi @thinkall, would this PR, #2046, help out with Automatically decide whether RAG is needed?

I was thinking if the agent adds a tag like <rag context="some context"> in the message, we can intercept that by one of the hooks or even a reply, and then perform some rag operations

Thank you @WaelKarkoub , interesting idea! Would adding mean RAG is already performed?

[WaelKarkoub]

@thinkall we could define what that tag means by adding attributes (e.g. <rag context="some context" task="search">could mean it needs to look through some databases) I'm not fully familiar with how rag works, but that tag system should be general enough for multiple use cases.

[ChristianWeyer]

Great initiative @thinkall.
How much are you thinking of reusing what has already been built and proven in other frameworks, like LangChain?

[thinkall]

Great initiative @thinkall. How much are you thinking of reusing what has already been built and proven in other frameworks, like LangChain?

Thank you @ChristianWeyer , I hope to support as many established features in the OSS as possible. So we're carefully thinking of the new design of the rag feature in AutoGen. Would you like to share your thoughts?

[ChristianWeyer]

Great initiative @thinkall. How much are you thinking of reusing what has already been built and proven in other frameworks, like LangChain?

Thank you @ChristianWeyer , I hope to support as many established features in the OSS as possible. So we're carefully thinking of the new design of the rag feature in AutoGen. Would you like to share your thoughts?

One thing is that there are so many connector & retriever implementations in LangChain, that it would not make sense to reinvent the wheel and trying to keep up. Same goes for embedding support.

[thinkall]

Great initiative @thinkall. How much are you thinking of reusing what has already been built and proven in other frameworks, like LangChain?

Thank you @ChristianWeyer , I hope to support as many established features in the OSS as possible. So we're carefully thinking of the new design of the rag feature in AutoGen. Would you like to share your thoughts?

One thing is that there are so many connector & retriever implementations in LangChain, that it would not make sense to reinvent the wheel and trying to keep up. Same goes for embedding support.

Agree!

Would you like to have a quick chat on this? It would be great to hear more from you!

[dsalas-crogl]

@thinkall Will the upcoming RAG update still require using message_generator in groupchat scenarios? It's my understanding that currently, the RAG agent has to initiate chat and message_generator has to be used, which results in all initial prompt messages being sent through retrieve_docs in RetrieveUserProxyAgent.

[ChristianWeyer]

Great initiative @thinkall. How much are you thinking of reusing what has already been built and proven in other frameworks, like LangChain?

Thank you @ChristianWeyer , I hope to support as many established features in the OSS as possible. So we're carefully thinking of the new design of the rag feature in AutoGen. Would you like to share your thoughts?

One thing is that there are so many connector & retriever implementations in LangChain, that it would not make sense to reinvent the wheel and trying to keep up. Same goes for embedding support.

Agree!

Would you like to have a quick chat on this? It would be great to hear more from you!

Sure. I am cethewe in AG Discord.

[thinkall]

Hello, I was wondering if there was support coming for pgvector, if not I would be happy to contribute.

Hi @Knucklessg1 , contribution is welcome, thank you for your interest!

Hi @Knucklessg1 , are you in our Discord channel? Could we have a quick chat? Thanks.

[thinkall]

@thinkall Will the upcoming RAG update still require using message_generator in groupchat scenarios? It's my understanding that currently, the RAG agent has to initiate chat and message_generator has to be used, which results in all initial prompt messages being sent through retrieve_docs in RetrieveUserProxyAgent.

Hi @dsalas-crogl , I'd like to remove the usage of message_generator, would that benefit your use case? Thanks.

Are you in our Discord channel?

[Knucklessg1]

Hello, I was wondering if there was support coming for pgvector, if not I would be happy to contribute.

Hi @Knucklessg1 , contribution is welcome, thank you for your interest!

Hi @Knucklessg1 , are you in our Discord channel? Could we have a quick chat? Thanks.

Yes absolutely. I reached out on Discord.

[jamesliu]

@thinkall any flow diagram regarding the rag?

[thinkall]

@thinkall any flow diagram regarding the rag?

Hi @jamesliu , there's one diagram here, you can find the workflow details in the Introduction section.

[Josephrp]

interesting roadmap , and i'm very happy with chromadb , looking forward to in memory vector store too , now. if anyone is interested it could be a good opportunity to collaborate and break down complex tasks .

i'll also consider creating + sharing an "advanced upsert" agent , which enriches the text chunks to improve retrieval performance.

[raolak]

Are RAG applications limited to document processing, or do they extend to code-related tasks as well? For instance:

• Existing code on which we need to run fix? or
• New codes (eg. new service) which goes through incremental development

Usecase:

• In a system comprised of numerous microservices that are regularly updated with new features or fixes, how can an agent acquire knowledge about these changes? Is RAG a viable method for this?
• For the development of new microservices within an existing system, how can knowledge be transferred to agents to enhance the design and implementation process using RAG?

[thinkall]

Are RAG applications limited to document processing, or do they extend to code-related tasks as well? For instance:

• Existing code on which we need to run fix? or
• New codes (eg. new service) which goes through incremental development

Usecase:

• In a system comprised of numerous microservices that are regularly updated with new features or fixes, how can an agent acquire knowledge about these changes? Is RAG a viable method for this?
• For the development of new microservices within an existing system, how can knowledge be transferred to agents to enhance the design and implementation process using RAG?

RAG can help if the documents are well organized. For pure code, currently, you can use 3rd party code chunk methods to help load the code into the vector dbs.

[ekzhu]

Let's also add documentation task to the roadmap? We should have a rag category under https://microsoft.github.io/autogen/docs/topics

[ChristianWeyer]

Do we also have a task on the roadmap for using custom embeddings? This is a very vital and important requirement for successful RAG.
Another would be to use a re-ranking model optionally to improve RAG results. @thinkall

[thinkall]

Do we also have a task on the roadmap for using custom embeddings? This is a very vital and important requirement for successful RAG. Another would be to use a re-ranking model optionally to improve RAG results. @thinkall

Custom embeddings are already supported and will also be supported in the new version.

Re-ranking may also be supported, but we may not implement the algorithms, instead we could support plugin different re-ranking models.

[maximedupre]

Are RAG applications limited to document processing, or do they extend to code-related tasks as well? For instance:

• Existing code on which we need to run fix? or
• New codes (eg. new service) which goes through incremental development

Usecase:

• In a system comprised of numerous microservices that are regularly updated with new features or fixes, how can an agent acquire knowledge about these changes? Is RAG a viable method for this?
• For the development of new microservices within an existing system, how can knowledge be transferred to agents to enhance the design and implementation process using RAG?

RAG can help if the documents are well organized. For pure code, currently, you can use 3rd party code chunk methods to help load the code into the vector dbs.

@thinkall Is it possible for you to provide an example of a 3rd party code chunk method? I'm very interested in extending the knowledge of an agent to my whole codebase :)

[raolak]

I’ve got some thoughts on how we could use agents to automate code generation +1 for Maxime, it would be really helpful to include a section on code chunking with examples to illustrate the working. Now, when it comes to generating code, agents shouldn't just spit out code, they should mimic the way engineers think and work in real life. Here’s what I mean: - Start by clearly defining the requirements—think OKRs that cascade from key results down to epics and stories. - Set up milestones and break down tasks among the agents involved. - Have each agent carry out their tasks and check in regularly to ensure everything’s on track, with room for human oversight when needed. - Provide visibility of OKR, milestones and tasks visibility at one place. Make a planner central for agent and human collaboration and progress tracking. - Keep the agent's execution isolated (in a separate process). May be a distributed workflow where each node can host one or more agents. Workflow is orchestrated through central planner contributed by agents and humans For the code workflow, we could see something like this: - Set up a new GitHub project with a clean, well-structured setup and isolated code (for new projects). - Handle resource creation, both on-premises and in the cloud. - Back it all up with a CI/CD system tailored for both on-premises and cloud environments. - Support incremental code commits through PR with CI/CD These steps would be helpful both for rolling out new features or fixes to existing projects and for starting fresh ones. I managed to get a mini reference implementation of a distributed key-value store (80%) using chatgpt(gpt4) and was able to build, test, and run the services locally (screenshot attached). I was experimenting with autogen to reproduce the steps that I have followed and see if I can achieve decent level autonomy (I am sure it will take many iterations :) ). I am still learning and experimenting. I will share my findings as I make progress. [image: Screenshot 2024-04-13 at 10.37.18 AM.png] Thanks for all the great work and support. Regards lnr

…

Message ID: ***@***.***>

[cforce]

@raolak

A solution for this has just been released
Checkout https://github.com/princeton-nlp/SWE-agent

[thinkall]

@thinkall Is it possible for you to provide an example of a 3rd party code chunk method? I'm very interested in extending the knowledge of an agent to my whole codebase :)

Hi @maximedupre , please check out an example of using 3rd party chunk method here: https://microsoft.github.io/autogen/blog/2023/10/18/RetrieveChat/#customizing-text-split-function

[tianlinzx]

Any plan to integrate with GraphRAG ??

[wammar]

@thinkall @qingyun-wu

Some of my team members recently started using AutoGen with RAG and are interested in contributing, but it is unclear what you're actively working on and what tasks do you need help with. It'd be great to collaborate if there's good alignment between this work stream and Holistic Intelligence for Global Good.

[thinkall]

Any plan to integrate with GraphRAG ??

We're working on it.
cc @randombet , @2152505

[thinkall]

@thinkall @qingyun-wu

Some of my team members recently started using AutoGen with RAG and are interested in contributing, but it is unclear what you're actively working on and what tasks do you need help with. It'd be great to collaborate if there's good alignment between this work stream and Holistic Intelligence for Global Good.

Thank you very much, @wammar, for your feedbacks! The tasks list contains RAG related issues and PRs we're working on. You're very welcome to raise PRs for resolving existing issues or propose new features such as new vector dbs, new retrieve util functions (file parsing, chunking, etc.), or review PRs.

Any thoughts, suggestions, comments are very welcome.

[zhwuwuwu]

Any plan to integrate with GraphRAG ??

We're working on it. cc @randombet

Is there any progress on it？

[devspacenine]

How will the RAG pattern change with the new v0.4 architecture?