AutoGen Studio and AutoGen v0.4 [Status Updates, Discussion]

[victordibia]

Hi all,

We are rewriting AutoGen Studio to take advantage of new features coming in AutoGen 0.4, as outlined in our readme. The goal of AutoGen Studio remains to create a tool to rapidly prototype, and debug multi-agent applications. This post updates developers on our progress and provides information for those interested in contributing or testing.

Installation Options

You can find the current version's code in autogen/python/packages/autogen-studio. There are two ways to install:

1. Recommended Method: Install the latest dev version from PyPI:

pip install autogenstudio==0.4.0.dev44

run the app

autogenstudio ui --appdir new

The command above tells AGS to store files in the new directory (avoiding conflicts with previous versions)

if you would like to force autoupgrade your db (still experimental)

autogenstudio ui --upgrade-database

2. Advanced Method: Install from source by following the AutoGen Studio readme. Note that this method is more complex and better suited for experienced users.

Development Status

While we're still in early development, you can view our planned features here.

Some Key Improvements

ags_update.mp4

The new AutoGen v0.4 API introduces asynchronous operation, which enables seamless message passing between agents. This enhancement particularly benefits end-user applications by adding several crucial features that were difficult to implement in the previous version:

New Features Already Available:

• Real-time agent updates streaming to the frontend
• Run control: You can now stop agents mid-execution if they're heading in the wrong direction, adjust the team, and continue
• Interactive feedback: Add a UserProxyAgent to get human input through the UI during team runs
• Message flow visualization: See how agents communicate with each other
• Ability to import specifications from external galleries
• Ability to wrap agent teams into an API using the AutoGen Studio CLI

Features Coming Soon:

See the roadmap here for planned features. Some include:

• Implement a drag and drop team builder interface for AGS #4202
• Ability to attach 3rd party or local gallery of components to AGS #4203

We welcome your feedback and contributions if you've had a chance to try the new version.

[SailorJoe6]

Just wanted to say, this looks pretty amazing!

[victordibia]

Thanks! @SailorJoe6, I seem to have gotten a notification on a session issue you reported, but cant seem to find that right now.
Any chance you can point me to it (assuming there was indeed an issue)?
Many thanks

[inayatwali]

This is great!

[SailorJoe6]

Last issue I reported was on the old version. Nothing reported against this one.

[victordibia]

Got it, thank you!

[mkarpovi-amd]

Attempting to create a team gives me an error reading: "POST /api/teams/ HTTP/1.1" 422 Unprocessable Entity. I made a clean Conda environment and installed ags v.0.4.0.dev37 with pip. Do you have any idea what might cause this?

[victordibia]

Thanks. Let me debug this a bit more in a new GH spaces env.
Will post any updates.

[victordibia]

I am unable to replicate this issue in fresh GH space.

autogenstudio ui --appdir new

Runs the app in a new directory, teams can be created and tasks run (after an openai_api_key is set).

Any chance you can share full logs from autogenstudio ui --appdir new

[mkarpovi-amd]

I seem to get a lot of messages like this:

INFO: Will watch for changes in these directories: ['C:\Users\mkarpovi']
INFO: Uvicorn running on http://127.0.0.1:7002 (Press CTRL+C to quit)
WARNING: --reload-include and --reload-exclude have no effect unless watchfiles is installed.
INFO: Started reloader process [636724] using StatReload
2024-11-22 14:20:49.704 | INFO | autogenstudio.web.initialization:init:38 - Initialized application data folder: new
INFO: Started server process [624616]
INFO: Waiting for application startup.
2024-11-22 14:20:49.738 | INFO | autogenstudio.web.app:lifespan:34 - Initializing application...
2024-11-22 14:20:49.746 | INFO | autogenstudio.database.schema_manager:init:64 - Initializing new Alembic configuration
2024-11-22 14:20:49.748 | INFO | autogenstudio.database.schema_manager:_initialize_alembic:141 - Initializing Alembic configuration...
2024-11-22 14:20:49.751 | INFO | autogenstudio.database.schema_manager:_initialize_alembic:152 - Created new env.py
2024-11-22 14:20:49.752 | INFO | autogenstudio.database.schema_manager:_initialize_alembic:158 - Created alembic.ini
2024-11-22 14:20:49.767 | INFO | autogenstudio.database.db_manager:init:56 - Schema status: Schema needs upgrade but auto_upgrade is disabled. Status: Unmigrated changes detected:
('add_table', Table('agent', MetaData(), Column('id', Integer(), table=, primary_key=True, nullable=False), Column('created_at', DateTime(timezone=True), table=, server_default=DefaultClause(<sqlalchemy.sql.functions.now at 0x287a457dd00; now>, for_update=False)), Column('updated_at', DateTime(timezone=True), table=, onupdate=ColumnElementColumnDefault(<sqlalchemy.sql.functions.now at 0x287a457df70; now>)), Column('user_id', AutoString(), table=), Column('version', AutoString(), table=, default=ScalarElementColumnDefault('0.0.1')), Column('config', JSON(), table=), schema=None))
('add_table', Table('model', MetaData(), Column('id', Integer(), table=, primary_key=True, nullable=False), Column('created_at', DateTime(timezone=True), table=, server_default=DefaultClause(<sqlalchemy.sql.functions.now at 0x287a4542450; now>, for_update=False)), Column('updated_at', DateTime(timezone=True), table=, onupdate=ColumnElementColumnDefault(<sqlalchemy.sql.functions.now at 0x287a457cb60; now>)), Column('user_id', AutoString(), table=), Column('version', AutoString(), table=, default=ScalarElementColumnDefault('0.0.1')), Column('config', JSON(), table=), schema=None))
('add_table', Table('team', MetaData(), Column('id', Integer(), table=, primary_key=True, nullable=False), Column('created_at', DateTime(timezone=True), table=, server_default=DefaultClause(<sqlalchemy.sql.functions.now at 0x287a457c9e0; now>, for_update=False)), Column('updated_at', DateTime(timezone=True), table=, onupdate=ColumnElementColumnDefault(<sqlalchemy.sql.functions.now at 0x287a457d340; now>)), Column('user_id', AutoString(), table=), Column('version', AutoString(), table=, default=ScalarElementColumnDefault('0.0.1')), Column('config', JSON(), table=), schema=None))
('add_table', Table('tool', MetaData(), Column('id', Integer(), table=, primary_key=True, nullable=False), Column('created_at', DateTime(timezone=True), table=, server_default=DefaultClause(<sqlalchemy.sql.functions.now at 0x287a4127140; now>, for_update=False)), Column('updated_at', DateTime(timezone=True), table=, onupdate=ColumnElementColumnDefault(<sqlalchemy.sql.functions.now at 0x287a442a000; now>)), Column('user_id', AutoString(), table=), Column('version', AutoString(), table=, default=ScalarElementColumnDefault('0.0.1')), Column('config', JSON(), table=), schema=None))
('add_table', Table('agentmodellink', MetaData(), Column('agent_id', Integer(), ForeignKey('agent.id'), table=, primary_key=True, nullable=False), Column('model_id', Integer(), ForeignKey('model.id'), table=, primary_key=True, nullable=False), Column('sequence', Integer(), table=, primary_key=True, nullable=False, default=ScalarElementColumnDefault(0)), schema=None))
('add_table', Table('agenttoollink', MetaData(), Column('agent_id', Integer(), ForeignKey('agent.id'), table=, primary_key=True, nullable=False), Column('tool_id', Integer(), ForeignKey('tool.id'), table=, primary_key=True, nullable=False), Column('sequence', Integer(), table=, primary_key=True, nullable=False, default=ScalarElementColumnDefault(0)), schema=None))
('add_table', Table('session', MetaData(), Column('id', Integer(), table=, primary_key=True, nullable=False), Column('created_at', DateTime(timezone=True), table=, server_default=DefaultClause(<sqlalchemy.sql.functions.now at 0x287a45c8080; now>, for_update=False)), Column('updated_at', DateTime(timezone=True), table=, onupdate=ColumnElementColumnDefault(<sqlalchemy.sql.functions.now at 0x287a45c9eb0; now>)), Column('user_id', AutoString(), table=), Column('version', AutoString(), table=, default=ScalarElementColumnDefault('0.0.1')), Column('team_id', Integer(), ForeignKey('team.id'), table=), Column('name', AutoString(), table=), schema=None))
('add_table', Table('teamagentlink', MetaData(), Column('team_id', Integer(), ForeignKey('team.id'), table=, primary_key=True, nullable=False), Column('agent_id', Integer(), ForeignKey('agent.id'), table=, primary_key=True, nullable=False), Column('sequence', Integer(), table=, primary_key=True, nullable=False, default=ScalarElementColumnDefault(0)), schema=None))
('add_table', Table('run', MetaData(), Column('id', Uuid(), table=, primary_key=True, nullable=False, default=CallableColumnDefault(<function uuid4 at 0x00000287A46107C0>)), Column('created_at', DateTime(timezone=True), table=, server_default=DefaultClause(<sqlalchemy.sql.functions.now at 0x287a45ca4b0; now>, for_update=False)), Column('updated_at', DateTime(timezone=True), table=, onupdate=ColumnElementColumnDefault(<sqlalchemy.sql.functions.now at 0x287a45ca7b0; now>)), Column('session_id', Integer(), ForeignKey('session.id'), table=, nullable=False), Column('status', Enum('CREATED', 'ACTIVE', 'COMPLETE', 'ERROR', 'STOPPED', name='runstatus'), table=, nullable=False, default=ScalarElementColumnDefault(<RunStatus.CREATED: 'created'>)), Column('error_message', AutoString(), table=), Column('run_meta', JSON(), table=), Column('version', AutoString(), table=, default=ScalarElementColumnDefault('0.0.1')), schema=None))
('add_index', Index('ix_run_id', Column('id', Uuid(), table=, primary_key=True, nullable=False, default=CallableColumnDefault(<function uuid4 at 0x00000287A46107C0>))))
('add_table', Table('message', MetaData(), Column('id', Integer(), table=, primary_key=True, nullable=False), Column('created_at', DateTime(timezone=True), table=, server_default=DefaultClause(<sqlalchemy.sql.functions.now at 0x287a457f6b0; now>, for_update=False)), Column('updated_at', DateTime(timezone=True), table=, onupdate=ColumnElementColumnDefault(<sqlalchemy.sql.functions.now at 0x287a457f3b0; now>)), Column('user_id', AutoString(), table=), Column('version', AutoString(), table=, default=ScalarElementColumnDefault('0.0.1')), Column('config', JSON(), table=), Column('session_id', Integer(), ForeignKey('session.id'), table=), Column('run_id', Uuid(), ForeignKey('run.id'), table=), Column('message_meta', JSON(), table=), schema=None))
2024-11-22 14:20:49.882 | INFO | autogenstudio.database.db_manager:create_db_and_tables:128 - Database tables created successfully
2024-11-22 14:20:49.884 | INFO | autogenstudio.web.app:lifespan:38 - Managers initialized successfully
2024-11-22 14:20:49.884 | INFO | autogenstudio.web.app:lifespan:41 - Application startup complete
INFO: Application startup complete.

The other issue is that I intend to do LLM inference through a local corporate gateway (which i accessed on 0.1.5 by modifying source to pass the relevant headers)--but I don't think I have got to the relevant point yet.

[victordibia]

I agree that we have not gotten to the LLM part yet (there is some work to be done there as its mostly the OpenAI client supported now).

I just made a dev release - 0.4.0.dev38 that fixes the Unmigrated changes detected issues above (though I dont think that should be what is preventing teams from being created).

pip install autogenstudio==0.4.0.dev38 

[victordibia]

@mkarpovi-amd .. still having issues?

[rajramakalai]

Can you help with how to define a ollama model on the json structure please
"model_client": {
"component_type": "model",
"model": "gpt-4o-2024-08-06",
"model_type": "OpenAIChatCompletionClient"
}

[ekzhu]

You need to add model capabilities.

[rajramakalai]

Got it working, Thank you, but running from source, and adding the model names to the _model_info.py
with model client as
"model_client": {
"api_key": ".",
"base_url": "http://localhost:11434/v1",
"component_type": "model",
"model": "phi3.5",
"model_type": "OpenAIChatCompletionClient"
}

[asheeshgarg]

I have pip install dev38 I am trying to use AzureOpenAIChatCompletionClient what all env varibale need to set or is there any way to set in studio directly?

[victordibia]

Hi @asheeeshgarg,

There is a bit of work to be done here.
Mostly around supporting the AzureOpenAI client Params as defined here https://microsoft.github.io/autogen/dev/user-guide/core-user-guide/framework/model-clients.html#azure-openai

I’ll take a look later today or tomorrow and provide updates

[lbalceda]

Hello @victordibia any updates on this one ? I also tried setting the various params but it seems to be requiring an OpenAIChatCompletionClient currently

[victordibia]

Hi @lbalceda ,

This took a while but now AOAI client spec with an api_key field is now supported.
Docs here

{
  "model": "gpt-4o-mini",
  "model_type": "AzureOpenAIChatCompletionClient",
  "azure_deployment": "gpt-4o-mini",
  "api_version": "2024-02-15-preview",
  "azure_endpoint": "https://your-endpoint.openai.azure.com/",
  "api_key": "your-api-key",
  "component_type": "model"
}

You will need the latest version https://pypi.org/project/autogenstudio/#history

[joslat]

As mentioned at https://www.linkedin.com/posts/dibiavictor_multiagentsystems-multiagentbook-autogen-activity-7271945150925299712-4g_u?utm_source=share&utm_medium=member_desktop I'd like to beta-test and provide feedback - Is there anything additional to the instructions already stated here?
Or just install and "let the games begin"?

If I've got the "old one" in place, any recommendation on how to properly "clean up" previously?

One additional question, is there a timeline where this will be "released"?

[harrywang]

I am also curious about the release date. thanks!

[victordibia]

Hi @joslat ,

The installation instructions are as above. A new conda env and using the --appdir flag is recommended.
Release date ~ Jan 15.
Roadmap is here #4006

[Knucklessg1]

This is so cool, I missed this update as I was working on the forked repository build-with-ag2. What are the plans on moving Studio to its own Repo?
It'll help avoid this confusion 😅

[victordibia]

@Knucklessg1 ,I agree that there should be an easy way to keep track of progress. Perhaps a separate repo might be valuable at some point. We will consider this.
In the meantime, please try out the new capabilities and share feedback!
v1 for the drag and drop composition feature just landed yesterday - what do you think?

ags_1080.mp4

[harrywang]

I can't agree more - a separate repo should be much better - it's hard to follow the updates in the current setup.

[Knucklessg1]

@victordibia I would like to add a section for RAG. Would that be a "Tool" item that can be added to only User Proxy Agents? Or would that be it's own section?

[victordibia]

Hi @Knucklessg1 ,
I agree that supporting some type of generalized RAG pattern would extremely useful.
What do you think of the proposal and plan in issue #4707
#4707

cc @ekzhu

[ekzhu]

A stateful Tool for RAG is a good start and super easy to integrate.

[lepulepu1198]

Hi there, I'm a university student from China. Can't wait to apply this amazing framework into my research!

I'm wondering if model: qwen by Aliyun could be supported in the future? I've tried to link it to this framework but failed in the end.

[lepulepu1198]

Thanks for reply, but the Qwen model is running by Alibaba Cloud rather than working on my local server. It used to work at previous version of 0.1.5.

What I did in the past to call this model is just fill the api key, model name and base_url in version of 0.1.5; But it seems that no place for me to do the same thing.

THANKS!

[victordibia]

Yes, the updated model client needs the model_capabilities field.
As long as your model is OpenAI compliant, the solution I mentioned above should work (after you add model_capabilities) ... it does not have to be a local model. Any model API endpoint that is openai compliant should work.
This link says the Alibaba ChatCompletion endpoints are openai compliant, so it should work

"model": "qwen-max-latext",
  "model_type": "OpenAIChatCompletionClient",
  "base_url": "https://dashscope-intl.aliyuncs.com/compatible-mode/v1", #..replace this
  "api_version": "1.0",
  "component_type": "model",
  "model_capabilities": {
    "vision": false,
    "function_calling": false,
    "json_output": false
  }

What errors are you seeing?
You can also open a new discussion thread for us to discuss this more

[lepulepu1198]

Thanks for the reply,

I've changed the code as you said, and firstly it requires me to allow the function calling capability.

Once I set "true" for function_calling, there is no reported errors in the playground, but I can see errors in the terminal of pycharm, which shows that my api_key is invalid.

Thanks for your help again!

[victordibia]

Good progress. I think we are almost there.

• The error suggests that it tried to hit the OpenAI endpoint ....(this is not what we want)
• To provide the right endpoint, we need to set the base_url. I see in your screenshot that you have not set the base URL.
• You will need to set base url to your alibaba cloud base url"base_url": "https://dashscope-intl.aliyuncs.com/compatible-mode/v1",

Can you confirm you have set it?

[lepulepu1198]

Appreciate it, but this is what happens after setting the "base_url" that you provided.

Then I tried to check the instructions on Alibaba cloud, it shows the "base_url" I need to use is "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1".

But here is what happened after changing the base_url.

[brandonscottbarney]

You mentioned on X that "If you’d like to beta test and provide feedback (or are already beta testing) , comment on the GitHub discussion thread in the comment section below," so I'm very interested as I just came across your research scrolling through NeurIPS2024 posts

[victordibia]

Hi @brandonscottbarney ,

Thanks! The installation instructions are exactly as above nothing else required ...

---

1. Recommended Method: Install the latest dev version from PyPI:

pip install autogenstudio==0.4.0.dev41

run the app

autogenstudio ui --appdir new

The command above tells AGS to store files in the new directory (avoiding conflicts with previous versions)

if you would like to force autoupgrade your db (still experimental)

autogenstudio ui --upgrade-database

[sean-josiah7]

Suggestion: on project/task executions... make Orcehstrator plans moderated/editable in interface by human in the loop.
On completion... Have the orchestator perform an afteraction analysis, looking to improve the prompt, the plan and product, convert this summarize where possible, abstract into a recipe/blueprint with configurable elements.

[ekzhu]

This is a great suggestion. Could you open an issue with the suggestions?

This requirement touches on the AgentChat library which runs the backend of studio. You can take a look at the orchestrator here:

python/packages/autogen-agentchat/src/autogen_agentchat/teams/_group_chat/_magentic_one

[victordibia]

I agree with Eric. There are several good ideas that could be come patterns that we support in AgentChat.

• A planner orchestrator (iterates with users on a plan before exploring the task to a set of fixed teams)
• An automatic orchestrator (iterates on plan with user, defines teams automatically etc)

[sean-josiah7]

done: #4764

…

On Thu, Dec 19, 2024 at 1:35 PM Victor Dibia ***@***.***> wrote: I agree with Eric. There are several good ideas that could be come patterns that we support in AgentChat. - A planner orchestrator (iterates with users on a plan before exploring the task to a set of fixed teams) - An automatic orchestrator (iterates on plan with user, defines teams automatically etc) — Reply to this email directly, view it on GitHub <#4208 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXSWEMPW3BI2FKXW4HFA4CL2GMGWLAVCNFSM6AAAAABR2JZNSCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNRSGA4DCNA> . You are receiving this because you commented.Message ID: ***@***.***>