FastAPI autogen asynchronous session not working

[gvijqb]

Hey guys. I am trying to implement an asynchronous thread based workflow for handling multiple chat sessions in parallel. But my fast API stalls when a code is being executed or a function is being run in a thread by autogen groupchat. The API starts giving 504 gateway timeout.

How should I handle this?

[rysweet]

@ekzhu any thoughts?

[ekzhu]

I am sorry can you provide code, model, package version, Python version

[gvijqb]

The first section in the link talks about docker...

For remote server, you can take a look at: https://microsoft.github.io/autogen/dev/user-guide/extensions-user-guide/azure-container-code-executor.html

It is also simple to integrate with any other providers, you just use their client directly.

Thanks. This appears to be useful. Will try it out.

Also, is there a way to completely segment the execution out of the chat? Such that we have a let's say monitoring agent whose job is to monitor the code execution and it can take necessary actions to stop or retry if needed. And this approach can be more microservice oriented. Where code execution is a separate microservice running outside the chat conversation. Any thoughts on this?

[ekzhu]

Do you mean running code execution as a tool? You can certainly wrap the executor into a function and use it s a tool

[gvijqb]

Do you mean running code execution as a tool? You can certainly wrap the executor into a function and use it s a tool

As a tool it can be helpful but it won't enable the agent to monitor what's happening in the code execution. Monitoring is especially helpful in long running code execution that might take 5-6 hours.

[ekzhu]

In that case the code execution should be performed on a long-running container that emits events to be tracked by some agents, which then make decisions such as whether to cancel the operations or send further events to the code execution container.

Ideally, your code execution container should be hosted in a separate service. Sounds like a long-running data processing or machine learning training stuff to me.

[gvijqb]

I see. That's interesting. So you're suggesting that I have a separate docker container that has like a pub-sub architecture workflow for publishing messages as the code snippets are getting executed? And the container itself has a subscriber collecting any messages published for code execution by an agent tool call?