Implement conversation API endpoint #188

streamer45 · 2024-05-27T20:17:04Z

Description

PR implements a bots-only /conversation API endpoint to allow other plugins to programmatically start and continue conversations with Copilot without resorting to hacks such as posting in DMs between bots.

The general idea is to leverage the AI plugin as much as possible while eliminating the requirement of a real thread existing in a channel.

The proposed handler expects a list of posts (i.e., a virtual thread) that will be fed as context. The last post in the list will be the current request.

Example

curl -v -H 'Authorization: Bearer BOT_AUTH_TOKEN' http://localhost:8065/plugins/mattermost-ai/conversation --data @payload.json

{
    "bot_name": "ai",
    "thread": [
    {
        "id": "a",
        "message": "There's an incident ongoing",
        "channel_id": "oi18bftr738ntn35xnyjyed9uh",
        "user_id": "zjm6x7y4n3gjzfj7kw3xp9c9se",
        "create_at": 1
    },
    {
        "id": "b",
        "message": "A customer is experiencing db query timeouts",
        "channel_id": "oi18bftr738ntn35xnyjyed9uh",
        "user_id": "zjm6x7y4n3gjzfj7kw3xp9c9se",
        "create_at": 2
    }
    ],
    "request":     {
        "id": "c",
        "message": "can you summarize the current conversation?",
        "channel_id": "oi18bftr738ntn35xnyjyed9uh",
        "user_id": "zjm6x7y4n3gjzfj7kw3xp9c9se",
        "create_at": 3
    },
    "use_system_role": true
}

streamer45 · 2024-05-27T20:17:29Z

server/api_conversation.go

+		return
+	}
+
+	bot := p.GetBotMentioned(post.Message)


I am keeping the message requirements intact, such as needing to mention a specific bot.

Maybe we should take the bot name as a parameter instead of requiring a mention in the post?

Yeah, given #188 (comment), I think it makes sense to make this a bit more structured. It may also be cleaner to decouple the current request/prompt from any past context.

streamer45 · 2024-05-27T20:18:01Z

server/api.go

@@ -26,6 +26,8 @@ func (p *Plugin) ServeHTTP(c *plugin.Context, w http.ResponseWriter, r *http.Req
 	router.GET("/ai_threads", p.handleGetAIThreads)
 	router.GET("/ai_bots", p.handleGetAIBots)

+	router.POST("/conversation", p.handlePostConversation)


I'm not overly worried about naming or scoping. Just started with this to get something working.

streamer45 · 2024-05-27T20:18:46Z

server/api_conversation.go

+			// Flushing lets us stream partial results without requiring the client to wait for the full response.
+			c.Writer.Flush()


For streaming to work correctly we'll need mattermost/mattermost@51f4271.

streamer45 · 2024-05-27T20:21:59Z

server/api_conversation.go

+		return
+	}
+
+	var conv []*model.Post


I thought a simple slice would be clean enough, but I'm happy to consider a more advanced approach, such as passing some sort of custom object that wraps the data we need.

Maybe we want to just wrap that in a struct so we can introduce other parameters if need be without a breaking change.

Yep, that's probably a healthier choice in the long run. Will update 👍

streamer45 · 2024-05-27T20:24:00Z

@crspeller Not urgent. I'd love to get some feedback. This is the minimum addition I could think of that would meet the basic requirements. Happy to review any of it.

crspeller

I suspect you will probably want a way to manipulate the system message using this API. Instructions to the bot are more powerful from the system message.

crspeller · 2024-05-27T23:14:06Z

server/api_conversation.go

+func (p *Plugin) handlePostConversation(c *gin.Context) {
+	userID := c.GetHeader("Mattermost-User-Id")
+
+	// We only allow bots to use this API handler for the time being.


If you are using the inter-plugin API to use this API we could restrict it to other plugins for now with the header: https://github.com/mattermost/mattermost-plugin-github/blob/5aa2450cc65254054872fc3ca6917b1c9354d543/server/plugin/api.go#L265

Thanks. I'll look into it although that would probably mean an additional roundtrip in my case (i.e. go through the plugin) since the bot is just a client running outside of Mattermost.

crspeller · 2024-05-29T22:06:42Z

server/api_conversation.go

+		return
+	}
+
+	var conv []*model.Post


Maybe we want to just wrap that in a struct so we can introduce other parameters if need be without a breaking change.

crspeller · 2024-05-29T22:07:59Z

server/api_conversation.go

+		return
+	}
+
+	bot := p.GetBotMentioned(post.Message)


Maybe we should take the bot name as a parameter instead of requiring a mention in the post?

streamer45 · 2024-06-14T14:40:58Z

I suspect you will probably want a way to manipulate the system message using this API. Instructions to the bot are more powerful from the system message.

@crspeller Slowly getting back to this. Are you referring to the prompt used?

crspeller · 2024-06-14T14:45:36Z

@streamer45 Yes having a way to specify the system prompt

server/api_conversation.go

streamer45 · 2024-06-14T16:13:41Z

@crspeller Updated. Please give it another pass whenever you get a chance.

crspeller · 2024-06-25T18:28:49Z

server/api_conversation.go

+	prompt.AppendConversation(p.ThreadToBotConversation(bot, threadData.Posts))
+
+	// Overriding post role if requested.
+	if reqData.UseSystemRole {


Sorry I think I was unclear about what this means. The system prompt is special instructions to the LLM that are weighted more highly then user instructions. Usually there is only one system prompt. You can look at OpenAI's explanation of what it means here: https://platform.openai.com/docs/guides/text-generation/chat-completions-api

I guess what I really want is to be able to call this API from other plugins and be able to customize what prompt is provided on line 122. So that we can provide some custom instructions to the LLM about how the rest of the conversation should be handled like telling it it's in a call and talking live, the participants, etc.

@crspeller Are you thinking to go as far as letting plugins pass/override a whole prompt template or more like letting them choose which prompt to use and possibly augment existing ones?

I am imagining them passing the whole prompt template. That way they can do whatever they want. Ideally they would be able to utilize the default personalities and whatnot as well.

Implement conversation API endpoint

60b7a34

streamer45 self-assigned this May 27, 2024

streamer45 commented May 27, 2024

View reviewed changes

crspeller requested changes May 29, 2024

View reviewed changes

Merge remote-tracking branch 'origin/master' into calls-ai

1b087cc

streamer45 added 2 commits June 14, 2024 17:11

Use structured request payload

7c11691

Allow overriding role

8043537

streamer45 commented Jun 14, 2024

View reviewed changes

server/api_conversation.go Show resolved Hide resolved

streamer45 requested a review from crspeller June 14, 2024 16:13

crspeller requested changes Jun 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement conversation API endpoint #188

Implement conversation API endpoint #188

streamer45 commented May 27, 2024 •

edited

Loading

streamer45 May 27, 2024

crspeller May 29, 2024

streamer45 May 29, 2024

streamer45 May 27, 2024

streamer45 May 27, 2024

streamer45 May 27, 2024

crspeller May 29, 2024

streamer45 May 29, 2024

streamer45 commented May 27, 2024

crspeller left a comment

crspeller May 27, 2024

streamer45 May 29, 2024

crspeller May 29, 2024

crspeller May 29, 2024

streamer45 commented Jun 14, 2024

crspeller commented Jun 14, 2024

streamer45 commented Jun 14, 2024

crspeller Jun 25, 2024

streamer45 Jul 26, 2024

crspeller Jul 29, 2024

		// Flushing lets us stream partial results without requiring the client to wait for the full response.
		c.Writer.Flush()

Implement conversation API endpoint #188

Are you sure you want to change the base?

Implement conversation API endpoint #188

Conversation

streamer45 commented May 27, 2024 • edited Loading

Description

Example

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

streamer45 commented May 27, 2024

crspeller left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

streamer45 commented Jun 14, 2024

crspeller commented Jun 14, 2024

streamer45 commented Jun 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

streamer45 commented May 27, 2024 •

edited

Loading