addTool in Relay Server Appears to Never Trigger Tool Handlers

[muradsofi]

Description

We are trying to use the addTool function in the relay server's instance of RealtimeClient, but it seems the tool handlers are never triggered when the tool is invoked. Instead, the tool logic appears to be bypassed entirely.

Here's a minimal code example demonstrating the issue:

---

Code Example

import { WebSocketServer, WebSocket } from "ws";
import { RealtimeClient } from "@openai/realtime-api-beta";
import { personalInstruction } from "../constants/personalInstruction";

export type RealtimeEvent = {
  type: string;
  [key: string]: unknown; // Allow for additional properties in the event object
};

export class RealtimeRelay {
  private apiKey: string;

  private wss: WebSocketServer | null;

  constructor(apiKey: string) {
    this.apiKey = apiKey;
    this.wss = null;
  }

  listen(port: number): void {
    this.wss = new WebSocketServer({ port });
    this.wss.on("connection", this.connectionHandler.bind(this));
    this.log(`Listening on ws://localhost:${port}`);
  }

  private async connectionHandler(
    ws: WebSocket,
    req: { url?: string; headers: { host?: string } },
  ): Promise<void> {
    if (!req.url) {
      this.log("No URL provided, closing connection.");
      ws.close();
      return;
    }

    const url = new URL(req.url, `http://${req.headers.host}`);
    const { pathname } = url;

    if (pathname !== "/") {
      this.log(`Invalid pathname: "${pathname}"`);
      ws.close();
      return;
    }

    // Instantiate new client
    this.log(`Connecting with key "${this.apiKey.slice(0, 3)}..."`);
    const client = new RealtimeClient({ apiKey: this.apiKey });

    client.addTool(
      {
        name: "get_weather",
        description: "Retrieve weather information for a given location.",
        parameters: {
          type: "object",
          properties: {
            location: {
              type: "string",
              description: "Name of the location to retrieve weather for.",
            },
          },
          required: ["location"],
        },
      },
      async ({ location }: { location: string }) => {
        console.log("get_weather tool called with location: ", location);
      },
    );

    client.updateSession({
      instructions: personalInstruction,
      input_audio_transcription: { model: "whisper-1" },
      turn_detection: { type: "server_vad" },
      tool_choice: "required",
    });

    // Relay: OpenAI Realtime API Event -> Browser Event
    client.realtime.on("server.*", (event: RealtimeEvent) => {
      this.log(`Relaying "${event.type}" to Client`);
      ws.send(JSON.stringify(event));
    });
    client.realtime.on("close", () => ws.close());

    // Relay: Browser Event -> OpenAI Realtime API Event
    const messageQueue: string[] = [];
    const messageHandler = (data: string) => {
      try {
        const event = JSON.parse(data) as { type: string };
        this.log(`Relaying "${event.type}" to OpenAI`);

        if (!client.isConnected()) {
          throw new Error("RealtimeClient is not connected");
        }

        client.realtime.send(event.type, event);
      } catch (e) {
        const error = e instanceof Error ? e.message : "Unknown error";
        console.error(error);
        this.log(`Error parsing event from client: ${data}`);
        this.log(`Error processing message from client: ${error}`);
      }
    };

    ws.on("message", (data: string | Buffer) => {
      const message = typeof data === "string" ? data : data.toString();
      if (!client.isConnected()) {
        messageQueue.push(message);
      } else {
        messageHandler(message);
      }
    });

    ws.on("close", () => {
      this.log("WebSocket closed. Disconnecting client...");
      messageQueue.length = 0; // Clear the message queue
      return client.disconnect();
    });

    // Connect to OpenAI Realtime API
    try {
      this.log(`Connecting to OpenAI...`);
      await client.connect();
    } catch (e) {
      const error = e instanceof Error ? e.message : "Unknown error";
      this.log(`Error connecting to OpenAI: ${error}`);
      ws.close();
      return;
    }

    this.log(`Connected to OpenAI successfully!`);
    while (messageQueue.length) {
      const message = messageQueue.shift()!;
      if (client.isConnected()) {
        messageHandler(message);
      } else {
        this.log("Client disconnected. Re-queuing message.");
        messageQueue.unshift(message); // Re-queue the message
        break; // Exit the loop until reconnection
      }
    }
  }

  public async testConnectionHandler(
    ws: WebSocket,
    req: { url?: string; headers: { host?: string } },
  ): Promise<void> {
    await this.connectionHandler(ws, req);
  }

  // eslint-disable-next-line class-methods-use-this
  private log(...args: unknown[]): void {
    console.log(`[RealtimeRelay]`, ...args);
  }
}

---

Expected Behavior

When the get_weather tool is invoked, the handler defined in addTool should be triggered, and its logic (e.g., logging "get_weather tool called with location: ") should execute.

---

Observed Behavior

The tool appears to be registered, but its handler is never triggered when the tool is called.

---

Questions

1. Is this behavior expected?
2. Should addTool be supported in relay server environments, or is there a limitation in how tools are executed server-side?
3. Is there a specific configuration required to enable addTool in the relay server?

Thank you for looking into this! Let me know if additional details or test cases are needed.

[muradsofi]

I previously reported an issue where tools registered using addTool in the server-side context (via the relay server's instance of RealtimeClient) were not triggering their handlers when invoked. I followed up with a patch as per the approach mentioned [here](#14 (comment)).

---

Steps Taken

1. Installed patch-package:

   • Command: npm i patch-package --save-dev.
   • Added @openai+realtime-api-beta+0.0.0.patch to the patches folder.
   • Updated package.json with the "postinstall": "patch-package" script.

2. Applied the patch:

   • This resolved the issue for tools registered via addTool when invoked from the client side.

---

Persisting Issue

• When addTool is used on the server side, the tool's handler is not triggered.

• However, if the tool description (not the function) is added client-side in the updateSession call, the server-side handler for the tool begins to work. Example:

  client.updateSession({
      turn_detection: { type: 'server_vad' },
      tool_choice: 'auto',
      tools: [
          {
              type: 'function',
              name: 'get_weather',
              description: 'Retrieve current weather information for a given location.',
              parameters: {
                  type: 'object',
                  properties: {
                      lat: {
                          type: 'number',
                          description: 'Latitude of the location.',
                      },
                      lng: {
                          type: 'number',
                          description: 'Longitude of the location.',
                      },
                      location: {
                          type: 'string',
                          description: 'Name of the location.',
                      },
                  },
                  required: ['lat', 'lng', 'location'],
              },
          },
      ],
  });

  • Adding only the description client-side appears to "activate" the tool's functionality server-side.

---

Next Steps

Could you provide clarification on the expected behavior for server-side tool registration using addTool? Additionally, any insights on resolving this issue without needing redundant client-side updates would be helpful.

Thank you for your support!