fix: adapt LLM client parameter response_format for Anthropic

[jtanningbed]

Description

Fixes(Occasional add_to_vector_store failure on Claude models)
Fix JSON formatting issue with Anthropic LLM responses in memory operations. The Anthropic API doesn't support the response_format parameter directly like OpenAI, (nor was this parameter injected into the request) causing JSON parsing errors occasionally when adding memories to the vector store. This change implements a workaround by injecting explicit JSON formatting requirements through the system message.

Motivation: Memory operations were failing with JSON parsing errors when using Anthropic's Claude model, preventing proper storage of memories in the vector store. While the FACT_RETRIEVAL_PROMPT does contain these formatting instructions, the prompt is not as explicit in its instruction as it could be (The response should be in json with a key as "facts" and corresponding value will be a list of strings, you can return an empty list corresponding to the "facts" key, etc. VS RESPONSE FORMAT: You MUST ALWAYS respond with a JSON object containing a 'facts' array.

Without wanting to make changes to the core prompt that powers vector storage operations, I opted to inject a more explicit system prompt when the response_format parameter of type json_format is passed from the memory client (system message is the . The more appropriate longer term solution may be a reevaluation of the core prompt or prefilling the Assistant responses to enforce the desired structure.

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

• previous:
  2024-11-24 18:07:09,488 - mem0.vector_stores.qdrant - INFO - Inserting 1 vectors into collection mem0
  2024-11-24 18:07:33,504 - httpx - INFO - HTTP Request: POST https://api.anthropic.com/v1/messages?beta=true "HTTP/1.1 200 OK"
  2024-11-24 18:07:53,148 - httpx - INFO - HTTP Request: POST https://api.anthropic.com/v1/messages?beta=true "HTTP/1.1 200 OK"
  2024-11-24 18:07:55,012 - httpx - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
  2024-11-24 18:07:55,218 - httpx - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
  2024-11-24 18:07:55,972 - httpx - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
  2024-11-24 18:07:56,476 - httpx - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
  2024-11-24 18:07:56,824 - httpx - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
  2024-11-24 18:07:57,763 - httpx - INFO - HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
  2024-11-24 18:07:57,766 - root - ERROR - Error in new_retrieved_facts: Expecting value: line 1 column 1 (char 0)
  2024-11-24 18:07:57,766 - root - INFO - Total existing memories: 0

• with change:
  2024-11-24 19:36:59,282 - httpx - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
  2024-11-24 19:37:01,046 - httpx - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
  2024-11-24 19:37:04,873 - httpx - INFO - HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
  2024-11-24 19:37:04,874 - root - INFO - {'id': '0', 'text': 'The capital of France is Paris', 'event': 'ADD'}
  2024-11-24 19:37:04,874 - root - INFO - Creating memory with data='The capital of France is Paris'
  2024-11-24 19:37:04,893 - mem0.vector_stores.qdrant - INFO - Inserting 1 vectors into collection mem0

• Unit Test

• test suite in tests/llms/test_anthropic.py covering:

  • Basic response generation
  • System message handling
  • JSON format injection
  • Fact retrieval message integration
  • Tool usage compatibility
    Test results show all scenarios passing, confirming proper JSON formatting across different use cases.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules
• I have checked my code and corrected any misspellings

Maintainer Checklist

• closes #xxxx (Replace xxxx with the GitHub issue number)
• Made sure Checks passed

[jtanningbed]

some non-structured outputs are still leaking through. updated this to only prefill the assistant response with "Here is the requested JSON:"

Seems like the expected method to force structured output on anthropic models is through tools. This isn't as straightforward with the current setup since there are many prompts that might expect different schemas. longer term solution would be to define pydantic models for the expected schemas and force the outputs via tools:

class FactRetrievalSchema(BaseModel): facts: List[str], Field(desc="", default=[])

facts_schema = FactRetrievalSchema.model_json_schema()

tools = [{"name": "send_to_manager", "description": "send the facts JSON to the manager", "input_schema": facts_schema}]
tool_choice={"type": "tool", "name": "send_to_manager"}

res = ...create(tools=tools, tool_choice=tool_choice, ...)

return res.content[0].input # return the tool call input
# or returnFactRetrievalSchema(**(res.content[0].input))

this would require defining models for all schemas and dynamically passing them in the tools request, not something i'm willing to do without getting some feedback first