Problem

When sending requests to Databricks model serving endpoints using Amazon Bedrock and Anthropic's Claude 3.5 Sonnet foundation model (claude-3-5-sonnet-20240620-v1:0), you encounter an error. 

HTTPError: 400 Client Error: INVALID_PARAMETER_VALUE: Content type in user messages can only be 'text' or 'image_url'. for url: <endpoint-url>/invocations. Response text: {"error_code": "INVALID_PARAMETER_VALUE", "message": "INVALID_PARAMETER_VALUE: Content type in user messages can only be 'text' or 'image_url'."}

The error occurs specifically when requests include messages with "type": "image" in the user content schema, as in the following example. 

{
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": <"Describe-this-image">},
        {"type": "image", "source": {"data": <"base64-encoded-image">}}
      ]
    }
  ]
}

Cause

Databricks has adopted a unified OpenAI-compatible schema for multimodal requests across AI providers. This standardization replaces previous provider-specific implementations to enable seamless interoperability between models like Anthropic Claude and OpenAI GPT. 

The official schema now enforces strict validation of content types. Images must use the standardized "image_url" format instead of "image" to ensure consistent request handling and future-proof integrations.

Solution

Update your requests to use the standardized OpenAI-compatible schema instead of Anthropic-specific formatting. 

1. Replace all instances of "type": "image" with "type": "image_url" and restructure the image payload to match the image_url object format. 
2. For base64-encoded images, prepend the data:image/<format>;base64, MIME type identifier to your encoded string.

# For image handling in Anthropic requests
content = [
    {"type": "text", "text": <"Explain-this-diagram">},
    {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{encoded_image}"}}
]

For more information, refer to the External models in Mosaic AI Model Serving documentation.