Problem

You’re attempting to serve Gemini models through Mosaic AI Model Serving using the built-in Google Cloud Vertex AI provider. You use a Google AI Studio-generated API key in your configuration, and authentication fails with a generic internal error.

("error_code": "INTERNAL_ERROR", "message": "Internal Error"}

Cause

You’re trying to use the Google Cloud Vertex AI provider with a Google AI Studio API key.

The Google Cloud Vertex AI provider in Mosaic AI Model Serving is designed to work specifically with Google’s service account keys. 

Solution

To serve a Gemini model continuing to use a Google AI Studio API key, set up the external model in Mosaic AI Model Serving with the Custom Provider configured in an OpenAI-compatible format.

Use the following configuration. 

Provider: Custom Provider

Custom Provider Model URL: https://generativelanguage.googleapis.com/v1beta/openai/chat/completions

Custom Provider Authentication Type: Bearer Token Authentication
  
Bearer Token: {{secrets/scope-name/gemini-api-key}}  # Use secrets!

Model Name: <Your gemini model> #gemini-2.0-flash

For further information on custom provider setup, refer to the “Custom provider” section of the External models in Mosaic AI Model Serving (AWS | Azure | GCP) documentation. You can also refer to the “Google Cloud Vertex AI” section of the same documentation.

For information on Google Gemini OpenAI compatibility, refer to the Google OpenAI compatibility documentation.