LLM Providers

LLM provider targets call language model APIs directly. These are used both as evaluation targets and as grader targets for scoring.

OpenAI

targets:
  - name: openai-target
    provider: openai
    api_key: ${{ OPENAI_API_KEY }}
    model: gpt-4o

Field	Required	Description
api_key	Yes	OpenAI API key
model	Yes	Model identifier
base_url	No	Custom base URL for OpenAI-compatible endpoints
api_format	No	API format: chat (default) or responses

api_format

Controls which OpenAI API endpoint is used:

Value	Endpoint	When to use
chat (default)	/chat/completions	All OpenAI-compatible endpoints (GitHub Models, local proxies, etc.)
responses	/responses	api.openai.com and Azure OpenAI when the deployment supports the Responses API

Most users should leave this unset. The default chat format is universally supported. Use responses when you need Responses API features on OpenAI or Azure OpenAI deployments that support it.

# OpenAI-compatible endpoint (default chat format works)
targets:
  - name: github-models
    provider: openai
    api_format: chat
    base_url: https://models.github.ai/inference/v1
    api_key: ${{ GH_MODELS_TOKEN }}
    model: ${{ GH_MODELS_MODEL }}

  # Opt in to Responses API for api.openai.com
  - name: openai-responses
    provider: openai
    api_format: responses
    api_key: ${{ OPENAI_API_KEY }}
    model: gpt-4o

Azure OpenAI

targets:
  - name: azure-base
    provider: azure
    endpoint: ${{ AZURE_OPENAI_ENDPOINT }}
    api_key: ${{ AZURE_OPENAI_API_KEY }}
    model: ${{ AZURE_DEPLOYMENT_NAME }}

Field	Required	Description
endpoint	Yes	Azure OpenAI endpoint URL or resource name
api_key	Yes	API key
model	Yes	Deployment name
version	No	Azure API version (defaults to v1)

Azure targets always route through the Responses API (/openai/v1/responses). The api version defaults to v1 and can be overridden via the version field.

Chat-completions-only deployments

If your Azure deployment only exposes /chat/completions (older deployments, certain regions), use provider: openai with a deployment-scoped base_url instead:

targets:
  - name: azure-chat
    provider: openai
    base_url: https://<resource>.openai.azure.com/openai/deployments/<deployment>
    api_key: ${{ AZURE_OPENAI_API_KEY }}
    model: <deployment>
    api_format: chat

The api_format field was previously available on provider: azure but has been removed — Azure targets always go through the Responses API.

Anthropic

targets:
  - name: claude_target
    provider: anthropic
    api_key: ${{ ANTHROPIC_API_KEY }}
    model: claude-sonnet-4-20250514

Field	Required	Description
api_key	Yes	Anthropic API key
model	Yes	Model identifier

Google Gemini

targets:
  - name: gemini_target
    provider: gemini
    api_key: ${{ GEMINI_API_KEY }}
    model: gemini-2.0-flash

Field	Required	Description
api_key	Yes	Google AI API key
model	Yes	Model identifier