Skip to content

Tested models

Every model on this page has been run through core-models against its real server. There are two kinds of test. They answer different questions, so read them separately.

Conformance: does the adapter speak this server correctly?

Section titled “Conformance: does the adapter speak this server correctly?”

Three scenarios per server: a plain answer, a full tool-call round trip (the model calls a tool, gets the result, answers from it), and a streamed answer. Each scenario is run once against the real server, and every HTTP exchange is recorded. The recording is then replayed offline on every pull request. The adapter must send exactly the request it sent when recorded, and produce the same answers from the same responses.

A pass means core-models handles that server’s dialect: its tool-call format, reasoning fields, usage fields, streaming and errors. It says nothing about how good the model is.

Tool-calling parity: is the model good enough for an agent?

Section titled “Tool-calling parity: is the model good enough for an agent?”

mast’s judged eval corpus: 31 Kubernetes incidents (CrashLoopBackOff, OOMKilled, DNS failures, NetworkPolicy, mTLS, and so on), ported from the LangChain SRE agent. The model works each incident with mast’s real tool catalog. The tools answer from per-incident fixtures, not a live cluster. A Claude model grades the final answer. The same corpus runs nightly against Claude, which is the baseline.

MetricWhat it measures
IntentShare of the investigative questions the corpus expects for an incident (is the pod crashing, what do its logs say, what changed) that the model’s tool calls actually answered. Any tool that answers a question counts, so it rewards investigating, not naming a particular tool. The headline number
QualityThe grader’s 1–5 score of the final answer against the expected diagnosis, normalized to 0–1. LLM-judged, so noisier than intent
Tool calls / incidentEvery tool call the model made per incident, repeats included
Distinct tools / incidentDifferent tools the model called per incident. It shows how a model investigates: broadly, or one look and an answer. A large gap from the previous column means repetition
Malformed callsCalls naming an unknown tool, missing or inventing an argument, passing a value outside a declared enum, or erroring. Counted against all calls made. This is the adapter-and-model correctness floor
Consequential missesQuestions the corpus expected answered, that a tool in the catalog would have answered, that the run never asked. Skipping a redundant tool is not a miss

Severity accuracy is also on the board, but it is a diagnostic: the corpus doesn’t define most of its severity labels, so it is not a comparison number.

All runs went through the built-in vertex-maas profile on all 31 incidents, graded by claude-haiku-4-5 on Vertex AI. Each managed model ran three times. The baseline is three consecutive claude-opus-5 nightlies. Values are means, with the range across runs in brackets.

ModelRunsIntentQualityTool calls / incidentDistinct tools / incidentMalformed calls
claude-opus-5 (nightly baseline)30.973 (0.973–0.973)0.855 (0.839–0.871)8.07.00 / 744
zai-org/glm-5.2-maas30.975 (0.957–0.984)0.941 (0.935–0.952)8.45.80 / 780
moonshotai/kimi-k2-thinking-maas30.941 (0.912–0.977)0.815 (0.806–0.823)5.24.30 / 487
qwen/qwen3-coder-480b-a35b-instruct-maas30.890 (0.880–0.896)0.672 (0.645–0.694)3.22.70 / 295
openai/gpt-oss-20b-maas30.663 (0.648–0.670)0.503 (0.468–0.524)1.51.40 / 136
meta/llama-4-maverick-17b-128e-instruct-maas10.2240.4600.30.30 / 8
ModelStatus
zai-org/glm-5.2-maasAt parity: intent matches Claude on every run, and quality is higher
moonshotai/kimi-k2-thinking-maasClose, below parity. One run matched Claude; the three-run mean is 0.03 behind
qwen/qwen3-coder-480b-a35b-instruct-maasWorks, below parity
openai/gpt-oss-20b-maasWorks, small tier
meta/llama-4-maverick-17b-128e-instruct-maasUnsupported for tool use (see below)

What it shows:

  • The provider path is correct. No malformed calls in 1,706 across the five open models’ runs.
  • GLM 5.2 is the one model at parity with Claude. Its intent ranged from 0.957 to 0.984 against Claude’s steady 0.973, with higher judged quality on every run.
  • Below parity, the difference is how broadly a model investigates. Claude reads about 7 distinct tools per incident and GLM about 6. Qwen and gpt-oss usually take one or two looks and answer, and lose intent on the complex incidents.
  • Tiers route on a new vendor. A profile mapping frontier to GLM 5.2, mid to Qwen3 Coder and small to gpt-oss-20b ran each tier on its own model. Checking that each tier is billed at its own rate waits until these models have prices.

Tool calls per incident counts every call, repeats included. Distinct tools per incident counts different tools. A large gap between the two means the model repeated itself.

Self-hosted on vLLM, and one model both ways: 2026-10-09

Section titled “Self-hosted on vLLM, and one model both ways: 2026-10-09”

These runs used vLLM 0.31.0 on one RTX PRO 6000 in GKE, reached over Private Service Connect (Self-hosting on GKE), with one run per configuration. Gemma 4 also ran on Vertex AI: the same weights, behind a different server.

Model and serverIncidents completedIntentQualityTool calls / incidentDistinct tools / incidentMalformed calls
openai/gpt-oss-120b, our vLLM31 / 310.8740.9117.73.41 / 241
google/gemma-4-26B-A4B-it, our vLLM, thinking off24 / 310.8580.7403.02.60 / 72
google/gemma-4-26B-A4B-it, our vLLM, thinking on, vLLM example template31 / 310.9330.68526.53.60 / 820
google/gemma-4-26B-A4B-it, our vLLM, thinking on, model’s own template31 / 310.8850.70211.33.60 / 351
google/gemma-4-26b-a4b-it-maas, Vertex AI, thinking on31 / 310.9170.6773.62.60 / 112

What it shows:

  • gpt-oss-120b works well self-hosted. Quality is above the Claude baseline. Its intent is lower, and part of its breadth is repetition (7.7 calls over 3.4 distinct tools). Its one malformed call named k8s_resource_spec.json: gpt-oss’s output format leaked a constrain-to-JSON marker into the tool name through vLLM’s parser. mast rejected it as an unknown tool, which is correct. core-models does not rewrite tool names.
  • Gemma 4 needs thinking on for agentic work. With thinking off (the default), it repeated one identical call (441 times on one incident) until the 65k context overflowed, on 7 of 31 incidents. Thinking is a per-request switch, set in the profile with extra_body: {"chat_template_kwargs": {"enable_thinking": true}}.
  • The same weights behaved differently on two servers, and the chat template explains much of it. With thinking on, Gemma 4 scored about the same on vLLM and Vertex AI, but on vLLM it repeated tools far more. vLLM’s bundled example template predates Google’s fix for tool-calling loops. Switching to the model’s own template (now the fixture’s default) cut model calls from 851 to 382 across the run. Vertex AI still made only 135. What’s left is repetition: 186 of vLLM’s 351 tool calls repeated an earlier call, against 9 of 112 on Vertex AI, and two incidents held most of them. Without the repeats, the two servers made about the same number of distinct calls. Sampling doesn’t explain it. vLLM was already sampling with Gemma’s published defaults (temperature 1.0, top_k 64, top_p 0.95), and lowering the temperature to 0.3 made things worse: the model fell into generating without end, and every repeat run of the four worst incidents timed out. The likeliest remaining difference is how each server renders the conversation, which we can’t inspect on Vertex AI. mast now tells a model inside the turn when it repeats a call, and ends the turn if it keeps going.

Read these numbers carefully:

  • Self-hosted results are one run each. The managed models’ ranges show how much a single run can move.
  • The baseline is the nightly board, not interleaved with these runs.
  • “At parity” means intent within Claude’s range on this corpus. That is evidence for unattended Kubernetes triage, not a general ranking of the models.

Llama 4 Maverick: unsupported for tool use

Section titled “Llama 4 Maverick: unsupported for tool use”

Llama 4 doesn’t emit structured tool calls itself. It writes a Python-style call list, such as [k8s_triage_workload(scope="web/frontend-v2")], and the server converts that text into tool calls. That only works when the whole reply is the call list.

Under mast’s tool catalog it mostly wasn’t. On 23 of 31 incidents no tool ran. In 22 of those the call was written inside a sentence (“First, I’ll check the snapshot using [k8s_triage_workload(...)]…”), and in one prose followed the call. Vertex AI’s parser rightly doesn’t treat that as a call. The model always knew which tool it wanted; it didn’t follow its own format.

core-models won’t recover such calls from text. A strict parser would rescue one incident in 23. A lenient one would execute calls the model only described as a plan, which is the wrong trade for an agent that can change a cluster. Llama 4 Maverick still passes conformance: short, single-tool requests work. It is left out for tool-using workloads. core-models #12 has the breakdown. It is also served only from us-east5.

ServerModelNotable behaviour the adapter handles
Ollama 0.9.6, localqwen3:1.7bReasoning inline as <think> tags, split out by reasoning_format: think_tags
Vertex AI, globalopenai/gpt-oss-20b-maasReasoning in reasoning_content; prompt-cache hits reported; rejects forced tool choice, so the built-in profile downgrades it to auto
Vertex AI, globalmoonshotai/kimi-k2-thinking-maasTool-call ids of the form functions.<name>:<n>
Vertex AI, globalqwen/qwen3-coder-480b-a35b-instruct-maasNo reasoning output; cached tokens reported
Vertex AI, globalzai-org/glm-5.2-maasText and a tool call in the same turn
Vertex AI, us-east5meta/llama-4-maverick-17b-128e-instruct-maas404 at global; needs a regional profile
vLLM 0.31.0 on GKE (RTX PRO 6000), over PSCopenai/gpt-oss-120bReports reasoning tokens separately. Cached tokens need --enable-prompt-tokens-details. Self-hosted with deploy/gke-vllm
vLLM 0.31.0 on GKE, over PSCgoogle/gemma-4-26B-A4B-itGemma’s own tool-call format, parsed by vLLM’s gemma4 parser. Recorded with thinking off
Vertex AI, global onlygoogle/gemma-4-26b-a4b-it-maasThinking on via extra_body. us-central1 refuses it as global-only

All nine replay on every pull request.

ServerProfileWhat exists
SAP AI Core orchestrationsap-ai-coreThe shared scenarios pass against a fake tenant built from SAP’s Orchestration v2 spec and SDK fixtures. No corpus from a real tenant yet (#83)
  • Conformance: go test -tags live -run TestLive -v . in core-models, with CORE_MODELS_LIVE_PROFILE and CORE_MODELS_LIVE_MODEL set. Add CORE_MODELS_LIVE_RECORD=<dir> to record a new corpus.

  • Parity: mast’s judged tier. For example:

    Terminal window
    go run ./internal/evals/cmd/evals --tier judge \
    --provider vertex-maas --model zai-org/glm-5.2-maas

    Add --rows for a cheap subset first. The board reports the tokens each model used.

A recorded corpus proves the adapter still speaks a dialect the way the server spoke it on the day it was recorded. To notice a provider changing underneath it, a monthly workflow (live-canary) runs the same scenarios live, one leg per provider and dialect:

LegProfileModel
openai-chatopenai, Chat Completionsgpt-5-mini
openai-responsesopenaigpt-5-mini
xai-chatxaigrok-4.20-0309-reasoning
xai-responsesxai, Responsesgrok-4.20-0309-reasoning
anthropicanthropicclaude-haiku-4-5
geminigeminigemini-3.8-flash
gemini-argongeminigemini-4-argon

Each leg makes four calls, so a month costs a few cents. It never runs on push or on pull requests, so no PR’s code can reach the keys. It reads the repository secrets OPENAI_API_KEY, XAI_API_KEY, ANTHROPIC_API_KEY and GEMINI_API_KEY. A leg whose secret isn’t set is skipped with a notice, not failed; use keys with a low spend limit. When a leg fails, the workflow opens an issue labelled live-canary, or comments on the open one, naming the leg and the failing tests. Run a leg locally with dev/ci/live-canary.sh <leg> and its key in the environment.