Model IDs are a dependency. Pin them like one.

Outline

• The bug that isn't a bug: your guessed model ID worked, so you shipped it.

• Three failure modes of a wrong ID, ranked by how much they'll cost you.

• The ten-second test for whether an endpoint guesses on your behalf.

• Dated vs floating IDs: two different risks, one policy.

• GET /v1/models is the only source of truth, and it's per-key.

• models.lock.json and a CI check that fails when a required ID vanishes.

• The alias layer: roles, not model names, everywhere in your code.

• A lint that stops model IDs leaking back into the codebase.

• Retirement playbook: six steps from "it's gone" to "merged".

• Channel variants on gateways: read the label, never pattern-match.

• Behavioural drift under a pinned ID — signals, not proof.

Draft

You typed claude-sonnet-4-5 from memory, it returned a completion, and you shipped. Six weeks later the outputs are subtly different, or a deploy fails with a 404 on a Tuesday morning, and the git blame points at a string literal in a file nobody has opened since.

Model IDs are a runtime dependency on someone else's release schedule, and almost nobody treats them like one. We pin requests==2.32.3 and then interpolate a model name from a config file we last checked in March.

Disclosure: I work on daoxe, a gateway that fronts hundreds of models from roughly 25 vendors — which means I get to watch upstream renames arrive from several directions at once. Everything below works against any endpoint that implements GET /v1/models.

Three ways a wrong ID fails

Mode A — a clean error. 404 or 400 with "code": "model_not_found". This is the good outcome. You find out in development, in the first second, and you fix a typo.

Mode B — alias resolution. You ask for gpt-4o, the provider resolves it to a dated build, and the response's model field says which one. Fine and normal — as long as you read the echo. If you never look, you've silently accepted whatever the alias points at this month.

Mode C — nearest-match routing. Some endpoints and proxies try to be helpful: they don't recognise your ID, so they route to the closest thing they do recognise. You asked for a flagship and got something else, with a 200 and no indication anywhere in the response that a substitution happened. This is the mode that costs real money and real trust, and it's invisible in exactly the way that matters.

The ten-second test for Mode C

Send an ID that cannot possibly exist, formed by mangling a real one:

curl -s https://api.daoxe.com/v1/chat/completions \
  -H "Authorization: Bearer $LLM_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"claude-sonnet-4-5-TYPO-NOT-A-MODEL",
       "max_tokens":8,"messages":[{"role":"user","content":"hi"}]}' \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); \
print("ERROR:", d["error"]["message"][:80]) if "error" in d else print("SERVED BY:", d.get("model"))'

If you get an error, every model ID you send is authoritative. If you get a completion, every model ID you send is advisory — the endpoint reserves the right to pick for you, and no amount of pinning on your side will help until you know the rule it uses. Run this before you build anything else in this article.

Dated versus floating

Two ID styles, two different risks:

	Floating (claude-sonnet-4-5)	Dated (claude-sonnet-4-5-20250929)
Behaviour	changes under you, no code change	stable while it exists
Lifespan	long	retired on a schedule
Fails as	silent quality drift	a loud 404
Use for	exploration, prototypes	anything with a recorded eval

The policy that follows: exploration floats, production is dated, and every eval artifact records the ID that actually served it — read from the response's model field, not from what you asked for. That last part is what lets you answer "was this eval run on the same build as that one" three months later.

GET /v1/models is the source of truth, and it's per-key

Not the docs, not the pricing page, not a blog post. The catalogue endpoint returns the IDs your key is entitled to call, which is frequently a different set from what the vendor sells:

curl -s https://api.daoxe.com/v1/models \
  -H "Authorization: Bearer $LLM_API_KEY" \
  | python3 -c 'import json,sys; [print(m["id"]) for m in json.load(sys.stdin)["data"]]' \
  | sort

One caveat worth knowing up front: most implementations return IDs and little else. Context windows, tool support, vision support and modality are usually not in there, so you still need a small capability sidecar of your own. Don't infer capabilities from the name.

Lock the catalogue

#!/usr/bin/env python3
"""check_models.py — fail CI when a model ID this repo depends on disappears.

    python3 check_models.py --write      # re-pin (review the diff!)
    python3 check_models.py              # verify
"""
import argparse, json, os, sys, urllib.request

BASE = os.environ["LLM_BASE_URL"].rstrip("/")
KEY = os.environ["LLM_API_KEY"]


def catalogue():
    req = urllib.request.Request(BASE + "/models")
    req.add_header("Authorization", "Bearer " + KEY)
    with urllib.request.urlopen(req, timeout=30) as resp:
        return sorted({m["id"] for m in json.load(resp).get("data", []) if m.get("id")})


def required_ids(path="models.json"):
    """The IDs this repo actually uses — derived from the alias map, never hand-listed."""
    with open(path) as fh:
        return sorted(set(json.load(fh)["roles"].values()))


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--lock", default="models.lock.json")
    ap.add_argument("--write", action="store_true")
    args = ap.parse_args()

    live = catalogue()
    if args.write or not os.path.exists(args.lock):
        with open(args.lock, "w") as fh:
            json.dump({"catalogue": live}, fh, indent=2)
            fh.write("\n")
        print(f"pinned {len(live)} ids to {args.lock}")
        return 0

    with open(args.lock) as fh:
        pinned = set(json.load(fh)["catalogue"])
    live_set, needed = set(live), set(required_ids())

    for mid in sorted(live_set - pinned):
        print(f"note   new in catalogue: {mid}")
    for mid in sorted(pinned - live_set - needed):
        print(f"warn   left the catalogue (unused here): {mid}")
    gone = sorted(needed - live_set)
    for mid in gone:
        print(f"ERROR  required by models.json but not in the catalogue: {mid}")
    return 1 if gone else 0


if __name__ == "__main__":
    sys.exit(main())

Run it nightly and on every pull request:

name: model-catalogue
on:
  schedule: [{ cron: "17 6 * * *" }]
  pull_request:
  workflow_dispatch:

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: python3 check_models.py
        env:
          LLM_BASE_URL: https://api.daoxe.com/v1
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}

The two failure signals mean different things. A nightly failure means an upstream retired or renamed something you depend on, and you found out before your users did. A pull-request failure means someone added a role pointing at an ID that isn't callable — a typo caught at review time instead of at 2am.

The alias layer

Nothing in your application code should contain a model name. Code refers to roles; one file maps roles to exact IDs.

{
  "pinned_at": "2026-08-12",
  "roles": {
    "fast":   "claude-sonnet-4-5",
    "deep":   "claude-sonnet-4-5",
    "bulk":   "claude-sonnet-4-5",
    "vision": "claude-sonnet-4-5",
    "embed":  "text-embedding-3-large"
  },
  "notes": {
    "deep": "dated build; re-pin only with an eval run linked in the PR"
  }
}

import functools, json, pathlib

@functools.lru_cache(maxsize=1)
def _roles():
    return json.loads(pathlib.Path("models.json").read_text())["roles"]

def model_for(role):
    try:
        return _roles()[role]
    except KeyError:
        raise KeyError(f"no model pinned for role {role!r} — add it to models.json") from None

Three things fall out of this for free. A rename becomes a one-line diff in a reviewable file instead of a search-and-replace across forty call sites. check_models.py can derive its required set from the same file, so the lock and the code can't disagree. And git log models.json becomes a readable history of what your product was running and when — which is the artifact you'll want the first time someone asks why output quality changed in July.

Then stop the IDs from leaking back in. One test, one line:

# tests/no_hardcoded_model_ids.sh — fails if a model-looking literal appears in src/
! rg -n --glob '!models.json' \
    -e '"(gpt|claude|gemini|deepseek|qwen|kimi|grok|llama)-[a-z0-9._-]+"' src/

When an upstream retires a model mid-project

1. Find out from CI, not from users. That's what the nightly is for.

2. Read the label before you pick a successor. Vendors publish migration notes; gateways publish catalogue metadata. Don't infer the replacement from the name — -mini, -fast, -pro and numeric bumps are marketing, not a capability contract.

3. Freeze a golden set. Twenty to fifty real requests from your own traffic with their recorded outputs. If you don't have one yet, this is the moment it becomes worth an afternoon; you will need it again.

4. Run the candidate against the golden set and diff the things that actually break in production: output format compliance, tool-call rate, refusal rate, response length, and latency. Not vibes.

5. One PR, one line. Change models.json, link the eval run in the description, let CI re-verify the lock.

6. Keep the corpse. Leave the old ID in the git history of models.lock.json and in your eval artifacts. When you're asked to reproduce a result from two quarters ago, that record is the only thing that will let you say what actually served it.

Variants and channels on a gateway

If you route through a gateway, two IDs that look like the same model may reach different upstream channels with different reliability characteristics. Some catalogues label this explicitly — the one I work on tags channels as full-power, official relay, reverse-engineered ("quality not guaranteed"), or promotional, and that labelling is the whole point: you can choose a cheaper or less stable path deliberately, or refuse to. What you should never do is pattern-match a suffix and assume you know what it means. Read the catalogue metadata, pick the exact string, pin it.

Drift under a pinned ID

Pinning protects you from renames. It does not protect you from an upstream changing what sits behind a name it didn't change. The only handle you have from the outside is behavioural: record how a pinned ID answers a fixed battery of prompts at temperature 0, re-run it on a schedule, and diff.

pipx install git+https://github.com/seven7763/llm-honesty-probe

Be precise about what that gives you: signals, not proof. No probe run from outside can prove which weights served a request, and a change in output can have several innocent explanations. What it does produce is a timestamped record that something changed on a date, which is a much stronger position to argue from than "it feels worse lately."

Pin the ID, alias the role, lock the catalogue, watch the behaviour. It's an afternoon of work, and it converts your loudest class of production surprise into a CI failure.