smrt
Open the smrt source on GitHub Switch to dark color scheme
← Capabilities

New capability

Choose which model answers, and what happens when it can’t

Your code asks for a tier, not a model. You bind each tier to a model or to an ordered list of models, and a call that hits a quota, a bad key or a renamed model moves to the next one. Every move is written to the usage row.

Verified against smrt 0.62.10

Tiers: ask for how good, not for which model

There are four tiers, cheapest first: cheap, good, great and best. A call can also ask for auto, which resolves to your default tier, or to good when you set none. Speed is not a tier. It is a separate hint, interactive or background, that a binding may honor.

  • With nothing configured, every tier uses the client you already configured. Calls behave exactly as they did before.
  • You bind a tier in config, in smrt.config, in environment variables, or per instance. The highest layer that binds a tier replaces the whole target, so a key never follows a target to another provider.
summarize.ts
typescript
await article.do('Summarize this', { tier: 'great' });

Fallback chains

Add @happyvertical/smrt-inference and import it once at startup. You can then bind a tier to an alias, which is an ordered list of candidate models. Importing the package with no aliases changes nothing.

  • A failed candidate is classified as quota, credential, missing-model, transient, refusal or missing-capability, and the next one is tried. A transient failure is retried once first.
  • When every candidate of a tier fails, the call moves down to the next cheaper tier, never up. Set degrade to false to stop that, or pass minimumTier on the call.
  • Text already streamed to the caller is never continued by another model.
  • The usage row of the answering call records the tier that answered and each substitution. A call that nothing answered leaves a zero-token row with the reason and throws InferenceExhaustedError.
startup.ts
typescript
import '@happyvertical/smrt-inference';
import { config } from '@happyvertical/smrt-core';

config({
  inference: {
    aliases: {
      frontier: {
        candidates: [
          { provider: 'anthropic', model: 'your-first-choice' },
          { provider: 'openai', model: 'your-second-choice' },
          { preset: 'lemonade', model: 'your-local-model' }
        ]
      }
    },
    tiers: { great: { alias: 'frontier' } },
    degrade: true // default
  }
});

Per-user and per-tenant choices

A person or a tenant can bring a provider and key of their own. For each tier the resolver looks at the user’s stored choice, then the tenant’s, then a tenant override the app allows, then the app’s own override, then config, then the configured client. This is not on until the host registers the resolver.

  • A tenant may bind a tier only to targets on the app’s allow-list. With no list, nothing is allowed.
  • Keys are stored once, in smrt-secrets, and are bound to the endpoint they were entered for. The server sends a key only to that origin.
  • Aliases and fusions are defined by the app. A tenant can pick an allowed one, never define one.
  • Each usage row records where the binding came from: the user’s own key, a local model, hosted inference or the app.
  • Sign in with ChatGPT is also in the package, for local apps only. Upstream marks it a preview. It is never a default and never a fallback.

The settings screen

InferenceSettingsPanel is a Svelte component from @happyvertical/smrt-inference/svelte. The default view is the whole screen for one provider. Advanced holds the tier ladder, the auto tier, aliases and fusions, and stays closed until someone opens it. A Usage tab shows usage by model, tier, prompt, fusion or template.

  • The panel only presents. Your host mounts it, supplies the load and save routes, and decides who may see and change what.
  • A key goes out of the panel once, when it is saved, and never comes back. The panel shows only whether a key is set.
  • At tenant scope, targets off the allow-list are disabled and aliases and fusions are read-only.
  • The Usage tab shows transcript counts and can link to a route of yours. It never shows transcript content.

A spend gate for hosted inference

If your app pays for inference on behalf of its users, the package can provision a gateway key per user and check each call first. The gate checks a kill switch, the tenant, the entitlement and then the plan’s spend caps. A blocked call throws InferenceUnavailableError and writes a blocked row, which reports show in their own column beside calls.

  • Off until the host registers the gate and configures a gateway. Nothing is gated otherwise.
  • A global disable on the kill switch always wins.

Fusions

A tier can be bound to a fusion: a small arrangement of prompts you define, in one of three modes (single, pair or handoff). Callers keep asking for the tier. smrt ships the interpreter and no fusions, so there is nothing to use until you define one. Each run of a fusion is recorded as a run with one usage row per seat.

Check the bindings before you need them

smrt doctor --ai checks each tier, alias and fusion without calling a model: the provider is recognized, the key variable is present, a model is named, aliases resolve and nothing loops. Add --probe and it also asks each endpoint for its model list. Keys never appear in the report.

terminal
bash
smrt doctor --ai
smrt doctor --ai --probe