Model CRD

apiVersion: krypton.ai/v1alpha1, kind: Model, namespaced (short name mdl).

The Model CRD declares “host this Hugging Face GGUF as an OpenAI-compatible model named X”. The controller turns it into a Deployment running llama.cpp’s HTTP server, plus a Service. The gateway aggregates every Model in the cluster at /v1/models and routes incoming /v1/chat/completions (etc.) requests by the model field in the body.

Minimal example

apiVersion: krypton.ai/v1alpha1
kind: Model
metadata:
  name: qwen2-0-5b
  namespace: models
spec:
  source:
    huggingface: Qwen/Qwen2.5-0.5B-Instruct-GGUF
    file: qwen2.5-0.5b-instruct-q4_k_m.gguf

That’s the smallest valid Model — every other field has a default.

Full example

apiVersion: krypton.ai/v1alpha1
kind: Model
metadata:
  name: qwen2-0-5b
  namespace: models
spec:
  # Weights
  source:
    huggingface: Qwen/Qwen2.5-0.5B-Instruct-GGUF
    file:        qwen2.5-0.5b-instruct-q4_k_m.gguf

  # Runtime
  runtime: llama.cpp           # only value supported today
  image: ""                    # blank = built-in llama.cpp:server image
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []

  # Networking
  port: 8080

  # Extra args appended to llama-server
  args:
    - "--ctx-size"
    - "4096"

  # Scaling (always-on; scale-to-zero on the roadmap)
  minReplicas: 1
  maxReplicas: 1

  # Pod
  resources:
    requests: { cpu: 500m, memory: 1Gi }
    limits:   { cpu: "2",   memory: 2Gi }
  env: []
  envFrom:
    - secretRef: { name: hf-token }   # for gated repos: HUGGINGFACE_HUB_TOKEN
  serviceAccountName: ""              # blank = auto-create

  # Lifecycle
  startupTimeout: 600s

Spec reference

Source

Gated repos require a token. Mount it as HUGGINGFACE_HUB_TOKEN via envFrom.

Runtime

Networking

Args

args is appended to the runtime entrypoint after the controller-owned flags (--host, --port, --hf-repo, --hf-file, --alias). Use it for tuning knobs like --ctx-size, --n-gpu-layers, --parallel. Don’t redefine the controller-owned flags from here.

Scaling

Pod

Lifecycle

Status (read-only)

Phases

status.url points at the in-cluster OpenAI base URL for the pod, e.g. http://qwen2-0-5b.models.svc:8080/v1. Most callers go through the gateway instead.

OpenAI compatibility

Once Ready, the gateway exposes the model under these standard paths:

The OpenAI-facing id is metadata.name. If two Model CRs share a name across namespaces, the gateway picks one and logs the collision — keep model names unique.