Concepts on Krypton Runtime

Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Krypton is composed of four binaries:

Component	Role
Manager	Kubernetes operator. Reconciles `Agent` CRs → Deployments + Services + ServiceAccounts. Runs the scaling decider.
Control plane	Read-only HTTP API + the operator UI. Optionally mirrors agents into Postgres for offline querying.
Gateway	Public ingress. Reverse-proxies invocations to the agent’s in-cluster Service.
Sidecar	Per-pod `krypton-proxy`. Enforces concurrency, surfaces in-flight count, exposes Prometheus metrics.

High-level diagram

%%{init: {"theme": "base", "flowchart": {"nodeSpacing": 70, "rankSpacing": 80, "diagramPadding": 24}, "themeVariables": {"fontFamily": "Inter, ui-sans-serif, system-ui, sans-serif", "primaryColor": "#eef2ff", "primaryTextColor": "#1f2937", "primaryBorderColor": "#6366f1", "lineColor": "#64748b", "secondaryColor": "#ecfeff", "tertiaryColor": "#f8fafc"}}}%%
flowchart TB
 client["Client"]
 ui["Krypton UI<br/>Operator console"]
 cp["Control plane<br/>REST API and cache"]
 gw["Gateway<br/>Ingress and activator"]
 mgr["Manager<br/>Controller"]
 scaler["Scaler<br/>Replica decisions"]

 subgraph pod["Agent pod"]
 proxy["krypton-proxy<br/>Concurrency and metrics"]
 app["User agent container"]
 proxy -->|"proxy"| app
 end

 ui -->|"REST"| cp
 client -->|"invoke"| gw
 gw -->|"proxy request"| proxy
 cp -->|"watch"| mgr
 mgr -->|"owns"| pod
 scaler -->|"scale"| pod
 scaler -. "in-flight" .-> proxy

 classDef external fill:#f8fafc,stroke:#94a3b8,color:#0f172a;
 classDef control fill:#eef2ff,stroke:#6366f1,color:#312e81;
 classDef traffic fill:#ecfeff,stroke:#0891b2,color:#164e63;
 classDef runtime fill:#f0fdf4,stroke:#16a34a,color:#14532d;
 classDef podbox fill:#ffffff,stroke:#cbd5e1,color:#0f172a;
 class client external;
 class ui,cp,mgr,scaler control;
 class gw,proxy traffic;
 class app runtime;
 class pod podbox;

Where state lives

State	Source of truth
Agent desired spec	The `Agent` CR (Kubernetes etcd)
`status.phase`, `replicas`	Manager writes; readers consume
`status.desiredReplicas`	Scaler (in manager)
`status.lastInvocationAt`	Gateway writes after each invocation
In-flight count	Sidecar’s `/_krypton/inflight` endpoint
Invocation history (later)	Postgres

CRDs are the source of truth. Postgres is a write-through mirror — the API serves directly from the informer cache (fresher, no DB hop).

Components

Mon, 01 Jan 0001 00:00:00 +0000

Manager

Source: cmd/manager + internal/controller.

Standard controller-runtime operator. Reconciles Agent CRs by owning three child resources per agent: a Deployment, a Service, and a ServiceAccount. The Deployment is injected with the krypton-proxy sidecar at template-render time.

Also runs the scaling decider as a manager.Runnable.

Key behavior:

CreateOrUpdate semantics: each child resource uses controllerutil.CreateOrUpdate wrapped in retry.RetryOnConflict so spec drift converges without hot-looping when the apps controller concurrently updates Deployment status.
Status writes use Patch with MergeFrom, not Update, so they don’t conflict with the scaler/gateway’s concurrent writes to other status fields.
Finalizer krypton.ai/cleanup blocks deletion until child resources have drained.

Control plane

Source: cmd/control-plane + internal/controlplane.

Request lifecycle

Mon, 01 Jan 0001 00:00:00 +0000

This page walks through what happens between curl POST .../invocations and the JSON coming back.

Hot path (pod already running)

%%{init: {"theme": "base", "themeVariables": {"fontFamily": "Inter, ui-sans-serif, system-ui, sans-serif", "primaryColor": "#eef2ff", "primaryTextColor": "#1f2937", "primaryBorderColor": "#6366f1", "lineColor": "#64748b", "secondaryColor": "#ecfeff", "tertiaryColor": "#f8fafc"}}}%%
sequenceDiagram
 autonumber
 participant Client
 participant Gateway
 participant Cache as Informer cache
 participant KubeProxy as kube-proxy
 participant Sidecar as krypton-proxy
 participant Agent as User container
 participant Status as Agent status

 Client->>Gateway: POST /v1/agents/agents/echo/foo
 Gateway->>Cache: Resolve Agent and ready Endpoints
 Cache-->>Gateway: echo.agents.svc:8080
 Gateway->>Gateway: Strip prefix, preserve traceparent, enable streaming flush
 Gateway->>KubeProxy: Proxy /foo
 KubeProxy->>Sidecar: Route to ready Endpoint
 Sidecar->>Sidecar: Check shutdown and acquire concurrency slot
 alt capacity available
 Sidecar->>Agent: Reverse proxy to 127.0.0.1:<spec.port>
 Agent-->>Sidecar: Streaming response
 Sidecar-->>Gateway: Release slot and forward response
 Gateway-->>Client: Response stream
 Gateway->>Status: Patch lastInvocationAt asynchronously
 else concurrency cap reached
 Sidecar-->>Gateway: 503 with Retry-After
 Gateway-->>Client: 503 with Retry-After
 end

Typical latency: P50 ~50ms, P95 ~200ms for a 100ms user-handler.