Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

docs/IR-CONTRACT.md: the frozen IR

This is the contract between the Nix library (Epic 1), codegen (Epic 2), and the executor (Epic 3 / 3.5). It is an API. Changing its shape requires an OpenSpec change to this document first, then downstream updates. Version it.

schemaVersion is bumped on any breaking change.

Top-level shape

{
  "schemaVersion": 1,
  "backend": {                                   // OPTIONAL; where state is stored
    "type": "s3",                                // required; the backend kind
    "bucket": "my-state", "key": "prod/app", "region": "eu-west-1"
    // backend-specific keys; STATIC only (no refs/unknowns); NO credentials here
    // (those come from the provider/AWS credential chain). Absent => local file store.
  },
  "providers": {
    "<provider-id>": {                 // e.g. "alpha", "registry.opentofu.org/x/alpha"
      "source": "<source-or-path>",    // for PoC: filesystem path to the binary
      "config": { /* provider config, may contain refs */ }
    }
  },
  "resources": [
    {
      "id": "<provider>.<type>.<name>",       // stable identity, unique
      "provider": "<provider-id>",
      "type": "<resource-type>",               // e.g. "alpha_token"
      "name": "<name>",
      "config": { /* attribute tree; leaves may be values, refs, or unknowns */ },
      "meta": {
        "dependsOn": ["<id>", ...],            // explicit edges (additive to implicit)
        "lifecycle": { "preventDestroy": false, "ignoreChanges": ["<path>"] }
        // count/for_each are NOT here: expansion already happened in Nix (see below)
      }
    }
  ],
  "dataSources": [                              // OPTIONAL; read, not created (see below)
    {
      "id": "data.<provider>.<type>.<name>",   // stable identity, unique, "data." prefix
      "provider": "<provider-id>",
      "type": "<datasource-type>",             // e.g. "alpha_lookup"
      "name": "<name>",
      "config": { /* attribute tree; leaves may be values, refs, or unknowns */ }
      // no meta/lifecycle: a datasource is read, never applied or destroyed
    }
  ],
  "edges": [
    { "from": "<id>", "to": "<id>", "via": "<config-path-in-to>" }  // dependency graph
  ],
  "nixConsumers": [
    // values Nix computed FROM resource outputs, surfaced for the round trip.
    // On a given phase these may still be unknown; they become concrete once
    // their inputs are resolved and Nix is re-evaluated.
    { "id": "<consumer-id>", "value": { /* tree; leaves may be values/refs/unknowns */ } }
    // A consumer whose id is "output.<name>" is a declared STACK OUTPUT (the
    // `outputs` arg to toIR), with value { "value": <expr> }. It is an ordinary
    // consumer (same resolution); the "output." prefix is a reserved convention
    // the executor reads for `nivis output`. No separate node type.
  ]
}

Backend (backend)

The OPTIONAL top-level backend object declares where state is stored. It is static configuration: it must be known before any evaluation (the executor has to know where state lives before it evaluates anything), so its leaves are plain JSON scalars/objects and may not contain a __ref, __derived, or other unknown leaf. It has a required non-empty string type naming the backend kind (e.g. "s3"); other keys are backend-specific and are interpreted by that backend, not the IR layer. Each backend defines and enforces the COMPLETE set of keys it accepts, so a key outside that set is an error rather than being ignored: for "s3" that set is type, bucket, key, region, endpoint, sseAlgorithm, kmsKeyId and the assumeRole block (roleArn, sessionName, externalId; see docs/REMOTE-STATE.md); for "local" it is type alone. Validation reaches inside a declared block, so a key misspelled one level down is reported rather than dropped.

Secrets are never in backend. It MAY name an identity to assume, such as a role ARN, which is a globally resolvable identifier rather than a credential; the credentials that authorize assuming it always come from the provider/AWS credential chain. It does NOT carry machine-local credential selectors such as a shared-config profile name, because backend is committed configuration and such a name resolves differently on each machine. Beyond an identity to assume, backend carries the location of state and the settings its backend defines for reaching it. When backend is absent, the executor uses the local file store (the default). Adding the optional field is additive: schemaVersion stays 1.

Reference encoding (the core of the contract)

A not-yet-known cross-resource or cross-domain value is a typed ref leaf:

{ "__ref": { "resource": "<id>", "path": ["attr"] } }                 // scalar attr
{ "__ref": { "resource": "<id>", "path": ["net", 0, "ip"] } }         // nested + list index
{ "__ref": { "resource": "<id>", "path": ["tags", "env"] } }          // map key
  • path is an ordered list of string keys / integer indices into the source resource's output object. This covers nested attributes, list/set indices, and map keys uniformly. (For sets, index is the post-apply stable ordering the provider returns.)
  • A ref inside an expanded for_each/count instance is just a normal ref whose resource is the concrete expanded id (e.g. alpha.alpha_token.web["a"] → id alpha.alpha_token.web__a). There is no special "expansion ref" because expansion is already done (below).
  • A ref whose target resource does not yet exist in state is unresolved, not an error, until fixpoint (Epic 3.5.3).

Ref classification (drives phase behavior, DESIGN D3)

The executor classifies each ref:

  • TF→TF: the ref appears in a resources[].config. Resolved in-executor when the target's output is known; does not require Nix re-eval.
  • *→Nix: the ref appears in a nixConsumers[].value, or a resources[].config leaf that Nix itself derived from another resource's output (Nix marks these, see "derived" below). Resolving these requires re-eval with the outputs ledger injected.

Unknown values (toward the provider)

When the executor calls PlanResourceChange with inputs that are still refs, it must present them to the provider as the protocol's unknown value sentinel, not as the __ref JSON (providers don't understand our refs). The mapping { "__ref": ... } → tfprotov6 unknown is the executor's responsibility (Epic 3a.5). Mine Pulumi's bridge for how it encodes unknowns at plan/diff time.

for_each / count expansion timing

Expansion happens in Nix. The IR contains concrete, already-multiplied resources with deterministic ids (<base>__<key>). The executor never sees count/for_each; it only sees resolved instances and edges between them. This keeps the Go ResourceNode simple and the graph explicit.

Datasources (dataSources)

A datasource reads existing infrastructure (an AMI by filter, a VPC, an availability zone) rather than creating it. The optional top-level dataSources array carries them, distinct from resources. A datasource node is { id, provider, type, name, config } with id data.<provider>.<type>.<name> (the data. prefix keeps it from colliding with a resource id). It has no meta/lifecycle: a datasource is read via the provider's ReadDataSource, never planned, applied, written to state, or destroyed.

A datasource is a first-class node in the dependency graph: a __ref/__derived in a resource (or another datasource) config MAY target a datasource id, and a datasource config MAY reference a resource or datasource, producing edges like any other node. So datasources participate in the phased fixpoint: the executor reads a datasource when its config inputs are fully known. A datasource with a fully-known config reads in the first phase; one whose config depends on a resource's apply-time output reads in a later phase, after that output lands in the ledger. Its read attributes enter the outputs ledger keyed by its id, so downstream nodes resolve against them exactly as they resolve resource outputs.

"Derived" Nix values

A config leaf that Nix computed from a resource output (e.g. "web-" + alpha.id) cannot be a plain __ref (it's a transformation). Nix emits such a leaf as unknown-pending until the inputs are available:

{ "__derived": { "inputs": ["<id>.attr", ...] } }   // value computed by Nix once inputs known

The executor treats __derived leaves as *→Nix: it cannot compute them; it records that the listed inputs are required, and once those outputs are in the ledger, the next Nix re-eval produces the concrete value. This is the mechanism that forces N>2 phases for chained Nix-mediated dependencies.

Build outputs (__build)

A config leaf that is the output of a Nix build (e.g. a resource source that is a built disk image) is a __build leaf carrying two store paths — the build output the provider must be given, and the derivation that produces it:

{ "__build": {
    "path": "/nix/store/<hash>-<name>/<file>",   // the output the provider reads
    "drv":  "/nix/store/<hash>-<name>.drv"       // the derivation that produces it
} }

Both are needed, and neither substitutes for the other, and the executor does two distinct things with them:

  • Substituting path into the config happens for every config handed to a provider — a plan, an apply, and a datasource read alike. It is a pure rewrite, performed once where configs are resolved, so no operation can hand a provider a raw leaf (a provider's encoder expects the value the leaf stands for).
  • Realising the leaf's derivation — building the output, or substituting it if the store prefers — happens only when a resource is applied, per resource, as it becomes ready. nivis evaluates; it does not build until it applies.

So a plan on a configuration that builds an artifact reports its diff without building anything: a plan compares a path, and whether the artifact exists yet does not change the comparison. A datasource carrying a build leaf is likewise substituted and never built — it reads existing infrastructure.

The derivation is what makes the output producible. An output path names a result, not a recipe: realising an output path can reuse a path that is already valid or fetch one from a substituter, but it cannot build one, and the derivation is not recoverable from it (the hashes are unrelated, and a store cannot report the deriver of a path that is not yet valid). A leaf carrying only path is therefore substitute-only; it remains valid IR for compatibility with a Nix library predating the field, and the executor says so when such a leaf cannot be realised.

Realising happens per resource as it becomes ready, so a build whose derivation depends on an earlier resource's apply-time output is realised in a later phase: the build participates in the phased fixpoint. That case is also why building from the derivation is required rather than convenient — there, the derivation does not exist until the earlier resource is applied, so the author cannot pre-build the path, and substitution can never satisfy it.

Unlike __ref/__derived, a __build leaf is a known value in one specific sense: it does not depend on the outputs ledger, so it passes through resolution unchanged and is neither an edge nor unknown-pending. That is not the same as its path existing. Evaluation fixes the path string; for a derivation that has never been built, the artifact it names does not exist at all. Conflating the two is what makes an output-path-only leaf look sufficient when it is not.

Authors emit it with the drv helper (source = drv image), or drvFile for a file inside the output.

Sensitive values across the boundary

Provider schema marks attributes sensitive. Sensitive outputs:

  • Must not be written into the IR JSON emitted by nix eval (that output and the Nix store are world-readable). The Nix side emits a ref/placeholder only.
  • Live only in the executor's outputs ledger, which is written with restricted permissions (0600) and is not a Nix store path.
  • When a sensitive output must feed a later Nix re-eval, it is injected via a private channel (file path passed as --argstr, file mode 0600), never baked into a derivation. The re-eval may use it but must not re-emit it into a world-readable output.

This is a hard requirement; getting it wrong leaks secrets into the store.

Outputs ledger (the phased-eval injection format)

The file the executor accumulates and injects on each re-eval:

{
  "phase": 2,
  "outputs": {
    "<resource-id>": { "<attr>": <value-or-{__sensitiveRef}>, ... }
  },
  "vars": {
    "<name>": <value>
  }
}

Nix reads this (path passed in via the flake plan argument) to resolve refs and compute __derived values on the next phase. __sensitiveRef points at the restricted-mode channel rather than embedding the secret.

outputs accumulates across phases. vars is the resolved configuration variables (see nivis.mkVars): a map of declared-variable name to its known value, resolved once by the executor before phase 0 and re-injected unchanged on every phase (variables are constant inputs, not resolved-across-phases outputs, so a vars value is never a ref or unknown). vars is optional: a ledger with no variables may omit it, and a plan that declares none may ignore it. The executor resolves vars from --var / --var-file / NIVIS_VAR_* with Terraform precedence (an explicit --var flag wins); the values travel only in this 0600 file, never on the Nix command line.

Validation

The contract is machine-checkable, not just prose. Two artifacts make it so:

  • docs/ir-schema.json: the normative JSON Schema (Draft 2020-12) encoding the structural rules of everything above: top-level shape, the __ref/__derived/__sensitiveRef leaf encodings, and the no count/ for_each in the IR rule. A leaf-marker object (__ref etc.) is dispatched to its exact subschema, so a malformed marker reports a precise, addressed error (e.g. at resources/1/config/label/__ref: 'path' is a required property) rather than a generic failure.
  • tests/ir-conformance/: the executable conformance suite. check.py layers (1) JSON-Schema structural validation over (2) the referential rules JSON Schema cannot express: unique ids, every provider declared, every edge endpoint present, every __ref/__sensitiveRef target existing. Fixtures under fixtures/valid and fixtures/invalid lock both directions; each invalid fixture asserts the error names the offending element.

Both producer/consumer sides MUST conform to these artifacts:

  • Nix (Epic 1.5 toIR): a property test that, for arbitrary valid resource graphs, toIR output passes check.py validate: every leaf is a value, a well-formed __ref, a __derived, or a __sensitiveRef; ids unique; every edge endpoint exists.
  • Go (Epic 3a.1 IngestIR): rejects malformed IR with an actionable error naming the offending resource/path (Epic 4c), matching the failure classes in tests/ir-conformance/fixtures/invalid. check.py is the reference behavior the Go validator is tested against.

Run the suite: python3 tests/ir-conformance/check.py test.