BigConfig manual
Four libraries that compose into a one-command cloud-provisioning CLI: a launcher (bc-pkg) that bootstraps a target package from GitHub, an infrastructure-automation CLI (Once) built on top of a workflow + render engine (SDK), which itself uses a Django-style template engine (Selmer). Each leaf is implemented in TypeScript, Python and Clojure. Pick a language at the top — the manual rewrites itself.
Writing your own launcher-compatible package? Read the BigConfig Package Spec →
Launcher · bc-pkg
PyPI · npm
bc-pkg is a tiny bootstrap CLI. You hand it <owner/repo@ref> once; it pins the package to a 40-character commit SHA, copies the package's root run file into the current directory, writes a language-native manifest, then forwards every subsequent argument to that pinned target. The Python and TypeScript implementations are deliberately equivalent — both consume the same GitHub-pinned target packages and both produce equivalent on-disk artifacts.
launcher/python PY
bc-pkg · stdlib-only · Python ≥ 3.11All logic in src/bc_pkg/cli.py. Run via uvx bc-pkg .... Build backend: hatchling. No tests.
launcher/typescript TS
bc-pkg · Node built-ins only · Node ≥ 18All logic in bin/bc-pkg.js (CommonJS). Run via npx bc-pkg .... No transpile step, no tests.
bc-pkg itself — but bc-pkg can launch Clojure target packages. When the pinned target ships a deps.edn, both launchers download a pinned Babashka and a Temurin JDK into a shared user cache, auto-install git on Linux if needed, and run bb run …. From the Clojure user's perspective, you still type uvx bc-pkg or npx bc-pkg.
Install & invoke
The launcher has no install step — invoke it on-demand:
# Bootstrap and run in one shot
npx bc-pkg bigconfig-ai/once@typescript package validate
# Subsequent runs in the same directory: ref is pinned, omit it
npx bc-pkg package validate
# For a Python target:
npx bc-pkg bigconfig-ai/once@python package validate
# For a Clojure target (Babashka + JDK + git auto-provisioned):
npx bc-pkg bigconfig-ai/once@clojure package validate
# Bootstrap and run in one shot
uvx bc-pkg bigconfig-ai/once@python package validate
# Subsequent runs in the same directory: ref is pinned, omit it
uvx bc-pkg package validate
# For a TypeScript target:
uvx bc-pkg bigconfig-ai/once@typescript package validate
# For a Clojure target (Babashka + JDK + git auto-provisioned):
uvx bc-pkg bigconfig-ai/once@clojure package validate
;; bc-pkg itself is not a Clojure program. Pick either of the two launchers
;; — they behave identically — and point them at a Clojure target package.
;;
;; Using the npm launcher:
npx bc-pkg bigconfig-ai/once@clojure package validate
;; Or the PyPI launcher:
uvx bc-pkg bigconfig-ai/once@clojure package validate
;; The first run pins the target, downloads Babashka (BB_VERSION, default 1.12.196)
;; and Temurin JDK (JDK_VERSION, default 21) into a shared cache, installs git
;; on Linux if needed, then runs `bb run ...` against the pinned package.
Required external commands per target language:
| Target language | Launcher requires | End-user requires |
|---|---|---|
| TypeScript | — | node + npm on PATH |
| Python | — | python3 (or python) + uv on PATH |
| Clojure | tar on POSIX (PowerShell fallback on Windows) | Nothing — bb, JDK and git are auto-provisioned |
Spec format
The first positional argument can be a target spec or part of the forwarded command. It's parsed against:
SPEC := "<owner>/<repo>@<ref>"
ref := branch | tag | 40-char SHA
Anything matching that exact regex (FULL_SHA_RE, SPEC_RE in both implementations) is treated as a target spec; everything else is forwarded to the pinned target. Re-invocations without a spec are normal: bc-pkg package validate uses whatever was pinned by the first invocation.
Ref resolution
The launcher needs a 40-char SHA to pin. Resolution rules:
- If
refalready matchesFULL_SHA_RE, it's lowercased and used verbatim — no network round-trip. - Otherwise the launcher calls
GET /repos/{owner}/{repo}/commits/{ref}on the GitHub REST API (application/vnd.github+json, headerx-github-api-version: 2022-11-28) and usesdata.sha. - If
$GITHUB_TOKENis set it's sent asAuthorization: Bearer …. Necessary for private repos and for raising the unauthenticated rate limit. - A 404 is reported as a clean error; a non-OK status surfaces the HTTP code.
Once resolved, the SHA is written into the manifest under bigconfig.sha (or :bigconfig/sha for deps.edn). Subsequent runs read it back rather than re-resolving.
Target detection
At pin time the launcher fetches three files from the target repo via GitHub raw API:
deps.edn # → Clojure target
package.json # → TypeScript target
pyproject.toml # → Python target
Exactly one of those must exist. Zero is rejected ("has no deps.edn, package.json, or pyproject.toml"); more than one is rejected as ambiguous ("is ambiguous; found clojure, typescript manifests"). The match selects the target language. For TypeScript and Python the published package name is read out of the manifest; for Clojure the coordinate is synthesised as io.github.{owner}/{repo}.
The target package's root run file is also required — if missing, initialisation fails. That file is what every subsequent forwarded command actually invokes.
On-disk artifacts
After pin, the launcher leaves these files in cwd:
Clojure target
;; deps.edn (metadata + git dep)
{:deps {io.github.bigconfig-ai/once
{:git/url "https://github.com/bigconfig-ai/once.git"
:git/sha "<40-char-sha>"}}
:bigconfig/repo "bigconfig-ai/once"
:bigconfig/ref "clojure"
:bigconfig/sha "<40-char-sha>"
:bigconfig/language "clojure"
:bigconfig/run "run"}
;; bb.edn (runtime-only — Babashka reads bb.edn, not deps.edn)
{:deps {io.github.bigconfig-ai/once
{:git/url "https://github.com/bigconfig-ai/once.git"
:git/sha "<40-char-sha>"}}}
;; run (verbatim copy from the pinned target)
TypeScript target
// package.json
{
"type": "module",
"scripts": { "run": "node run" },
"dependencies": {
"once": "github:bigconfig-ai/once#<40-char-sha>"
},
"bigconfig": {
"repo": "bigconfig-ai/once",
"ref": "typescript",
"sha": "<40-char-sha>",
"language": "typescript",
"run": "run",
"packageName": "once"
}
}
// run (verbatim copy)
Python target
# pyproject.toml
[project]
name = "bigconfig-cli"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = [
"once @ git+https://github.com/bigconfig-ai/once.git@<40-char-sha>",
]
[tool.bigconfig]
repo = "bigconfig-ai/once"
ref = "python"
sha = "<40-char-sha>"
language = "python"
run = "run"
package-name = "once"
# run (verbatim copy)
Forwarded commands invoke that run file via the language-appropriate runner: node run (TypeScript), uv run python run (Python), or bb run (Clojure, with auto-provisioned Babashka + JDK).
Re-entry & validation
On every subsequent invocation:
read_metadata/readMetadatalooks for exactly one ofdeps.edn,package.json,pyproject.tomlwith abigconfig/:bigconfig/*/[tool.bigconfig]block. More than one is a hard error.- If the metadata is incomplete (missing repo, ref, sha, or language), bail.
- If a spec is also passed,
validate_existing_metadatacompares repo / ref / SHA. Any mismatch is fatal; the directory is not silently re-initialised. - If the
runfile is missing on disk, refetch it from the pinned SHA (restore_run_if_missing/restoreRunIfMissing). - Forward the rest of
argvto the target.
Babashka + JDK cache (Clojure targets only)
For Clojure targets the launcher provisions Babashka and a Temurin JDK into a user-wide cache so multiple projects share the download:
cacheRoot (macOS/Linux) = "$XDG_CACHE_HOME/bc-pkg" or "~/.cache/bc-pkg"
cacheRoot (Windows) = "%LOCALAPPDATA%/bc-pkg"
cacheRoot/
├── bb/<version>/bb # Babashka native binary
└── jdk/<feature>/… # Temurin JDK extracted tree
└── .javahome # marker pointing to JAVA_HOME inside the tree
Versions are pinned via the platform-resolver. Override with environment:
| Variable | Default | Effect |
|---|---|---|
BB_VERSION | 1.12.196 | Babashka release tag downloaded from github.com/babashka/babashka/releases. |
JDK_VERSION | 21 | Temurin feature version pulled from api.adoptium.net/v3/binary/latest/<v>/ga/<os>/<arch>/jdk/hotspot/normal/eclipse. |
GITHUB_TOKEN | — | Sent as Bearer for the metadata + commit-resolve API calls; needed for private repos. |
XDG_CACHE_HOME / LOCALAPPDATA | OS default | Cache root. |
Both launchers use install_once/installOnce: write into a sibling <final>.tmp-<pid>-…, then rename atomically. Safe under concurrent installs. Don't replace it with naive mkdir/rename.
Platform matrix (resolve_platform / resolvePlatform):
| OS | arch | Babashka asset | JDK asset |
|---|---|---|---|
| macOS | x64 | babashka-<v>-macos-amd64.tar.gz | mac/x64 |
| macOS | arm64 | babashka-<v>-macos-aarch64.tar.gz | mac/aarch64 |
| Linux | x64 | babashka-<v>-linux-amd64.tar.gz | linux/x64 |
| Linux | arm64 | babashka-<v>-linux-aarch64-static.tar.gz | linux/aarch64 |
| Windows | x64 | babashka-<v>-windows-amd64.zip | windows/x64 |
| Windows | arm64 | unsupported (no prebuilt babashka) | |
Linux git auto-install
For Clojure targets git is also required (Babashka resolves Git deps through it). On Linux only, if git isn't on PATH the launcher tries to install it:
- If
uid == 0, run commands directly; otherwise prefix withsudo. If neither root norsudois available, abort. - Probe for the first available package manager — in order
apt-get,dnf,yum,zypper,pacman,apk. - Run its install step (e.g.
apt-get update -ythenapt-get install -y git). Forapt-get, a failedupdateis non-fatal (the network refresh can flake; the subsequent install retries). - Verify
git --versionworks after the install.
macOS and Windows are not auto-installed: install git manually and re-run.
Parity guarantee
The PyPI launcher and the npm launcher must produce equivalent on-disk artifacts for the same <owner/repo@ref>. When you change one, mirror the other in the same change set. The contract is enforced by hand against bigconfig-ai/once@{clojure,typescript,python}; there is no automated CI gate.
| Concern | Source of truth |
|---|---|
| Spec / SHA regex | FULL_SHA_RE, SPEC_RE in both files |
| Manifest shapes | write_{clojure,typescript,python}_manifest in cli.py / write{Clojure,TypeScript,Python}Manifest in bc-pkg.js |
| Cache root layout | cache_root / cacheRoot |
Default BB_VERSION / JDK_VERSION | Constants at top of cli.py / bc-pkg.js |
| Linux git auto-install matrix | ensure_git / ensureGit |
| Re-init error semantics | validate_existing_metadata / validateExistingMetadata |
.venv/.../site-packages/resources to ./resources after uv sync, so the SDK renderer can find template data shipped via wheels. The TS launcher does not do this for Python targets. If a Python target ever ships resources/ via its npm-equivalent path, mirror the symlink in bc-pkg.js too.
Once · Package CLI
depends on SDKOnce is the user-facing CLI that bc-pkg bootstraps. Each invocation drives a profile through a six-stage provisioning pipeline: OpenTofu for compute, SMTP, DNS and SMTP post-verification, then Ansible to configure the freshly provisioned host and deploy applications listed under the profile's once.applications. Profiles pin one cloud provider (DigitalOcean / hcloud / OCI / no-infra) plus shared SMTP (Resend) and DNS (Cloudflare) sub-profiles.
All three implementations expose the same external behavior — same CLI shape, same profile structure, same templates, same step contracts. The Clojure version is the byte-equivalent reference for the once package build artifact.
CLI commands
The package-level commands chain steps together; individual tool runners drive one Tofu or Ansible invocation. Both forms work from the launcher (npx bc-pkg ... / uvx bc-pkg ...) and from a checked-out leaf.
# Package-level (chained workflows)
once package validate # pre-flight: schema, tools, creds, image refs, ssh-agent
once package describe # providers + SSH reachability + deployed-apps report
once package build # render all stages into .dist/ without applying
once package create # run all six stages
once package delete # reverse the four Tofu stages
once package validate create # validate first, create only if it passes
once validate # shortcut for `once package validate`
# Individual tool runners (each needs `render` first)
once tofu render tofu:init tofu:apply:-auto-approve
once tofu-smtp render tofu:init tofu:apply:-auto-approve
once tofu-dns render tofu:init tofu:apply:-auto-approve
once tofu-smtp-post render tofu:init tofu:apply:-auto-approve
once ansible render -- ansible-playbook main.yml
once ansible-local render -- ansible-playbook main.yml
# During dev (from once/typescript checkout):
npm run once -- package validate
# Package-level (chained workflows)
once package validate # pre-flight: schema, tools, creds, image refs, ssh-agent
once package describe # providers + SSH reachability + deployed-apps report
once package build # render all stages into .dist/ without applying
once package create # run all six stages
once package delete # reverse the four Tofu stages
once package validate create # validate first, create only if it passes
once validate # shortcut for `once package validate`
# Individual tool runners (each needs `render` first)
once tofu render tofu:init tofu:apply:-auto-approve
once tofu-smtp render tofu:init tofu:apply:-auto-approve
once tofu-dns render tofu:init tofu:apply:-auto-approve
once tofu-smtp-post render tofu:init tofu:apply:-auto-approve
once ansible render -- ansible-playbook main.yml
once ansible-local render -- ansible-playbook main.yml
# During dev (from once/python checkout):
uv run once -- package validate
;; The Clojure leaf uses a Babashka `run` script with task shortcuts.
bb run validate ;; strict shortcut for `bb run once package validate`
bb run once package validate ;; same, via workflow step (can be chained)
bb run once package describe ;; opt-in post-provisioning report
bb run once package build ;; render all stages into .dist/
bb run once package create ;; run all six stages
bb run once package delete ;; reverse the four Tofu stages
bb run once package validate create ;; validate, then create if it passes
bb run once package delete create ;; clean-slate redeploy
;; Individual tool runners (each needs `render` first)
bb run tofu render tofu:init tofu:apply:-auto-approve
bb run tofu-smtp render tofu:init tofu:apply:-auto-approve
bb run tofu-dns render tofu:init tofu:apply:-auto-approve
bb run tofu-smtp-post render tofu:init tofu:apply:-auto-approve
bb run ansible render -- ansible-playbook main.yml
bb run ansible-local render -- ansible-playbook main.yml
once package create does not call validate for you. Validation and describe are workflow steps you opt into — chain them explicitly (package validate create) or accept that create will fail late if a credential is missing.
Six-stage create pipeline
once package create threads opts through six stages. The first four are Tofu (and produce outputs consumed by later stages); the last two are Ansible.
~/.ssh/config so the host is reachable as Host oncedeploy user; deploy appsStage 6 is where the user's application list shows up: it installs Docker and the ONCE runtime on the host, sets up s-nail with the SMTP credentials so applications can send mail, then iterates over once.applications and runs each one. The deploy user gets a restricted shell (a ForceCommand Babashka script at /usr/local/bin/deploy) authorized by deploy-pubkey.
Delete pipeline
once package delete reverses the four Tofu stages: tofu-smtp-post → tofu-dns → tofu-smtp → tofu. The Ansible stages have nothing to destroy — the remote host is going away.
lifecycle { prevent_destroy = true } by default, so a casual delete is a no-op on the VM itself. To actually tear the box down, set BC_PAR_COMPUTE_PREVENT_DESTROY=false in the environment before invoking delete.
BC_PAR_COMPUTE_PREVENT_DESTROY=false once package delete
Profiles
Profiles are plain language-native data composed in options (options.ts / options.py / options.clj). Two layers compose into a profile: private sub-profile maps (cloud-provider × SMTP × DNS × backend × deploy keys) and public application profiles that pin a domain, package name and the list of apps deployed by Ansible.
Sub-profiles
- Compute:
oci,hcloud,digitalocean,no-infra-compute - SMTP:
resend(andno-infra-smtpfor already-provisioned mail) - DNS:
cloudflare(andno-infra-dns) - State backend:
s3,r2,local - Deploy:
compute-pubkey(private half lives inssh-agentso Ansible can reach the freshly provisioned VM) anddeploy-pubkey(authorised on the remotedeployuser)
Application profiles
| Profile | Compute | Backend | Apps deployed |
|---|---|---|---|
profile-alpha | DigitalOcean | R2 | once-bigconfig + once-caddy-redirect + once-forms |
profile-beta | OCI | R2 | once-bigconfig |
profile-gamma | OCI | R2 | once-pocketbase |
profile-no-infra | Existing server | Local | once-bigconfig |
const profileAlpha = compose(
resend, cloudflare, r2, digitalocean, deploy,
{
profile: "profile-alpha",
params: {
"domain": "alpha.example.com",
"package": "profile-alpha",
"once": {
applications: [
{ host: "www.alpha.example.com",
image: "ghcr.io/bigconfig-ai/once-bigconfig:latest" },
{ host: "alpha.example.com",
image: "ghcr.io/bigconfig-ai/once-caddy-redirect:latest" },
{ host: "forms.alpha.example.com",
image: "ghcr.io/bigconfig-ai/once-forms:latest",
env: ["[email protected]"] },
],
},
},
},
);
profile_alpha: Opts = compose(
resend, cloudflare, r2, digitalocean, deploy,
{
"profile": "profile-alpha",
"params": {
"domain": "alpha.example.com",
"package": "profile-alpha",
"once": {
"applications": [
{"host": "www.alpha.example.com",
"image": "ghcr.io/bigconfig-ai/once-bigconfig:latest"},
{"host": "alpha.example.com",
"image": "ghcr.io/bigconfig-ai/once-caddy-redirect:latest"},
{"host": "forms.alpha.example.com",
"image": "ghcr.io/bigconfig-ai/once-forms:latest",
"env": ["[email protected]"]},
]
},
},
},
)
(def profile-alpha
(merge-with merge resend cloudflare r2 digitalocean deploy
{::render/profile "profile-alpha"
::workflow/params
{:domain "alpha.example.com"
:package "profile-alpha"
:once {:applications
[{:host "www.alpha.example.com"
:image "ghcr.io/bigconfig-ai/once-bigconfig:latest"}
{:host "alpha.example.com"
:image "ghcr.io/bigconfig-ai/once-caddy-redirect:latest"}
{:host "forms.alpha.example.com"
:image "ghcr.io/bigconfig-ai/once-forms:latest"
:env ["[email protected]"]}]}}}))
(merge-with merge ...); TypeScript and Python ship a compose helper that does the same. Later layers win on conflicting param keys.
The active profile binding (bb)
Each leaf has a single top-level binding called bb (named after Babashka, the script runner) that selects which profile is "live". Switching profiles is one edit in options.*.
// once/typescript/src/once/options.ts
export const bb = profileAlpha; // change to profileBeta, profileGamma, profileNoInfra
# once/python/src/once/options.py
bb: Opts = profile_alpha # change to profile_beta, profile_gamma, profile_no_infra
;; once/clojure/src/clj/io/github/bigconig_ai/once/options.clj
(def bb profile-alpha) ;; change to profile-beta, profile-gamma, profile-no-infra
The profile name is also the leading segment of the build output directory: once package build writes into .dist/profile-alpha-<hash>/, so switching bb changes the rendered artifact path.
Parameter flow
Each stage receives an opts map whose params were built by composing three transformations:
BC_PAR_* env-var overrides on top of the profile defaultstofu-smtp stage's outputtofu stage's output// once/typescript/src/once/params.ts (sketched)
export const optsFn = (opts) =>
tofuParams(tofuSmtpParams(readBcPars(opts)));
# once/python/src/once/params.py (sketched)
def opts_fn(opts: Opts) -> Opts:
return tofu_params(tofu_smtp_params(read_bc_pars(opts)))
(def opts-fn (comp tofu-params tofu-smtp-params workflow/read-bc-pars))
In once package create and once package delete, the global opts feed into each stage; the per-template params at each stage are reconstructed from BC_PAR_* env vars plus the previous stage's Tofu output, not from options.* directly. The profile's params map is what validate / describe and the individual tool runners read.
BC_PAR_* environment-variable overrides
Any param can be overridden at runtime by exporting an env var. The variable name is the param's kebab-case key, uppercased, with hyphens and dots converted to underscores, prefixed with BC_PAR_.
provider-backend → BC_PAR_PROVIDER_BACKEND
hcloud-token → BC_PAR_HCLOUD_TOKEN
oci-availability-domain → BC_PAR_OCI_AVAILABILITY_DOMAIN
compute.prevent-destroy → BC_PAR_COMPUTE_PREVENT_DESTROY
export BC_PAR_DOMAIN="alpha.example.com"
export BC_PAR_PROVIDER_BACKEND="s3" # or "r2" / "local"
export BC_PAR_S3_BUCKET="my-tf-state-bucket"
export BC_PAR_HCLOUD_TOKEN="xxx"
.envrc.private (gitignored) per leaf — the file is sourced by direnv via the leaf's .envrc. Never put a token, key or password in options.* — replace them with REPLACE_ME and let .envrc.private supply the real value.
Validation step
once package validate (or the shortcut once validate) is an opt-in workflow step. It runs the following checks against the active profile and fails fast on any of them:
- Schema: the profile is structurally valid (Clojure uses Malli; TS/Python use equivalent runtime checks).
- Tools: the CLIs required by the active providers are on
PATH—tofu,ansible-playbook,ssh, plus per-provider helpers (doctl,hcloud,oci, etc.). - Credentials: the credentials referenced by the profile (
do-token,hcloud-token, OCI config, Cloudflare token, Resend API key, R2/S3 keys) are non-placeholder. - Image refs: the application images named in
once.applicationsresolve viaskopeo. - ssh-agent: the private half of
compute-pubkeyis loaded in the activessh-agent(cloud providers only —no-infrais skipped).
validate / describe are workflow steps that thread side-effecting collaborators through validate-report / validateReport / validate_report — pure builders that accept the side-effect helpers as injected parameters so tests can stay process-free.
The CLI shortcut once validate exits non-zero if any extra args are passed; chain validate into a workflow with once package validate create when you want "validate, then create only if validation passes".
Describe step
once package describe is an opt-in post-provisioning report. For the active profile it prints:
- The configured providers (compute / SMTP / DNS / backend).
- SSH reachability for the provisioned host.
- For every entry in
once.applications: image, tag, currently-running digest, the registry's digest, and an "update available?" flag.
It's read-only and idempotent — describe never changes provider state. The pure helpers (describeReport / describe_report / describe-report) accept an injected runner so tests can replay canned tofu output --json + skopeo inspect + ssh responses instead of hitting the network.
Template structure
Per-stage templates live under each leaf at:
src/resources/io/github/bigconig-ai/once/tools/
├── tofu/ # Multi-cloud .tf templates
│ ├── digitalocean/
│ ├── hcloud/
│ ├── oci/
│ └── no-infra/
├── tofu-backend/ # Remote state backend templates
│ ├── s3/
│ ├── r2/
│ └── local/
├── tofu-smtp/ # SMTP (Resend) setup
├── tofu-dns/ # DNS (Cloudflare)
├── tofu-smtp-post/ # SMTP post-verification
├── ansible/ # Remote host playbooks (incl. files/deploy bb script)
└── ansible-local/ # Local machine playbooks
The Selmer parser is configured with non-default delimiters in tools.*:
| Selmer setting | Value in Once | What it does |
|---|---|---|
tag-open / tag-close | < / > | Tag delimiters are <% … %> — but tags aren't used in Once templates. |
filter-open / filter-close | { / } | Variables in file content are written as <{ var }>. The literal {{ … }} in templates passes through untouched, so downstream Ansible (which uses {{ … }} for its own variables) is happy. |
Provider switching is directory-level. Each render step passes a :transform [[<provider> delimiters]] entry ([["digitalocean", delimiters]], [["oci", delimiters]], …) and the SDK renderer copies only the matching subdirectory under tools/<step>/. The rendered output lands in .dist/<profile-name>-<hash>/<module>/.
Plugin system
Once registers one pluggable step with the SDK's step registry: render-tofu-backend. After every render step, the handler reads provider-backend from params ("s3" / "r2" / "local") and writes the matching backend block into the just-rendered Tofu directory. That keeps state-backend logic out of the per-provider tofu/ templates and isolates it in one place.
// once/typescript/src/once/tools.ts (sketched)
registerHandleStep("io.github.bigconig-ai.once.tools/render-tofu-backend", handler);
# once/python/src/once/tools.py (sketched)
register_handle_step("io.github.bigconig-ai.once.tools/render-tofu-backend", handler)
(pluggable/handle-step ::render-tofu-backend ...)
Build artifact contract
once package build renders every stage into <leaf>/.dist/<profile-name>-<hash>/ without applying anything. For the canonical profile-alpha profile, the expected output is:
once/<lang>/.dist/profile-alpha-d2264632/
bb once build in once/clojure) is the reference. The Python and TypeScript implementations should produce byte-equivalent output at this path. This is the test suite's smoke target for cross-language compatibility.
.dist/ is generated output — never edit it as source. The renderer overwrites everything inside it on every build.
SDK · Engine
depends on SelmerThe SDK is the workflow and rendering engine that once sits on top of. The Clojure SDK (big-config/clojure), Python SDK (big-config/python), and TypeScript SDK (big-config/typescript) expose the same public surface: workflow primitives, a step-fn middleware system, a pluggable step dispatcher, a Selmer-based file-tree renderer, a swappable shell runner, Git-tag pessimistic locking, and OpenTofu construct helpers (big-tofu). Reserved keys are verbatim string identifiers (e.g. "big-config/exit") that flow through unchanged across all three languages.
Each implementation pulls in its own Selmer port: Clojure uses selmer/selmer 1.13.1 from Maven, Python and TypeScript use the GitHub-pinned ports under ../selmer/.
Concepts
A workflow is a state machine over an opts map. Each step is a function (opts) → opts. The state machine's wire function looks at the current step name and decides which implementation function to run next; the implementation returns a new opts map; the wire function looks at the result to pick the next step. Steps signal success/failure via reserved keys in the opts map, not exceptions.
opts
A plain map/dict/object keyed by namespaced strings. Engine keys, workflow keys, step-specific keys, and profile params all live in this single map.
step-fn
A wrapper of shape (f, step, opts) → opts that runs f in the middle and gets before- and after-hook access. Used for logging, tap-into-REPL, exit-code-on-error, etc.
Reserved opts keys
All three implementations preserve namespaced string keys verbatim. They are never rewritten to snake_case or camelCase at the boundary.
| Key | Meaning |
|---|---|
big-config/exit | Exit code; 0 means the step (or chain) succeeded. |
big-config/err | Human-readable error message (set when exit ≠ 0). |
big-config/stack-trace | Stack trace (when the failure had one). |
big-config/env | Environment variables snapshot relevant to the run. |
big-config/procs | Subprocess records (set by the shell runner). |
big-config/steps | Trace of step names visited. |
big-config/test-mode | Truthy when running under test harness (skips certain side effects). |
big-config.workflow/steps | Pipeline of step keywords for the current workflow. |
big-config.workflow/params | Profile / per-stage parameters for rendering. |
big-config.workflow/name | Workflow identifier. |
big-config.workflow/prefix | Path prefix for nested workflows. |
big-config.workflow/{create,build,delete,validate,describe}-fn | Workflow-level handlers for top-level commands. |
big-config.render/{templates,module,profile} | Renderer inputs and output naming. |
big-config.run/{shell-opts,cmds,run-cmd,dir} | Shell runner inputs and accumulator. |
big-config.lock/{owner,lock-keys,lock-details,lock-name,tag-content} | Git-tag locking state. |
big-config.git/{prev,current,origin}-revision, upstream-name | Git helpers state. |
Per-language access
import { EXIT, ERR, WF_STEPS, WF_PARAMS } from "big-config/keys";
if (opts[EXIT] !== 0) handle(opts[ERR]);
from big_config import EXIT, ERR
from big_config.workflow import STEPS as WF_STEPS, PARAMS as WF_PARAMS
if opts[EXIT] != 0:
handle(opts[ERR])
(require '[big-config.workflow :as workflow])
(when (pos? (:big-config/exit opts))
(handle (:big-config/err opts)))
Workflow primitives
Lives in core (core.ts / core.py / core.clj):
| Concept | Clojure | Python | TypeScript |
|---|---|---|---|
| Workflow constructor | create-workflow | workflow | createWorkflow |
| Step-fn constructor | create-step-fn | step_fn | createStepFn |
| Success terminal | ok | ok | ok |
| Branch on success/failure | choice | choice | choice |
A minimal workflow
import { createWorkflow, ok } from "big-config/core";
const START = "example/start";
const END = "example/end";
function wire(step, _stepFns) {
if (step === START) return [ok, END];
return [(opts) => opts, null];
}
const wf = createWorkflow({ firstStep: START, wireFn: wire });
const result = wf([], {});
from big_config.core import ok, workflow
START = "example/start"
END = "example/end"
def wire(step, _step_fns):
if step == START:
return ok, END
return (lambda opts: opts), None
wf = workflow({"first_step": START, "wire_fn": wire})
result = wf([], {})
(require '[big-config.core :as core])
(def START :example/start)
(def END :example/end)
(defn wire [step _step-fns]
(if (= step START)
[core/ok END]
[identity nil]))
(def wf (core/create-workflow {:first-step START :wire-fn wire}))
(wf [] {})
Composition layer
The workflow namespace adds higher-level orchestration on top of core:
| Concept | Clojure | Python | TypeScript |
|---|---|---|---|
| Compose a named pipeline | create-workflow-star | workflow_star | createWorkflowStar |
| Run named-step sequence | run-steps | run_steps | runSteps |
| Parse CLI args into steps | parse-args | parse_args | parseArgs |
Read BC_PAR_* overrides | read-bc-pars | read_bc_pars | readBcPars |
| Merge per-tool params | merge-params | merge_params | mergeParams |
| Compute new step prefix | new-prefix | new_prefix | newPrefix |
| Step name → output path | path | path | path |
| Register a workflow | register-workflow | register_workflow | registerWorkflow |
The recognised step names in CLI argument parsing (parse-args-steps / PARSE_ARG_STEPS / parseArgSteps):
lock · git-check · render · create · build · delete
validate · describe · exec · git-push · unlock-any
An argument like tofu:init becomes tofu init; the trailing -- appends the rest of the tokens as a single raw command string consumed by the exec step.
Renderer
The render module copies a templated directory tree into .dist/, evaluating Selmer in file contents and selecting subdirectories by variable.
- File content uses the SDK delimiter customisation (e.g.
<{ var }>) — the literal{{ … }}is left intact for downstream tools. - Directory names use the standard Selmer
{{ var }}delimiters — a directory called{{ provider }}gets replaced by the value ofproviderat render time, and (more often) a top-level:transform [[<val> delimiters]]entry selects which subdirectory to copy. - The render data is the merged
paramsplustarget-object/module/profileread out of the opts map. - Reserved keys:
big-config.render/templates(input root),big-config.render/module(output sub-path),big-config.render/profile(profile name used to derive the output directory).
The renderer is what powers Once's six-stage build — each stage is a separate render step that lands a different module under .dist/<profile>-<hash>/.
Shell runner
Shell-command execution is gated behind a single seam so tests can swap in a fake runner instead of spawning processes:
| Concept | Clojure | Python | TypeScript |
|---|---|---|---|
| Default runner | big-config.run/default-runner | big_config.run.default_runner | defaultRunner |
| Active runner var | big-config.run/*runner* | big_config.run.runner | runner (mutable export) |
| Swap runner | binding | set_runner / context manager | setRunner / withRunner |
| Generic command | generic-cmd | generic_cmd | genericCmd |
| Mktemp create / remove | mktemp-create-dir / mktemp-remove-dir | mktemp_create_dir / mktemp_remove_dir | mktempCreateDir / mktempRemoveDir |
Each command produces a ProcResult (exit code, stdout, stderr); handleCmd / handle_cmd writes it into opts under big-config.run/run-cmd and updates big-config/exit.
big-config/exit and big-config/err; the next step's wire function decides what to do. Don't add try/catch around step functions — let the engine route the failure.
Locking
big-config.lock implements pessimistic locking via Git tags. A workflow that needs exclusive access pushes a tag with a content payload (the lock owner) under big-config.lock/lock-name; concurrent runs see the existing tag and bail. big-config.unlock is the force-release counterpart.
| Step | Clojure | Python | TypeScript |
|---|---|---|---|
| Generate lock id | generate-lock-id | generate_lock_id | generateLockId |
| Create local tag | create-tag | create_tag | createTag |
| Push tag to remote | push-tag | push_tag | pushTag |
| Delete local + remote tag | delete-tag / delete-remote-tag | delete_tag / delete_remote_tag | deleteTag / deleteRemoteTag |
| Read remote tag content | get-remote-tag / read-tag | get_remote_tag / read_tag | getRemoteTag / readTag |
| Verify ownership | check-tag / check-remote-tag | check_tag / check_remote_tag | checkTag / checkRemoteTag |
The composed workflow is big-config.lock/lock (assembled in lock.{ts,py,clj}). It runs the steps above end-to-end; chain it before any expensive provisioning step that should not run concurrently.
Git helpers
big-config.git provides workflow steps that capture the current Git revision, the upstream branch, and the origin revision — useful before/after a provisioning run to record the exact code state. Outputs land under big-config.git/{prev,current,origin}-revision and big-config.git/upstream-name.
Pluggable steps
The pluggable-step system lets downstream packages (like Once) hook into a step's dispatch by name. The wire function calls handle-step; if the step name has been registered with register-handle-step (Clojure's defmethod-style multimethod), the registered handler runs instead of (or alongside) the default implementation.
| Operation | Clojure | Python | TypeScript |
|---|---|---|---|
| Register handler | defmethod handle-step | register_handle_step | registerHandleStep |
| Remove handler | remove-method | remove_handle_step | removeHandleStep |
| Clear all handlers | (clear-handle-steps) | clear_handle_steps | clearHandleSteps |
The pluggable layer also re-exports create-workflow-star so a downstream caller can construct a plugin-aware workflow with one import.
Step-fn middleware
big-config.step-fns ships a small library of step-fn wrappers used by the CLI and by Once:
create-exit-step-fn/createExitStepFn— convert a non-zerobig-config/exitintoprocess.exitat the named terminal step.create-print-error-step-fn/createPrintErrorStepFn— printbig-config/errto stderr at the named terminal step.tap-step-fn/tapStepFn— tap each opts map into a REPL/inspector for debugging.log-step-fn/logStepFn— log step names as they run.bling-step-fn/blingStepFn— emit pretty terminal output (the CLI's default).
SDK Selmer filters
big-config.selmer-filters exposes SDK-flavoured helpers around Selmer's delimiters, including normalize-delimiters / normalizeDelimiters (default delimiter resolution, including tagOpen, tagClose, filterOpen, filterClose, tagSecond, shortCommentSecond) and whitespace-control / whitespaceControl (Selmer's whitespace-trim helper applied with the SDK's delimiters). Downstream packages call these instead of hard-coding Selmer's defaults.
big-tofu (OpenTofu construct helpers)
The big-tofu/ subpackage models OpenTofu resources as language-native data, then serialises them into the rendered Terraform/Tofu modules. Two files: core (the Construct type, name encoding, reference helpers) and create (small library of opinionated resource constructors: bucket, sqs, kms, provider, …).
| Helper | Clojure | Python | TypeScript |
|---|---|---|---|
| Resource type | big-config.big-tofu.core/->Construct | big_tofu.core.Construct | Construct |
| Reference an attribute | reference | reference | reference |
| FQN → name | fqn->name | fqn_to_name | fqnToName |
| AWS caller identity datasource | caller-identity | caller_identity | callerIdentity |
| Construct an S3 bucket | bucket | bucket | bucket |
| SQS queue | sqs | sqs | sqs |
| KMS key | kms | kms | kms |
| Backend provider block | provider | provider | provider |
CLI
Each language SDK exposes the SDK DSL through its own CLI binary (used directly when working on the engine, or indirectly through once):
# From a big-config/typescript checkout
npm run build
./dist/cli.js render lock tofu:init -- tofu apply -auto-approve
# From a big-config/python checkout
uv sync
uv run big-config render lock tofu:init -- tofu apply -auto-approve
;; From a big-config/clojure checkout
clojure -M:run -- render lock tofu:init -- tofu apply -auto-approve
Parsing rules (shared across the three implementations):
- Tokens that match a registered step name (
lock,render,create, …) become steps in the workflow. tool:subcommandexpands to two tokens (tool subcommand).- The literal
--separator collects everything after it into a single raw command string consumed by theexecstep.
Selmer · Templates
Django-styleSelmer is a Django-inspired template engine. The Clojure implementation (originally yogthos/Selmer, here as selmer/clojure) is the upstream reference; selmer/python and selmer/typescript are dependency-free ports that preserve the public API shape (variables, filters, tags, includes, template inheritance, validation, caching, custom tags/filters, custom delimiters). License: EPL-1.0.
selmer/clojure CLJ
Selmer 1.13.1 on Maven (selmer/selmer). Has build.clj + Kaocha tests + :deps/prep-lib for AOT-compiled selmer.node when consumed as a Git dep.
selmer/python PY
No runtime deps. Pytest. Mutating-name convention: both add_filter and add_filter_bang exported. py.typed marker shipped.
selmer/typescript TS
No runtime deps. Vitest. Mutating-name convention: both addFilter and addFilter$ exported. ESM-only.
Quick start
import { render } from "selmer";
render("Hello {{ name|upper }}!", { name: "Ada" });
// → "Hello ADA!"
from selmer import render
render("Hello {{ name|upper }}!", {"name": "Ada"})
# → "Hello ADA!"
(require '[selmer.parser :refer [render]])
(render "Hello {{ name|upper }}!" {:name "Ada"})
;; ⇒ "Hello ADA!"
Rendering a file from disk:
import { renderFile, setResourcePath } from "selmer";
setResourcePath("./templates"); // pins the lookup root
renderFile("page.html", { title: "Home" });
from selmer import render_file, set_resource_path
set_resource_path("./templates")
render_file("page.html", {"title": "Home"})
(require '[selmer.parser :refer [render-file set-resource-path!]])
(set-resource-path! "./templates")
(render-file "page.html" {:title "Home"})
Template language
Variables
Variable expressions are wrapped in {{ … }} by default. Dotted paths walk maps and array indices; the double-dot escape preserves a literal namespace slash:
{{ name }}
{{ person.address.0.street }}
{{ foo..bar/baz }} # namespaced key: foo {"bar/baz" ...}
Filters
A pipe applies a filter; filters chain left to right and take arguments separated by colons:
{{ name|upper }}
{{ price|currency-format:"en-US" }}
{{ items|sort-by:"priority"|first }}
Tags
Tags are wrapped in {% … %}. Block tags have a matching closing tag ({% endif %}, {% endfor %}, …):
{% if user.admin %}admin{% else %}guest{% endif %}
{% for item in items %}- {{ item }}
{% endfor %}
{% block sidebar %}…{% endblock %}
{% extends "base.html" %}
{% include "partial.html" %}
Public API
| Operation | Clojure | Python | TypeScript |
|---|---|---|---|
| Render a template string | selmer.parser/render | selmer.render | render |
| Render from disk | render-file | render_file | renderFile |
| Parse to AST | parse / parse-input | parse_string / parse_file / parse_input | parse / parseString / parseFile |
| Render parsed AST | render-template | render_template | renderTemplate |
| Register custom filter | add-filter! | add_filter / add_filter_bang | addFilter / addFilter$ |
| Register custom tag | add-tag! | add_tag / add_tag_bang | addTag / addTag$ |
| Toggle template cache | cache-on! / cache-off! | cache_on / cache_off (+ _bang) | cacheOn / cacheOff (+ $) |
| Clear template cache | (reset! selmer.parser/templates {}) | clear_cache | clearCache |
| Pin resource lookup root | set-resource-path! | set_resource_path | setResourcePath |
| Toggle validation | (validate-on!) / (validate-off!) | validate_on / validate_off | validateOn / validateOff |
| Static analysis | known-variables | known_variables | knownVariables |
! bang for side-effecting functions. Python ports it as _bang (since ! isn't a valid identifier character), TypeScript as $. Both forms are exported by Python and TypeScript — pick whichever reads better. Code that wants to look like a Clojure port can use add_filter_bang / addFilter$; code that wants idiomatic Python or TS can use add_filter / addFilter.
Built-in filters
The bundled filter library, grouped by intent. Filter names are kebab-case in all three implementations (TypeScript keeps the kebab-case keys verbatim in the filter map):
Strings
strupperlowercapitalizetitlenamesubsaddslashescenterreplaceremoveremove-tagsurlescapeabbreviateabbr-leftabbr-middleabbr-rightabbr-ellipsislinebreakslinebreaks-brlinenumbers
Numbers
addmultiplydivideroundnumber-formatcurrency-formatdouble-formatget-digitbetween?
Collections
firstlasttakedropdrop-lastjoinsortsort-bysort-reversedsort-by-reversedlengthcountlength-iscount-isempty?not-emptyrangerand-nthgetpluralize
Dates & misc
datedefaultdefault-if-emptyhashjsonemailphonesafe
Each implementation lets you register more via add-filter! / add_filter / addFilter.
Custom delimiters
The four delimiters are the most-used escape hatch: pass them via the per-call options object to make Selmer's tag/filter delimiters something other than {% / %} and {{ / }}. This is exactly what the SDK does to keep its templates compatible with downstream Ansible (which itself uses {{ … }}).
| Delimiter | Clojure | Python | TypeScript |
|---|---|---|---|
Tag opener (default {%) | :tag-open | tag_open | tagOpen |
Tag closer (default %}) | :tag-close | tag_close | tagClose |
Filter opener (default {{) | :filter-open | filter_open | filterOpen |
Filter closer (default }}) | :filter-close | filter_close | filterClose |
import { render } from "selmer";
render("hello <{ name }>",
{ name: "world" },
{ tagOpen: "<", tagClose: ">",
filterOpen: "{", filterClose: "}" });
// → "hello world" ; literal {{ ... }} in the template passes through
from selmer import render
render("hello <{ name }>",
{"name": "world"},
tag_open="<", tag_close=">",
filter_open="{", filter_close="}")
# → "hello world" ; literal {{ ... }} in the template passes through
(require '[selmer.parser :refer [render]])
(render "hello <{ name }>"
{:name "world"}
{:tag-open \< :tag-close \>
:filter-open \{ :filter-close \}})
;; ⇒ "hello world" ; literal {{ ... }} in the template passes through
Extensibility
Custom filter
import { addFilter, render } from "selmer";
addFilter("shout", (s) => String(s).toUpperCase() + "!");
render("{{ x|shout }}", { x: "hi" }); // → "HI!"
from selmer import add_filter, render
add_filter("shout", lambda s: str(s).upper() + "!")
render("{{ x|shout }}", {"x": "hi"}) # → "HI!"
(require '[selmer.filters :refer [add-filter!]])
(require '[selmer.parser :refer [render]])
(add-filter! :shout (fn [s] (str (.toUpperCase (str s)) "!")))
(render "{{ x|shout }}" {:x "hi"}) ;; ⇒ "HI!"
Custom tag
import { addTag, render } from "selmer";
addTag("greet", (args) => `hello ${args[0]}!`);
render("{% greet world %}", {}); // → "hello world!"
from selmer import add_tag, render
add_tag("greet", lambda args, _ctx: f"hello {args[0]}!")
render("{% greet world %}", {}) # → "hello world!"
(require '[selmer.parser :refer [add-tag! render]])
(add-tag! :greet (fn [args _context-map] (str "hello " (first args) "!")))
(render "{% greet world %}" {}) ;; ⇒ "hello world!"
Caching & validation
Selmer caches parsed templates by source. Toggle the cache while debugging:
import { cacheOff, cacheOn, clearCache } from "selmer";
cacheOff(); // useful in dev — re-parse every render
cacheOn();
clearCache(); // drop the cache without disabling it
from selmer import cache_off, cache_on, clear_cache
cache_off()
cache_on()
clear_cache()
(require '[selmer.parser :refer [cache-off! cache-on!]])
(cache-off!)
(cache-on!)
Validation runs at parse time and catches mismatched {% if %} / {% endif %}, unknown tags, etc. Disable with validateOff() / validate_off() / (validate-off!) when you want render-time errors instead (useful when programmatically registering tags after parse).
Validation errors raise a typed exception:
| Concept | Clojure | Python | TypeScript |
|---|---|---|---|
| Validation error type | selmer.validator/->ValidationException | SelmerValidationError | SelmerValidationError |
| Pre-baked error page template | resources/error.html | selmer/resources/error.html | resources/error.html |
Internal seams
If you're extending the engine (or porting changes from upstream into the Python/TypeScript siblings), these are the internal modules behind the public surface:
| Concern | Clojure | Python | TypeScript |
|---|---|---|---|
| Tokenizer | (implicit in parser.clj) | scanner.py | scanner.ts |
| Node tree | selmer.node (AOT-compiled) | node.py | node.ts |
| Filter expression parser | selmer.filter-parser | filter_parser.py | filter-parser.ts |
| Tag handlers | selmer.tags | tags.py | tags.ts |
| Built-in filters | selmer.filters | filters.py | filters.ts |
| Inheritance + include resolver | selmer.template-parser | template_parser.py | template-parser.ts |
| Validator | selmer.validator | validator.py | validator.ts |
| Safe-string handling | (implicit in util.clj) | safe.py | safe.ts |
| Ring / middleware glue | selmer.middleware | middleware.py | middleware.ts |
| Shared helpers | selmer.util | util.py | util.ts |
selmer/clojure first and port across to selmer/python and selmer/typescript in the same change set. Stdlib-only / no-dependency constraints on the ports are load-bearing — adding a runtime dep breaks downstream SDK users of big-config.