Run pravo in a pipeline

A pipeline needs four answers: does the specification evaluate, what changes, what exactly was applied and did anything change outside pravo. Each is a command with an exit code.

Work in an empty directory. It stands for the repository. The root is app/ and the policy is policy/plan.cue, outside the root. Every command uses -C app, which runs pravo as if it started in app/.

package ci

import (
	rnd "pravo.dev/providers/hashicorp/random"
	loc "pravo.dev/providers/hashicorp/local"
)

// Pinned, so that a pipeline cannot pick up a new major version unnoticed.
$random: rnd.#Provider & {$version: "~> 3.6"}
$local: loc.#Provider & {$version: "~> 2.5"}

server: $random.#Resource.random_pet & {length: 2}
port: $random.#Resource.random_integer & {min: 8000, max: 8999}

note: $local.#Resource.local_file & {
	filename: "note.txt"
	content:  "\(server.id)\n"
}

// A managed resource is neither deleted nor replaced.
changes: [...{
	addr:   string
	mode:   string
	action: string
	if mode == "managed" && (action == "delete" || action == "replace") {
		error("\(addr): \(action) is not allowed")
	}
}]

1. Check without credentials

pravo init installs the providers and generates the packages that validate, plan and apply need. pravo validate evaluates the specification against them. It starts no provider and needs no credentials.

formatted
Success! 3 resources, 0 data sources, 0 ephemeral, 0 function calls, 0 outputs across 2 providers.

pravo fmt prints the name of each file it rewrites, so a clean tree prints nothing and test -z passes. A mistake in the specification gives exit code 6 from validate.

package ci

server: length: "two"

Error: evaluating ./app:
server.length: conflicting values "two" and 2 (mismatched types string and int):
    ./app/bad.cue:3:17
    ./app/main.cue:12:49

plan and apply report the same error with exit code 6. A pipeline tells a mistake in the code from a failed plan by the code.

2. Ask whether a change is pending

Apply once. Then plan -detailed-exitcode exits 0 when the plan is empty and 2 when it has changes.

Apply complete! Resources: 3 added, 0 changed, 0 destroyed. [txn <id>]
exit=0

Add a resource.

package ci

extra: $random.#Resource.random_pet & {length: 2}

exit=2

A job reads the code from a command that is not in a pipe, because a pipe hides it.

Exit code 0 means that the plan found no change among the resources it looked at: the ones a change reaches and the ones they read. It does not mean that every resource matches its object. A plan does not read the other records, unless it is asked to with -refresh=expired or -refresh=all or a resource sets a refresh age of its own that has run out, so it is not a drift detector. Step 5 finds drift.

3. Read a plan as data

Every command accepts --json. Standard output carries the data, and progress goes to standard error. plan prints one document.

{"add":1,"change":0,"destroy":0,"read":0,"blast_radius":1}
[{"addr":"extra","action":"create"}]

The addr of a change is the address of the resource. A pipeline can post the summary on a pull request or compare a number with a budget. apply prints one JSON event per line and ends with a summary. A plain apply --json that has changes to make needs -auto-approve, because there is no prompt.

4. Plan, then apply the file

The plan job saves a plan under the policy. The apply job applies that file and names the policy itself.

Plan: 1 to add, 0 to change, 0 to destroy.
Policy ../policy/plan.cue (sha256:<digest>) is satisfied.
Saved the plan to change.plan. To apply exactly these actions, run:

--actor and --reason are recorded on the plan and on the transaction. They are labels that the caller supplies and pravo does not verify. pravo show prints the plan with its label, and --json gives the same as data.

Saved plan created <time> by "pravo -C app plan --out change.plan --actor ci --reason build 42 --policy ../policy/plan.cue" (actor ci)

A saved plan is checked against the specification, the provider versions and the state it touches before anything runs. Change the specification after the plan was made.

package ci

extra: $random.#Resource.random_pet & {length: 3}

Error: saved plan is stale; run "pravo plan" again:
  - the configuration changed since the plan was made

Exit code 3 means that the plan is stale. Plan again and apply the new file.

Policy ../policy/plan.cue (sha256:<digest>) is satisfied.
extra: Creating...
extra: Creation complete after 0s [id=<id>]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed. [txn <id>]

Under a policy an apply is of a saved plan. A plain pravo apply --policy ... is refused with exit code 5. The second plan that it makes is not the one that was judged. Write and roll out plan policies covers the policy.

5. Find drift on a schedule

pravo drift reads every resource from its provider, ignores refresh ages and exits 2 when something changed outside pravo. Delete the file behind pravo's back.

Drift. 1 of 4 resources changed outside pravo:
  note: create
    - refresh found the real state differs from the state (drift)

With -webhook <url> the report is also posted as JSON, so a scheduler can hand drift to whoever triages it.

pravo drift reads every record, and an estate of many records takes long to read. Two other jobs read fewer.

• pravo drift --due --limit N reads the records that are past their refresh age, the oldest first, at most N of them, and plans nothing. Each run costs what it reads. A job that runs it every few minutes reads the whole estate in turn, and a record that was never read goes first. It exits 2 when it finds drift.
• pravo plan -refresh=expired reads every record that is past its refresh age and plans the specification as well. Use it where a job must see both.

A read that finds drift leaves a mark on the record. The next plan, saved plan and apply act on it, whichever job found it, and it stays until an apply writes the resource.

6. Test without providers or credentials

A job that only asks whether a change to the specification still produces the expected plan needs no provider and no credentials. Record the responses once. Replay them in every run.

Record a plan from an empty state in a scratch copy of the root.

Plan: 4 to add, 0 to change, 0 to destroy.
local
random

The recordings directory holds the provider's responses, one directory per provider. A response carries the attributes of the objects the provider returned, the sensitive ones included, and the result of an ephemeral resource. A recording is as sensitive as state, so handle it as you handle state. pravo creates its directories with mode 0700 and its files with mode 0600. It does not write the values of a request, so the credentials in a provider configuration and write-only attributes are not in it.

Version control keeps no file modes. Commit the recording with the specification only when no resource in it returns a secret. Otherwise keep it where the pipeline keeps state, and fetch it before the job. A fresh clone of the repository with an empty provider store replays the recordings.

Configuration does not load yet; generating schemas for the declared providers first.
Created cue.mod/module.cue
Providers:
  local = hashicorp/local v2.9.1
  random = hashicorp/random v3.9.1
Up to date cue.mod/gen/pravo.dev/providers/hashicorp/local (2 resource types, 3 data sources, 1 ephemeral, 1 functions)
Up to date cue.mod/gen/pravo.dev/providers/hashicorp/random (10 resource types, 0 data sources, 2 ephemeral, 0 functions)

pravo has been initialised. Run "pravo plan" to see what it would do.
Plan: 4 to add, 0 to change, 0 to destroy.

The clone has an empty provider store, and replay answers from the recordings. No provider binary was downloaded:

the provider store is empty

A change that alters what pravo asks the provider is a missing recording.

Error: extra: planning: PlanResourceChange random_pet: no recording for this request (replaying from recordings/random)

When the change is intended, record again and store the new recordings where the old ones are. When it is not, you caught it before a real provider saw it. Replay shows that pravo and the specification still agree on the requests. It does not show that a real provider behaves the same; pravo drift on a schedule covers that.

7. Refuse a replacement

Change server so that the provider has to replace it.

package ci

import (
	rnd "pravo.dev/providers/hashicorp/random"
	loc "pravo.dev/providers/hashicorp/local"
)

// Pinned, so that a pipeline cannot pick up a new major version unnoticed.
$random: rnd.#Provider & {$version: "~> 3.6"}
$local: loc.#Provider & {$version: "~> 2.5"}

server: $random.#Resource.random_pet & {length: 3}
port: $random.#Resource.random_integer & {min: 8000, max: 8999}

note: $local.#Resource.local_file & {
	filename: "note.txt"
	content:  "\(server.id)\n"
}

exit=5
Plan: 2 to add, 0 to change, 1 to destroy.
Error: the plan violates the policy ../policy/plan.cue (sha256:<digest>):
changes.1: server: replace is not allowed:
no plan file

The plan is printed first and judged afterwards. Exit code 5 means that a policy refused the run, and no plan file was written. With --json, standard output carries the plan document and then the error object, so a consumer takes the object whose kind is error.

Exit codes

With --json, an error also prints {"code":N,"kind":"error","message":"..."} on standard output. The lookup page is Exit codes.

One pipeline

The commands of this page, in the jobs that run them:

pull request   pravo init; test -z "$(pravo fmt)"; pravo validate
               PRAVO_REPLAY=recordings pravo plan --json > plan.json
plan job       pravo plan -out change.plan --actor ci --reason "$REV" --policy "$POLICY"
apply job      pravo apply change.plan --policy "$POLICY"
schedule       pravo drift --due --limit 500 --actor drift-cron

validate, plan and apply need the generated packages under cue.mod/gen. A job runs pravo init first, or the repository commits cue.mod/gen. The $POLICY file is checked out from a protected branch, and the apply job is defined where the authors of the roots cannot change it unreviewed.

What next

• Write and roll out plan policies writes the policy file and tests it.
• Let an agent plan and apply adds reservations and the tool server.
• Find out why a run fails reads the log of a job that failed.