State

pravo records what it manages in a state graph. Every resource and every data source is a node with its own record, its own revision and its own lock. A dependency is an edge between two nodes. There is no state file and no lock on the whole state. Cost follows the change says what that gives a large estate.

A directory of records

The default state is a directory next to the specification, pravo.state. Work in an empty directory. The specification below has two resources and two outputs. One output is sensitive.

package estate

import (
	"pravo.dev/pravo"
	rnd "pravo.dev/providers/hashicorp/random"
	loc "pravo.dev/providers/hashicorp/local"
)

$settings: pravo.#Settings & {refresh: "1h"}

$random: rnd.#Provider
$local:  loc.#Provider

port: $random.#Resource.random_integer & {min: 20000, max: 29999, seed: "web"}

web: $local.#Resource.local_file & {
	filename: "web.conf"
	content:  "port = \(port.result)\n"
}

url: pravo.#Output & {value: "http://localhost:\(port.result)"}
digest: pravo.#Output & {value: web.content_sha256, sensitive: true}

pravo has been initialised. Run "pravo plan" to see what it would do.

Outputs:
  digest: (sensitive)
  url: "http://localhost:26720"

pravo.state/meta.json
pravo.state/outputs.json
pravo.state/resources/port.json
pravo.state/resources/web.json
pravo.state/txn/<id>.json

The lineage identifies the store for its whole life. A saved plan records it, and an apply refuses the plan when the lineage differs.

In a file name, every other character of an address is written as % and four hexadecimal digits. The characters that stay are letters, digits, ., _ and -. The address replicas[0] is stored as resources/replicas%005B0%005D.json.

Each node is written on its own, to a temporary file that replaces the old one. A crash never leaves a half-written node. Two commands that write different nodes never conflict. The files are written with mode 0600 in directories with mode 0700, and the attributes of a node can include secrets. Secrets and ephemeral values says what the state holds.

A record

A node holds what the provider reported and what pravo needs to plan.

{
  "addr": "web",
  "mode": "managed",
  "type": "local_file",
  "name": "web",
  "provider": "registry.opentofu.org/hashicorp/local",
  "provider_key": "local",
  "provider_version": "<v>",
  "schema_version": 0,
  "dependencies": [
    "port"
  ],
  "config_hash": "<hash>",
  "revision": 1,
  "generation": "<id>",
  "refreshed_at": "<time>",
  "applied_at": "<time>",
  "updated_at": "<time>",
  "txn": "<id>"
}

An ephemeral resource and a function call have no node. pravo state show prints the bookkeeping as comments and the attributes as CUE.

// web (provider registry.opentofu.org/hashicorp/local v<v> as local, schema version 0, revision 1)
// refreshed <n>s ago, applied <n>s ago, config hash <hash>
// depends on port
{

The graph is queryable. pravo state graph prints the edges, or Graphviz with -dot. pravo state dependents ADDRESS prints everything that a change to the node can reach. pravo state query EXPRESSION evaluates a CUE expression with resources, edges and outputs in scope.

web -> port

["web"]

Revisions

A revision counts the writes of a node. A new node has revision 1.

• A write that changes the node raises the revision by one.
• A refresh that returns the attributes already recorded updates the timestamps and keeps the revision. Looking at the real world does not invalidate the saved plans that touch the node.
• A refresh that returns other attributes, or finds that the object is gone, writes the mark and raises the revision, so a saved plan made against the node is stale.
• A read of a data source that returns the attributes and edges already recorded keeps the revision. With the refresh age always or never it writes nothing. With a duration it writes the time of the read, as the refresh of a resource does.
• A replacement deletes the record and writes a new one that continues the count: the new record is one revision above the old one. A saved plan made against the old record is stale.
• A move writes the record at its new address, one revision above the old one, in the same generation.
• A node that is removed and created again, by pravo state rm or by a destroy and a later apply, starts at revision 1 again. Its generation is the transaction that created the new record, so it differs from the generation of the record it replaced. A record written before generations were recorded has none, and gets one when it is next written.

A saved plan records the revision and the generation of every node it touches. Plans describes how an apply uses them. A plan made against a record that has since been removed and created again is stale, although the new record may be at the same revision.

Transactions

A transaction is the record of one command that read providers, changed a record or changed the infrastructure. pravo state log lists them, newest first.

Change the specification so that it replaces port, then rename web to site with $from.

package estate

import (
	"pravo.dev/pravo"
	rnd "pravo.dev/providers/hashicorp/random"
	loc "pravo.dev/providers/hashicorp/local"
)

$settings: pravo.#Settings & {refresh: "1h"}

$random: rnd.#Provider
$local:  loc.#Provider

port: $random.#Resource.random_integer & {min: 20000, max: 29998, seed: "web"}

site: $local.#Resource.local_file & {
	$from:    "web"
	filename: "web.conf"
	content:  "port = \(port.result)\n"
}

url: pravo.#Output & {value: "http://localhost:\(port.result)"}
digest: pravo.#Output & {value: site.content_sha256, sensitive: true}

Saved the plan to change.plan. To apply exactly these actions, run:
  pravo apply change.plan

Apply complete! Resources: 2 added, 0 changed, 2 destroyed. [txn <id>]

The saved plan replaced port and site, which reads it. A plan that moves a record writes the move when it runs, so the plan is a transaction of its own.

<id>  apply    applied   <time> <took>
    pravo apply change.plan
    2 added, 0 changed, 2 destroyed
    replace  port  r1 -> r2
    replace  site  r2 -> r3
<id>  plan     planned   <time> <took>
    pravo plan --out change.plan
    2 to add, 0 to change, 2 to destroy; 2 of 2 reached, 0 skipped, 0 refreshed, 2 within refresh age, 0 not read, 0 ephemeral opened, 0 functions called, 0 imported, 1 moved
    move-to  site  r0 -> r2
    move-from web
<id>  apply    applied   <time> <took>
    pravo apply --auto-approve
    2 added, 0 changed, 0 destroyed
    create   port  r0 -> r1
    create   web  r0 -> r1

A transaction records:

• the kind: plan, apply, destroy, import or edit;
• the command line as it was run, flags included;
• the user, the host, the actor and the reason, the time and the duration;
• the outcome: planned, applied, rejected or failed. An apply is running from the moment it holds its locks until it ends; a running apply whose command was killed is listed by pravo state log as interrupted;
• every node that it touched, with the revision before and after.

A plan that read nothing and changed no record is not recorded. A plan that refreshed or moved records is. An apply that a lock rejected is recorded too, with no changes. An apply that has no record to write and no output to write or remove takes no lock and is not recorded.

The command line is the one that ran. An apply of a saved plan records the --policy flags it was given, so the log shows whether a policy was named. The log is evidence. It prevents nothing. It is as writable as the rest of the state, and a refusal by a policy is not recorded in it.

Locks and reservations

An apply locks every node that its plan writes, and no other, all of them before it changes anything, and releases them when it finishes. A node that the apply only reads is not locked, and neither is a data source whose read returned its record. A lock held by another process rejects the apply before it starts, with exit code 4. An apply never holds some of its locks while it waits for the others. When it finds a lock held it releases the ones it has taken. If the command that holds the lock has not begun to apply, because it is taking locks of its own and may give them up, the apply looks at the lock for a short time, between 25 milliseconds and 3 seconds, and takes its locks again when the lock is free. A command that applies, or a reservation, rejects it at once. Of applies that start together and need some of the same nodes, one takes them and the others are rejected because of it. Once the apply holds its locks, it reads the revisions and generations of the nodes it writes and of the records it reads that the plan lists from the state, and a node that another command wrote, removed or created again since the plan was made stops it with exit code 3. Applies that need different nodes run at the same time. Outputs are one record that each apply updates when it ends. An apply writes the outputs whose value it changed and leaves the others as they are recorded at that moment, so two applies that run at the same time both keep their own changes. Two applies that change the same output leave the value of the one that ends last.

The rejection names the node, the command, the process, the host, the holder and the time. It reads apply rejected: ADDRESS is locked by COMMAND (pid PID on HOST, HOLDER, since TIME), followed by ; N other nodes are held as well when the apply needs more nodes that are held. The smallest address is named. There is no global lock and no queue.

A lock is an object that exists while the command runs. It is created whole, or not at all, and it is released only by the command that holds it: a lock that another command took over is neither renewed nor deleted by the former holder. The command renews it every 2 minutes, so a command that ends sooner renews nothing. A lock whose process has stopped on the same host is reclaimed at once. A lock that has not been renewed for 10 minutes is reclaimed. Of commands that find the same lock left behind, one reclaims it. pravo state locks lists the locks and marks the ones that may be reclaimed with [stale]. pravo state unlock ADDRESS releases one by hand, and pravo state unlock --stale releases every stale lock and reservation.

A plan holds the lock of a node for the instant that it writes a refreshed record. When another process holds the lock, the plan leaves the record alone. A plan that refreshes a node another process is applying does not wait for it. With the lock held, the plan reads the record from the state once more, and writes the refreshed record only if it is the one the plan read. When another command wrote it, removed it or created it again in the meantime, the plan leaves it as it is and prints a warning.

Reservations

pravo plan --reserve DURATION takes a reservation on every node with a pending change. A reservation is a lock that outlives the command. Another actor cannot plan or apply a reserved node until the reservation expires. The actor who holds it applies the nodes, and the apply ends the reservation.

Change the number again. Two nodes have a pending change.

package estate

import (
	"pravo.dev/pravo"
	rnd "pravo.dev/providers/hashicorp/random"
	loc "pravo.dev/providers/hashicorp/local"
)

$settings: pravo.#Settings & {refresh: "1h"}

$random: rnd.#Provider
$local:  loc.#Provider

port: $random.#Resource.random_integer & {min: 20000, max: 29997, seed: "web"}

site: $local.#Resource.local_file & {
	$from:    "web"
	filename: "web.conf"
	content:  "port = \(port.result)\n"
}

url: pravo.#Output & {value: "http://localhost:\(port.result)"}
digest: pravo.#Output & {value: site.content_sha256, sensitive: true}

Reserved 2 nodes until <time>.

port  reserved by alice until <time>: pravo plan --reserve 10m --actor alice
site  reserved by alice until <time>: pravo plan --reserve 10m --actor alice

Error: plan rejected: port is reserved by alice (pravo plan --reserve 10m --actor alice until <time>, since <time>)

Apply complete! Resources: 2 added, 0 changed, 2 destroyed. [txn <id>]

No locks or reservations held.

An actor is a label that pravo does not verify. A reservation coordinates the people and agents who tell the truth about who they are. Policies says what that means for a rule.

Outputs

An output is a struct built from pravo.#Output. An apply evaluates every output and records it in outputs.json. pravo output prints them. The first apply of a root creates outputs.json, empty when the root declares no output, so that a root that reads these outputs finds it. Later applies write it only when an output changed.

digest: (sensitive)
url: "http://localhost:29960"

An output with sensitive: true is hidden where outputs are listed. That is a plan, the summary of an apply, and pravo output without a name. The state holds it as {"$sensitive": VALUE}. pravo output NAME and --json print it.

{"digest":["$sensitive"],"url":"http://localhost:29960"}

pravo destroy removes every output. A part of a saved destroy plan removes the outputs that read the resources it destroys and leaves the others; see Outputs of a partial destroy.

An output is not a copy of the state. Another root can read outputs.json alone with pravo.#RemoteOutputs, so a reader needs access to that one object and not to the resources. An output that the other root marked sensitive is not given out.

Backends

The state graph lives in a backend. A backend stores objects by key, reads them, deletes them, lists them, and creates one only if it does not exist. The local backend is a directory. The S3 backend keeps the same keys as objects in a bucket.

$settings: pravo.#Settings & {state: pravo.#S3State & {
	bucket:     "infra-state"
	prefix:     "teams/platform/app"
	region:     "eu-central-1"
	endpoint:   "https://minio.internal:9000" // an S3-compatible store
	path_style: true
}}

The block is typed by backend. Local fields with an S3 backend, S3 fields with the local backend, and backend: "s3" without a bucket are errors at evaluation. A bucket with no backend means s3. Credentials come from the usual AWS chain: the environment, a profile, or a role. Static access_key and secret_key fields end up in the specification.

Every property of the local store holds in a bucket. Each node is one object, written on its own. The bucket holds a log of transactions and a lock for each node. A lock is an object created with the If-None-Match: * conditional write. Two commands on different machines cannot both create it, so the store must honour that header, and If-Match on renew, release and reclaim. S3-compatible stores says how to test a store. The prefix lets several roots share a bucket.

pravo state pull DIR copies the configured graph into a local directory. pravo state push DIR copies a local directory into the configured backend, objects and transaction log included.

Cost follows the change

The state keeps one record for each resource, and the record names the resources it depends on. A command does not have to read, plan or lock the estate to find what a change affects. Four parts of the design make the work of a command follow the change and not the number of records.

Reach and the refresh age work together. Reach decides which records a plan looks at: the resources it plans and the resources they read. The refresh age decides whether a record that the plan looks at is read again. A short default age does not make a plan read a resource it does not look at, and reach does not make a plan trust a record that is older than its age.

In practice

A change to one resource that no other resource reads reads that resource and the resources it reads, takes one lock, writes one record and saves a plan with one change, whether the state holds twenty records or twenty thousand. A change that other resources read reaches them as well, and the work follows the number it reaches. The printed plan is as long as the change. A saved plan stays current while other commands write nodes that it neither changes nor reads.

People and jobs that change different parts of one state apply at the same time. A job that needs a node that another job holds is refused with exit code 4 and is not queued. A change reads what it reaches shows the reads, as the Graph: line counts them, and Lab 2 shows applies that do not meet.

An estate therefore does not need cutting into several states to keep the reads, the locks and the writes of a change small. A split also has a price: the edges stop at the boundary of a state. A change in one root does not reach a root that reads its outputs with pravo.#RemoteOutputs, so the roots are planned and applied one after the other, in an order that someone keeps.

Two costs follow the size of the estate. pravo evaluates the whole specification on every run, and it opens every record of the state before it plans. A change that reaches every resource, such as a change to a resource that all the others read, costs what the estate costs, and so does creating the estate. -v prints the time of each phase: see Debugging. A split makes each of these smaller, because each state holds fewer resources.

A few records are read whether or not the change reaches them. A data source is read once its record is older than its refresh age, so with the default age an estate with many data sources reads all of them on every plan, and a refresh age set with $settings: refresh keeps a plan from reading them again within that age. A resource that sets $lifecycle: refresh is read once its record is older than that age. Refresh age lists every case.

A plan does not promise more than it read. A record that a plan did not read is trusted. For a record that the plan looked at, the refresh age says for how long. For one that it did not look at, the trust lasts until something reads it. The Graph: line counts the records that were not read. pravo drift compares them with the infrastructure, and a job that runs pravo drift --due --limit N reads the estate in turns. Plans says how to choose the age.

When to split

Split a state for ownership, permissions or blast radius.

• Ownership and permissions. A state is the unit of access. Whoever may write the bucket may write every record in it, and the records hold the attributes of the resources, secrets included. Teams that must not change each other's resources keep separate states.
• Blast radius. A command acts on one state. pravo destroy deletes every node in it, and a wrong edit to the specification, or to the credentials of a provider, can reach every node that uses it. A policy can bound the blast radius of a plan; separate states bound what any command can touch.

Remote state describes the bucket that one state lives in.

Recap

• The state is a directory or a bucket of independent records. Each has an address, a revision and a lock.
• A revision counts writes. A saved plan records the revision of every node it touches.
• An apply locks every node it writes. A plan holds a lock only while it writes a refreshed record. A reservation is a lock that outlives its plan.
• A transaction records who ran what, and which nodes changed. The log is evidence and prevents nothing.
• The cost of a plan, an apply and a lock follows the change. Ownership, permissions and blast radius decide whether to split a state.

Next: Providers describes the processes behind the records.