§ How-to · Providers and state

Keep the state in a shared bucket

Store the state in an S3 bucket so that a team shares it, move an existing local state into the bucket and read another root's outputs.

Time: 10 min
Level: intermediate
Needs: A specification that you can plan, and an S3 bucket. This page uses the in-memory bucket from pravo's source.
Done when: pravo state list prints the same addresses from two checkouts, and pravo plan in the second one reports no changes.

View as Markdown

On this page

A bucket holds the state for every checkout, and the locks that keep two people from changing one resource. This page puts a state in a bucket, reads it from a second checkout and moves a local state into it.

Note

Every command below runs against tools/fakes3, the in-memory S3 server in pravo's source, on port 9000. For a real bucket, change step 1 and the endpoint and path_style lines.

1. Provide a bucket

For AWS, create the bucket first and turn on versioning, so that an overwrite can be undone. pravo does not create the bucket. It needs permission to get, put, delete and list objects in it. A store other than Amazon S3 must honour conditional requests: S3-compatible stores says how to test it. Credentials come from the standard AWS chain: environment variables, a shared profile or an instance role. Never write keys in the specification.

To follow along without an account, start the in-memory server in a second terminal from a checkout of pravo's source. It forgets everything when it stops.

go run ./tools/fakes3 -addr 127.0.0.1:9000

Give the shell test credentials. The server accepts any value.

export AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test

2. Point the root at the bucket

Work in an empty directory. The state block of $settings selects the backend.

main.cue

package remote

import (
	"pravo.dev/pravo"
	rnd "pravo.dev/providers/hashicorp/random"
)

$settings: pravo.#Settings & {
	state: pravo.#S3State & {
		backend:    "s3"
		bucket:     "pravo-state"
		prefix:     "demo/app"
		region:     "us-east-1"
		endpoint:   "http://127.0.0.1:9000"
		path_style: true
	}
}

$random: rnd.#Provider & {$version: "~> 3.6"}

server: $random.#Resource.random_pet & {length: 2}
port: $random.#Resource.random_integer & {min: 8000, max: 8999}

bucket is required. prefix lets several roots share one bucket.
endpoint and path_style are for S3-compatible servers. For AWS, remove both and keep region.
prefix may be computed. prefix: "demo/\(env)" gives each environment its own state.

3. Apply and look at the state

A store that ignores conditional requests lets two commands hold one lock, so pravo probes an S3 store before it first writes to it. pravo state probe runs the same checks on demand:

terminal

$pravo init > /dev/null

$pravo state probe

create only (If-None-Match)      ok
replace (If-Match on a put)      ok
delete (If-Match on a delete)    ok
concurrent creates               ok
s3://pravo-state/demo/app honours the conditional requests that the locks need.

terminal

$pravo init > /dev/null

$pravo apply -auto-approve | grep '^Apply complete'

$ls

Apply complete! Resources: 2 added, 0 changed, 0 destroyed. [txn <txn>]
cue.mod
main.cue
pravo.lock.cue

No pravo.state directory appears. The state is in the bucket, and every command reads it from there.

terminal

$pravo state list

$pravo state log

port
server
<txn>  apply    applied   <date> <time> <took>
    pravo apply --auto-approve
    2 added, 0 changed, 0 destroyed
    create   port  r0 -> r1
    create   server  r0 -> r1

The addresses are the paths of the resources in the specification, whichever backend holds the state.

4. Use it from a second checkout

A colleague clones the repository. Here the clone is a new directory with the same main.cue.

terminal

$mkdir second

$cp main.cue second

$pravo -C second init > /dev/null

$pravo -C second state list

$pravo -C second plan | grep -E '^(No changes|Graph)'

port
server
No changes. The specification matches the state. 2 records were not read; run pravo drift to compare them with the infrastructure.
Graph: planned 0 of 2, 2 unchanged and skipped, 2 not read.

The second checkout sees the same state. The resources are random values, which exist only in the state, so nothing differs. A resource that lives on one machine's disk, such as local_file, is a different case. Another machine finds the file missing and plans to create it again. Remote state suits resources that live somewhere shared, as cloud resources do.

5. Coordinate through the bucket

Locks and reservations are objects in the bucket. A reservation made in one checkout stops another actor in a second checkout. Change length to 3 in main.cue and reserve the node:

main.cue

package remote

import (
	"pravo.dev/pravo"
	rnd "pravo.dev/providers/hashicorp/random"
)

$settings: pravo.#Settings & {
	state: pravo.#S3State & {
		backend:    "s3"
		bucket:     "pravo-state"
		prefix:     "demo/app"
		region:     "us-east-1"
		endpoint:   "http://127.0.0.1:9000"
		path_style: true
	}
}

$random: rnd.#Provider & {$version: "~> 3.6"}

server: $random.#Resource.random_pet & {length: 3}
port: $random.#Resource.random_integer & {min: 8000, max: 8999}

terminal

$pravo plan -out change.plan -reserve=10m --actor alice | grep '^Reserved'

$pravo state locks

Reserved 1 node until <time>.
server  reserved by alice until <time>: pravo plan --out change.plan --reserve=10m --actor alice

The second checkout takes the same change and plans as another actor. -refresh=false skips reading the resources, so the output is the refusal alone.

terminalexit 4

$cp main.cue second

$pravo -C second plan -refresh=false --actor bob

Error: plan rejected: server is reserved by alice (pravo plan --out change.plan --reserve=10m --actor alice until <time>, since <time>)

Exit 4 means that a node is locked or reserved. A command that changes a resource holds a lock on it. A running command renews the lock every 2 minutes, and a lock that is not renewed for 10 minutes is reclaimed. pravo state locks marks the locks that may be reclaimed with [stale]. pravo state unlock <address> releases a lock that a crashed run left behind, and pravo state unlock --stale releases all of them. A lock or a reservation covers one resource, and pravo state locks lists them by address. Applying the plan ends the reservation:

terminal

$pravo apply change.plan --actor alice | grep '^Apply complete'

$pravo state locks

Apply complete! Resources: 1 added, 0 changed, 1 destroyed. [txn <txn>]
No locks or reservations held.

An apply locks the resources it writes and no others. After it holds the locks, it reads the revision of each resource it writes, and of each record it reads, from the bucket, and stops with exit code 3 if another command wrote one of them since the plan was made. The bucket must return the object that was last written when it is read back. S3 does. Opening the state reads one object for each record before a command starts, 64 at a time, so the time to open grows with the size of the estate and with the latency of the bucket. A plan then reads from the providers only the records it looks at.

6. Move a local state into the bucket

A root that has used the local pravo.state directory is moved by copying the directory up. Start from such a root. Its settings.cue is not written yet.

moved/main.cue

package moved

import rnd "pravo.dev/providers/hashicorp/random"

$random: rnd.#Provider & {$version: "~> 3.6"}

server: $random.#Resource.random_pet & {length: 2}
port: $random.#Resource.random_integer & {min: 8000, max: 8999}

terminal

$pravo -C moved init > /dev/null

$pravo -C moved apply -auto-approve | grep '^Apply complete'

$ls moved

Apply complete! Resources: 2 added, 0 changed, 0 destroyed. [txn <txn>]
main.cue
pravo.lock.cue
pravo.state

Add the state block in a file of its own, with a new prefix.

moved/settings.cue

package moved

import "pravo.dev/pravo"

$settings: pravo.#Settings & {
	state: pravo.#S3State & {
		backend:    "s3"
		bucket:     "pravo-state"
		prefix:     "demo/moved"
		region:     "us-east-1"
		endpoint:   "http://127.0.0.1:9000"
		path_style: true
	}
}

The bucket is empty at this prefix, so a plan creates everything. pravo state push copies the local directory into the configured backend, and the plan is then empty.

terminal

$pravo -C moved plan | grep -E '^(Plan|No changes)'

$pravo -C moved state push pravo.state

$pravo -C moved plan | grep -E '^(Plan|No changes)'

Plan: 2 to add, 0 to change, 0 to destroy.
Copied 5 objects from pravo.state to s3://pravo-state/demo/moved
No changes. The specification matches the state. 2 records were not read; run pravo drift to compare them with the infrastructure.

Archive the local directory when you are satisfied. pravo state pull <dir> copies the other way, from the bucket into a local directory.

7. Read another root's outputs

A second root reads the values that the first one published, and nothing else from its state. The first root publishes an output:

url.cue

package remote

import "pravo.dev/pravo"

url: pravo.#Output & {value: "http://\(server.id).internal:\(port.result)"}

terminal

$pravo apply -auto-approve | grep -A1 '^Outputs'

Outputs:
  url: "http://<pet>.internal:<port>"

In the second root, pravo.#RemoteOutputs points at the first root's state. The settings are those of the state itself.

app/main.cue

package app

import (
	"pravo.dev/pravo"
	rnd "pravo.dev/providers/hashicorp/random"
)

$random: rnd.#Provider & {$version: "~> 3.6"}

net: pravo.#RemoteOutputs & {
	state: {
		backend:    "s3"
		bucket:     "pravo-state"
		prefix:     "demo/app"
		region:     "us-east-1"
		endpoint:   "http://127.0.0.1:9000"
		path_style: true
	}
}

worker: $random.#Resource.random_pet & {length: 2, prefix: net.outputs.url}

terminal

$pravo -C app init > /dev/null

$pravo -C app plan | grep -E 'prefix|^Plan'

  +     prefix:    "http://<pet>.internal:<port>"
Plan: 1 to add, 0 to change, 0 to destroy.

pravo reads only the object outputs.json under the prefix. A bucket policy that allows reading that object gives the second root the outputs and none of the resources. Outputs marked sensitive: true are not given out. pravo does not order the two roots: apply the first before the second.

8. Decide who may write the bucket

The bucket holds the record of what exists. Whoever can write to it can change what the next plan believes: pravo state rm, pravo state mv and pravo import are not plans, so no policy sees them. The locks and the transaction log are objects in the same bucket. Give read, write, delete and list permission to the identities that run pravo against this root, and treat the right to write the bucket as the right to change what pravo knows.

What next

Environments computes the prefix from the environment.
CI runs plan and apply as separate jobs against the shared state.
State describes what the bucket holds.
state and Settings are the reference for pravo state push, pull and the state block. Environment variables and remote outputs covers pravo.#RemoteOutputs.