Lab 2: two applies at once

OpenTofu holds one lock for the whole state. pravo holds one lock for each resource that an apply writes. Two applies that write different resources run at the same time. An apply that needs a resource another process holds is refused.

Outputs are one record that each apply updates when it ends: an apply writes the outputs it changed and keeps the rest as recorded. The specification of this lab has no outputs.

This lab starts three applies on one state while one of them is busy. It uses a key that takes seconds to generate, which gives the first apply time to hold its lock. The time varies between machines and runs.

Prepare the plans

Work in an empty directory. Start with one file and apply it, so that the state exists.

package main

import (
	loc "pravo.dev/providers/hashicorp/local"
	tlsp "pravo.dev/providers/hashicorp/tls"
)

$local: loc.#Provider
$tls:   tlsp.#Provider

base: $local.#Resource.local_file & {filename: "base.txt", content: "base\n"}

Configuration does not load yet; generating schemas for the declared providers first.
Created cue.mod/module.cue
Providers:
  local = hashicorp/local v2.9.1
  tls = hashicorp/tls v4.4.1
...
pravo has been initialised. Run "pravo plan" to see what it would do.
...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed. [txn <txn>]

Add two resources. slow is an RSA key of 8192 bits, which takes seconds to create. quick is a file.

package main

import (
	loc "pravo.dev/providers/hashicorp/local"
	tlsp "pravo.dev/providers/hashicorp/tls"
)

$local: loc.#Provider
$tls:   tlsp.#Provider

base: $local.#Resource.local_file & {filename: "base.txt", content: "base\n"}

slow:  $tls.#Resource.tls_private_key & {algorithm: "RSA", rsa_bits: 8192}
quick: $local.#Resource.local_file & {filename: "quick.txt", content: "quick\n"}

Save one plan. Three people each get a copy: Ana, Bob and a CI job.

...
  # slow (tls_private_key) will be created
  + slow: {
  +     algorithm:                     "RSA"
...
  +     rsa_bits:                      8192
    }
...
Plan: 2 to add, 0 to change, 0 to destroy.
Graph: planned 2 of 3, 1 unchanged and skipped, 1 not read.

Saved the plan to all.plan. To apply exactly these actions, run:
  pravo apply all.plan

Each copy creates two resources. A plan may be applied in parts, as in B7. The parts here are slow and quick.

Wait for a lock

A script that waits until a resource is locked makes the run repeatable. It asks pravo state locks ten times a second.

for i in $(seq 1 200); do
	pravo state locks | grep -q "^$1 " && exit 0
	sleep 0.1
done
exit 1

Run three applies

Ana starts in the background. The script returns when slow is locked. Then the CI job and Bob run, in that order.

slow  locked by ana, pid <pid> on <host> since <time>: pravo apply a.plan --only slow --actor ana
Applying 1 of 2 remaining changes from c.plan.

Error: apply rejected: slow is locked by pravo apply a.plan --only slow --actor ana (pid <pid> on <host>, ana, since <time>)
exit 4
Applying 1 of 2 remaining changes from b.plan.
quick: Creating...
quick: Creation complete after 0s [id=<id>]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed. [txn <txn>]
1 change remains in b.plan; apply with "pravo apply b.plan".
exit 0
ana is still running
Applying 1 of 2 remaining changes from a.plan.
slow: Creating...
slow: Creation complete after 0s [id=<id>]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed. [txn <txn>]
1 change remains in a.plan; apply with "pravo apply a.plan".

Read it from the top.

• pravo state locks lists slow: the actor, the process, the host, the time and the command line of the holder.
• The CI job's apply stops with exit code 4. The error names the holder. Nothing was written.
• Bob's apply wrote quick and ended. Ana's was still running at that moment.
• Ana's apply ended after the key was generated, and its lock was released.

The locks are gone.

No locks or reservations held.

Read the log

pravo state log lists every transaction, newest first. The refused apply is recorded as well, with no changes.

<txn>  apply    applied   <time> <duration>
    pravo apply b.plan --only quick --actor bob
    actor bob
    1 added, 0 changed, 0 destroyed
    create   quick  r0 -> r1
<txn>  apply    rejected  <time> <duration>
    pravo apply c.plan --only slow --actor ci
    actor ci
    apply rejected: slow is locked by pravo apply a.plan --only slow --actor ana (pid <pid> on <host>, ana, since <time>)
<txn>  apply    applied   <time> <duration>
    pravo apply a.plan --only slow --actor ana
    actor ana
    1 added, 0 changed, 0 destroyed
    create   slow  r0 -> r1

Bob's apply started after Ana's and is listed above it. The refusal is a transaction with the status rejected, and its message is the error that the CI job saw. The actor label comes from --actor. pravo does not verify it.

Break it: a plan made too early

The CI job still has its copy. The plan was made before Ana and Bob applied. It expects to create slow and quick, and both exist now.

Error: saved plan is stale; run "pravo plan" again:
  - slow changed in state (revision 1, plan expected 0)
  - quick changed in state (revision 1, plan expected 0)

Exit code 3 is a stale plan. Each reason names a record, the revision it has now, and the revision the plan expected. A record that did not exist has revision 0. The CI job plans again. It finds No changes.

...
No changes. The specification matches the state. 3 records were not read; run pravo drift to compare them with the infrastructure.
Graph: planned 0 of 3, 3 unchanged and skipped, 3 not read.

Remove what the lab created.

Destroy complete! Resources: 0 added, 0 changed, 3 destroyed. [txn <txn>]

Recap

• An apply locks the resources its plan writes, all of them before it changes anything. There is no lock on the state as a whole.
• An apply that needs a locked resource is refused with exit code 4. The log records the refusal.
• A plan that was made before another apply touched its resources is stale. Exit code 3 says so.

A lock left behind by a command that crashed on another host is released with pravo state unlock <address>. pravo state locks marks the locks that may be reclaimed with [stale], and pravo state unlock --stale releases all of them. A state shared by several machines is the subject of Remote state.