Skip to content

Approvals and break-glass

Some operations change infrastructure. MEHO does not let those run silently: an operation the connector marks requires_approval — every dangerous write, and any caution write a policy elevates — does not execute when you call it. It parks in an approval queue and returns awaiting_approval, and a second person has to approve it before it runs. That is the four-eyes rule, and it is on by default.

Baseline: self-approval is off by default

config.approvalAllowSelfApproval defaults to "false" (APPROVAL_ALLOW_SELF_APPROVAL=false) — the secure baseline. With it off, the identity that requested a gated write can never approve it, and GET /ready reports approval_queue.effective_posture: "four_eyes_enforced". Leave it off. Setting it "true" is a posture-wide, audited emergency break-glass (covered in Option 2 below) that still never reaches a dangerous or destructive operation. The endorsed single-operator answer is the agent-requester pattern, not the flag.

This guide is about the case the rule does not obviously cover: you are the only operator. A solo deploy still has to be able to make gated writes, so MEHO ships two ways for one person to clear the queue — one you should reach for every day, and one reserved for emergencies. This page covers both, and how to tell an auditor which one your deploy is running.

Prerequisites

What you see when a write parks

Call a gated operation — from the CLI, an agent, or the operator console — and instead of a result you get a parked request:

meho operation call vault-1.x vault.kv.put --target rdc-vault --params '{...}'
# → status: awaiting_approval
#   approval_request_id: 4b1c…

The parked request is side-effect-free: nothing was written to the target. It is now waiting in the queue, which you can list and inspect:

meho approvals list                 # pending requests
meho approvals show <id>            # the parked operation + its proposed effect

To act on it you approve or reject it — on any surface, and they all share one backplane path (same policy, same audit row):

meho approvals approve <id>         # runs the parked write now
meho approvals reject <id>          # terminal; the requester must re-file

After approval, the person who sent the request reads what the operation printed with one command. Only that person can read it:

meho approvals result <id>          # the output of the approved run

The operator console exposes the same queue at /ui/approvals: a bell with the live pending count, a request-detail modal with the proposed effect, and Approve / Deny buttons.

When several requests belong to one change ticket and depend on each other, approve them as a group, oldest first:

meho approvals approve --filter-work-ref <ticket> --in-order

Each step is an ordinary approval and runs at once. MEHO goes on to the next step only when the run was OK. It stops at the first step that fails or is refused, and names that step and the steps it did not run. At each step it shows what the step does and asks you to type yes first. In the console, filter the Pending tab to the change ticket and use Approve group in order. The sign-in rules hold for every step: when a step needs a new sign-in (your sign-in is too old, or the step is destructive and your sign-in already approved something), the run stops there and shows Sign in again. Sign in, then start the run again; the steps that ran OK are skipped.

You can also approve several requests by id in one run: meho approvals approve <id> <id> .... The command shows each request (what it does, the operation, the connector and its version, the target, the risk level and the parameters) and asks you to type yes for each one. Anything else skips that request.

Approving asks you to sign in again. An approval must come from a person who is present, so MEHO checks the sign-in behind every approval. In the console, your sign-in must be at most 15 minutes old and must have asked for your password. When it is older, the request shows Sign in to approve instead of Approve: click it, sign in with your password (and your authenticator code or passkey), then open the request again and approve. On the command line and REST, MEHO needs a fresh sign-in made only for approvals. So meho approvals approve opens your browser and asks for your password (and your authenticator code, if your realm asks for one).

  • The browser always asks for your password, even when you are signed in to the console in the same browser.
  • One sign-in covers approvals for 15 minutes. In that time one run of the command can approve several requests.
  • An operation marked destructive needs a sign-in of its own. The command opens the browser again for it.
  • The command keeps the sign-in in memory for this one run. It never stores it.
  • Your normal meho login stays as it is, but it never approves. An MCP token or an agent token never approves either.
  • Reject needs no new sign-in, in the console or on the command line.
  • The same limits hold in the console: 15 minutes for one sign-in, and a sign-in of its own for a destructive operation.

Without a browser (for example over SSH), the command prints the sign-in link and stops. Approve in the console instead: it is the normal place to approve.

The rule: requester and approver must differ

Approval keys on your subject — the stable sub claim on your token, not your display name or your role. The guard is simple: the identity that requested a parked write may not be the identity that approves it. Try to approve your own request and every surface refuses with the same error:

self_approval_forbidden: operator '<your-sub>' may not approve approval_request
<id>: requester and approver must differ
(set APPROVAL_ALLOW_SELF_APPROVAL=true for audited single-operator break-glass)

That is a REST 403, an MCP invalid-params error, and a disabled Approve button in the console — all naming the same flag. To take back your own pending request, withdraw it: meho approvals withdraw <id> [--reason TEXT] (or POST /api/v1/approvals/{id}/withdraw). The request then shows "withdrawn by the sender", nobody can approve it, and the operation does not run. A person, an agent or a service account can each withdraw its own request.

On a team this is invisible: your colleague approves your write. On a one-operator deploy it looks like a dead end — you are both the requester and the only identity that could approve. It is not a dead end. Pick one of the two paths below.

The everyday answer is to make sure the write is not requested under your own subject in the first place. Park it under an agent principal instead — a first-class non-human identity with its own stable sub. Then, when you approve as yourself, requester and approver are already different subjects, the gate clears, and you never touch the break-glass flag. The full audit lineage stays intact: the request row records the agent as requester, the decision row records you as approver.

An agent principal is a Keycloak client_credentials client that carries principal_kind=agent. You create one, attach it to an agent definition, and drive the gated write through a scheduled run of that definition. Four steps:

# 1. Register the agent principal (mints its client_credentials client;
#    the output includes the client id you pass to --identity-ref below).
meho agent-principal register nightly-writer

# 2. Author an agent definition bound to that principal. Its toolset and
#    prompt are what perform the gated write when the definition runs.
meho agent create nightly-writer --identity-ref <client-id-from-step-1>

# 3. Schedule the definition. The autonomous run parks the write under the
#    agent's sub (principal_sub=<agent-sub>). Use --kind one_off with
#    --fire-at <ISO8601> instead for a single run rather than a cron.
meho scheduler create --kind cron --agent-definition nightly-writer \
  --cron-expr "0 2 * * *"

# 4. The write is now parked under the agent. Approve as yourself — your
#    sub differs from the agent's, so the gate clears and it dispatches.
meho approvals list
meho approvals approve <id>

It must be an autonomous run, not a delegated one

Only an autonomous / scheduled agent run parks the request under the agent's own subject. A run you launch interactively binds the agent as your delegate (RFC 8693 token exchange): you stay the subject, the agent is only the actor, and the parked request still carries your sub — so you still cannot approve it. Routing a solo write through an interactive agent run does not break the deadlock. The scheduled trigger is what makes the agent the requester.

Which identity ends up on the request — and therefore whether you can approve it — comes down to how the write was dispatched:

How the write was dispatched Requester sub You can approve it?
You call it directly (human operator) your sub No — requester == approver
You launch an agent interactively; it makes the write your sub (agent is only the actor) No — you are still the subject
A scheduled / autonomous agent run makes the write the agent's sub Yes — distinct subject, no flag needed

This is the endorsed posture for a single-operator tenant: it keeps the four-eyes invariant genuinely intact (two distinct subjects really do touch every write) instead of switching it off.

Option 2 — the audited break-glass flag (emergencies)

When you genuinely cannot stage an agent-requester — an incident, a first bring-up, a write you must make in the next five minutes — a deployment admin can enable self-approval posture-wide:

# Helm values overlay
config:
  approvalAllowSelfApproval: "true"   # renders APPROVAL_ALLOW_SELF_APPROVAL=true

With the flag on, the requester == approver guard is lifted and you can approve your own parked writes. Understand exactly what you traded away:

  • It is posture-wide, not per-write. The flag re-opens the single-account request-and-grant hole for every lower-risk operation across the whole tenant, not just the one you are trying to clear. There is no scoping to a single write.
  • It never reaches a dangerous or destructive operation. Break-glass self-approval applies only to the caution and safe tiers. A dangerous or destructive operation can never be self-approved — flag or no flag, it always needs a second subject to approve it.
  • It is emergency-grade, not the solo default. Reach for Option 1 first. Leaving this on turns four-eyes into no-eyes for lower-risk writes across the whole tenant.
  • It is still audited. Self-approval is not silent (see below). A reject never needs the flag.

Break-glass is a lever an admin holds, not a per-write toggle

Prefer the agent-requester pattern for anything you do more than once. Set APPROVAL_ALLOW_SELF_APPROVAL=true for a real emergency, make the write, and turn it back off.

A third path: a segregated approver

The two options above are about one person clearing the queue. A different need is separation of duties — someone who may approve gated writes but may never run an operation. MEHO expresses that directly: a principal can carry an approver capability, independent of its tenant role. An approver may list, inspect, approve, and reject parked requests, yet dispatch stays gated on the operator role — so an approver provisioned with the read_only role plus approver=true clears a four-eyes gate but is refused call_operation outright. "Approve, but never operate" is now a role, not a matter of discipline.

The four-eyes rule still holds for an approver: it cannot approve a request it filed itself. And the capability grants only the queue — an approver does not gain the right to read the result of a dispatch it did not request. Provision the approver claim in your realm with the Keycloak realm setup.

Permanent deletes are gated hardest

A permanent delete — a VM, a DNS record, a firewall rule, a secret — runs on the most restrictive tier, destructive (the safety order is safe < caution < dangerous < destructive). A destructive operation runs in a fixed order: it is previewed, a person approves it, the approval is bound to that exact preview, the system states exactly what will be removed, and only then does it run — with the whole sequence written to the audit log. Concretely:

  • The approval is bound to the preview. The preview returns a hash of the exact resolved request. That hash is presented on the call and re-checked at approve time, so the approval is tied to the preview a human actually saw. If the parameters changed between preview and approval, the approve is refused.
  • The run checks the object again. Right before an approved delete runs, MEHO reads the object once more and compares it with the approval card: the same id or key, or, for an object that has none, the same fields. If someone deleted the object and created a new one under the same name while the request waited, the run is refused and nothing is deleted. The same happens when the card said there was nothing to delete and an object with that name is there now, and when MEHO cannot read the object right before the run. The requester sends a new request, and the approver sees the new object. If the object is simply gone, the run reports it as already absent.
  • A blast-radius statement is mandatory. A destructive operation cannot park without a block naming the object, its enumerated child objects, and how irreversible the change is. The console renders it as a "what this destroys" card above the request detail. An operation that cannot produce that statement cannot be parked at all — it fails closed.
  • No self-approval, ever. Break-glass self-approval never applies to a dangerous or destructive operation; only lower-risk operations can be self-approved. A destructive delete always needs a second subject.
  • The record shows who decided. The decision names the approver, so the audit answers "who signed off on this delete?" without inference.
  • An add-on can propose a delete. The tier refuses only agent principals. An add-on that signs in as a service principal previews the delete and calls it with the preview hash. The call waits for a person to approve it; no standing grant can approve it in advance. After approval the delete runs under the approver, and the add-on reads the result with GET /api/v1/approvals/{id}/result. Only the caller that started the delete can read that result.

Sandbox mode: one exception, for disposable test systems only

A MEHO instance can run in sandbox mode. It is only for an instance that holds disposable test systems: systems built for one test and deleted after it. On such an instance a tenant admin may give a service principal a standing grant that also covers deletes, a delete grant. So an automated test run can build, change and delete its own test systems with no wait for a person. Every other instance keeps the rules above; it shows "sandbox_mode": "off" on /ready and in meho status.

What stays the same on a sandbox:

  • A person's delete still waits for a second person. An agent login is still refused a delete. A satellite runner never gets a delete.
  • A permanent delete under a delete grant still needs the preview hash of its own preview and a blast-radius statement. MEHO checks both before it records the grant as used, so a refused call leaves no "auto-approved" record. The record of a used grant holds the hash and the blast radius.

How delete grants work:

  • Only a tenant admin creates one, in the web console. The REST API and the CLI refuse it, also with a person's token, unless the instance has the chart value sandboxMode.deleteGrantsFromApi: true. Then a tenant admin who signed in with meho login may create one through REST or meho service-principals grants create, with the same checks.
  • Every delete grant expires within 30 days. MEHO never renews one by itself; the banner on every console page counts the delete grants that expire within 7 days. A tenant admin renews the delete grants of one service principal as a set, on the grant page: one new expiry for all of them, and each one is checked like a new grant.
  • Delete grants are kept in their own table. An older MEHO build never reads it, so after a rollback every delete waits for a person again.

How the mode is set:

  • At install time, with the chart values sandboxMode.enabled: true and sandboxMode.confirmHost equal to the instance's host name. The host name must have a part that is exactly sandbox (for example meho.sandbox.example). MEHO refuses to start otherwise.
  • MEHO stores the mode in the database at the first start and refuses to start when the stored mode differs from the setting. To change the mode, install a new instance with a new database.
  • Keep a sandbox on its own site: its own MEHO, Keycloak and Vault, only disposable systems as targets, and no network route to any other system. Those walls take the place of a person's approval.

Proving which posture a deploy is running

Break-glass leaves a trail, so an auditor never has to trust a claim about the values file.

Is self-approval even enabled? Read it off the health surface — no cluster access to the chart needed:

curl -s https://<your-backplane>/ready | jq .features.approval_queue.effective_posture
# "four_eyes_enforced"            → the default; self-approval is OFF
# "single_operator_break_glass"   → APPROVAL_ALLOW_SELF_APPROVAL=true is set

/ready returns the features block only when you send the metrics token. Add -H "Authorization: Bearer <metrics-token>" to the curl call. A chart install makes the token on the first install; read it with kubectl get secret <release>-metrics-auth -o jsonpath='{.data.token}' | base64 -d.

Did a specific write self-approve? Two signals:

  • A structured approval_self_approval_break_glass log event (WARNING) is emitted every time a self-approval goes through, carrying the approval_request_id, op_id, operator_sub, and tenant_id.
  • On the audit ledger, a self-approval is the approval.decision row where the requester equals the approver — principal_sub == reviewed_by. There is no separate self_approved field to filter on; the equality is the marker.

So a break-glass self-approval is visible three ways — the live posture on /ready, a WARNING log at the moment it happens, and the requester-equals-approver decision row on the permanent ledger.

Deciding on someone else's written instruction

Sometimes you approve or reject a request because another person told you to, in writing. You still decide as yourself. MEHO records who told you, their exact words, and where the work is tracked.

End your reason with three lines, after a blank line:

meho approvals approve 4b1c… --reason "Ran the cleanup in the change window.

Instructed-by: lead@example.com
Instruction: \"please run the cleanup tonight\"
Refs: #12, CHG-9"
  • Instructed-by: the person who told you (their login or email).
  • Instruction: their words, in quotes, exactly as they wrote them.
  • Refs: the issue or change ticket, comma-separated.

Only on lab-class deployments

On production, MEHO refuses a decision made on someone else's instruction. You must decide on your own judgement. A lab-class deployment turns this on at install time with config.approvalAllowDelegatedDecision: "true" (env var APPROVAL_ALLOW_DELEGATED_DECISION). The default is "false". No request can change it.

When it is off, MEHO answers with a 403 error whose message starts with delegated_decision_forbidden:. The request stays pending. Remove the three lines and decide yourself, or ask the person who instructed you to decide. MEHO also writes a WARNING log event named approval_delegated_decision_refused.

Check the stance of a deploy on /ready:

curl -s https://<your-backplane>/ready | jq .features.approval_queue.delegated_decision_posture
# "production_own_hand_only"      → the default; delegated decisions are refused
# "lab_class_delegation_allowed"  → APPROVAL_ALLOW_DELEGATED_DECISION=true is set

The same lines work in the console reason box and in the REST reason field.

If you write Instructed-by: but leave out Instruction: or Refs:, MEHO refuses the decision with a 422 error and the request stays pending. Add the missing line and try again. A reason can be at most 2000 characters.

Where it shows up afterwards:

  • meho approvals show <id> prints Instructed by, Instruction and Refs under Reviewed by.
  • The console shows the same three lines on the decided request.
  • The audit log keeps your full reason. The decision row also has a delegation block with the three values, and the backplane writes a WARNING log event named approval_delegated_decision.

Which one should I use?

Situation Use
Solo deploy, recurring or planned gated writes Agent-requester (Option 1) — no flag, four-eyes stays real
Solo deploy, one-off write you can schedule Agent-requester (Option 1)
Genuine emergency, no time to stage an agent Break-glass (Option 2), then turn it back off
More than one operator Neither — a second operator approves the first's request

Troubleshooting

approve returns self_approval_forbidden and I am the only operator. The write was parked under your subject — a direct human call, or an interactive agent run (where you stay the subject). Re-park it under an agent-requester: wire it as a scheduled trigger under an agent definition (Option 1) so the autonomous run parks it with the agent's sub, then approve as yourself. Reserve APPROVAL_ALLOW_SELF_APPROVAL for genuine emergencies — it re-opens the single-account hole for every operation.

The Approve button is greyed out in the console. Same cause: you are the requester. The modal names both options inline and links back here. Deny stays enabled. To take back your own request, use meho approvals withdraw <id> instead.

approve fails with fresh_sign_in_required or fresh_sign_in_already_used. The command line or REST call did not show a fresh, unused sign-in. Run meho approvals approve <id> in a terminal where a browser can open, and sign in when asked. If it still fails right after you signed in, the realm is not set up for it yet: see Keycloak realm setup, Step 2. Or approve in the console.

I set the flag but /ready still says four_eyes_enforced. The backplane reads APPROVAL_ALLOW_SELF_APPROVAL at startup. Confirm the value reached the pod (config.approvalAllowSelfApproval: "true" in your overlay renders the APPROVAL_ALLOW_SELF_APPROVAL env var) and that the pod restarted onto the new ConfigMap.

When a guide and your deploy disagree

These guides are written against the product version this site version documents (see the version selector). If a command or error message differs on your deploy, check that your CLI and backplane versions match the docs version you are reading — then file a docs issue: a guide a fresh user cannot execute verbatim is a bug.