`gj06dZddlmZddlZddlZddlZddlmZmZddlm Z ddl m Z m Z ddl mZddlmZej"eZd Zej*d dd Zddd Zdddd ddZddddZddZddddZddddZddddZy)u External drain-control marker contract (dashboard → gateway). Task 2.2 of the safe-shutdown plan (decisions.md Q-B, option A): the dashboard has no way to call into a running gateway — there is no HTTP control channel into the gateway process (guardrails: "there is NO external control channel into a running gateway"). Restart/drain is driven only by the gateway reacting to its own inputs: slash commands, process signals, and file markers it writes itself (``.restart_notify.json``). So the begin/cancel-drain dashboard endpoint communicates with the running gateway the same way: it writes (or removes) a marker file, and a gateway background watcher reacts to it. This module owns that marker contract so both sides — the dashboard endpoint (writer) and the gateway watcher (reader) — share one definition and can never disagree. Contract (presence-based, mirroring ``.restart_notify.json``): * begin-drain → write ``{HERMES_HOME}/.drain_request.json`` with ``{"action": "drain", "requested_at": , "principal": , "epoch": , "suppress_notification": }``. * cancel-drain → remove the marker. * The gateway watcher treats **presence of a marker stamped with the current instantiation epoch** as "external drain active": flip ``gateway_state -> "draining"`` and stop accepting new turns. Absence (or a marker from a *prior* instantiation) means "not draining" (revert to ``running`` if we had flipped it). Why the epoch (NS-570). ``HERMES_HOME`` is a **durable** store — on Hermes Cloud it is a persistent Fly volume (``/opt/data``). A begin-drain marker written there *survives a machine restart*. But the disruptive lifecycle actions a drain protects (auto-update / image migrate / env edit / profile change) all **restart the machine**, which is exactly the signal that the drain is over. Without the epoch, a freshly-restarted gateway re-reads the orphaned marker on boot and parks itself right back in ``draining`` forever (NS-570: an auto-updated instance refused every turn for ~52 min). Stamping the marker with an identity of *this* container/VM instantiation, and ignoring a marker whose epoch doesn't match, makes "a deliberate restart clears the drain" true by construction — while a marker written during the *current* instantiation (the live drain) still matches, and an s6 respawn of just the gateway (PID 1 / init unchanged) still honours an in-flight drain. Reading the marker never raises: a malformed/half-written file reads as "present but contentless", which the watcher still treats as drain-active (fail-safe toward quiescing — a corrupt begin marker must not be ignored). The epoch check is deliberately **lenient**: it ignores a marker only on a *definite* epoch mismatch. A marker with no epoch (legacy/corrupt/contentless), or an environment where the epoch cannot be computed (non-Linux, no ``/proc``), both degrade to the original presence-only behaviour — never fail-closed. ) annotationsN)datetimetimezone)Path)AnyOptional)get_hermes_home)atomic_json_writez.drain_request.json)maxsizecFd} tdjdj}d} tdjd}|j dddj }|d}|s|sy|d |S#t$rY]wxYw#tt f$rY,wxYw) u!Identity of THIS container / VM instantiation. Stable for the life of the PID-1 init process — so an s6 respawn of just the gateway keeps the same epoch and an in-flight drain is honoured — but changes when the machine/container is recreated (a fresh PID 1 → a fresh epoch). Composed from two ``/proc`` facts: * the kernel **boot id** (``/proc/sys/kernel/random/boot_id``) — changes on a VM / microVM reboot (e.g. a Fly Firecracker machine restart); * **PID 1's start time** (field 22 of ``/proc/1/stat``) — changes on a plain ``docker restart`` (the host kernel, hence boot_id, is unchanged, but ``/init`` is a brand-new process). Together they discriminate every restart mode that matters: | event | boot_id | pid1 start | epoch | marker | |--------------------------------|---------|------------|--------|--------| | Fly microVM reboot (auto-upd.) | changes | changes | NEW | reject | | plain ``docker restart`` | same | changes | NEW | reject | | s6 respawn of the gateway only | same | same | SAME | honour | | host ``hermes gateway restart``| same | same(init) | SAME | honour | The last row is intentional: a host install has no durable-volume drain bug, and honouring a drain across a deliberate process restart is the intended reversible behaviour (D4a) — PID 1 there is the long-lived init (systemd/launchd), so the epoch is stable. Returns ``""`` when neither identity source is readable (non-Linux, no ``/proc``). An empty epoch disables the staleness check downstream, degrading to the released presence-only behaviour — never fail-closed. Memoised: the epoch is constant for the life of the process. z/proc/sys/kernel/random/boot_idutf-8encodingz /proc/1/stat)r :)r read_textstripOSErrorrsplitsplit IndexError)boot_id pid1_startstattails H/root/.hermes/venv/lib/python3.12/site-packages/gateway/drain_control.pycurrent_instantiation_epochr CsDG 2 3 YY ( UW J  N#--w-?{{3"1%++-"X  :Ya | $$%     Z    s#)A?AB? B  B B B cB||n t}t|tz S)zBAbsolute path to the drain-request marker, respecting HERMES_HOME.)r r_DRAIN_REQUEST_FILENAME)homebases rdrain_request_pathr%s!#4):D :/ //z drain-controlF) principalsuppress_notificationr#cdtjtjj |t t |d}tt|||S)u Write the begin-drain marker. Returns the payload written. Atomic write so the gateway watcher never reads a half-written file. Idempotent: re-writing while a drain is already in progress just refreshes ``requested_at`` (harmless — the watcher keys off presence, not content). Stamps the marker with :func:`current_instantiation_epoch` so a marker that later survives a machine restart on the durable HERMES_HOME volume can be recognised as stale and ignored (NS-570). ``suppress_notification`` is a generic "be quiet on the shutdown that ends this drain" flag. When the drain culminates in a process exit (e.g. NAS recreates the machine for an auto-update image migration), the gateway's shutdown path reads it via :func:`drain_notification_suppressed` and skips the *home-channel* "gateway shutting down" broadcast — the operator-flavoured ping that would otherwise fire on every routine auto-update, potentially dozens of times a day. It NEVER suppresses the per-active-session interrupt ping. The gateway stays agnostic about *why* the drain is quiet; the policy of which drain causes set the flag lives entirely in the caller (NAS). The field defaults False so legacy/operator drains behave exactly as before. drain)action requested_atr'epochr() rnowrutc isoformatr boolr r%)r'r(r#payloads rwrite_drain_requestr3sP8 X\\2<<>,.!%&;!< G(.8 Nr&r#ct|} |jy#t$rYyt$r!}tj d||Yd}~yd}~wwxYw)zRemove the drain marker (cancel-drain). Returns True if one existed. Best-effort: a missing file is not an error (cancel is idempotent). TFz&drain-control: failed to remove %s: %sN)r%unlinkFileNotFoundErrorr_logwarning)r#pathes rclear_drain_requestr<sN d #D    =tQGs AAA  AcNt}|sy|jd}|sy||k7S)u True iff ``body``'s epoch is a *definite* mismatch with this process. Lenient by design — returns False (i.e. "not stale, honour it") whenever it can't be sure: * the current epoch can't be computed ("" fallback, no /proc), OR * the marker carries no epoch (legacy marker, or a corrupt/contentless ``{}`` body). Only a marker whose epoch is present AND differs from the current instantiation epoch is considered stale. This preserves the fail-safe-toward-quiescing contract for malformed markers. Fr-)r get)bodycurrent marker_epochs r_marker_epoch_is_stalerBs1*+G 88G$L  7 ""r&c:t|}|yt|ryy)udTrue iff a begin-drain marker for THIS instantiation is present. A marker whose ``epoch`` does not match the current instantiation epoch is treated as absent: it survived a container/VM restart (HERMES_HOME is a durable Fly volume on Hermes Cloud) and the lifecycle action that triggered the drain has already completed — honouring it would wedge the freshly-restarted gateway in ``draining`` (NS-570). The staleness check is lenient (see :func:`_marker_epoch_is_stale`): a legacy/corrupt marker with no epoch, or an environment without ``/proc``, still reads as drain-active. r4FT)read_drain_requestrBr#r?s rdrain_requestedrFs$ 4 (D |d# r&clt|}|yt|ryt|jdS)uTrue iff an ACTIVE drain marker asks to suppress the shutdown broadcast. "Active" means exactly what :func:`drain_requested` means — a marker present AND stamped with the current instantiation epoch. A stale (other-epoch) marker that survived a machine restart on the durable HERMES_HOME volume is ignored here just as it is for drain state (NS-570): we must never let an orphaned marker's flag silence a *fresh* gateway's legitimate shutdown broadcast. Only honours the flag when it is explicitly truthy in the marker body. A legacy marker without the field, a corrupt/contentless ``{}`` body, or an absent marker all read as "not suppressed" (False) — fail toward the louder, more-visible behaviour, consistent with :func:`read_drain_request`'s never-raise contract. The gateway's shutdown path uses this to skip ONLY the home-channel broadcast; the per-active-session interrupt ping is unaffected. r4Fr()rDrBr1r>rEs rdrain_notification_suppressedrHs6" 4 (D |d# 01 22r&c.t|} |jd} t j|}t|tr|SiS#t$rYyt$r!}tj d||Yd}~yd}~wwxYw#ttf$ricYSwxYw)aReturn the marker payload, or ``None`` if absent. A present-but-unparseable marker returns ``{}`` (truthy-presence preserved via :func:`drain_requested`; callers that need the body get an empty dict rather than an exception). Never raises. rrNz$drain-control: failed to read %s: %s) r%rr7rr8r9jsonloads ValueError TypeError isinstancedict)r#r:rawr;datas rrDrDs d #Dnngn. zz#dD)41r1   ;T1E  " s.A B A=A=A88A=BB)returnstr)N)r#Optional[Path]rRr)r'rSr(r1r#rTrRdict[str, Any])r#rTrRr1)r?rUrRr1)r#rTrRzOptional[dict[str, Any]])__doc__ __future__r functoolsrJloggingrrpathlibrtypingrrhermes_constantsr utilsr getLogger__name__r8r" lru_cacher r%r3r<rBrFrHrDr&rrbs0b# ' ,#w"/Q:% :%z0%"' ## #  #  #L37  #*/3&=A32262r&