`gjredZddlmZddlZddlZddlZddlmZddlm Z ddl m Z m Z ejeZedhZedd hZd Ze d Zed GddZddd ddZ d dZd!dZd"dZd#dZd#dZd$dZd%dZd&dZ d'dZdZ d(dZ!edk(r e"e!y))aContainer-boot reconciliation of per-profile gateway s6 services. Service directories under /run/service/ live on **tmpfs** and are wiped on every container restart. Profile directories under ``$HERMES_HOME/profiles//`` live on the persistent VOLUME, and each one records its gateway's last state in ``gateway_state.json``. This module bridges the two: on every container boot, walk the persistent profiles, recreate the s6 service slots, and auto-start only those whose last recorded state was ``running``. Wired into the image as /etc/cont-init.d/02-reconcile-profiles by the Dockerfile (Phase 4 Task 4.0). Runs as root after 01-hermes-setup (the stage2 hook) has chowned the volume and seeded $HERMES_HOME, but before s6-rc starts user services. Without this module, every ``docker restart`` would silently wipe every per-profile gateway, even though the user's profiles still exist on disk. ) annotationsN) dataclass)Path)LiteralSequencerunningdrainingdegraded)z gateway.pidzprocesses.json)started registeredskippedT)frozenc0eZdZUdZded<ded<ded<y) ReconcileActionz8One profile's outcome from a single reconciliation pass.strprofile str | None prior_stateReconcileActionLabelactionN)__name__ __module__ __qualname____doc____annotations__L/root/.hermes/venv/lib/python3.12/site-packages/hermes_cli/container_boot.pyrrPsB L  rrF)dry_runcontainer_argvcg}ddlm}|tjj d}t |||}|xs t |}|tv} |st|t|d| |jtd|| rdnd |d z } | jrt| jD]} | js| d z js(| j dk(rt"j%d Mt | } | xr| tv} |s#t| t|| j | |jt| j | | rdnd |s t'|||S) ubRecreate s6 service registrations for every persistent profile. Always registers a ``gateway-default`` slot for the root profile (the implicit profile that lives at the top of ``$HERMES_HOME``, not under ``profiles/``). The dispatcher in ``hermes_cli.gateway`` maps an empty profile suffix to ``gateway-default``, so this slot is what ``hermes gateway start`` (no ``-p``) targets. Without it, bare ``hermes gateway start`` inside the container would land on ``s6-svc -u /run/service/gateway-default`` → uncaught ``CalledProcessError`` → traceback to the user (PR #30136 review). The default slot's prior state is read from ``$HERMES_HOME/gateway_state.json`` (sibling to the profile root, not under ``profiles/``); stale runtime files there are swept the same way as for named profiles. Args: hermes_home: The container's HERMES_HOME (typically /opt/data). Profiles live under ``/profiles//``; the default profile lives at ```` itself. scandir: The s6 dynamic scandir (typically /run/service). Service directories are created at ``/gateway-/``. dry_run: When True, walk and return the action list without touching the filesystem. For tests and `--dry-run` debug. container_argv: Optional container PID 1 argv override. Production reads ``/proc/1/cmdline``; tests inject it directly. Returns: One :class:`ReconcileAction` per profile, in this order: ``default`` first, then named profiles in directory order. r)is_truthy_valueGATEWAY_MULTIPLEX_PROFILES)r rdefault)startr r )rrrprofileszSOUL.mdu_profiles/default/ exists — skipping to avoid colliding with the reserved root-profile s6 slot)utilsr"osenvironget'_maybe_migrate_legacy_gateway_run_state_read_desired_state_AUTOSTART_STATES_cleanup_stale_runtime_files_register_serviceappendris_dirsortediterdirexistsnamelogwarning_write_reconcile_log) hermes_homescandirrr actionsr"multiplex_profileslegacy_default_statedefault_prior_statedefault_should_start profiles_rootentryr should_starts rreconcile_profile_gatewaysrCXsL&(G &( 34C% /R2Ek2R.2CC $[1'94HI NN?'0yl  *,MM1134# E<<> I%--/zzY& =-e4K&&K;:K+K ,U3!'5::\J NN? '$0yl ?# J ['2 Nrc j|dz }|jrytjjddj dvry| t |n t }t|sy|sGddl}|jtjddt|jdd d zy) aSeed root gateway_state for pre-s6 `gateway run` containers. The tini image let Docker users run the gateway as the container command (`docker run ... gateway run`). After the s6 migration, profile gateways are restored from persisted gateway_state.json; a legacy container with no state file would therefore register the default service down and never start. Only synthesize state when no root gateway_state.json exists so explicit stopped/failed states keep winning across restarts. gateway_state.jsonNHERMES_GATEWAY_NO_SUPERVISE)1trueyesrrzlegacy-container-cmd) gateway_state desired_state timestamp migrated_from ) r4r(r)r*lowertuple_read_container_argv_is_legacy_gateway_run_requesttime write_textjsondumpsint)r9r r state_fileargvrTs rr+r+s 33J zz~~3R8>>@DXX$2$>5 DXDZD )$ / djj&&TYY[)3 *     rc tdj}td|jdD}t d|Dr|S td}|j D]h}|jjs |dz j}td|jdD}t d|Dsf|cS y #t $rYwxYw#t $rYwxYw#t $rYy wxYw) aBest-effort read of the container's main program argv. Under s6-overlay v2, PID 1 is ``/init`` and its argv contains the ``main-wrapper.sh`` path. Under s6-overlay v3, PID 1 is ``s6-svscan`` and the actual command (``rc.init top main-wrapper.sh ...``) lives on a different PID. We try PID 1 first (fast path, covers v2 and pre-s6 images), then fall back to scanning ``/proc/*/cmdline`` for a process whose argv contains ``main-wrapper.sh`` (the rc.init-launched PID in v3). z/proc/1/cmdlinec3FK|]}|s|jddywutf-8replaceNdecode.0parts r z'_read_container_argv..s" 04DDKK + s!!c3$K|]}d|v ywzmain-wrapper.shNrrbs rrez'_read_container_argv..s:T D(:z/proccmdlinec3DK|]}|r|jddywr]r`rbs rrez'_read_container_argv..s& GY/s c3$K|]}d|v ywrhrrbs rrez'_read_container_argv..s>$,>rir) r read_bytesrQsplitanyOSErrorr3r5isdigit)rawrZproc_dirrAs rrRrRs $%002 8; %8H   :T: :K ; =%%' E::%%' y(446IIe,D >>>  " 1          sTA C9C8 C)3C8C8C8 C&%C&) C52C84C55C88 DDct|}tdt|Dd}| ||dzd}n"|r t|djdk(r|dd}|r t|djdk(r|dd}|S)u|Strip the s6/wrapper prefix off the container argv, leaving the hermes args. Two container-command argv shapes are handled: * **s6-overlay v2 / tini:** PID 1 argv is ``/init /opt/hermes/docker/main-wrapper.sh [args...]``. * **s6-overlay v3:** PID 1 is ``s6-svscan`` and the command lives on the rc.init-launched process as ``/bin/sh -e /run/s6/basedir/scripts/rc.init top /opt/hermes/docker/main-wrapper.sh [args...]`` (see :func:`_read_container_argv`). Rather than peel each leading token positionally (which silently breaks the moment s6 changes its launcher shape again — exactly what happened in the v2→v3 bump), drop everything up to and including the ``main-wrapper.sh`` token: that wrapper path is the stable boundary the image owns, and the subcommand always follows it. Pre-s6 / direct ``hermes`` invocations carry no wrapper, so fall back to peeling a bare ``init`` prefix. The wrapper re-execs ``hermes ``, so an explicit leading ``hermes`` is peeled too. Shared by the legacy-gateway and dashboard role detectors. c3JK|]\}}|jds|ywrh)endswith)rcias rrez/_strip_container_argv_prefix..=s Htq!!**5F*GHs##Nrinithermes)listnext enumeraterr5)rZargs wrapper_idxs r_strip_container_argv_prefixr!s, :D HyH KK!O%& $tAw-$$.ABx T!W ""h.ABx Krcft|}d|vryt|dk\xr|ddk(xr|ddk(S)z Gd1g2 GtAw%7GGrcBt|}t|xr|ddk(S)uReturn True when the container's command is the dashboard. A dashboard-only container (``hermes dashboard ...``) never spawns or supervises per-profile gateways — that is the gateway container's job. Reconciling profile gateway s6 slots there is not just wasted work: when the gateway and dashboard containers share a bind-mounted HERMES_HOME, both race to ``flock()`` the same ``logs/gateways//lock`` files, producing "Resource busy" failures and an s6-log restart storm. So the dashboard container skips reconciliation entirely. Detected from PID 1 argv (``/proc/1/cmdline``) rather than an operator flag: the role is a fact about the container's command, not a tunable, and a flag can be forgotten in a hand-written compose/k8s manifest — reintroducing the exact storm this prevents. Mirrors the argv handling in :func:`_is_legacy_gateway_run_request`. r dashboard)rboolrs r_is_dashboard_containerrTs&" ( -D : 0$q'[00rcB|dz }|jsy tj|j}|j d}||S|j d}|t vry|S#t tjf$rtjd|YywxYw)uRead the persisted gateway desired state for reconciliation. Newer state files carry ``desired_state``: operator intent written by s6 lifecycle commands. Older files only carry ``gateway_state``; keep that as a compatibility fallback so existing running/stopped profiles preserve their behavior until the next explicit start/stop. When falling back to ``gateway_state`` (no explicit ``desired_state``), a transient running sub-state (``draining``) is normalised to ``running`` — see ``_TRANSIENT_RUNNING_STATES``. A gateway hard-killed mid-drain leaves ``draining`` as its last persisted value; without this it would be treated as a non-autostart state and the gateway would stay DOWN forever. An explicit ``desired_state`` is always honoured verbatim (it is the operator's durable intent), so this normalisation only affects the legacy/transient fallback path. Missing or unparseable files count as "no desired state" so we don't bork the whole reconciliation on a corrupt file. rENrLrKrz-could not read %s; treating as no prior state) r4rVloads read_textr*_TRANSIENT_RUNNING_STATESrpJSONDecodeErrorr6r7) profile_dirrYdatarLrKs rr,r,is(33J     zz*..011  $ 1 5 5 T)) * ;Z  s7A,A,*A,,/BBcDtD]}||z jdy)uRemove gateway.pid and processes.json — they reference PIDs in the dead container's process namespace and would otherwise confuse the newly-started gateway's process-mismatch checks.T) missing_okN)_STALE_RUNTIME_FILESunlink)rr5s rr.r.s)%5 t ##t#45rcfddl}ddlm}m}m}|||d|z }|j d|j zdz}|jr|j|d|jd  |d z jd |d z } | j|j|i | jd|dz } | j|j| jd|dz } | j| d z } | j|j|| jd|s|dz j|||jr|j||j!|y#t"$r|j|dwxYw)uRecreate the s6 service slot for one profile. Mirrors the rendering in :func:`S6ServiceManager.register_profile_gateway`, but here we control the start state directly via the ``down`` marker file (s6-svscan honors it on rescan). Cannot use the manager directly because the cont-init.d phase runs as root before s6-svscan starts scanning the dynamic scandir — the manager's ``s6-svscanctl -a`` call would fail with no control socket. Atomicity: build the new layout in a sibling temp directory and rename it into place via :meth:`Path.replace`. This matches :meth:`S6ServiceManager.register_profile_gateway` (PR #30136 review item O4) — even though cont-init.d runs before s6-svscan starts scanning, an atomic publication keeps the contract uniform between the two registration paths and protects against a half-populated dir if the script is interrupted mid-write. rN)S6ServiceManager_seed_supervise_skeletonvalidate_profile_namezgateway-.z.tmpT) ignore_errors)parentstypezlongrun r) extra_envifinishr6down)shutilhermes_cli.service_managerrrr with_namer5r4rmtreemkdirrU_render_run_scriptchmod_render_finish_script_render_log_runtouchr_ Exception) r:rr%rrrr service_dirtmp_dirrr log_subdirlog_runs rr/r/s$ '"hwi00K##C+*:*:$:V$CDG~~ gT 2 MM$M3 6 %%k2o '::7b:QR %8#*@@BC Uu_ u$+;;GDE e v  $ $ & !)     MM+ & $  gT 2 s 6DFF0c $ddl}|dz }|jdd|dz } |jr5|jjt k\r|j |dz |jd }|jd d 5}|D]=}|j|d |jd|jd|jd? dddy#t$r!}tjd||Yd}~d}~wwxYw#1swYyxYw)uAppend one line per profile to $HERMES_HOME/logs/container-boot.log. Operators inspect this to debug "why didn't my profile come back up". Keeping a separate log file (vs. mixing into agent.log) lets troubleshooters grep for "profile=foo" without wading through unrelated activity. Size-bounded: when the file exceeds ``_LOG_ROTATE_BYTES`` (defaults to 256 KiB ≈ 3000 reconcile lines), the current file is renamed to ``container-boot.log.1`` (replacing any previous rotation) before the new entries are appended. This gives long- lived containers a soft cap of ~512 KiB across the two files without pulling in logrotate or s6-log machinery just for this one append-only file (PR #30136 review item O3). rNlogsT)rexist_okzcontainer-boot.logzcontainer-boot.log.1zcould not rotate %s: %sz%Y-%m-%dT%H:%M:%S%zrxr^)encodingz profile= prior_state= action=rO)rTrr4statst_size_LOG_ROTATE_BYTESr_rpr6r7strftimeopenwriterrr) r9r;rTlog_dirlog_pathexctsfrxs rr8r8s$F"G MM$M.--H> ?? !8!8 , -B sW - A GG$i {- G((2'   > -x==> s%AC AD D"C>>DDic rttr tdyttj j dd}ttj j dd}t||}|D]4}td|jd |jd |j6y) z@Entry point invoked from /etc/cont-init.d/02-reconcile-profiles.uPreconcile: skipping (dashboard container — does not need per-profile gateways)r HERMES_HOMEz /opt/dataS6_PROFILE_GATEWAY_SCANDIRz /run/service)r9r:zreconcile: profile=rr) rrRprintrr(r)r*rCrrr)r9r:r;rxs rmainr,s356  $ rzz~~m[ABK2::>>">OPG(G  !!))-==/!(( =  r__main__) r9rr:rrrr Sequence[str] | Nonereturnlist[ReconcileAction])r9rr rrrrr)rztuple[str, ...])rZ Sequence[str]rz list[str])rZrrr)rrrr)rrrNone)r:rrrr%rrr)r9rr;rrr)rrX)#r __future__rrVloggingr( dataclassesrpathlibrtypingrr getLoggerrr6 frozensetr-rrrrrCr+rRrrSrr,r.r/r8rr SystemExitrrrrsf&#  !$g!yk*<&z:&>? 9AB $!!!+/ rrr r ) r  rj##)# #  #L+\(VH1*$N5_D'' 5' '^ > z TV r