`gjdZddlmZddlZddlZddlZddlmZmZddl m Z ejdZ dZ dZd Zdd Zdd Zefdd  dd Ze efdd  ddZddZe efdd  ddZy)uLAuto-resume restart-loop breaker (#30719, defense-3). Defenses 1 and 2 (the ``_HERMES_GATEWAY`` guard on ``hermes gateway stop|restart`` + ``terminal_tool``, and the cron-creation lifecycle filter) stop the agent from scheduling its own restart via the cron and CLI paths. They do NOT cover every SIGTERM source: an agent running a raw ``terminal("launchctl kickstart -k gui//ai.hermes.gateway")``, an external monitor with a bad trigger, or any other repeated crash can still drive the supervisor (launchd ``KeepAlive`` / systemd ``Restart=``) into a tight respawn loop. On each boot the gateway auto-resumes the restart-interrupted session, whose next turn re-runs the offending logic — SIGTERM every ~10 seconds until manually broken. This module is the last-resort circuit breaker: it records a timestamp each time the gateway boots with restart-interrupted sessions pending, keeps a rolling window of recent boots persisted across processes (each boot is a fresh process, so in-memory state is useless), and reports the loop as "tripped" once too many such boots happen inside a short window. When tripped, the caller SKIPS auto-resume for that boot — the gateway still starts and serves real inbound messages, it just stops replaying the session that keeps killing it, which breaks the cycle and puts a human back in the loop. State lives in ``/gateway/restart_loop.json`` so it is profile-scoped and survives process death. It is intentionally tiny and best-effort: any read/write failure fails OPEN (no false trip) because a broken breaker must never wedge a healthy gateway. ) annotationsN)ListOptionalget_hermes_homez gateway.run<c"tdz dz S)Ngatewayzrestart_loop.jsonrM/root/.hermes/venv/lib/python3.12/site-packages/gateway/restart_loop_guard.py _state_pathr/s  y (+> >>r c( tjd}tj|}|j dg}|Dcgc]$}t |t tfst|&c}Scc}w#tttf$rgcYSwxYw)Nutf-8encodingboots) r read_textjsonloadsget isinstanceintfloatOSError ValueError TypeError)rawdatarts r _load_bootsr"3sym%%w%7zz#"%"'GQ:a#u+FaGGG Z + s*AA8A3# A30A83A88BBc t}|jjdd|jt j d|idy#t $rYywxYw)NT)parentsexist_okrrr)rparentmkdir write_textrdumpsr)rpaths r _save_bootsr+=sU } $6  GU#34wG    sAA AAnowc|tjn|}|td|z }tDcgc] }||k\s | }}|j|t ||Scc}w)u)Record that the gateway just booted with restart-interrupted sessions. Prunes boots older than ``window_seconds`` and appends the current time. Returns the pruned+appended list (most recent last). Best-effort — a persistence failure returns the in-memory list without raising. )timemaxr"appendr+)window_secondsr-tscutoffr!rs rrecord_restart_interrupted_bootr6Fs_ B #a( (F# 51fQ 5E 5 LL L 6s A#A#c|dkry|tjn|}|td|z } tDcgc] }||k\s | }}t ||k\Scc}w#t$rYywxYw)usReturn True if the gateway has restarted ``>= max_restarts`` times with restart-interrupted sessions inside the last ``window_seconds``. Reads the persisted boot log written by ``record_restart_interrupted_boot`` and counts boots within the window. Fails OPEN (returns False) on any error — a broken breaker must never wedge a healthy gateway. rFr/)r0r1r" Exceptionlen) max_restartsr3r-r4r5r!recents ris_restart_loop_trippedr<Ysyq B #a( (F(]:a6k!:: v;, &&; s( A AA AA A+*A+cX tjdy#t$rYywxYw)zBRemove the persisted boot log (used on clean shutdown / by tests).T) missing_okN)runlinkrr r rclearr@rs,  -    s  ))ct||}|dkDrt||k\nd}|r*tjdt|||t |S)aERecord this restart-interrupted boot and report whether the loop is now tripped. This is the single entry point the gateway calls: it appends the current boot, then checks whether the (now-updated) window has reached the threshold. Returns True when auto-resume should be SKIPPED to break the loop. r,rFa0Restart-loop breaker TRIPPED: %d restart-interrupted gateway boots within %ds (threshold %d). Skipping auto-resume to break a suspected SIGTERM-respawn loop (#30719). Restart-interrupted sessions stay resume-pending and will continue on the next real user message. If this is a false positive, delete %s.)r6r9loggerwarningr)r:r3r-rtrippeds rcheck_and_recordrEzsW ,N DE,81,rSs:#  !,   = )? 1   (-0' '''  '  '2 -0     r