`gj5%dZddlmZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl m Z m Z ddl mZddlmZmZmZmZddlmZddlmZej0eZd Zd Zd Zd Zdd ZdddZ dddZ!ddddd ddZ"ed ddZ# d dZ$dddddd d!dZ%edddd d"dZ&y)#uOut-of-loop shutdown backstop + event-loop liveness heartbeat (#66892). When the asyncio loop freezes mid-drain, every asyncio-based recovery path is structurally unable to fire: the drain deadline, status rewrites, and forensics all need the same loop that is stuck. launchd/systemd KeepAlive only restarts a *dead* process, so a wedged-but-alive gateway sits as a zombie until manual SIGKILL. This module provides: 1. A plain OS-thread shutdown watchdog armed at ``stop()``. If shutdown has not completed within ``restart_drain_timeout + grace``, it dumps all-thread stacks via ``faulthandler`` plus a metadata snapshot, then ``os._exit`` so the service manager can revive the process. 2. An event-loop heartbeat file at ``/state/gateway.heartbeat`` so external supervision can distinguish "process alive" from "loop frozen" (``gateway_state.json`` alone can't — it only rewrites on transitions/turns). ) annotationsN)datetimetimezone)Path)AnyCallableDictOptional)get_hermes_home)atomic_json_writegN@g>@)statezgateway.heartbeat)logszgateway-shutdown-watchdog.logctjjddj}|r t |St S)zHHERMES_HOME for process-level identity files (ignore profile overrides). HERMES_HOME)osenvirongetstriprr )vals L/root/.hermes/venv/lib/python3.12/site-packages/gateway/shutdown_watchdog.py_process_hermes_homer/s5 **.. + 1 1 3C Cy  cD||n t}|jtS)z1Return ``/state/gateway.heartbeat``.)rjoinpath_HEARTBEAT_RELATIVEhomebases rget_loop_heartbeat_pathr 7s$#4)=)?D 4==- ..rcD||n t}|jtS)zBReturn the faulthandler / metadata dump path for a fired watchdog.)rr_WATCHDOG_DUMP_RELATIVErs rget_shutdown_watchdog_dump_pathr#=s$#4)=)?D 4==1 22r)pid start_timerextract|}t||ntjt j t jjtjd}|t||d<|r|j| t||d|S#t$rtj!ddY|SwxYw)uAtomically rewrite the loop-liveness heartbeat file. ``start_time`` is the gateway process start (``time.time()`` epoch seconds) so supervisors can detect PID reuse. Best-effort — never raises. N)r$ updated_at monotonicr%)indentz&Failed to write gateway loop heartbeatTexc_info)r intrgetpidrnowrutc isoformattimer)floatupdater Exceptionloggerdebug)r$r%rr&pathpayloads rwrite_loop_heartbeatr:Cs #4 (D#/3ryy{;ll8<<0::<^^%G  %j 1  uN$5 K N = M KNsB&& C  C )grace_sc tt|d} tt|d}||zS#ttf$rd}Y.wxYw#ttf$r t}Y||zSwxYw)z=Return the wall-clock leash for the shutdown watchdog thread.)maxr3 TypeError ValueError!DEFAULT_SHUTDOWN_WATCHDOG_GRACE_S) drain_timeoutr;draingraces rresolve_shutdown_watchdog_delayrE`sx E-(#.2E'NC( 5= z " z "21 5=2s 2A AA A%$A%c |jjdddtj|t j tjj|xsid} t|dd5}|jtj|t d z|jd |j t!j"|d |jd|jddd t&j(jd|ddtjdt&j(jt!j"dy#t$rYywxYw#t$$r|jd YwxYw#1swYxYw#t$$rYwxYw#t$$rYywxYw)z:Best-effort faulthandler + metadata dump before hard-exit.T)parentsexist_okNshutdown_watchdog_fired)eventr$delay_sfired_atsnapshotazutf-8)encoding)default z(--- faulthandler dump (all threads) --- )file all_threadsz%(faulthandler.dump_traceback failed) z--- end dump --- z&Gateway shutdown watchdog fired after z.0fzs (pid=z); dumping all thread stacks. )rS)parentmkdirOSErrorrr.rr/rr0r1openwritejsondumpsstrflush faulthandlerdump_tracebackr5sysstderr) dump_pathrKrMheaderfhs r_write_watchdog_dumprdqstd; +yy{LL.88:N F  )S7 3 r HHTZZ4t; < HH@ A HHJ C++F HH) * HHJ   4WSMBIIK= ? A  ##5A " CAB C          ssE:-F5;AF) F "!F)F5 A-G: FF F&#F)%F&&F))F2.F55 GG GGzgateway-shutdown-watchdog) done_event snapshot_fn exit_coderanamec`||ntj tt|ddkrSdfd } tj|d|jS#tt f$r t YQwxYw#t$rtjddYSwxYw) aArm a daemon-thread hard-exit backstop for a wedged shutdown path. If ``done_event`` is set before ``delay_s`` elapses, the thread exits quietly (normal / progressing shutdown completed). Otherwise it dumps diagnostics and calls ``os._exit(exit_code)``. Never raises. Returns the ``done_event`` (creating one when omitted) so the caller can disarm on successful completion. r=rc>tj z}tj|krL|tjz } jt|drytj|krL j ryd}   }  n t}t| | tjd |tjtjfD]} |j ddlm}m}|| ddlm}|dt)j* y#t $r}dt |i}Yd}~d}~wwxYw#t $rYwxYw#t $rYwxYw#t $rYrwxYw#t $rYqwxYw) N?)timeoutsnapshot_error)rKrMuhShutdown watchdog fired after %.0fs — forcing process exit (asyncio drain path appears wedged; see %s)r)remove_pid_filerelease_gateway_runtime_lock)drain_log_queue)r2r)waitminis_setr5reprr#rdr6criticalr_stdoutr`r\gateway.statusrorphermes_loggingrqr_exit)deadline remainingrMexctargetstreamrorprqdelaydonerarhrgs r _watchdogz(arm_shutdown_watchdog.._watchdogs>>#e+nn) 4>>#33IyyY!4y5nn) ;;= -1  " 9&=(39X9ZVUXF  OO>  zz3::. F     T   ( *  6 C ( I 9,d3i8 9             s`E6E#1E2FF E  EE # E/.E/2 E>=E> F  F  FFT)r~daemonrizFailed to arm shutdown watchdogr+)returnNone) threadingEventr>r3r?r@rAThreadstartr5r6r7) rKrfrgrhrarirrrs ``` @@rarm_shutdown_watchdogrs$$/:Y__5FD2E'NC( z 33jG $TBHHJ K z "212z G 6 F KGs#A.&B .BB B-,B-) interval_sr%rshould_continuec K tt|d}t || ||syt j|d{||syt ||?#ttf$r t}YdwxYw77w)uRewrite the loop heartbeat file on a cadence until cancelled / gated off. Runs as an asyncio task on the gateway loop — if the loop freezes, this task stops and the file mtime/updated_at goes stale for external monitors. rl)r%rN)r>r3r?r@DEFAULT_HEARTBEAT_INTERVAL_Sr:asynciosleep)rr%rrintervals rloop_heartbeat_foreverrs0uZ(#. JT:   &/@ mmH%%%  &/@  >  z "0/0 &s3BA&0B B B&A>;B=A>>B)rr)N)rOptional[Path]rr) r$z Optional[int]r%Optional[float]rrr&Optional[Dict[str, Any]]rr)rBr3r;r3rr3)rarrKr3rMrrr)rKr3rfzOptional[threading.Event]rgz&Optional[Callable[[], Dict[str, Any]]]rhr-rarrir[rzthreading.Event) rr3r%rrrrzOptional[Callable[[], bool]]rr)'__doc__ __future__rrr]rYloggingrr_rr2rrpathlibrtypingrrr r hermes_constantsr utilsr getLogger__name__r6rArrr"rr r#r:rErdrrrrrs&#   '00,#   8 $%)!#4C/ 3"&&*      $   @7 "+ + + ' +  + b-1:> $+T T*T8 T  T  T TTr5"&48 ?? ?  ? 2 ?  ?r