`gjN<dZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddlmZddlmZddlmZddlmZej,eZe j2Ze j6ZdZdZeeegdfZ ee!e"eefgdfZ#eegdfZ$ee!egdfZ%ege!e"eeffZ&dd ed e!e"eeffd Z'd e(efd Z)d e!ed efdZ*de!ed efdZ+de!e"eefd e!efdZ,ded efdZ-dZ.dZ/dZ0GddZ1y#e $rdZ YwxYw)aShared file sync manager for remote execution backends. Tracks local file changes via mtime+size, detects deletions, and syncs to remote environments transactionally. Used by SSH, Modal, and Daytona. Docker and Singularity use bind mounts (live host FS view) and don't need this. N)Path)Callable)get_hermes_home)_file_mtime_keyg@HERMES_FORCE_FILE_SYNCcontainer_basereturnc*ddlm}m}m}g}|D].}|dj d|d}|j |d|f0||D]}|j |d|df||D]}|j |d|df|S)a[Enumerate all files that should be synced to a remote environment. Combines credentials, skills, and cache into a single flat list of (host_path, remote_path) pairs. Credential paths are remapped from the hardcoded /root/.hermes to *container_base* because the remote user's home may differ (e.g. /home/daytona, /home/user). r)get_credential_file_mountsiter_cache_filesiter_skills_filescontainer_path /root/.hermes host_path)r)tools.credential_filesr r r replaceappend)rr r r filesentryremotes O/root/.hermes/venv/lib/python3.12/site-packages/tools/environments/file_sync.pyiter_sync_filesr5s $&E+-3'(00 ^Q   eK(&12 3 #.AD eK(%0@*ABCD!@D eK(%0@*ABCD Lc  ddlm}t} |}|D]h}t |t r|j dnd}|s) |jtt|jjj|S#t$r tcYSwxYw#t$r tcYSwxYw#t$r4|jtt|jYwxYw)zBReturn credential files that are upload-only for remote sandboxes.r)r rN) rr Exceptionset isinstancedictgetaddstrr expanduserresolveOSError)r pathsmountsrrs r_credential_host_pathsr(RsEeE+-9.8.EEIIk*4   9 IIc$y/446>>@A B 9 L! u  u  9 IIc$y/4467 8 9s4B B"?B; BB"B87B8;:C87C8 remote_pathsc8ddjd|DzS)zz$quoted_rm_command..ksD!u{{1~D!join)r)s rquoted_rm_commandr7is chhD|DD DDrdirsc8ddjd|DzS)z>Build a shell ``mkdir -p`` command for a batch of directories.z mkdir -p r+c3FK|]}tj|ywr-r.)r1ds rr3z'quoted_mkdir_command..ps!?Q%++a.!?r4r5)r8s rquoted_mkdir_commandr<ns !?$!?? ??rrc pt|Dchc]\}}tj|c}}Scc}}w)zCExtract sorted unique parent directories from (host, remote) pairs.)sorted posixpathdirname)r_rs runique_parent_dirsrBss* eDF9$$V,D EEDs2 pathctj}t|d5tfddD]}|j | ddd|j S#1swY|j SxYw)z$Return hex SHA-256 digest of a file.rbc&jdS)Ni)read)fsrz_sha256_file..|s!&&-rrN)hashlibsha256openiterupdate hexdigest)rChchunkrHs @r _sha256_filerRxsiA dD Q/5 E HHUO  ;;= ;;=s &A!!A9)lc<eZdZdZeddfdedededede dzde dzf d Z d d d e d dfdZ ddedzd dfdZded dfdZded dfdZddZ ddedeeeefdzd edzfdZ ddddedeeeefdzdeedzd edzfdZededeed e fdZy)FileSyncManageruTracks local file changes and syncs to a remote environment. Backends instantiate this with transport callbacks (upload, delete) and a file-source callable. The manager handles mtime-based change detection, deletion tracking, rate limiting, and transactional state. Not used by bind-mount backends (Docker, Singularity) — those get live host FS views and don't need file sync. N get_files_fn upload_fn delete_fn sync_intervalbulk_upload_fnbulk_download_fnc||_||_||_||_||_i|_i|_t|_d|_ ||_ y)Ng) _get_files_fn _upload_fn_bulk_upload_fn_bulk_download_fn _delete_fn _synced_files_pushed_hashesr_upload_only_host_paths_last_sync_time_sync_interval)selfrYrZr[r\r]r^s r__init__zFileSyncManager.__init__sT*#-!1#;=.014$&)+rF)forcerlr c|sJtjjts't }||j z |j kry|j}|jjt|Dchc]\}}| }}}g}t|j}|D]J\} } t| } | |jj| | k(r3|j| | f| || <L|jD cgc] } | |vs|  } } |s| st |_yt|j}t|j}|rt j#dt%|| rt j#dt%|  |r=|j&1|j'|t j#dt%|n3|D].\} } |j)| | t j#d| | 0| r'|j+| t j#d| |D]\} } t-| |j| <| D]0} |j/| d|jj/| d2||_ t |_ycc}}wcc} w#t0$r.}||_ ||_t j3d|Yd}~yd}~wwxYw)aTRun a sync cycle: upload changed files, delete removed files. Rate-limited to once per ``sync_interval`` unless *force* is True or ``HERMES_FORCE_FILE_SYNC=1`` is set. Transactional: state only committed if ALL operations succeed. On failure, state rolls back so the next cycle retries everything. Nzfile_sync: uploading %d file(s)z+file_sync: deleting %d stale remote file(s)z#file_sync: bulk-uploaded %d file(s)zfile_sync: uploaded %s -> %szfile_sync: deleted %sz-file_sync: sync failed, rolled back state: %s)osenvironr _FORCE_SYNC_ENV _monotonicrhrir`rgrNr(rrerrrfloggerdebuglenrbrardrRpoprwarning)rjrlnow current_filesrArcurrent_remote_paths to_upload new_filesr remote_pathfile_keyr2 to_delete prev_files prev_hashesexcs rsynczFileSyncManager.syncsRZZ^^O<,CT)))D,?,??**,  $$++,B,DE8EF91fFF,. ++, &3 . "I{&y1H!!%%k2h>   i5 6%-Ik " .!% 2 2T1a?S6SQT T#-@VX`a "3a77.w7ENNL! S%5M "s?B// C?8=C::C?rchtjtju}gd}|rLtjtj }fd}tjtj | |j ||rN|Ktjtj |r$tjtj yyyy#|rN|Ktjtj |r$tjtj wwwwxYw)z>Single sync-back attempt with SIGINT protection and file lock.NcVj||ftjdy)Nz/sync_back: SIGINT deferred until sync completes)rrrrs)signumframedeferred_sigints r _defer_sigintz6FileSyncManager._sync_back_once.._defer_sigint.s"&&7 NOr) threadingcurrent_thread main_threadsignal getsignalSIGINT_sync_back_locked raise_signal)rjron_main_threadoriginal_handlerrrs @rrzFileSyncManager._sync_back_once"s #113y7L7L7NN(* %// >  P MM&-- 7 7  " "9 -"2"> fmm-=>"'' 6##?~~"2"> fmm-=>"'' 6##?~s =CAD1ct|jyt|dd} tj|tj|j tj|tj |jy#t tf$rY"wxYw# tj|tj n#t tf$rYnwxYw|jwxYw)z;Sync-back under file lock (serializes concurrent gateways).Nwzutf-8)encoding) fcntl_sync_back_implrLflockLOCK_EXLOCK_UNr%IOErrorclose)rjrlock_fds rrz!FileSyncManager._sync_back_lockedEs =  " y#8  KK /  "  GU]]3 MMOW%    GU]]3W%   MMOsA4B&$BB#"B#&C3($C  C3 CC3CC3c |j td t|j}t j d5}|jt|j tjj|j}|tkDr%tjd|t dddyt j d5}t#j$|j5}|j'|d dddd}|j(t+z}tj,|D]\}} } | D]x} tjj/|| } tjj1| |} d | z}|j2j5|}|t7| }||k(rwd}|j9||}|-|j;||| }|tj=d ||j?||rtj=d|tjjA|r(|&t7|}||k7rtjd|tjBtjjE|dtGjH| ||dz }{|rtjKd|ntj=dddddddy#t$rg}YwxYw#t$rd}YwxYw#1swY$xYw#1swYCxYw#1swYyxYw)z1Download, diff, and apply remote changes to host.Nz/_sync_back_impl called without bulk_download_fnz.tar)suffixruBsync_back: remote tar is %d bytes (cap %d) — skipping extractionzhermes-sync-back-)prefixdata)filter/upload_only_host_pathsz(sync_back: skipping %s (no host mapping)z2sync_back: skipping upload-only credential file %suwsync_back: conflict on %s — host modified since push, remote also changed. Applying remote version (last-write-wins).T)rrz%sync_back: applied %d changed file(s)z%sync_back: no remote changes detected)&rc RuntimeErrorlistr`rtempfileNamedTemporaryFilernamernrCgetsizer%_SYNC_BACK_MAX_BYTESrrrvTemporaryDirectorytarfilerL extractallrgr(walkr6relpathrfr rR_resolve_host_path_infer_host_pathrs_is_upload_only_host_pathexistsmakedirsr@shutilcopy2info)rj file_mappingtftar_sizestagingtarappliedrdirpath _dirnames filenamesfname staged_filerelr| pushed_hash remote_hashr host_hashs rrzFileSyncManager._sync_back_implVs&  ! ! )PQ Q  2 2 45L ( ( 7O J2  " "4= 1 77??2773..X2O JO J ,,4GH? JG\\"''*;cNN76N:;003I3KK'68WWW5E2%1GY !*1%&(ggll7E&B  ggook7C&)Ci &*&9&9&=&=k&J '2*6{*CK*k9 (*.K%)$;$;K$V $,(,(=(= + ,7M)>)I )0 & $N$/!"!)99)E[\"LL T +%77>>)49P(4Y(?I(K7 &%H%0 !" BGGOOI$>N [)<1 c1%2%hKK GQLL!HI? J!O JO J L   ;;? J? J!O JO JsxL %M .)L%M M  L>;L1G-L><M  LL L.*M -L..M 1L; 6L>>M M  Mr|rc6||ng}|D]\}}||k(s |cSy)zAFind the host path for a known remote path from the file mapping.N)rjr|rmappinghostrs rrz"FileSyncManager._resolve_host_paths5#/":,# LD&$  rrrc0||ng}|xs t}|D]}\}}|j||rtt|j}|j |dzsLtt|j}|t |d} || zcSy)uInfer a host path for a new remote file by matching path prefixes. Uses the existing file mapping to find a remote->host directory pair, then applies the same prefix substitution to the new file. For example, if the mapping has ``/root/.hermes/skills/a.md`` → ``~/.hermes/skills/a.md``, a new remote file at ``/root/.hermes/skills/b.md`` maps to ``~/.hermes/skills/b.md``. Nr)rrr"rr startswithrt) rjr|rrrrr remote_dirhost_dirrs rrz FileSyncManager._infer_host_paths#/":,!7!@35# )LD&--d4JKT&\001J%%j3&67tDz001$S_%56&(( )rrc tt|jj}||vS#t$r(tt|j}Y||vSwxYwr-)r"rr#r$r%)rrresolveds rrz)FileSyncManager._is_upload_only_host_pathsi 94 ?557??ABH111 94 ?5578H111 9s06+A'&A'r-)r N)__name__ __module__ __qualname____doc___SYNC_INTERVAL_SECONDS GetFilesFnUploadFnDeleteFnfloat BulkUploadFnBulkDownloadFnrkboolrrrrrrr"rtuplerrr staticmethodrrrrrXrXs 6.226, ,, ,  , %t+ ,)4/,(%*OQTOQdOQj&bTD[&bD&bP!7!7$!7F4D"ZJzIMc)-eCHo)>)EQTW[Q[GKDHC'+E#s(O'rs   ,3   8 $ ^^ * S#J$ %eCHo./56 4&$,' T#YK% & b$uS#X// 0 Cd5c?>S:C.EDIE#E @tCy@S@ Fd5c?3FS F ss-S2S2_ EsDDD