`gj fdZddlmZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlZddlmZmZmZmZmZddlmZmZmZmZej6eZdd d+dZe j>jAdd Z!d gZ"hd Z#d Z$d Z%d,dZ&d-d.dZ'dd d/dZ(e jRde jTZ+d,dZ,d,dZ-ddd0dZ.d1dZ/da0d2dZ1d3dZ2d4dZ3d5dZ4d6dZ5d7dZ6d8d Z7Gd!d"Z8Gd#d$Z9d9d%Z:d:d&Z;d;d'Zy)=uvCua-driver backend (macOS, Windows, Linux). Speaks MCP over stdio to `cua-driver`. The Python `mcp` SDK is async, so we run a dedicated asyncio event loop on a background thread and marshal sync calls through it. The same `cua-driver call ` surface (click, type_text, hotkey, drag, scroll, screenshot, launch_app, list_apps, list_windows, get_window_state, move_cursor, wait) works identically across macOS, Windows, and Linux — cua-driver's PARITY matrix marks the action tools VERIFIED on macOS and Windows in the cross-platform Rust port (`cua-driver-rs`). Linux is the most recent runtime (X11 today, Wayland via XWayland; pure- Wayland progress tracked upstream). It is enabled in `check_computer_use_requirements` alongside macOS and Windows. The plumbing in this file is OS-agnostic; per-host gaps (no DISPLAY, missing AT-SPI, etc.) surface as specific blocked checks via `hermes computer-use doctor` rather than failing silently. Install: - **macOS**: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh)" - **Windows** (PowerShell): irm https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.ps1 | iex After install, `cua-driver` is on $PATH and supports `cua-driver mcp` (stdio transport) which is what we invoke. The macOS path uses private SkyLight SPIs (SLEventPostToPid, SLPSPostEventRecordTo, _AXObserverAddNotificationAndCheckRemote) that aren't Apple-public and can break on OS updates. The Windows path in cua-driver-rs uses stable Win32 APIs (SendInput + UI Automation) — not subject to the same SPI breakage class. ) annotationsN)AnyDictListOptionalTuple) ActionResult CaptureResultComputerUseBackend UIElementrequested_deliveryc t|tr|nid fd }|d}t|tsd}|d}t|tsd}|d} t| tsd} |d} t| tsd} |d} t| tsd} |dxs|d } t| tsd} t|tr|nd} t |||||| | | | | S) aBuild an ActionResult, lifting cua-driver's structured verdict. All structured fields are additive: a driver that omits ``structuredContent`` (or any individual field) leaves the corresponding ActionResult attribute ``None``, so callers and tests see unchanged behavior on old drivers. See the action response shape in cua-driver's mcp-tool-notes and NousResearch/hermes-agent#67052. cP|vrj|Sj|SNget)keymetascs Q/root/.hermes/venv/lib/python3.12/site-packages/tools/computer_use/cua_backend.py_pickz"_action_result_from.._pickQs& "966#; xx}verifiedNeffect escalationpathdegradedcode reason_code) okactionmessagerrrrrr delivery_moder)rstrreturnr) isinstancedictboolr%r )namer!r#r structuredrrrrrrrrr$rs ` @r_action_result_fromr,>s""*d3B Z H h % 8_F fc "|$J j$ ' =D dC Z H h % = 0E-0D dC +55G*M&SWM    #   rHERMES_CUA_DRIVER_CMDz cua-drivermcp> full screenallscreendesktop fullscreen)progmanworkerwprogram manager shell_traywndtaskbarfinderr2dockCUA_DRIVER_RS_TELEMETRY_ENABLEDc ddlm}|xsi}|jdxsi}t|jdd S#t$rYywxYw)uTrue when Hermes should disable cua-driver telemetry for this user. Reads ``computer_use.cua_telemetry`` from config.yaml. Default is False (telemetry off). Any failure to read config fails SAFE — toward the privacy-preserving default of telemetry disabled. r) load_config computer_use cua_telemetryFT)hermes_cli.configr=rr) Exception)r=cfgcus r_cua_telemetry_disabledrDsT 1m!r WW^ $ *6777 sAA AAcft||ntj}tr d|t<|S)aReturn the environment dict for spawning cua-driver. Starts from ``base_env`` (defaults to ``os.environ``) and, when telemetry is disabled (the default), injects ``CUA_DRIVER_RS_TELEMETRY_ENABLED=0``. When the user has opted in, the var is left untouched so cua-driver uses its own default. Used by every cua-driver spawn site (MCP backend, status, doctor, install) so the policy is applied consistently. 0)r(osenvironrD_CUA_TELEMETRY_ENV_VAR)base_envenvs rcua_driver_child_envrLs/ 8/xRZZ @C &) "# Jrg@timeoutc > ddlm}tj|dgdd|tj|t }|jxsdj}|jdk7s|s|ttfS tj|}t!|t"s|ttfS|j%d}t!|t"s|ttfS|j%d}|j%d }t!|trt'd |Ds|ttfSt!|t(r|s||fS||fS#t $r|ttfcYSwxYw#ttf$r|ttfcYSwxYw) uReturn ``(command, args)`` that spawn cua-driver's stdio MCP server. Surface 8 of NousResearch/hermes-agent#47072: instead of hardcoding ``["mcp"]`` we ask the driver itself via ``cua-driver manifest`` (trycua/cua#1961). The manifest carries a stable ``mcp_invocation`` pointer with both ``command`` and ``args``, so a future cua-driver that renames or relocates the subcommand keeps working without a Hermes patch. Falls back to ``(driver_cmd, ["mcp"])`` for older drivers that don't expose ``manifest``, or any indeterminate failure — the wrapper must not refuse to start just because the discovery hop failed. r_sanitize_subprocess_envmanifestTcapture_outputtextrNstdinrKmcp_invocationargscommandc3<K|]}t|tywr)r'r%).0as r z*_resolve_mcp_invocation..s,NAZ3-?,Ns)tools.environments.localrQ subprocessrunDEVNULLrLrAlist_CUA_DRIVER_ARGSstdoutstrip returncodejsonloads ValueError TypeErrorr'r(rr0r%) driver_cmdrNrQprocoutrR invocationrYrZs r_resolve_mcp_invocationrps~$ 2E~~  $dG$$))=)?@  ;; " # # %C !34 01112::c? h %4 0111./J j$ '4 0111 >>& !DnnY'G dD !,N,N)N4 0111 gs #74 D=- 24 01112  "24 01112s#?EE9E65E69 FFz~^\s*(?:-\s+)?\[(\d+)\]\s+(\w+)(?:\s*=\s*"([^"]*)"|\s+"([^"]*)"|\s+\((?!\d+\))([^)]*)\))?(?:\s+(?:\(\d+\)\s+)?id=([^\s\[\]]+))?c(tjdk(S)Ndarwinsysplatformrr _is_macosrw(s <<8 ##rcFttjtS)zCTrue if `cua-driver` is on $PATH or HERMES_CUA_DRIVER_CMD resolves.)r)shutilwhich_CUA_DRIVER_CMDrvrrcua_driver_binary_availabler|,s  _- ..rg @c  ddlm}tjtddgdd|tj |t }|jxsdj}|sy tj|}t|tr|j!d ry|S#t$rYywxYw#ttf$rYywxYw) aRun ``cua-driver check-update --json`` and return its parsed state. The payload mirrors the ``check_for_update`` MCP tool: ``{current_version, latest_version, update_available, ...}``. Returns ``None`` (callers should stay quiet) when the result is indeterminate: the binary is missing, the driver is too old to support the verb (it predates trycua/cua#1734), the GitHub check failed (an ``error`` field is set), or the output didn't parse. Best-effort; never raises. rrPz check-updatez--jsonTrSNrWerror)r_rQr`rar{rbrLrArerfrhrirjrkr'r(r)rNrQrmrndatas rcua_driver_update_checkr1sE~~ nh 7dG$$))=)?@   ;; " # # %C zz# dD !TXXg%6 K   "s$AB!(B0! B-,B-0CCct}|r|jdsy|jdxsd}|jdxsd}d|d|dS) z~One-line "an update is available" message, or ``None`` when up to date, indeterminate, or the driver is too old to report.update_availableNlatest_version?current_version cua-driver z is available (you have z7); update with `hermes computer-use install --upgrade`.)rr)statelatestcurrents rcua_driver_update_nudgerZsd $ %E  "45 YY' ( /CFii)*1cG fX5gY?? @rFcjtrydadd}tj|ddjy)zEmit an update nudge at most once per process, off-thread so the (cached, ~20h) GitHub poll never blocks the first computer_use action.NTcj t}|rtjd|yy#t$rYywxYw)Nzcomputer_use: %s)rrAloggerinfo)msgs r_runz!_maybe_nudge_update.._runss< )+C  KK*C 0    s & 22zcua-driver-update-check)targetr*daemonr&None)_update_checked threadingThreadstart)rs r_maybe_nudge_updaterks3O13D egrc>tjdk(rd}nd}d|dS)Nwin32za irm https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.ps1 | iexzs /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh)"zxcua-driver is not installed. Install with one of: hermes computer-use install Or run the upstream installer directly: zW Or run `hermes tools` and enable the Computer Use toolset to install it automatically.rs) installers rcua_driver_install_hintrs? ||w 8   3   4 +a arc dg}tj|D]}|jdxs;|jdxs(|jdxs|jdxsd}|jt t |jd|jd|d |S) uParse UIElement list from get_window_state AX tree markdown. Last-resort fallback for cua-driver builds that don't carry the canonical ``structuredContent.elements`` array (see ``_parse_elements_from_structured`` — Surface 2 of #47072 prefers that path). Captures the label whichever form cua-driver used: ``= "value"``, ``"quoted"``, ``(parenthesised)``, or ``id=Label``. Bounds always come back ``(0, 0, 0, 0)`` because the markdown surface doesn't carry them — yet another reason to prefer the structured path; element-index clicks don't need them (the driver resolves the index to a frame internally). rWrrrr)indexrolelabelbounds)_ELEMENT_LINE_REfinditergroupappendr int)markdownelementsmrs r_parse_elements_from_treersH  & &x 0  JaggajJAGGAJJ!''!*J aggaj/     Orc Fg}|D]}t|ts|jd}t|ts7t|jdtr|jdnd}t|jdtr|jdnd}t|jdtr|jdnd}d}|rk t|jdd t|jd d t|jd d t|jd d f}|jd }t|tr|r|nd} |jt||||| |S#t t f$rd}Y^wxYw)uGSurface 2 of NousResearch/hermes-agent#47072: read the canonical ``structuredContent.elements`` array cua-driver-rs emits on every ``get_window_state`` response (trycua/cua#1961). Each entry has at minimum ``element_index``, ``role``, ``label``; ``frame`` (``{x, y, w, h}``) is included whenever the AT-SPI / AXFrame call returned usable bounds. Older code parsed the same information out of the markdown tree via a regex (lossy: bounds were always ``(0, 0, 0, 0)``) — this path preserves the real frame so downstream consumers (e.g. ``UIElement.center()``) work against pixel coordinates instead of just the index lookup. Unknown / malformed entries are skipped rather than failing the whole walk — the wrapper degrades to "fewer elements" rather than "no elements" on a bad row. element_indexrrWrframeNrxrywh element_token)rrrrr) r'r(rrr%rkrjrr ) raw_elementsrrawidxrrrr raw_tokentokens r_parse_elements_from_structuredrsk"!#H #t$ ggo&#s# ",SWWV_c"Bswwv$.swww/?$E 2$.swww/?$F D,8  & #q)* #q)* #q)* #q)* GGO, ' 37I 4    3 @ Oz* &% &sA*F  F F c|jdrNt|dk\r@tj|ddd}tj|ddd}|dkDr |dkDr||fS|jdrd}t|}|d z|kr||d k7r|d z }||d z}|dz }|d vsd |cxkrdkrnn6|dz|kDr ytj|||dzd}|dks||z|kDr y|dvrS|dk\rLtj||dz|dzd}tj||dz|dzd}|dkDr |dkDr||fS y||z }|d z|kry)zCBest-effort PNG/JPEG dimension sniffing without extra dependencies.sPNG  bigrsr r>> rr)rr) startswithlenr from_bytes)rwidthheightinmarker segment_lens r_image_dimensions_from_bytesrs ~~*+CBs2bz51Br E2 19!&=  ~~k"  H!eai1v~QQZF FA%)?4)?1uqy ..Qq1uu=KQ!k/A"5 !# ^^CAa!e,fnaltcmdctrlshiftoptionrZcontrolrrr)rZrrz[+\-]N)rerrflowerrr) keysMODIFIER_NAMES KEY_ALIASESpparts modifiersrpart normalizeds r_parse_key_combors [N#HHK(*4(@ N1AGGIQWWY__  NE NI C __T40  '   Z (C   > Os B Bc2eZdZdZddZddZdd dZddZy) _AsyncBridgezNRuns one asyncio loop on a daemon thread; marshals coroutines from the caller.cRd|_d|_tj|_yr)_loop_threadrEvent_readyselfs r__init__z_AsyncBridge.__init__1s:> 37 oo' rcPjrjjryjjdfd }t j |dd_jj jjds tdy) Nctj_tjjjj  jj  jjy#t$rYywxYw# jjw#t$rYwwxYwxYwr) asyncionew_event_looprset_event_looprset run_forevercloserArsrrz _AsyncBridge.start.._run;s //1DJ  " "4:: . KKOO   &&(JJ$$& JJ$$& sBB0B BBCB76C7 CCCCTzcua-driver-loop)rrr*@rMz)cua-driver asyncio bridge failed to startr) ris_aliverclearrrrwait RuntimeError)rrs` rrz_AsyncBridge.start6s <zzT\\5J5J5L""4( >? ?&tTZZ8 ;>? ?zz'z**rc|jrI|jjr/|jj|jj|jr|jj dd|_d|_y)Ng@rM)r is_runningcall_soon_threadsafestoprjoinrs rr!z_AsyncBridge.stopWs\ ::$**//1 JJ + +DJJOO < << LL  c  *  rNr>@)rNOptional[float]r&r)__name__ __module__ __qualname____doc__rrrar!rvrrrr.sX( L, +rrceZdZdZddZddZddZddZddZddZ ddZ dd Z dd Z dd Z ddd ZddZeddZed dZed!dZed!dZddZd"dZeddhZd#d"dZy )$_CuaDriverSessionuHolds the mcp ClientSession. Spawned lazily; re-entered on drop. Lifecycle ownership: a single long-running coroutine (`_lifecycle_coro`) opens both the stdio_client and ClientSession contexts, populates capabilities, sets `_ready_event`, and then waits on `_shutdown_event`. When shutdown is signalled the same coroutine closes the contexts — keeping anyio's cancel-scope task-identity invariant intact (the bridge schedules each `bridge.run(coro)` as a NEW task, so opening contexts in one and closing them in another raises "Attempted to exit cancel scope in a different task"). Tool calls run in their own short-lived tasks; they only touch the session object, never the surrounding contexts. c||_d|_tj|_d|_i|_d|_tj|_ d|_ d|_ d|_ y)NFrW) _bridge_sessionrLock_lock_started _capabilities_capability_versionr _ready_event_shutdown_event_lifecycle_future _setup_error)rbridges rrz_CuaDriverSession.__init__ss^  ^^%  .0(* %OO-8<!%59rc2|js tdy)Nzcua-driver session not started)r1rrs r_require_startedz"_CuaDriverSession._require_starteds}}?@ @rc Kddl}ddlm}m}ddlm}ddlm}tj|_ |j}d|_ tsttd|_ t!t"\}}|j} ||||t%} || 4d{\} } d |_ || | 4d{} | j'd{|j}d |_ |j)| d{| |_d |_ |j,j/t0j3d |j|z | |z || z |jj5d{dddd{dddd{d|_d |_y77777<7.#1d{7swY>xYw75#1d{7swYExYw#t6$r'}||_|j,j/d}~wwxYw#d|_d |_wxYww)u}Long-lived owner of the stdio MCP contexts. Opens, signals ready, blocks on shutdown, then cleans up. enter + exit happen in the SAME asyncio task, so anyio's cancel-scope invariant holds — fixing the "Attempted to exit cancel scope in a different task than it was entered in" warning emitted by the previous _aenter/_aexit split. rN) ClientSessionStdioServerParameters) stdio_clientrPz binary-checkzmanifest-discovery)rZrYrKzmcp-initializezcapability-discoveryreadyzBcua-driver session ready in %.1fs (manifest=%.1fs, mcp_init=%.1fs)F)timer.r<r=mcp.client.stdior>r_rQr rr5 monotonic_startup_phaser|rrrpr{rL initialize_populate_capabilitiesr.r4rrrr BaseExceptionr7r1)r_timer<r=r>rQ_t0rZrY _t_manifestparamsreadwritesession_t_inites r_lifecycle_coroz!_CuaDriverSession._lifecycle_coros) <1E '}}oo-@ ".0"#:#<== #7D 3ODMGT//+K*--A-CD F$F+ 6 6}e&6#(u566!,,...#oo/G+AD'55g>>>$+DM*1D'%%))+KK;)C/#c)+- ..33555+66 6 6H!DM"DM] 66. ?6+6666 6 6 6 60 !"D     ! ! #   !DM"DMsA I A+H7G8H;G,G G,G+G ,/GGA8GGG G,$G%G,) H4G*5H9IH G, GGGG,G' GG' #G,*H,G>2G5 3G>:H H1 "H,,H11H44IIcK |jd{}t|dgxsgD]}t|dd}t|ts!t|dd}|"t|ddxsi}|j d}t|t r/|Dchc]}t|ts|c}|j |<t|j |<t|dd}|"t|ddxsi}|j d}t|tr||_yy7 cc}w#t$r } tjd| Yd} ~ yd} ~ wwxYww)uSurface 4: cache per-tool capability sets + capability_version from tools/list. Soft prerequisite — discovery failure leaves the map empty and supports_capability degrades to False.Ntoolsr* capabilities model_extracapability_versionz5cua-driver tools/list capability discovery failed: %s) list_toolsgetattrr'r%rrcr2rr3rArdebug) rrM tools_listtool tool_namecapsextraccvrOs rrEz(_CuaDriverSession._populate_capabilitiessQ U&1133J GR8>B :#D&$7 !)S1t^T:<$D->D"E 99^4DdD)#'5:a+=5D&&y158ED&&y1 :(%94@Bz M4@FBYY34"c"+-(#345 U LLPRS T T UsXED)D!A;D)D$)D$-A2D)E!D)$D)) E2E E EEc|j5|jr dddy|jj|j d|_dddy#1swYyxYw)NT)r0r1r-r_start_lifecycle_lockedrs rrz_CuaDriverSession.startsW ZZ !}} ! ! LL    ( ( * DM  ! ! !sA1AA'ctj|_d|_d|_|j j }| tdtj|j||_ |jjds:|jt|dd}ddlm}td |d |d |j#td |j|jy) z^Spawn the lifecycle owner and wait for it to reach ready. Caller must hold self._lock.Nrr$rMrCunknownr)display_hermes_homezEcua-driver session never reached ready (timeout 30s; stuck in phase: z.). Run `hermes computer-use doctor` and check z&/logs/agent.log for the phase timings.z!cua-driver session setup failed: )rrr4r7r5r-rrr run_coroutine_threadsaferPr6r_signal_shutdown_lockedrWhermes_constantsrd)rloopphaserds rraz)_CuaDriverSession._start_lifecycle_lockeds&OO- #||!! <>? ?!(!A!A  "D"   %%d%3  ( ( *D"2I>E <##('*>&())OQ     (3D4E4E3FG$$ % )rc|j5|js dddyd|_|jdddy#1swYyxYw)NF)r0r1_stop_lifecycle_lockedrs rr!z_CuaDriverSession.stop3sG ZZ *== * *"DM  ' ' )  * * *sAAA cL|j|j}|y |jdd|_y#tjj $rt jdY > ? &*D "s3;2B-B/B7B BBB B#c|jj}|j}|1|.|jr |j |j yyyy#t $rYywxYw)z8Set the asyncio shutdown event from the caller's thread.N)r-rr5rr rr)rrhevents rrfz)_CuaDriverSession._signal_shutdown_lockedMsg||!!$$   1doo6G ))%))47H 1    sA A#"A#clK|jj||d{}t|S7wr)r. call_tool_extract_tool_result)rr*rYrs r_call_tool_asyncz"_CuaDriverSession._call_tool_asyncXs/}}..tT::#F++;s 424Nc|&|jj|tvStfd|jj DS)uReturn True when the connected cua-driver advertises the given capability token (trycua/cua#1961 capability vocabulary). When ``tool`` is given, scope the check to that specific tool's advertised capability set. When omitted, return True if ANY tool advertises the capability — useful for "is this feature available anywhere on the driver" probes. Always returns False before the session is started (so consumers on a dead/uninitialised wrapper degrade rather than crash). c3&K|]}|v ywrrv)r\r\ capabilitys rr^z8_CuaDriverSession.supports_capability..ksN$:%N)r2rranyvalues)rryrZs ` rsupports_capabilityz%_CuaDriverSession.supports_capability]sK  !3!3!7!7ce!DD DN$2D2D2K2K2MNNNrc||jvS)umReturn True when ``tools/list`` advertised a tool by this name. Used to route capture(): cua-driver dropped the standalone ``screenshot`` tool and folded full-window PNG capture into ``get_window_state`` (whose own description notes it "Also captures a PNG screenshot of the specified window"). Older drivers that still expose ``screenshot`` keep using it; newer ones fall through to ``get_window_state``. Returns False when discovery hasn't populated the map yet — callers treat that as "unknown" and probe defensively rather than trusting it. )r2)rr*s r _has_toolz_CuaDriverSession._has_toolmst))))rc,t|jS)zTrue once ``tools/list`` populated the per-tool map. When False, ``_has_tool`` answers are not trustworthy (discovery failed or the session hasn't started) and capture() should probe defensively.)r)r2rs rcapabilities_discoveredz)_CuaDriverSession.capabilities_discovered|s D&&''rc|jS)uDriver-advertised capability vocabulary version (empty string when the driver predates the field — older builds had no version).)r3rs rrUz$_CuaDriverSession.capability_versions'''rc|jj}t|jdd}|dvxs/|jdxrd|vxst |t t fS)zHReturn True for MCP/stdio failures that are recoverable by reconnecting.r'rW> EndOfStreamBrokenResourceErrorClosedResourceErroranyioResource) __class__r&rWrr'BrokenPipeErrorEOFError)excr*modules r_is_closed_session_errorz*_CuaDriverSession._is_closed_session_errorsd}}%% b9 Q Q <!!'*AzT/A <#:; rcDt|}d|vxsd|vxs d|vxsd|vS)uReturn True for the cua-driver daemon-proxy EAGAIN congestion error. On macOS the ``cua-driver mcp`` bridge forwards calls to the CuaDriver daemon over a non-blocking unix socket. Heavier ops (notably ``get_window_state``, which walks the AX tree and captures a PNG) can come back as an ``McpError`` carrying ``Resource temporarily unavailable (os error 35)`` — POSIX EAGAIN — when the socket buffer is momentarily full. This is transient by definition: the same call succeeds when retried after a short pause (which is why spaced-out single calls work while rapid/large ones intermittently fail). Detect it by message so we can retry with backoff rather than surfacing an empty 0x0 capture to the model. See the EAGAIN diagnosis in references/catalog-add-troubleshooting (apple-music skill) and the cua-driver daemon-proxy note. z Resource temporarily unavailablez os error 35zdaemon transport errorz daemon proxy)r%)rrs r_is_transient_daemon_errorz,_CuaDriverSession._is_transient_daemon_errorsC"#h .# 5 %# %'3. %$  rc|jr |jd|_i|_d|_|jd|_y#t$r }tj d|Yd}~Qd}~wwxYw)zRecreate the MCP session after the daemon/stdin transport was closed. Caller must hold self._lock (the reconnect-once retry path holds it).z6cua-driver session cleanup before reconnect failed: %sNFrWT)r1rkrArrXr2r3rarrOs r_restart_session_lockedz)_CuaDriverSession._restart_session_lockedsp == Z++- #%  $$&  Z UWXYY ZsA A4A//A4c ddl}ddl}ddl}ddlm}t |}d} |dk(r4d|vr0|j dd\} } tj| | |d<td |tj|g} d } d } d}d } t| D]} |j| d d td||t}|j$xsd j'}|ddxs|j(xsd dd}t+d|j-d|j-dfDd}|dk7r tj.||d}||}n=|| dz kst2j5d||dz| | |j7| | dz} |t#d|d| d|g}d}t9|t r|nd}d}t9|t r|j;dd uxs|j;dd u}|j;d }|s~|j;d!xs| }|rgtj<j?|rH tA|d"5}tCjD|jGjId#}ddd|r|jM||j;d%}|%|j;d&}||d'nd }|r|d(|n|}||||d)| r7tj<j?| r tjN| SSS#t $r}t#d|d||d}~wwxYw#tj0$rd}YwxYw#1swYxYw#t $r!}t2jKd$||Yd}~d}~wwxYw#tP$rYSwxYw#| rFtj<j?| r& tjN| w#tP$rYwwxYwwwxYw)*uFallback transport: invoke ``cua-driver call `` as a subprocess instead of going through the stdio MCP bridge. The ``cua-driver mcp`` stdio bridge can persistently fail to forward heavier calls (notably ``get_window_state``) to the daemon with POSIX EAGAIN, while the plain ``cua-driver call`` path — which talks to the daemon over its own socket — keeps working. When the MCP path gives up, we retry over the CLI and remap the JSON into the same dict shape that ``_extract_tool_result`` produces, so callers (capture(), _action(), list_windows parsing) are transport-agnostic. For ``get_window_state`` we route the screenshot to a temp file via ``screenshot_out_file`` so the daemon returns a tiny JSON body (a path) instead of a multi-megabyte base64 blob — the large payload is what congests the daemon socket and triggers EAGAIN in the first place. We read the PNG back from disk and base64-encode it ourselves. The CLI call is itself retried a few times with backoff, since the underlying daemon socket can still be momentarily busy. rNrPget_window_statescreenshot_out_file cua_shot_z.png)prefixsuffixcallrg?rWTg.@)rTrUrNrKzcua-driver CLI fallback for z failed to spawn: c3,K|] }|dk7s |yw)Nrv)r\rs rr^z7_CuaDriverSession._call_tool_via_cli..sJ1!r'QJ {[r)defaultrzMcua-driver CLI fallback for %s got no JSON (attempt %d/%d); retrying in %.1fsrz returned no JSON after z attempts: FisErroris_errorscreenshot_png_b64screenshot_file_pathrbasciiz.cua-driver CLI fallback: failed reading %s: %s tree_markdown element_countz elementsr)rimagesstructuredContentr))r`tempfiler@r_rQr(mkstemprGrr{rhdumpsrangeramaxrLrArrerfstderrminfindriJSONDecodeErrorrrpsleepr'rrexistsopenbase64 b64encoderKdecoderXrremoveOSError) rr*rYrN _subprocess _tempfilerGrQ call_args shot_filefdrattemptsbackoffparsedlast_errattemptrmrOrnr candidaterrr+rshotfpathfhrecrs r_call_tool_via_cliz$_CuaDriverSession._call_tool_via_clis( )$EJ #' % %*?y*P%--[-PMB HHRL/8I+ ,djj.CDN  ? !j&??DtSwEW45I5KL+D{{(b//1t9A):DS(AJ# >JB;)$(JJs56{$; !,!*X\)NN=gk8W KK(qLG? !B~"24&8Pj H:7 !#FD3=fd3KQUJH&$'"::i0D8ZFJJzK)E!6e!%eT!2Sb'-'7'7 'B'I'I''RSMM$'zz/2#O4B24.I.bG3:gYb/D %/# RWW^^I6IIi(7yG!j&)EdVK]^_]`'abhiij //)$( )JSS(e"LL)Y[`bcdde&RWW^^I6IIi(7ys?N.L9N>NN NN"O8O O OOOO start_session end_sessionc|js4||jvr&tjd||j |j  |j j|j|||S#t$r}|j|r/tjd|||j|||cYd}~S|j|stjd||j5|jdddn #1swYnxYw|j j|j|||cYd}~Sd}~wwxYw)Nz=cua-driver session not active on %s; (re)starting before callrMzIcua-driver MCP transport failed on %s (%s); falling back to CLI transportz:cua-driver MCP session closed during %s; reconnecting once)r1_LIFECYCLE_CALLSrrprr:r-rarvrArrrr0r)rr*rYrNrOs rrtz_CuaDriverSession.call_tool6s-}}T-B-B!B NNOQU  JJL  X<<##D$9$9$$Ew#W W X..q14591..tT7CC003 NNWY] ^ /,,. / / /<<##D$9$9$$Ew#W W XsB,A?? E:EE4E<D ED 3EEE)r8rr&rr)rMrr&rr*r%rYDict[str, Any]r&rr)ryr%rZ Optional[str]r&r))r*r%r&r)r&r)r&r%)rrAr&r))r*r%rYrrNfloatr&rr#)r&r'r(r)rr:rPrErrar!rkrfrvr}rpropertyrrU staticmethodrrrr frozensetrrtrvrrr+r+ds :(AV"p!UF!#%J**& , O *(( ((     0 sp!/=!AB'Xrr+c^d}g}g}t|dddu}t|ddxsd}g}t|dgxsgD]~}t|dd}|dk(r!|jt|dd xsd 6|d k(s, "images": [b64, ...], "image_mime_types": [mime, ...], # parallel to `images`, "" when absent "structuredContent": , "isError": bool, } structuredContent is populated from the MCP result's structuredContent field (MCP spec §2024-11-05+) and takes precedence for structured data like list_windows window arrays. `image_mime_types` is the explicit `mimeType` cua-driver emits on every image part as of trycua/cua#1961 (Surface 7 of NousResearch/hermes-agent#47072). Each entry corresponds index-for-index with `images`; an empty string entry signals the part carried no mimeType (older cua-driver build), and the caller should fall back to base64-prefix sniffing. NrFTrcontenttyperUrWimagermimeTyperc3&K|] }|s| ywrrv)r\ts rr^z'_extract_tool_result..s7Q17s)rr)rrimage_mime_typesrr)rWrr"rfrrhrir) mcp_resultrrrrr+ text_chunksrptypeb64mimejoineds rruru`sP.DF"$z9e4B ? g $-C c"tZ6<" ''- .7k77 )/)B)B:)N4::f%TZD,'  ## D s6DD,+D,c^|jdxsg}|r/|dr*|jdxsg}|r |dr|dnd}|d|fS|jdxsi}|jdxs|jd}|r,|jdxs|jd xsd}||fSy ) uPull a (png_b64, mime_type) pair out of a flattened tool result. cua-driver delivers window screenshots in two shapes depending on tool + transport: * As an MCP ``image`` content part — surfaced by ``_extract_tool_result`` in ``out["images"]`` with a parallel ``image_mime_types`` entry. This is what ``get_window_state`` emits over the stdio MCP transport. * As a base64 field inside ``structuredContent`` — ``screenshot_png_b64`` (+ ``screenshot_mime_type``). This is what ``get_window_state`` returns when its structured payload carries the image instead of a content part (newer driver builds; also the shape seen via the ``cua-driver call`` CLI surface). Checking both makes capture() robust to either delivery shape, so the image never silently drops just because the driver moved it between the content list and structuredContent. Returns ``(None, None)`` when neither location carries an image. rrrNrrpng_b64screenshot_mime_type mime_type)NNr)rnrmimesrr+rs r_image_from_tool_resultrs(WWX  $"F &)*+1r U1XuQx4ay$,-3J ..- . K*..2KC NN1 2 ~~k*  Dy rct|tst|ttfsy t|}|dkDr|SdS#t$rYywxYw)zCReturn a positive integer, rejecting booleans and malformed values.Nr)r'r)rr%rj)valuers r _positive_intrsQ%jc &CUaZ6)T) s = A A c g}|D]}t|jd}t|jd}||<|jd}t|ttfrt|t s|nd}|j |jdd|||jdd |jd d|d |S) u Normalise cua-driver ``list_windows`` entries, dropping unusable ones. Every downstream operation needs both an integer ``pid`` (for get_window_state / action tools) and ``window_id`` (for screenshot / element clicks), so a window missing either is uncapturable. Crucially, on X11 a window's PID comes from the *optional* ``_NET_WM_PID`` property — the desktop root, panels, and override-redirect popups routinely omit it, so the driver reports ``pid: null`` for them. Coercing every entry unconditionally (``int(w["pid"])``) let one such window abort enumeration of the real, targetable windows. We skip the unusable entries instead so capture() and focus_app() still find the windows that matter. ``z_index`` follows CUA Driver semantics: higher = closer to front. Wayland may return ``z_index: null`` (undefined stacking order); we treat null as the lowest priority so real windows still sort above desktop/root windows, and the backmost never ends up selected as the capture target. pid window_idz_indexrapp_namerW is_on_screenTtitlerrr off_screenrr)rrr'rrr)r) raw_windowswindowsrpid_int window_id_intz_rawrs r_ingest_windowsrs*%'G  e -%aeeK&89 ?m3 i %ec5\::eUYCZ%`aj"-&eeND99UU7B'     Nrc 6eZdZdZdCdZdCdZdCdZdDdZdCdZdEdFdZ dGdZ dHd Z dId Z dJ dKd Z dLd Zd d d ddd d dd dMdZd d d d dd d dd dNdZdd d d d d dd dOdZd dd dPdZd dd dQdZdRdSdZdHdZdHdZdTdUdZd d d d dd dVd ZdWd!Zd d" dXd#ZdYd$ZdZd%Zd[d&Zd'd(d)d* d\d+Zd d, d]d-Zd d d d d. d^d/Z d d d d d0 d_d1Z!d d, d`d2Z"dd3 dad4Z#d[d5Z$d[d6Z%dd'd7 dbd8Z&d[d9Z'd[d:Z(dcd;Z)d[d<Z* ddd=Z+dRd>d? ded@Z,dfdAZ-dgdBZ.y )hCuaDriverBackendz@Default computer-use backend. Cross-platform via cua-driver MCP.ct|_t|j|_d|_d|_d|_d|_i|_dtjjdd|_ y)Nzhermes- ) rr-r+r. _active_pid_active_window_id _last_app _last_target_snapshot_tokensuuiduuid4hex _session_idrs rrzCuaDriverBackend.__init__sm#~ )$,,7 *.04(,AE13$#*$**,*:*:3B*?)@ Arc>tddlm}|ddddl}|j |j j  |j jdd|jiy#t$r }tjd|Yd}~yd}~wwxYw) Nr)ensureztool.computer_useF)promptrrMz:cua-driver start_session failed (continuing anonymous): %s) rtools.lazy_depsr importlibinvalidate_cachesr.rrtr rArrX)r _lazy_ensurerrOs rrzCuaDriverBackend.starts ;(7 ##%  Z MM # #OiAQAQ5R S Z LLUWX Y Y Zs (A33 B<BBc|jjr) |jjdd|ji |jj|jjy#t$r }t j d|Yd}~Zd}~wwxYw#|jjwxYw)NrrMz7cua-driver end_session failed (continuing teardown): %s) r.r1rtr rArrXr!r-rs rr!zCuaDriverBackend.stop?s == ! ! [ '' 4CSCS7TU MM   LL     [ VXYZZ [ LL   s#(A6B"6 B?BB"B>c<tjdvrytS)N)rrrlinuxF)rtrur|rs r is_availablezCuaDriverBackend.is_availableOs <<; ;*,,rcJd|_d|_d|_d|_i|_y)zGForget a capture/focus target so a failed lookup cannot misroute input.N)rrrrrrs r_clear_active_targetz%CuaDriverBackend._clear_active_targetXs)!%  "rc H|jt|dddgd|dS)zAReturn an empty capture after disarming any prior target context.rNrW)moderrrrapp window_title png_bytes_len)rr )rrr#s r_failed_capturez CuaDriverBackend._failed_capture`s3 !!#   rcB |jj||}|j ddurP|j d}|jt d|dt |tr |r d|zdz|S#t$r|jwxYw)zKCall a capture-stage tool and disarm state on transport or logical failure.rTrrz failedz: rW)r.rtrArrrr'r%)rr*rYrnr#s r_call_capture_toolz#CuaDriverBackend._call_capture_toolns --))$5C 779  %ggfoG  % % 'dV7+%/%='Ry>S OQS     % % '  s BBc|jdd|jd}|jdxsijdxsg}t|}|j dd|r|St j d |jjdd|jdd }|jd dur't jd |jgS|jdxsijdxsg}t|}|j dd|S#t$r"}t jd |gcYd }~Sd }~wwxYw)uLoad normalized visible windows, with the shared CLI recovery path. Windows are sorted by ``z_index`` **descending**: CUA Driver defines higher values as closer to the front, so the frontmost window ends up at index 0 — which is what ``capture()`` and ``focus_app()`` pick as the default target. ``_ingest_windows`` already normalised null ``z_index`` (Wayland) to 0, so those windows sort to the back. list_windowsT)on_screen_onlyrMrrc |dSNrrvrs rz0CuaDriverBackend._load_windows.. 1Y<r)rreversezScua-driver list_windows returned no windows over MCP; re-fetching via CLI transportg4@z3cua-driver CLI re-fetch for list_windows failed: %sNrz:cua-driver CLI re-fetch for list_windows returned an errorc |dSr&rvr's rr(z0CuaDriverBackend._load_windows..r)r) r!r rrsortrrpr.rrAr~r)rrnrrcli_outrs r _load_windowszCuaDriverBackend._load_windows~sI%% #0@0@ A ww239r>>yIOR !+. / > N ,  mm66#'D4D4DEG ;;y !T ) LLU V  % % 'I{{#67=2BB9MSQS !+. / >  LLNPS TI s*D.. E7EEEc |jjsgS|Dcgc]>}t|jddjjk(r|@}}|r|S |j }t}t}|D] } t| tr| jddur(| jd} t| tst| ttfs` t| } | dkrrd D chc]Q} t| j| x} tr.| jr| jjS}} |vr|j| tfd |Ds|j|  |Dcgc]}|jd|vs|}}|r|S|Dcgc]/}t|jddjvr|1}}|r|S|Dcgc]}|jd|vs|}}|r|S|Dcgc]X}t|jddjs-t|jd djvr|Zc}Scc}w#t $r$}t jd||g}Yd}~@d}~wwxYw#t$rY>wxYwcc} wcc}wcc}wcc}wcc}w) aResolve ``app=`` through exact names before convenience substrings. Linux ``list_windows`` can omit an app name while ``list_apps`` retains name/bundle-ID metadata. Exact direct names and exact metadata aliases must win over substring matches: querying ``Code`` must not silently select ``Visual Studio Code`` merely because it is frontmost. rrWz1computer_use list_apps fallback failed for %r: %sNrunningFrr) bundle_idbundleIdr*r display_namec3&K|]}|v ywrrv)r\alias app_lowers rr^z:CuaDriverBackend._match_windows_for_app..s=EY%'=rzr)rfrr%r list_appsrArrXrr'r(r)rrjaddr{)rrrr direct_exact running_appsr exact_pids partial_pidsraw_appraw_pidrrraliasesmetadata_exactdirect_partialmetadata_partialr6s @r_match_windows_for_appz'CuaDriverBackend._match_windows_for_appsIIK%%' I Cj" 56<<>DDFF      >>+L #u !$ # &Ggt, I0F%0Okk%(G'4( 7S#J0O 'laxY C(88u3?EKKM ##%G G#s#=W==  %+ &.&-Ke 0J!KK ! ! Cj" 56<<>>    ! !'.O!!%%,,2NAOO # #  quuZ,-335Sw!34::<<   w   LLLcSV WL  "   L P s\AJ2J9 K AK)KK4K$K)'K)5AK. K#KK KKNc  ||Q|||j|dSt|}t|}|||j|dS|xsd||ddddg}n$ |j}|s|j|S|k|i|rg|j j tvrCd/d}|D cgc] } || s | } } | s|j|d |d St| d }n2|0|.|r,|j||} | s|j|d |dS| }td|D|d} | d|_ | d|_ i|_ | d} |s |js| |_|j|jd|_d}d}g}dx}}d}|dk(r|j j#dxs|j j$ }d}|r=|j'd|jdd|j(d}t+|\}}|sd}||j'd|j|j|j(d}t+|\}}t-|dt.r|dnd}t1|\}}t3j4d|}|r|j7d}|svt8j;d|j |j j=d|j|j|j(dd}|j?d d!ur|j n|j?d"r |d"d}d#}n|j'd|j|j|j(d}d0d%}||rt8j;d&|j|j |j j=d|j|j|j(dd}|j?d d!ur|j n ||s|}t-|dt.r|dnd}t1|\}}t3j4d(|}|j?d)xsij?d*} t-| tBr| r tE| }n|r tG|ng}|D!cic]%}!|!jHr|!jJ|!jH'c}!|_ t+|\}}t3j4d|}|r|j7d}d}"|r9 tMjN|d+}#tQ|#}"tS|#\}$}%|$r|%r|$}|%}tU|||||| ||"|. S#t$r|j wxYwcc} w#t$r }t8jAd$|Yd}~d}~wwxYw#t$r!}t8jAd'|Yd}~d}~wwxYwcc}!w#t$rtQ|d,zd-z}"YwxYw)1uCapture the frontmost on-screen window or an exact known target. Maps hermes `capture(mode, app)` → cua-driver `list_windows` + `get_window_state` (ax/som) or `screenshot` (vision). Nz3z?rWFrrc|jddd|jddjtfdtDS)NrrW rc3&K|]}|v ywrrv)r\r*haystacks rr^zGCuaDriverBackend.capture.._is_desktop_window..9sN48+Nrz)rrr{_DESKTOP_WINDOW_NAMES)rrHs @r_is_desktop_windowz4CuaDriverBackend.capture.._is_desktop_window7sFeeJ34AaeeGR6H5IJPPRN8MNNNrz'c4tfddDrdSdS)Nc3K|];}|jdddjddjv=yw)rrWrFrN)rr)r\rrs rr^z=CuaDriverBackend.capture....MsJ'AEE*b12!AEE'24F3GHNNPP'sAA)r4r5r6r9r2rr)r{r's`rr(z*CuaDriverBackend.capture..Ms%3'['$ar)rz!c3,K|] }|dr |yw)rNrv)r\rs rr^z+CuaDriverBackend.capture..bsAQ<qArrrrrrvision screenshotjpegU)rformatqualityrMr)rrrMrzAXWindow\s+"([^"]+)"rzbcua-driver vision capture returned no image over MCP (window_id=%s); re-fetching via CLI transportr$rTrz image/pngz8cua-driver CLI re-fetch for vision screenshot failed: %scH|jdry|jdxsi}|jds|jdryt|jdtr|jdnd}t|xsd\}}|xr|j S)NrFrrrrrW)rr'r%rrf)rnsc_txt_trs r _gws_is_emptyz/CuaDriverBackend.capture.._gws_is_emptys778$ gg128b77:&#''2F*G )3CGGFOS)Icggfor(32-288:..rzrcua-driver get_window_state returned an empty result over MCP (pid=%s window_id=%s); re-fetching via CLI transportz7cua-driver CLI re-fetch for get_window_state failed: %sz(\d+)\s+elements?rr)validaterr) rrrrrrrrimage_mime_type)rrr&r))rnrr&r))+rrr.rArrfr_SCREEN_CAPTURE_SENTINELSsortedrCnextrrrrrr.rrr!r rr'r%rrsearchrrrprrr~rcrrrrr b64decoderrr )&rrrrr target_pidtarget_window_idrrJrr2filteredrrrr\rrrruse_screenshotsc_outgws_outrUrXrwtr-cli_excrZrr sc_elementsrOrrdetected_widthdetected_heights& rcapturezCuaDriverBackend.captures, ?i3{i/++O's+J,Y7 !%5%=++[ I2!-# G ,,.++D11 ;9,9J9J9LPi9i O#*CQ-?-BqCGC++A#IIJ  G[Y.3227C@H++;C7C;<  GA'A71:N!%=!' !4!#*% dnn%DN##//  "&)-$& 8  '' 5=}}<<< 04F00 %)%;%;"(#%#'#3#3 ,C6+J("F~11&#//%)%;%;#'#3#3,C7+K(+5WV_c*JwvPR*404YY6=#%88A;LD** "mm>>*#'#3#3)-)?)?'+'7'7 G{{9-5113 X.")("3A"6*5--"++!%!7!7#//G" /W%K$$d&<&< "mm>>*#'#3#3)-)?)?'+'7'7 G{{9-5113*73") '1#&F76?BD,T2MGT .8A#;;':;ArFFzRK+t,:;G>B4T:"%??(%D !(?w'G $G_2D9B!xx{   6&&w? #C 2Ns2S/!o*E,F%'+  g ))+ .DZ!LLRT[p!LLQSZ6%2 6 #G q 0A 5  6sbU(, V:VA>V .A-V7.*W$8W)(V V4V//V47 W!WW!)XXc|r|dk(ry|dk7rtd|dd|dS|jjd | std|d dd Sd|d<|rd|d<y)a8Attach delivery_mode to an input-action args dict. Background is the default and never needs a flag. Foreground is only sent when the driver advertises support for it; on an older driver that lacks the capability we refuse with a structured ``foreground_unsupported`` result instead of silently downgrading to background (which would land the input somewhere the model didn't expect). Returns an ActionResult to short-circuit on refusal, or None to proceed. See NousResearch/hermes-agent#67052 phase B. backgroundN foregroundFbad_delivery_modezunknown delivery_mode u — use background|foreground.)r!r"rr#zinput.delivery_moderZforeground_unsupportedzThis cua-driver build does not support foreground delivery (no `input.delivery_mode` capability). Update cua-driver to escalate to the foreground rung.)r!r"rr$r#r$Tbring_to_front)r r.r})rr"rYr$rts r_apply_deliveryz CuaDriverBackend._apply_deliveryDs"  = L (.A00AA`a  }}00 !1  .F*E  !-_ %)D! "rleftrF)elementrrbutton click_countrr$rtc|j} | tdddS|xsdj} | dvrtddd|dS|d k(rd nd} | | d } |/|jtd| d S|| d <|j| d<nF|6|4|jtd| dS|| d<|| d<|j| d<ntd| dS|r|| d<|j | | ||} | | S|j | | S)NFclick*No active window — call capture() first.r!r"r#rv>rvrightmiddlezunknown button u" — expected left, right, middle.r double_click)rrxz,No active window_id for element_index click.rrz)No active window_id for coordinate click.rrzclick requires element= or x/y.modifier)rr rrru_action)rrwrrrxryrr$rtr button_normrZrYrefusals rr{zCuaDriverBackend.clicknsd ;5(TV V'..0 9 95*9&Ce(fh h!,!1~w'*kB  %%-#uT,Z\\$+D ! $ 6 6D  ]q}%%-#uT,WYYDIDI $ 6 6D 5(IK K (D &&tT=.Q  N||D$''r) from_element to_elementfrom_xyto_xyrxrr$rtc |j} | tdddSd| i} |6|4|jtdddS|| d<|| d<|j| d <n|r|p|jtddd St|d t|d c| d <| d<t|d t|d c| d<| d<|j| d <ntdddS|j d| ||} | | S|j d| S)NFdragr|r}rz+No active window_id for element-based drag.rrrz(No active window_id for coordinate drag.rrfrom_xfrom_yto_xto_yzGdrag requires from_element/to_element or from_coordinate/to_coordinate.)rr rrrur) rrrrrrxrr$rtrrYrs rrzCuaDriverBackend.dragsQ ;5(TV V %s|  # (>%%-#uV,Y[[#/D !+D  $ 6 6D   U%6%%-#uV,VXX-0_c'!*o *DNDN),U1XE!H &DL$v, $ 6 6D 5(qs s&&vt]NS  N||FD))rr)amountrwrrrr$rtc |j} | tdddS| |tdtd|d} |!|j|| d<|j| d <nT|R|P|jtddd S|j j d d r || d <|| d<|j| d <|jd| ||} | | S|jd| S)NFscrollr|r}r2)r directionrrrz*No active window_id for coordinate scroll.zinput.scroll.coordinatesrrrr) rr rrrr.r}rur) rrrrwrrrr$rtrrYrs rrzCuaDriverBackend.scrolls ;5(TV V"!SV_-   4#9#9#E$+D ! $ 6 6D  ]q}%%-#uX,XZZ}}00*1S S $ 6 6D &&x}nU  N||Hd++r)r$rtc|j}|j}||tdddS|||d}|jd|||}||S|j d|S)NF type_textr|r})rrrU)rrr rur)rrUr$rtrrrYrs rrzCuaDriverBackend.type_textsv** ;)+5(TV V'*DQ&&{D-X  N||K..rcf|j}|j}||tdddSt|\}}|stddd|dS|r4||||gzd}|j d|||} | | S|j d|S|||d }|j d |||} | | S|j d |S) NFrr|r}zCould not parse key from ''.)rrrhotkey)rrr press_key)rrr rrur) rrr$rtrrkey_namerrYrs rrzCuaDriverBackend.keys** ;)+5(TV V/t4)5*DTF"(MO O +.Y,5 ,B$DD**8T=.YG"<<$/ /YxHD**;m^\G"<< T2 2rc|j}|j}||tdddS|tdddS||||d}|jd|S)zBSet a value on an element. Handles AXPopUpButton selects natively.F set_valuer|r}z,set_value requires element= (element index).)rrrr)rrr r)rrrwrrrYs rrzCuaDriverBackend.set_value%s{** ;)+5(TV V ?5(VX X"$   ||K..rc |jjdd|ji}|jd}t |t r$t |jdt r|dS|jd}t |t r|St |t r$t |jdt r|dSt |tr{g}|jD]d}tjd|}|s|j|jdjt|jdd f|SgS) Nr7rMrappsrz(.+?)\s+\(pid\s+(\d+)\)rr)r*r)r.rtr rr'r(rcr% splitlinesrr`rrrfr)rrnr+rrliners rr7zCuaDriverBackend.list_apps8smm%%kIt?O?O3PQWW01 j$ 'Jz~~f7Mt,Tf% %wwv dD !K dD !j&1A4&H<  dC D) VII8$?KK)9)9);CPQ O TU VK rc"|jSr)r.rs rr#zCuaDriverBackend.list_windowsNs!!##rc  |j}|j||}|r|dnd}|rr|d|_|d|_i|_|d|_|j|j d|_tddd |dd |jd |j d  S|jtddd|d S#t$r|jwxYw)u`Target an app for subsequent actions without stealing system focus. cua-driver background-automation never needs to bring a window to the front: capture(app=...) already selects the right window via list_windows. We implement focus_app as a pure window-selector — enumerate on-screen windows, find the best match for *app*, and store its pid/window_id so that subsequent click/type calls hit the right process. raise_window=True is intentionally ignored: stealing the user's focus is exactly what this backend is designed to avoid. rNrrrrNT focus_appz Targeted z (pid z , window z) without raising window.r}Fz#No on-screen window found for app 'r) r.rArrCrrrrrr )rr raise_windowrmatchedrs rrzCuaDriverBackend.focus_appQs ((*G --gs; 'D %e}D %+K%8D "$&D !#J/DN''!33!D   #F:$6#7vd>N>N=OP""&"8"8!99RT  !!#u[&I#b$QS S3   % % '  s CC)r1r*urlsadditional_arguments creates_new_application_instancec|s |s tdd|ji}|r||d<|r||d<|rt||d<|rt||d<|rd|d<|jj d |}|d xsd |d iS) uIdempotent launch. Returns ``{pid, bundle_id, name, windows[]}`` so callers can skip an extra ``list_windows`` round-trip before ``get_window_state``. ``creates_new_application_instance=True`` forces a new instance even if the app is already running — use it when concurrent runs may touch the same app so each session gets its own isolated window.z,launch_app requires either bundle_id or namerMr1r*rrTr launch_apprr)rjr rcr.rt)rr1r*rrrrYrns rrzCuaDriverBackend.launch_apps"KL L )4+;+;<  )D  DL :DL +/0D+ED' ( +7;D3 4mm%%lD9&'@FCK+@@rc<|jddt|iS)zYTerminate by pid. Equivalent to ``kill -9`` on POSIX, ``taskkill /F`` on Windows.kill_apprrr)rrs rrzCuaDriverBackend.kill_apps||JC(9::r)rc`dt|i}|t||d<|jd|S)zActivate a window so subsequent foreground-dispatched input lands on it. cua-driver's docstring notes this is the cheaper path than per-call SetForegroundWindow flashes.rrrt)rr)rrrrYs rrtzCuaDriverBackend.bring_to_fronts8 !&s3x0  #ID ||,d33rcP|jdt|t|dS)uHMove the agent-cursor *overlay* to a screen point. This is a visual hint — it does NOT move the real OS pointer (cua-driver explicitly avoids stealing pointer focus). The overlay glides smoothly to the target, so consumers use it before a click to give a visible "where the agent is going" cue. move_cursor)rrr)rrrs rrzCuaDriverBackend.move_cursors" ||MQc!f+EFFrc|jjdd|ji}|jdxsi}t |jddt |jddfS)zPReturn the *real* OS cursor position in screen points (origin top-left).get_cursor_positionrMrrrr)r.rtr rr)rrnrs rrz$CuaDriverBackend.get_cursor_positionsgmm%% !It/?/?#@ WW( ) /R266#q>"CsA$777rc||jjdd|ji}|jdxsiS)zReturn the logical size of the main display in points plus its backing scale factor. Shape: ``{width, height, backing_scale_factor}``.get_screen_sizerMrr.rtr rrrns rrz CuaDriverBackend.get_screen_sizes?mm%%  4+;+;< ww*+1r1rg?rQrR)factorrSrTc |jjdt|t|t|t|t|t||t||jd S)zReturn a JPEG / PNG of a sub-region of a window, optionally scaled. cua-driver supports zoom-to-rect for callers that need a higher-resolution view of a specific element.zoom) rrrrrrrSrTrM)r.rtrrr ) rrrrrrrrSrTs rrzCuaDriverBackend.zooms[ }}&&vYqauQxeAhFmW'' 0   r) cursor_idcNdt|i}|r||d<|jd|S)z:Toggle the agent cursor overlay's visibility for this run.enabledrset_agent_cursor_enabled)r)r)rrrrYs rrz)CuaDriverBackend.set_agent_cursor_enableds1!*4=9  )D ||6==r)glide_msdwell_ms idle_hide_msrci}|t||d<|t||d<|t||d<|r||d<|jd|S)uTune the overlay's motion timings — glide duration, post-click dwell, idle-hide delay. Each None means "leave at current value".rrrrset_agent_cursor_motion)rr)rrrrrrYs rrz(CuaDriverBackend.set_agent_cursor_motionse "  $XD   $XD   ##(#6D  )D ||5t<Eww*+1r1r) record_videoc|jjd|t||jd}|j dxsiS)a;Enable trajectory recording (per-turn screenshots + action JSON) to ``output_dir``. ``record_video=True`` ALSO captures the main display to ``/recording.mp4`` (H.264). Recording ownership is keyed by this run's session id so concurrent runs don't fight over the recorder.start_recording) output_dirrrMr)r.rtr)r r)rrrrns rrz CuaDriverBackend.start_recording sKmm%%&7$ .'':   ww*+1r1rc||jjdd|ji}|jdxsiS)zDisable recording and finalise the mp4 (if video was on). Returns the recorder's final state including ``last_video_path``.stop_recordingrMrrrs rrzCuaDriverBackend.stop_recording, sBmm%%&6 t''9  ww*+1r1rc||jjdd|ji}|jdxsiS)zReturn the current recorder state without changing it. Shape: ``{recording, enabled, output_dir, next_turn, last_video_path, last_error, owner, video_active}``.get_recording_staterMrrrs rrz$CuaDriverBackend.get_recording_state4 s?mm%% !It/?/?#@ ww*+1r1r)dry_run speed_factorc||jjd|t|t||jdS)zReplay a prior recording's turn stream by re-invoking each turn's tool call in lexical order. ``dry_run=True`` logs without actually firing the tools.replay_trajectory)trajectory_dirrrrM)r.rtr)rr )rrrrs rrz"CuaDriverBackend.replay_trajectory= s> }}&&':,G}!,/'' =   rcR|jjdd|jiS)zBootstrap ffmpeg for ``start_recording(record_video=True)`` on Linux / Windows. macOS records natively via ScreenCaptureKit and doesn't need ffmpeg.install_ffmpegrM)r.rtr rs rrzCuaDriverBackend.install_ffmpegJ s+}}&& y$*:*:;  rc||jjdd|ji}|jdxsiS)z-Return the current cua-driver runtime config. get_configrMrrrs rrzCuaDriverBackend.get_configT s=mm%% 9d&6&67 ww*+1r1rc 8|jdt|S)uSet cua-driver config keys. Common keys include ``max_image_dimension`` (image-output resizing), recording flags, etc. Unknown keys are passed through verbatim — cua-driver validates against its own schema. set_config)rr()rconfigs rrzCuaDriverBackend.set_config[ s ||L$v,77rc|jjdd|ji}|jdxsd|diS)a.Return a lightweight snapshot of running regular apps + on-screen visible windows with bounds, z-order, owner pid. Roughly the data ``list_windows`` exposes, in one call. Most callers should prefer ``capture()`` / ``focus_app()`` which already use this shape internally.get_accessibility_treerMrrrrs rrz'CuaDriverBackend.get_accessibility_treed sH mm%% $y$2B2B&C ww*+DF /DDrc t|||jd}|j||jj d|S)azInteract with a browser page loaded in a running app (Chrome, Safari, Edge, ...). cua-driver routes through CDP / Apple Events / AX tree depending on the target. ``action`` + ``page_args`` shape depends on the requested operation (e.g. ``action="eval"`` takes ``js: str``); see cua-driver's ``page`` tool description for the full grammar.)rr"rMpage)rr updater.rt)rrr" page_argsrYs rrzCuaDriverBackend.pageq sCs8''  I}}&&vt44rr$rMc|r t|ni}|jd|j|jj |||S)uCall any cua-driver MCP tool by name with arbitrary args. ``session`` is injected (preserves the caller's explicit one via setdefault). For tools the wrapper doesn't already type- wrap, this is the supported escape hatch — preferred over reaching for ``self._session.call_tool`` directly because it keeps the session-id contract consistent with everything else.rMrM)r( setdefaultr r.rt)rr*rYrNpayloads rrtzCuaDriverBackend.call_tool sC!%$t*"9d&6&67}}&&tWg&FFrc|jd}t|tsy|jj|}|sy|jj d|sy||d<y)aSurface 6: when the wrapper is about to call a token-capable tool with `element_index`, look up the matching `element_token` from the last snapshot and attach it. cua-driver-rs's contract for combined args is documented in trycua/cua#1961: "element_token takes precedence over element_index when both supplied. Returns an explicit 'stale' error if the snapshot has been superseded." Gated on the per-tool capability claim so we don't send the field to drivers that predate the surface (which would reject the schema with `additionalProperties: false`). rNzaccessibility.element_tokensrrr)rr'rrr.r})rrZrYrrs r_maybe_attach_element_tokenz,CuaDriverBackend._maybe_attach_element_token sfhh'#s# %%))#. }}00 *1   %_rc |j|||jd|j |jj ||}|d }|d}|jdxsi}d }t|trt|jd d }nt|tr|}|s+t|trt|jd d }i} t|tr| j|t|tr| j|t|||| ||jd  S#t $r1}t jd|td|d|cYd}~Sd}~wwxYw) NrMzcua-driver %s call failedFzcua-driver error: r}rrrrWr#r$r )rrr r.rtrAr exceptionr rr'r(r%rr,) rr*rYrnrOr!rr+r#rs rrzCuaDriverBackend._action s[ ((t4  4#3#34 Y--))$5CY 6{WW017R  dD !$((9b12G c "G:j$7*..B78G " dD ! KK  j$ ' KK #"4WdJ6:hh6OQ Q) Y   8$ ?5ASTUSV?WX X YsD== E7&E2,E72E7rr)rW)rr%r#r%r&r r)r&List[Dict[str, Any]])rrrr%r&r)somNNN) rr%rrr Optional[int]rrr&r ) r"r%rYrr$rrtr)r&zOptional[ActionResult])rwrrrrrrxr%ryrrOptional[List[str]]r$rrtr)r&r )rrrrrOptional[Tuple[int, int]]rrrxr%rrr$rrtr)r&r )rr%rrrwrrrrrrrr$rrtr)r&r )rUr%r$rrtr)r&r )rr%r$rrtr)r&r r)rr%rwrr&r )F)rr%rr)r&r ) r1rr*rrrrrrr)r&r)rrr&r )rrrrr&r )rrrrr&r )r&Tuple[int, int])r&r)rrrrrrrrrrrrrSr%rTrr&r)rr)rrr&r ) rr%rr%rr%rrr&r ) rrrrrrrrr&r )rrr&r)rr%rr)r&r)rr%rr)rrr&r)r&r )rrr"r%rrr&r)r*r%rYOptional[Dict[str, Any]]rNrr&r)rZr%rYrr&r)r*r%rYrr&r )/r&r'r(r)rrr!rrrr!r.rCrmrur{rrrrrr7r#rrrrtrrrrrrrrrrrrrrrrrrtrrrvrrrrsJ#BLZ@ -#   (TM +M 25M M d!!#' I I I  I ! I  I X (((% (  ( (Z"&)-'+$6(6(  6(  6(  6(6('6(%6(6( 6(v'+$(-1+/)-'+$%*$%*" %* + %* ) %*%*'%*%%*%* %*V!%)-'+$+,+, +,  +,  +, +,'+,%+,+, +,\FJ). /"& /3? /@D#(3 3-93:/&,$)Sh$("$(4816A!A A " A 2 A+/A AB; 374"/4;G4G82!    *-    #1  .=A>,9>EQ>=A<@@D;? =*9=*9=/>=,9 =EQ =&GK<@;?:> <0C<,9<,9<+8 <DP <*;?2*72CQ2.3 2&* 27E 222+003  #'  (-  8F   28 E55"05$ G&* G# G/= G&4 Qrr)r*r%r!r)r#r%rrr+rrrr&r rr)rJzOptional[Dict[str, str]]r&zDict[str, str])rlr%rNrr&zTuple[str, List[str]])rNrr&r)r&rrr)rr%r&List[UIElement])rrr&r)rbytesr&r)rr%r&zTuple[str, str])rr%r&zTuple[Optional[str], List[str]])rrr&r)rnrr&z#tuple[Optional[str], Optional[str]])rrr&r)rrr&r)?r) __future__rr rconcurrent.futuresrmrhloggingrGrryr`rtrr typingrrrrrtools.computer_use.backendr r r r getLoggerr&rr,rHrr{rdr]rIrIrDrLrpcompile MULTILINErrwr|rrrrrrrrrrrr+rurrrrrvrrrs!F#     33   8 $)-< < <<  <  <&<<\**..!8,G7V;& $333 3P2::.LL  $/ 14&R **82j$N2//lyXyXx5p$N*%XTQ)TQr