`gjdZddlmZddlZddlmZmZmZeje Z d dZ d dZ d d dZ ddZ dd Zd gZy)u Vision-routing decisions for ``computer_use`` capture results. Background ---------- ``computer_use(action='capture', mode='som'|'vision')`` returns a ``_multimodal`` envelope containing the captured screenshot. That envelope is delivered back to the **active session model** as the tool result. When the active main model has no vision capability (e.g. text-only or text+code-only models), or when the active provider rejects multimodal content inside tool-result messages, the screenshot trips a 404 / 400 at the provider boundary and the agent loop reports a hard tool failure. Issue #24015 reports this regression for the ``cua-driver`` backend: configuring ``auxiliary.vision`` (a dedicated vision-capable model) in ``config.yaml`` was silently ignored — the screenshot was still routed at the *main* model and failed with HTTP 404 ``No endpoints found that support image input`` even though a perfectly good vision backend was sitting in config waiting to be used. This module centralises the small policy decision: should a captured screenshot be returned as multimodal content (main model handles vision natively) or pre-analysed via the auxiliary vision pipeline so the main model only ever sees text? Behaviour (mirrors ``vision_analyze`` for consistency) ------------------------------------------------------ * If the user explicitly configured ``auxiliary.vision`` (any of ``provider``, ``model``, or ``base_url`` non-empty / not ``"auto"``), the screenshot is routed through the aux vision pipeline. Users who pay for a dedicated vision model usually want it used. * Otherwise, if the user explicitly declared the active model vision-capable via ``model.supports_vision`` / provider model config, return ``False``. This is the escape hatch for custom/local OpenAI-compatible VLM routes that are absent from models.dev and provider allowlists. * Otherwise, if the active main model+provider can carry an image inside a tool-result message AND the model reports ``supports_vision=True`` in models.dev metadata, return ``False`` (use the multimodal path). * In every other case (non-vision main model, provider that does not accept multimodal tool results, lookup failure), route through aux vision so the main model receives a text description it can act on. The decision intentionally fails *closed* (i.e. towards aux routing) when metadata is missing or ambiguous: returning a screenshot to a model that cannot read it is a hard tool failure, while routing it through aux costs one extra LLM call and yields a usable description. ) annotationsN)AnyDictOptionalct|tsy|jdxsi}t|tsy|jdxsi}t|tsyt|jdxsdj j }t|jdxsdj }t|jdxsdj }|dvr|s|syy ) arTrue when ``auxiliary.vision`` carries a non-default user override. Mirrors ``agent.image_routing._explicit_aux_vision_override`` so the capture path and the user-attached-image path agree on what counts as an explicit user request for the aux vision pipeline. ``provider: "auto"``, blank values, or a missing block all count as *not* explicit. F auxiliaryvisionprovidermodelbase_url)r autoT) isinstancedictgetstrstriplower)cfgauxr r r r s T/root/.hermes/venv/lib/python3.12/site-packages/tools/computer_use/vision_routing.py_explicit_aux_vision_overrider8s c4  ''+  $"C c4  WWX  $"F fd #6::j)/R0668>>@H  7#)r * 0 0 2E6::j)/R0668H<h c ddlm} ||||S#t$r }tj d|Yd}~yd}~wwxYw#t$r }tj d|Yd}~yd}~wwxYw)z@Return config-declared ``supports_vision`` for the active route.r)_supports_vision_overridezEcomputer_use vision_routing: config override lookup import failed: %sNz>computer_use vision_routing: config override lookup failed: %s)agent.image_routingr Exceptionloggerdebug)r r rrexcs r%_lookup_user_declared_supports_visionr!Sso A(h>>  S     L   s% ? <7< A(A##A(c`|r|sy ddlm}| ||||S ddlm}|||}|ytt|ddS#t$rd}YCwxYw#t$r"}tj d|||Yd}~yd}~wwxYw#t$r"}tj d|||Yd}~yd}~wwxYw) zEReturn config/models.dev ``supports_vision`` for *(provider, model)*.Nr)_lookup_supports_visionuNcomputer_use vision_routing: image-routing caps lookup failed for %s:%s — %s)get_model_capabilitiesu@computer_use vision_routing: caps lookup failed for %s:%s — %ssupports_visionF) rr#rrragent.models_devr$boolgetattr)r r r_lookup_image_supportsr r$capss rr#r#ks 5&Y) )(E3? ?;%h6 | /7 88/ &!%&   LL#%      N eS  s?A AB AA A?A::A? B- B((B-c|sy ddlm}t |||S#t$r }tj d|Yd}~yd}~wwxYw)aZReturn whether *provider*+*model* carries images inside tool-result messages. Reuses ``tools.vision_tools._supports_media_in_tool_results`` so the capture-routing decision stays in lockstep with the ``vision_analyze`` native fast path. Returns None on import failure so callers fall back to aux routing rather than guessing. Nr)_supports_media_in_tool_resultszBcomputer_use vision_routing: tool-result support lookup failed: %s)tools.vision_toolsr,rrrr')r r r,r s r(_provider_accepts_multimodal_tool_resultr.sP F /%@ AA  P   s AAAct|ryt|||}|dury|duryt||}||duryt|||}|duryy)avReturn True iff the captured screenshot should be pre-analysed via aux vision. Args: provider: active inference provider id (e.g. ``"openrouter"``, ``"anthropic"``, ``"openai-codex"``). Lower-case canonical id. model: active main model slug as it would be sent to the provider. cfg: loaded ``config.yaml`` dict (or None). Returns: ``True`` when the caller should hand the screenshot to the aux vision pipeline (and surface a text-only tool result). ``False`` when the caller should keep the existing multimodal envelope (main model handles vision natively). TF)rr!r.r#)r r r user_declaredaccepts_tool_imager%s r"should_route_capture_to_aux_visionr2sm&%S)9(E3OMA(ER!%75%@-hsCO$ rr2)rOptional[Dict[str, Any]]returnr')r rr rrr3r4Optional[bool])N)r rr rr4r5)r rr rrr3r4r')__doc__ __future__rloggingtypingrrr getLogger__name__rrr!r#r.r2__all__rrr>s-^#&&   8 $6  " 6%)!9!9 !9 "!9 !9HB*## # "# #N) r