`gjvbdZddlmZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z m Z mZmZej eZehdZdZdj-d eDZej0d ezd zej2Zej0d ezd zej2Zd dZehdZehdZd!dZ d"dZ d#dZ!d$dZ"d%dZ#d&dZ$ d' d(dZ% d)dZ&d*dZ'ehdZ(d+dZ)d'd,dZ*d-dZ+ d' d.dZ,gdZ-y)/uRouting helpers for inbound user-attached images. Two modes: native — attach images as OpenAI-style ``image_url`` content parts on the user turn. Provider adapters (Anthropic, Gemini, Bedrock, Codex, OpenAI chat.completions) already translate these into their vendor-specific multimodal formats. text — run ``vision_analyze`` on each image up-front and prepend the description to the user's text. The model never sees the pixels; it only sees a lossy text summary. This is the pre-existing behaviour and still the right choice for non-vision models. The decision is made once per message turn by :func:`decide_image_input_mode`. It reads ``agent.image_input_mode`` from config.yaml (``auto`` | ``native`` | ``text``, default ``auto``) and the active model's capability metadata. In ``auto`` mode: - If the active model reports ``supports_vision=True`` (via config override or models.dev metadata), we attach natively — vision-capable main models should always see the original pixels, even when an auxiliary vision backend is configured. That auxiliary backend then acts as a *fallback* for sessions whose main model can't take images. - Otherwise, if the user has explicitly configured ``auxiliary.vision`` (provider/model/base_url not ``auto``/empty), we route through the text pipeline so the auxiliary vision backend can describe the image for the text-only main model. - Otherwise (non-vision model, no explicit override), we fall back to text via the default vision_analyze flow. This keeps ``vision_analyze`` surfaced as a tool in every session — skills and agent flows that chain it (browser screenshots, deeper inspection of URL-referenced images, style-gating loops) keep working. The routing only affects *how user-attached images on the current turn* are presented to the main model. ) annotationsN)Path)AnyDictListOptionalTuple>autotextnative) .png.jpg.jpeg.gif.webp.bmpz.tiffz.tifz.heic|c#>K|]}|jdyw).N)lstrip).0es F/root/.hermes/venv/lib/python3.12/site-packages/agent/image_routing.py r?sAahhsmAsz/(?\"']+?\.(?:z)(?:\?[^\s<>\"']*)?c& t|tr|sggfSg tjd|tjD]1} j |j |jf3tjd|D]1} j |j |jf3d fd }g}t}tj|D]}||j r|jd}tjj|} tjj|sk ||vrq|j!||j |g}t} t"j|D]b}||j r|jd} | j%d} | | vrA| j!| |j | d||fS#t$rY+wxYw)uScan free-form text for image references the model should see. Returns ``(local_paths, urls)``: * ``local_paths`` — absolute (``/``) or home-relative (``~/``) paths whose suffix is an image extension AND whose expanded form exists on disk as a file. Order-preserving, deduplicated. * ``urls`` — ``http(s)://…`` URLs whose path ends in an image extension (a ``?query`` is allowed after the extension). Order-preserving, deduplicated. Matches inside fenced code blocks (``` ``` ```) and inline backticks (`` `…` ``) are skipped so that snippets pasted into a task body for reference aren't mistaken for live attachments. This mirrors the behaviour of ``gateway.platforms.base.BaseAdapter.extract_local_files``. Local paths are validated against the filesystem; URLs are not (the provider fetches them at request time). z```[^\n]*\n.*?```z `[^`\n]+`c.tfdDS)Nc3DK|]\}}|cxkxr|kncywN)rsrposs rrz7extract_image_refs.._in_code..rs7DAq1._in_codeqs7J777rz .,;:!?)]>)r!intreturnbool) isinstancestrrefinditerDOTALLappendstartendset_LOCAL_IMAGE_PATH_REgroupospath expanduserisfileOSErroradd _IMAGE_URL_RErstrip) r mr$ local_paths seen_pathsmatchrawexpandedurls seen_urlsurlr#s @rextract_image_refsrERs( dC 2v )+J [[-tRYY ?01779aeeg./0 [[t ,01779aeeg./08 K5J%..t4% EKKM " kk!n77%%c* 77>>(+, z ! x 8$% D%I''-  EKKM " kk!njj% )   c C   -   s/H HH>1onyestrue>0noofffalsect|tr|St|tr|dvr t|Syt|tr0|j j }|t vry|tvryy)z@Return True/False for recognised boolean values, None otherwise.)rNTF)r)r(r&r*striplower _TRUE_TOKENS _FALSE_TOKENS)r@r s r_coerce_capability_boolrTsf#t #s &=9 #s IIK      r%ct|tsy|jd}t|tr|ni}t|jd}||St |jdxsdj }d}|j dr|tdd}|jd}t|tr|ni} tjtd|||fD]} | j| } t| tr| ni} | jd} t| tr| ni}|j|}t|tr|ni}t|jd}||cS|jd }t|tr$t}td||fD]V} |j| | j dr|j| tddC|jd| X|D]} t| tst | jd xsdj }||vrE| jd} t| tr| ni}|j|}t|tr|ni}t|jd}||cSy) uResolve user-declared vision capability from config.yaml. Resolution order, first hit wins: 1. ``model.supports_vision`` (top-level shortcut for the active model) 2. ``providers..models..supports_vision`` (named custom providers — ``provider`` may be the runtime-resolved value ``"custom"`` and/or the user-declared name under ``model.provider``; both are tried. For ``custom:`` syntax, the stripped ```` is also tried as a provider key.) Returns None when no override is set, so the caller falls through to models.dev. Returns False explicitly only when the user wrote a recognised boolean false token. Nmodelsupports_visionprovidercustom: providersmodelscustom_providersname) r)dictgetrTr*rP startswithlenfromkeysfilterlistr1r9)cfgrXrV model_cfg_raw model_cfgtopconfig_providerstripped_suffix providers_raw providers_cfgp entry_rawentry models_raw models_cfg per_model_raw per_modelcoercedr]candidate_names entry_names r_supports_vision_overriderxs& c4 GGG$M1;M41P VXI !)--0A"B CC  )-- 39r:@@BOO!!),)#i./:GGK(M5? t5TMZ\M ]]6$?O(TU V !%%a( -7 4-H bYYx( 3=j$3OZUW "u- 5? t5TMZ\ ))--8I*JK  N ww12"D) #ux9: 3A    "||I&##Ac)no$67##gaSM2  3 * Ii.Y]]628b9??AJ0"x0J'1*d'CJ&NN51M)3M4)H bI-imm)*5;<BBD2:>D"EKKMSSU R0668>>@ .2DHX2XN c4 GGG$M1;M41P VXI9==,2399;H)-- 39r:@@BO #O D8_5 6/A 779   *   Q 2 3   '! . /GGK(M-&# D!%%d+E%&:.4"5;;=I  ww12"D)&56177966) Ii.Y]]628b9??AJ0Z5E5E5Gw5VY]]:.4"5;;=B   W    @7sBK2 L2 K?>K?c|xsdjj}|dk(ry|sy ddlm}||dk(S#t$rYywxYw)zBTrue when the active provider likely fronts a local Ollama server.rYollamaTFr)detect_local_server_type)rPrQagent.model_metadatarr~)rXr{rnrs r_should_probe_ollama_visionrCsW R &&(AH} A'1X== s> A  A cvt|tsy|jj}|tvr|Sy)z5Normalize a config value into one of the valid modes.r )r)r*rPrQ _VALID_MODES)r@vals r _coerce_moderRs3 c3  ))+   C l r%ct|tsy|jdxsi}t|tsy|jdxsi}t|tsyt|jdxsdj j }t|jdxsdj }t|jdxsdj }|dvr|s|syy ) ucTrue when the user configured a specific auxiliary vision backend. An explicit override means the user has a dedicated vision backend available; it's used as a *fallback* when the main model can't take images natively. In ``auto`` mode, native vision on a vision-capable main model still wins over this fallback — see issue #29135. F auxiliaryvisionrXrYrVr{>rYr T)r)r_r`r*rPrQ)rfauxrrXrVr{s r_explicit_aux_vision_overrider\s c4  ''+  $"C c4  WWX  $"F fd #6::j)/R0668>>@H  7#)r * 0 0 2E6::j)/R0668H<h r%ct|||}||S|r|syd} ddlm}|||}|t |jSt||}|s'|xsdjjdk(rd}t||r ddl m }|||} | | S yy#t$r"}tj d|||Yd}~d}~wwxYw#t$r"}tj d |||Yd}~yd}~wwxYw) aReturn True/False if we can resolve caps, None if unknown. Consults the user's ``supports_vision`` override in config.yaml first (so custom/local models declared as vision-capable don't fall through to text routing in ``auto`` mode), then falls back to models.dev. Nr)get_model_capabilitiesu2image_routing: caps lookup failed for %s:%s — %srYrzhttp://localhost:11434/v1)query_ollama_supports_visionu:image_routing: ollama vision probe failed for %s:%s — %s)rxagent.models_devrr~loggerdebugr(rWrrPrQrrr) rXrVrfoverridecapsrexcr{r ollama_visions r_lookup_supports_visionrws)h>H 5 Da;%h6 D(())*39H R..0668HD."8X6  I8IM($$) - a I8UZ\_``a  LLL     s/BC C!B>>C C1C,,C1cd}t|tr?|jdxsi}t|trt|jd}|dk(ry|dk(ryt |||}|duryt |ryy)a1Return ``"native"`` or ``"text"`` for the given turn. Args: provider: active inference provider ID (e.g. ``"anthropic"``, ``"openrouter"``). model: active model slug as it would be sent to the provider. cfg: loaded config.yaml dict, or None. When None, behaves as auto. r agentimage_input_moder r T)r)r_r`rrr)rXrVrfmode_cfg agent_cfgsupportss rdecide_image_input_modersH#tGGG$* i &#IMM2D$EFH86 'xGIF87aGIF89a image/gif sRIFFsWEBP image/webpsBM image/bmpsftyp>avifavisz image/avif>heicheimheisheixhevchevxmif1msf1z image/heic>II*MM*z image/tiffsz image/x-iconisrrrrc ddlm} ddl}|j  ddl} ddl m }|j||5}|jdvr|jd}|}|j|dd |jcdddS#t$rtj dYywxYw#t$rYwxYw#t$rYwxYw#1swYyxYw#t$r }tj d |Yd}~yd}~wwxYw) aDecode arbitrary image bytes with Pillow and re-encode as PNG. Returns None if Pillow isn't installed or can't decode the input (rare formats, corrupted bytes, missing optional decoder plugin for HEIC/AVIF, or vector formats like SVG). Caller falls back to skipping the image so the rest of the turn still works. HEIC/HEIF and AVIF need optional Pillow plugins; we try to register them on demand and swallow ImportError so a missing plugin just looks like 'Pillow can't decode this' rather than crashing. r)Imagezimage_routing: Pillow not installed; cannot transcode non-standard image format to PNG. Install with `pip install Pillow` (and `pillow-heif` / `pillow-avif-plugin` for those formats).N)BytesIO>LPLARGBRGBArPNGF)formatoptimizez " "-   L           " "  JC  siBB8CC"A C C"B54B58 CC CCCC"C"" D +DD c|t|}|r|Stjt|\}}|r|j dr|S|j j }dddddddj|dS)zReturn image MIME type for *path*. If *raw* bytes are provided, magic-byte sniffing wins (authoritative). Otherwise we fall back to ``mimetypes`` then suffix-based defaults. zimage/rrrrr)rrr rrr)r mimetypes guess_typer*rasuffixrQr`)r5r@sniffedmime_rs r _guess_mimerMs  (- N""3t9-GD! ) [[   F   c&, r%c2 ddlm}|t| |j}t||}|tvrJt|}|tj d||ytjd|j||}d }tj|jd }d |d |S#t$r!}tj d||Yd}~yd}~wt $rYwxYw#t $r!}tj d||Yd}~yd}~wwxYw) uWEncode a local image as a base64 data URL at its native size. Size limits are NOT enforced here — the agent retry loop (``run_agent._try_shrink_image_parts_in_messages``) shrinks on the provider's first rejection. Keeping this simple means providers that accept large images (OpenAI 49 MB+, Gemini 100 MB) don't pay a silent quality tax just because one other provider is stricter. Format compatibility IS handled here: if the sniffed MIME isn't one of ``_UNIVERSALLY_SUPPORTED_MIMES`` (i.e. it's something like AVIF, HEIC, BMP, TIFF, or ICO that some providers reject outright), we transcode to PNG with Pillow before declaring media_type. This fixes the user-visible "Could not process image" HTTP 400 from Anthropic on Discord-attached AVIF/HEIC/BMP files. Returns None if the file can't be read OR if the format isn't universally supported AND Pillow can't transcode it (Pillow missing, HEIC/AVIF plugin missing, vector format like SVG, corrupt bytes). The caller reports those paths in ``skipped`` and the rest of the turn proceeds. r)raise_if_read_blockedz6image_routing: blocked local image attachment %s -- %sNu'image_routing: failed to read %s — %s)r@zimage_routing: %s is %s which is not accepted by all major vision providers and could not be transcoded to PNG; skipping this attachment.zIimage_routing: transcoded %s (%s) -> image/png for provider compatibilityrasciizdata:z;base64,)agent.file_safetyrr* ValueErrorrwarningr~ read_bytesr_UNIVERSALLY_SUPPORTED_MIMESrrr^base64 b64encodedecode)r5rrr@r transcodedb64s r_file_to_data_urlrgs,  ;c$i(oo t %D //&s+   NN,d    W IIt    3  & &w /C 4& &&= OQUWZ[    @$Ls4B5C,5 C)>C C)(C), D5DDc*g}g}g}g}|D]}t|}|jr|js|jt |It |} | s|jt |q|jdd| id|jt ||xsgD]@} | xsdj } | s|jdd| id|j| B|xsdj } |s|rj| xsd} g} | jd|D| jd|D| dd j| z}d |d g}|j|||fSg}| r|jd | d ||fS) uBuild an OpenAI-style ``content`` list for a user turn. Shape: [{"type": "text", "text": "...\n\n[Image attached at: /local/path]"}, {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}, {"type": "image_url", "image_url": {"url": "https://example.com/a.png"}}, ...] Local paths are read from disk and embedded as base64 ``data:`` URLs. Remote URLs (``http(s)://``) are passed through verbatim — the provider fetches them server-side. The model still sees the pixels either way. For each successfully attached image, a hint is appended to the text part: * local path → ``[Image attached at: ]`` * URL → ``[Image attached: ]`` The hint gives the model a string handle so MCP/skill tools that take an image path or URL argument can be invoked on the same image without an extra round-trip. This parallels the text-mode hint produced by ``Runner._enrich_message_with_vision`` (``vision_analyze using image_url: ``) so behaviour is consistent across both image input modes. Images are attached at their native size. If a provider rejects the request because an image is too large (e.g. Anthropic's 5 MB per-image ceiling), the agent's retry loop transparently shrinks and retries once — see ``run_agent._try_shrink_image_parts_in_messages``. Returns (content_parts, skipped). Skipped entries are local paths that couldn't be read from disk; URLs are never skipped (they're not validated here). image_urlrD)typerrYzWhat do you see in this image?c3(K|] }d|d yw)z[Image attached at: ]Nr)rrns rrz-build_native_content_parts..sN!015Nc3(K|] }d|d yw)z[Image attached: rNr)rus rrz-build_native_content_parts..sJq-aS2Jrz  r )rr ) rexistsis_filer.r*rrPextendjoin) user_text image_paths image_urlsskipped image_partsattached_paths attached_urlsraw_pathrndata_urlrDr base_text hint_lines combined_textpartss rbuild_native_content_partsr sLG(*K "N!M - Nxxz NN3x= ) $Q' NN3x= ) *   c(m, -R"yb!    S!" O " " $D<< " N~NNJMJJ$+T*TYYz-BB 06 'N&O [!g~ E  fd34 '>r%)rr rE)r r*r'zTuple[List[str], List[str]])r@rr'Optional[bool])rfOptional[Dict[str, Any]]rXr*rVr*r'r )rfrrXr*r'r*)rXr*r{r*r'r()r@rr'r*)rfrr'r(r)rXr*rVr*rfrr'r )rXr*rVr*rfrr'r*)r@bytesr' Optional[str])r@rr'Optional[bytes])r5rr@rr'r*)r5rr'r)rr*rz List[str]rzOptional[List[str]]r'z&Tuple[List[Dict[str, Any]], List[str]]).__doc__ __future__rrloggingrr4r+pathlibrtypingrrrrr getLogger__name__r frozensetr _IMAGE_EXTSr_IMAGE_EXT_PATTERNcompile IGNORECASEr2r:rErRrSrTrxrrrrrrrrrrrr __all__rr%rrs$L#  33   8 $34  XXA[AA "rzz69KKfTMM !$669OOMM BZ34 56 "Q !QQ Q Qh8 !88 8v <%)** * "* *Z     "  d0| )*  1h 48'|'+VVV$V, Vr r%