`gj|O zUdZddlZddlZddlZddlZddlZddlZddlmZm Z ddl m Z m Z m Z mZmZmZmZddlmZej(eZdedefdZehd Zdede efd Zdedefd Zed d hZeej>dej>dej>dej>dej>dej>dej>dej>dej>dh Z ejBdejBdfZ"edhZ#ejBdZ$da%da&ee'd<defdZ(d)dZ)dejTejVzdefd Z,dedefd!Z-d"ed#edefd$Z.dedefd%Z/dedefd&Z0d'ede efd(Z1y)*uLURL safety checks — blocks requests to private/internal network addresses. Prevents SSRF (Server-Side Request Forgery) where a malicious prompt or skill could trick the agent into fetching internal resources like cloud metadata endpoints (169.254.169.254), localhost services, or private network hosts. The check can be globally disabled via ``security.allow_private_urls: true`` in config.yaml for environments where DNS resolves external domains to private/benchmark-range IPs (OpenWrt routers, corporate proxies, VPNs that use 198.18.0.0/15 or 100.64.0.0/10). Even when disabled, cloud metadata hostnames (metadata.google.internal, 169.254.169.254) are **always** blocked — those are never legitimate agent targets. Limitations (documented, not fixable at pre-flight level): - DNS rebinding (TOCTOU): an attacker-controlled DNS server with TTL=0 can return a public IP for the check, then a private IP for the actual connection. Fixing this requires connection-level validation (e.g. Python's Champion library or an egress proxy like Stripe's Smokescreen). - Redirect-based bypass is mitigated by httpx event hooks that re-validate each redirect target in vision_tools, gateway platform adapters, and media cache helpers. Web tools use third-party SDKs (Firecrawl/Tavily) where redirect handling is on their servers. N)AnyOptional) parse_qslquoteunquoteurljoinurlparseurlsplit urlunsplit)is_truthy_valueurlreturncxt|ts|S|j}|s|Stjdd|} t |}|jjdvr|S|j}|j}|r9 |jdjd}||k7r|j||d}t|j d}t|j"d }t|j$d }t'|j||||fS#t $r|cYSwxYw#t$r|}YwxYw) uReturn an ASCII-safe HTTP URL for Hermes-owned URL tools. Browsers and HTTP clients expect URIs, but users and models often provide IRIs such as ``https://wttr.in/Köln``. Preserve URL syntax and existing percent escapes while encoding non-ASCII host/path/query/fragment text. This is intentionally for URL tool inputs only; arbitrary shell commands must not be rewritten. z ^([A-Za-z][A-Za-z0-9+.-]*://)\s+z\1>httphttpsidnaasciiz/%:@!$&'()*+,;=)safez/%:@!$&'()*+,;=?) isinstancestrstripresubr ValueErrorschemelowernetlochostnameencodedecode UnicodeErrorreplacerpathqueryfragmentr ) r rawparsedrr ascii_hostr$r%r&s C/root/.hermes/venv/lib/python3.12/site-packages/tools/url_safety.pynormalize_url_for_requestr+(s- c3  ))+C   &&4eS AC#}}$55 ]]FH "!077@J  !^^Hj!H v}}fdE8D EE)   "!J "s# D D+ D('D(+ D98D9>x-amz-signaturex-amz-security-tokenjwttokenapikeypasswdsecretapi_keypassword signature auth_token credential session_id credentials access_token authorization client_secretawsaccesskeyidx_amz_signaturex_amz_security_tokenc\t|trd|vry t|j}|j j dvs |jsyt|jdD]+\}}|s t|j tvs)|cSy#t$rYywxYw)alReturn the first sensitive query parameter name in ``url``, if any. Used before handing URLs to third-party fetch/browser backends. Prefix-based token redaction catches known credential shapes; this catches opaque magic links, OAuth codes, signed URL signatures, and custom ``?token=...`` values that do not have a recognizable vendor prefix. ?N>rrT)keep_blank_values) rrr rrrrr%rr_SENSITIVE_QUERY_PARAM_NAMES)r r(keyvalues r*sensitive_query_param_namerFws c3 3c>#))+&}}$55V\\ E U WS\'')-IIJ  sB B+*B+ct|duS)zHReturn True when ``url`` carries likely credential-bearing query params.N)rFr s r*has_sensitive_query_paramsrIs %c *$ 66zmetadata.google.internalz metadata.googz169.254.169.254z 169.254.170.2z169.254.169.253z fd00:ec2::254z100.100.100.200z::ffff:169.254.169.254z::ffff:169.254.170.2z::ffff:169.254.169.253z::ffff:100.100.100.200z169.254.0.0/16z::ffff:169.254.0.0/112zmultimedia.nt.qq.com.cnz 100.64.0.0/10F_cached_allow_privatectrtSdadatjddj j }|dvrdatS|dvrtS ddlm}|}|jd i}t|tr$t|jd d rdatS|jd i}t|tr$t|jd d rdatStS#t$rYtSwxYw) acReturn True when the user has opted out of private-IP blocking. Checks (in priority order): 1. ``HERMES_ALLOW_PRIVATE_URLS`` env var (``true``/``1``/``yes``) 2. ``security.allow_private_urls`` in config.yaml 3. ``browser.allow_private_urls`` in config.yaml (legacy / backward compat) Result is cached for the process lifetime. TFHERMES_ALLOW_PRIVATE_URLS>1yestrue>0nofalser)read_raw_configsecurityallow_private_urls)defaultbrowser) _allow_private_resolvedrKosgetenvrrhermes_cli.configrUgetrdictr Exception)env_valrUcfgsecrYs r*_global_allow_private_urlsrds$$"!ii3R8>>@FFHG&& $$$&&$$ 5ggj"% c4 _ GG( )5& %) !( ('')R( gt $ KK, -u* %) !( ( !     sAC9-AC99 D  D c daday)u+Reset the cached toggle — only for tests.FN)rZrKrJr*_reset_allow_private_cachergs$!rJipct|tjrt|jh|j}|jxsN|j xs@|j xs2|jxs$|jxs|jxs|tvS|js$|j s|j s |jry|js |jry|tvryy)zz(is_always_blocked_url..Js0!c 0zDBlocked request to cloud metadata address (always-blocked floor): %sr%u>Unparseable IP address %r for hostname %s — skipping addressc3&K|]}|v ywryrf)r{r|resolveds r*r}z(is_always_blocked_url..gs6$'C6r~zJBlocked request to cloud metadata address (always-blocked floor): %s -> %sz&is_always_blocked_url error for %s: %s)r rrrrstrip_BLOCKED_HOSTNAMESloggerwarningrj ip_addressr_ALWAYS_BLOCKED_IPSany_ALWAYS_BLOCKED_NETWORKSsocket getaddrinfo AF_UNSPEC SOCK_STREAMgaierrorsplitr`debug) r r(r addr_info_family_sockaddrip_strexcrhrs @@r*is_always_blocked_urlrs:C#OO)r00288:AA#F ) ) NNQ  %%h/B >((C0%=0-1  **$ 0 0&2D2DI +4  &GQ1ha[Ff}c*1- $//7..#6+C637  % (Y B *   _aijkalnvw    =sCH sAF> F>-E+8F>=4E<1(F>F/ F>F>)F>+ E96F>8E99F><FF>FF>#F;8F>:F;;F>> G(G##G(rrc |dk(xr|tvS)zGReturn True when a trusted HTTPS hostname may bypass IP-class blocking.r)_TRUSTED_PRIVATE_IP_HOSTS)rrs r*_allows_private_ip_resolutionr{s W  F-F!FFrJc t|}|jxsdjjj d}|j xsdjj}|dvrt jd|xsdy|sy|tvrt jd|yt}t||} tj|dtjtj}|D]\}}}}} | d } d | vr| j!d d } t#j$|  t(vst+ fd t,Drt jd|| y|rx|r{t/ st jd|| y|rt j1d|y|rt j1d|y#tj$rt jd |YywxYw#t&$rt jd | d |YywxYw#t2$r!} t jd|| Yd} ~ yd} ~ wwxYw)uReturn True if the URL target is not a private/internal address. Resolves the hostname to an IP and checks against private ranges. Fails closed: DNS errors and unexpected exceptions block the request. When ``security.allow_private_urls`` is enabled (or the env var ``HERMES_ALLOW_PRIVATE_URLS=true``), private-IP blocking is skipped. Cloud metadata endpoints (169.254.169.254, metadata.google.internal) remain blocked regardless — they are never legitimate agent targets. rNrw>rru.Blocked request — unsupported URL scheme: %szFz(Blocked request to internal hostname: %sNu1Blocked request — DNS resolution failed for: %srru=Blocked request — unparseable IP address %r for hostname %sc3&K|]}|v ywryrfrzs r*r}zis_safe_url..s/^cc /^r~z3Blocked request to cloud metadata address: %s -> %sz5Blocked request to private/internal address: %s -> %szKAllowing private/internal resolution (security.allow_private_urls=true): %szAAllowing trusted hostname despite private/internal resolution: %sTu5Blocked request — URL safety check error for %s: %s)r rrrrrrrrrdrrrrrrrrjrrrrrrurr`) r r(rrallow_all_privateallow_private_iprfamilyrrrrrhs @r* is_safe_urlrsFH#OO)r00288:AA#F--%2,,.446 * * NNKVM`W` a ) ) NNEx P7886J **8T6;K;KVM_M_`I*3  %FAq!Xa[Ff}c*1- ))&1((C/^E]/^,^If$-=.QSBTKf1 4  LL]   LLS  W  NNNPX Y   ^`hij`kmuv B  NPSUXY sBH/H/H/6H/ 4G(H/*H?8H/8H/;H/> H/ H/#H/=H/)H?H/HH/#H,(H/+H,,H// I8IIcRKtjt|d{S7w)zSame rules as :func:`is_safe_url`, but run the DNS work off the event loop. ``socket.getaddrinfo`` can block; call this from async code paths (gateway, ``web_extract_tool``, vision download hooks) instead of ``is_safe_url``. N)asyncio to_threadrrHs r*async_is_safe_urlrs! "";4 44 4s '%'responsec t|ddsyt|dixsi}|jd}|r)ttt|ddt|St|dd}|rt|jSy) aReturn the redirect target visible from inside an httpx response hook. In ``httpx.AsyncClient`` response event hooks, ``response.next_request`` is frequently ``None`` even for a genuine redirect (it is populated later by the redirect-following machinery). Relying on ``next_request`` alone means an SSRF redirect guard silently never fires: a public URL that 302s to ``http://169.254.169.254/`` gets followed anyway. The ``Location`` header, however, is already present on the response, so resolve the target from it first (handling relative Locations via ``urljoin``) and only fall back to ``next_request`` when no ``Location`` header is set. is_redirectFNheaderslocationr rN next_request)getattrr^rrr )rrrrs r*redirect_target_from_responsers{ 8]E 2h 2.4"G{{:&Hs78UB78#h-HH8^T:L<##$$ rJ)rN)2__doc__rjloggingr[rrrtypingrr urllib.parserrrrr r r utilsr getLogger__name__rrr+ frozensetrCrFboolrIrrr ip_networkrrrsrZrK__annotations__rdrg IPv4AddressrkrurrrrrrfrJr*rsg2  [[[!   8 $-F3-F3-Fn )* 0CHSM,7C7D7  I*+I)I*+I)I*+I12I/0I12I12 ! I)*I12&'&%%o6  #t#0!D0!f"y,,y/D/DD,`s`t`FGCGGG SSSTSl555CHSMrJ