`gjJdZddlmZddlZddlZddlZddlmZeje Z ejdZ ddZ ddZddZdd Zddd Zddd Zdd Zdd ZddZddZddZgdZy)atMessage and tool-payload sanitization helpers. Pure functions extracted from ``run_agent.py`` so the AIAgent module can stay focused on the conversation loop. These walk OpenAI-format message lists and structured payloads, repairing or stripping problematic characters that would otherwise crash ``json.dumps`` inside the OpenAI SDK or be rejected by upstream APIs. All helpers are stateless and side-effect-free except for in-place mutation of their input (where documented). Backward-compatible re-exports from ``run_agent`` remain in place so existing imports ``from run_agent import _sanitize_surrogates`` keep working. ) annotationsN)Anyz[\ud800-\udfff]c\tj|rtjd|S|S)zReplace lone surrogate code points with U+FFFD (replacement character). Surrogates are invalid in UTF-8 and will crash ``json.dumps()`` inside the OpenAI SDK. This is a fast no-op when the text contains no surrogates. �) _SURROGATE_REsearchsubtexts M/root/.hermes/venv/lib/python3.12/site-packages/agent/message_sanitization.py_sanitize_surrogatesr s* D!  400 Kc*dfd|S)uuReplace surrogate code points in nested dict/list payloads in-place. Mirror of ``_sanitize_structure_non_ascii`` but for surrogate recovery. Used to scrub nested structured fields (e.g. ``reasoning_details`` — an array of dicts with ``summary``/``text`` strings) that flat per-field checks don't reach. Returns True if any surrogates were replaced. Fc$t|trz|jD]f\}}t|tr2tj |s,tj d|||<dHt|ttfs_|hyt|trut|D]f\}}t|tr2tj |s,tj d|||<dHt|ttfs_|hyy)NrT) isinstancedictitemsstrrrr list enumerate)nodekeyvalueidx_walkfounds r rz-_sanitize_structure_surrogates.._walk4s dD !"jjl ! UeS)$++E2$1$5$5h$FS $d|4%L  !d #'o ! UeS)$++E2$1$5$5h$FS $d|4%L  !$rpayloadrrs @@r _sanitize_structure_surrogatesr *s E!& 'N Lrc*d}|D] }t|ts|jd}t|tr1tj |rtj d||d<d}n{t|trk|D]f}t|ts|jd}t|ts6tj |sLtj d||d<d}h|jd}t|tr0tj |rtj d||d<d}|jd}t|tr2|D],}t|ts|jd} t| tr0tj | rtj d| |d<d}|jd } t| ts| jd} t| tr0tj | rtj d| | d<d}| jd } t| tstj | stj d| | d <d}/|jD]q\} }| d vr t|tr2tj |s1tj d||| <d}Mt|ttfsdt|spd}s |S) aSanitize surrogate characters from all string content in a messages list. Walks message dicts in-place. Returns True if any surrogates were found and replaced, False otherwise. Covers content/text, name, tool call metadata/arguments, AND any additional string or nested structured fields (``reasoning``, ``reasoning_content``, ``reasoning_details``, etc.) so retries don't fail on a non-content field. Byte-level reasoning models (xiaomi/mimo, kimi, glm) can emit lone surrogates in reasoning output that flow through to ``api_messages["reasoning_content"]`` on the next turn and crash json.dumps inside the OpenAI SDK. FcontentrTr name tool_callsidfunction arguments>r#roler"r$) rrgetrrrr rrr )messagesrmsgr"partr r#r$tctc_idfnfn_namefn_argsrrs r _sanitize_messages_surrogatesr2Ks E3!#t$ '')$ gs # (<(Repaired unescaped control chars in tool_call arguments for %sz ,\s*([}\]])z\1{}[]r2Nu8Repaired malformed tool_call arguments for %s: %s → %sPuARepaired control-char-laced tool_call arguments for %s: %s → %suPUnrepairable tool_call arguments for %s — replaced with empty object (was: %s))rrstriploggerwarningjsonloadsdumpsJSONDecodeError TypeError ValueErrorrer countrangeendswithrE) raw_args tool_name raw_strippedparsed reserialisedfixed open_curly open_bracket_escapeds r _repair_tool_call_argumentsrjs(2(C'@8>>#bL CYOvI9U  L7zz&Z@ < ' NNP   )Z 8    E FF>5% 0ES!EKK$44J;;s#ekk#&66LA~ z!!a |## 2Y   JJu  ## ~~c"u{{3'7%++c:J'Jcr $S)9EKK e  JJw  NNS<,gcrl N   )Z 8    NN /<$ sKA B$$CCE''BG43G494H..IIAJJ/.J/c|sy|d}t|tr|jddk7ryt|tr|nd}|j d|j xsddy ) u"Append a synthetic assistant turn when an interrupted tail is a tool result. A turn cut short by ``/stop`` can leave the transcript ending on a raw ``tool`` message (a tool finished, or its execution was cancelled, but the model never streamed a closing assistant turn). Persisting that tail means the next user message lands as ``… tool → user`` — a role-alternation violation that strict providers (Gemini, Claude) react to by hallucinating a continuation of the user's message and ignoring prior context, which reads to the user as "lost context" (#48879). ``finalize_turn`` closes this on the happy interrupt path, but the retry/backoff/error interrupt aborts in ``conversation_loop`` ``return`` early and never reach it — this shared helper closes the sequence on all of them. ``final_response`` is usually empty on an interrupt, so an explicit placeholder is used rather than an empty-content assistant turn. Mutates ``messages`` in place. Returns True if a closing turn was appended. FrQr(toolr: assistantzOperation interrupted.)r(r"T)rrr)rr<rS)r*final_responselastr s r close_interrupted_tool_sequencerpsh&  B"D OO::<;#; rcF|jddjdS)zRemove non-ASCII characters, replacing with closest ASCII equivalent or removing. Used as a last resort when the system encoding is ASCII and can't handle any non-ASCII characters (e.g. LANG=C on Chromebooks). asciiignore)errors)encodedecoder s r _strip_non_asciirw:s" ;;wx; 0 7 7 @@rcd}|D]}t|ts|jd}t|trt |}||k7rj||d<d}nbt|t rR|D]M}t|ts|jd}t|ts6t |}||k7sG||d<d}O|jd}t|trt |}||k7r||d<d}|jd}t|t ru|D]p} t| ts| jdi} t| ts7| jd} t| tsYt | }|| k7sj|| d<d}r|j D]3\} } | d vr t| tst | }|| k7s-||| <d}5|S) aStrip non-ASCII characters from all string content in a messages list. This is a last-resort recovery for systems with ASCII-only encoding (LANG=C, Chromebooks, minimal containers). Returns True if any non-ASCII content was found and sanitized. Fr"Tr r#r$r&r'>r#r(r"r$)rrr)rrwrr)r*rr+r" sanitizedr,r r#r$r-r/r1rrs r _sanitize_messages_non_asciirzCs E/!#t$ '')$ gs #(1IG#!*I  & )dD)88F+D!$,$4T$: $,+4DL$(E )wwv dC (.ID 'F WW\* j$ ' -b$' B/B!"d+"$&&"5%gs3(8(AI(G32;;(, -))+ !JC??%%,U3 %(CH E !Q/!` Lrct|S)z7Strip non-ASCII characters from tool payloads in-place.)_sanitize_structure_non_ascii)toolss r _sanitize_tools_non_asciir~~s ( //rcd}g}t|D]\}}t|ts|jd}t|ts9g}|D]9}t|tr|jddvrd})|j |;t |t |ks|r||d<|jddk(rd|d<|j |t|D]}||=|S) uBRemove image_url content parts from all messages in-place. Called when a server signals it does not support images (e.g. "Only 'text' content type is supported."). Mutates messages so the next API call sends text only. Preserves message alternation invariants: * ``tool``-role messages whose content was entirely images are replaced with a plaintext placeholder, NOT deleted — deleting them would leave the paired ``tool_call_id`` on the prior assistant message unmatched, which providers reject with HTTP 400. * Non-tool messages whose content becomes empty are dropped. In practice this only hits synthetic image-only user messages appended for attachment delivery; real user turns always include text. Returns True if any image parts were removed. Fr"type>image image_url input_imageTr(rlu:[image content removed — server does not support images])rrrr)rr<r;reversed)r*r to_deleterBr+r" new_partsr,s r _strip_images_from_messagesrs$ EIH%$3#t$ '')$'4(   'D$%$((6*:>c*c  &  ' y>CL (!*IF*"^I  #-$.i  QK Lrc*dfd|S)zCStrip non-ASCII characters from nested dict/list payloads in-place.Fct|tra|jD]M\}}t|trt |}||k7s'|||<d/t|tt fsF|Oyt|t r\t |D]M\}}t|trt |}||k7s'|||<d/t|tt fsF|Oyy)NT)rrrrrwrr)rrrryrrrs r rz,_sanitize_structure_non_ascii.._walks dD !"jjl ! UeS) 0 7I E)$-S $d|4%L !d #'o ! UeS) 0 7I E)$-S $d|4%L !$rrrs @@r r|r|s E!* 'N Lr) rrpr r r2rErjrwrzr~rr|)r rreturnr)rrrbool)r*rrr)r?rrr)?)r`rrarrr)N)r*rrnrrr)r}rrr)__doc__ __future__rrVloggingr\typingr getLogger__name__rTcompilerr r r2rErjrprwrzr~rr|__all__rrr rs #     8 $  -. BAH'T^B@A8v0 -`: r