`gj 6dZddlZddlZddlZddlZddlZddlZddlmZddl m Z m Z m Z m Z mZddlmZmZddlmZddlmZmZddlZddlmZmZmZmZmZmZmZdd lm Z dd l!m"Z"m#Z#dd l$m%Z%dd l&m'Z'e#Z(ee)jTd z Z+e'e(e+ d&de,de-de e,de e-fdZ.eGddZ/eGddZ0eGddZ1GddZ2 d'de,de,de,de3d e,d!e-d"e3d#e4fd$Z5e6d%k(rejne5yy)(a Trajectory Compressor Post-processes completed agent trajectories to compress them within a target token budget while preserving training signal quality. Compression Strategy: 1. Protect first turns (system, human, first gpt, first tool) 2. Protect last N turns (final actions and conclusions) 3. Compress MIDDLE turns only, starting from 2nd tool response 4. Compress only as much as needed to fit under target 5. Replace compressed region with a single human summary message 6. Keep remaining tool calls intact (model continues working after summary) Usage: # Compress a directory of JSONL files python trajectory_compressor.py --input=data/my_run # Compress a single JSONL file python trajectory_compressor.py --input=data/trajectories.jsonl # Compress 15% sample of a file python trajectory_compressor.py --input=data/trajectories.jsonl --sample_percent=15 # Compress with custom output and token target python trajectory_compressor.py --input=data/trajectories.jsonl --output=compressed.jsonl --target_max_tokens=16000 # Compress 10% sample from a directory python trajectory_compressor.py --input=data/my_run --sample_percent=10 N)Path)ListDictAnyOptionalTuple) dataclassfield)datetime)base_url_host_matchesbase_url_hostname)Progress SpinnerColumn TextColumn BarColumnTaskProgressColumnTimeElapsedColumnTimeRemainingColumn)Console)OPENROUTER_BASE_URLget_hermes_home)jittered_backoff)load_hermes_dotenvz.env) hermes_home project_envmodelrequested_temperaturebase_urlreturnc^ ddlm}m}|||}||ury||S|S#t$r|cYSwxYw)zApply fixed model temperature contracts to direct client calls. Returns ``None`` when the model manages temperature server-side (Kimi); callers must omit the ``temperature`` kwarg entirely in that case. r)_fixed_temperature_for_modelOMIT_TEMPERATUREN)agent.auxiliary_clientr!r" Exception)rrrr!r"fixed_temperatures H/root/.hermes/venv/lib/python3.12/site-packages/trajectory_compressor.py _effective_temperature_for_modelr';sN%Y5UHE,,$    %$$%s  ,,ceZdZUdZdZeed<dZeed<dZ e ed<dZ e ed <dZ eed <dZ eed <dZeed <dZeed <dZe ed<dZeed<eZeed<dZeed<dZeed<dZe ed<dZe ed<dZeed<dZeed<dZeed<dZe ed <d!Ze ed"<dZeed#<dZeed$<d%Z e ed&<dZ!eed'<dZ"eed(<d)Z#eed*<e$d+ed,dfd-Z%y.)/CompressionConfigz)Configuration for trajectory compression.zmoonshotai/Kimi-K2-Thinkingtokenizer_nameTtrust_remote_codei;target_max_tokensisummary_target_tokensprotect_first_systemprotect_first_humanprotect_first_gptprotect_first_toolprotect_last_n_turnszgoogle/gemini-3-flash-previewsummarization_modelrOPENROUTER_API_KEY api_key_envg333333? temperature max_retries retry_delayadd_summary_noticezM Some of your previous tool responses may be summarized to preserve context.summary_notice_text _compressed output_suffix num_workers2max_concurrent_requestsskip_under_targetsave_over_limiti,per_trajectory_timeoutmetrics_enabledmetrics_per_trajectoryzcompression_metrics.jsonmetrics_output_file yaml_pathrct|dd5}tj|xsi}ddd|}dvrH|djd|j|_|djd|j |_d|vrH|djd |j |_|djd |j|_d |vr|d jd |j|_|d jd |j|_ |d jd|j|_ |d jd|j|_ |d jd|j|_ d|vr|djd|j|_ |djdxs |j|_|djd|j|_|djd|j |_|djd|j"|_|djd|j$|_d|vrl|djd|j&|_|djd|j(|_|djd|j*|_d|vr|djd|j,|_|djd|j.|_|djd|j0|_|djd |j2|_d!|vrl|d!jd"|j4|_|d!jd#|j6|_|d!jd$|j8|_|S#1swYxYw)%z"Load configuration from YAML file.rutf-8encodingN tokenizernamer+ compressionr,r-protected_turns first_system first_human first_gpt first_tool last_n_turns summarizationrrr6r7r9r;outputr<r=r? processingr@rBrCrDmetricsenabledper_trajectory output_file)openyaml safe_loadgetr*r+r,r-r.r/r0r1r3r4rr6r7r9r;r<r=r?r@rBrCrDrFrGrH)clsrIfdataconfigs r& from_yamlzCompressionConfig.from_yaml}s)S7 3 +q>>!$*D + $ $($5$9$9&&BWBW$XF !'+K'8'<'<=PRXRjRj'kF $ D '+M':'>'>?RTZTlTl'mF $+/ +>+B+BCZ\b\x\x+yF (  $*./@*A*E*EnV\VqVq*rF ')-.?)@)D)D]TZTnTn)oF &'+,='>'B'B;PVPhPh'iF $(,->(?(C(CLRXRkRk(lF %*./@*A*E*EnV\VqVq*rF ' d ")-o)>)B)B7FLfLf)gF &"?377 CVvFO!%o!6!:!:=&J\J\!]F !%o!6!:!:=&J\J\!]F !%o!6!:!:=&J\J\!]F !%o!6!:!:=&J\J\!]F  t (,X(:(:;OQWQjQj(kF %)-h););#5#5ovG[G[#\F  4 !%l!3!7!7 vGYGY!ZF -1,-?-C-CD]_e_}_}-~F *'+L'9'='=>QSYSkSk'lF $%),%7%;%;'Hc'+K+KKK $#pp&M3&K#%S%"t" OT "%C%!OT #'D'995#5*=55rsr)ceZdZUdZdZeed<dZeed<dZeed<dZ e ed<dZ eed<dZ eed <dZ eed <d Zeed <d Zeed <dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<deeeffdZy)TrajectoryMetricsz,Metrics for a single trajectory compression.roriginal_tokenscompressed_tokens tokens_saved?compression_ratiooriginal_turnscompressed_turns turns_removedturns_compressed_start_idxturns_compressed_end_idxturns_in_compressed_regionFwas_compressedstill_over_limitskipped_under_targetsummarization_api_callssummarization_errorsrcj|j|j|jt|jd|j |j |j|j|j|jd|j|j|j|j|jd S)Nr2) start_idxend_idx turns_count) rvrwrxrzr{r|r}compression_regionrrrrr)rvrwrxroundrzr{r|r}rrrrrrrr)selfs r&to_dictzTrajectoryMetrics.to_dicts#33!%!7!7 --!&t'='=q!A"11 $ 5 5!//!<<88#>># #11 $ 5 5$($=$='+'C'C$($=$=#  rsN)rhrirjrkrvrormrwrxrzrpr{r|r}rrrrrnrrrrrrlrrrrrsr&rurus6OSsL#"u"NCcM3&(($&c&&'' ND "d"!&$&#$S$ !#! c3h rsruczeZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed<dZ eed<dZ eed <dZ eed <dZeed <dZeed <dZeed <dZeed<dZeed<eeZeeed<eeZeeed<eeZeeed<dZeed<dZeed<dZeed<defdZde ee!ffdZ"y)AggregateMetricsz*Aggregate metrics across all trajectories.rtotal_trajectoriestrajectories_compressed!trajectories_skipped_under_targettrajectories_still_over_limittrajectories_failedtotal_tokens_beforetotal_tokens_aftertotal_tokens_savedtotal_turns_beforetotal_turns_aftertotal_turns_removedtotal_summarization_callstotal_summarization_errors)default_factorycompression_ratiostokens_saved_listturns_removed_listprocessing_start_timeprocessing_end_timegprocessing_duration_secondsr[c|xjdz c_|xj|jz c_|xj|jz c_|xj |j z c_|xj|jz c_|xj|jz c_ |xj|jz c_ |xj|jz c_ |xj|j z c_|j"r|xj$dz c_|j&j)|j*|j,j)|j |j.j)|j|j0r|xj2dz c_|j4r|xj6dz c_yy)z,Add a trajectory's metrics to the aggregate.N)rrrvrrwrrxrr{rr|rr}rrrrrrrappendrzrrrrrr)rr[s r&add_trajectory_metricsz'AggregateMetrics.add_trajectory_metricssl 1$   G$;$;;  7#<#<< 7#7#77 7#9#99 '":"::   G$9$99  &&'*I*II& ''7+G+GG'  ! !  ( (A - (  # # * *7+D+D E  " " ) )'*>*> ?  # # * *7+@+@ A  ' '  2 2a 7 2  # #  . .! 3 . $rsrc:|jr+t|jt|jz nd}|jr+t|jt|jz nd}|jr+t|jt|jz nd}|j |j |j|j|jt|j t|j dz dd|j|j|jt|jt|jdz dd|j|j |j"dt|dt|dt|dd |j$|j&td|j&t|j$dz z dd |j(|j*t|j,dd d S) Nryrrr2)rrrrrcompression_rate) total_before total_after total_savedoverall_compression_ratio)rr total_removedr:)avg_compression_ratioavg_tokens_saved_per_compressed avg_turns_removed_per_compressed)total_api_calls total_errors success_rate) start_timeend_timeduration_seconds)summarytokensturnsaveragesrXrZ)rsumlenrrrrrrrrmaxrrrrrrrrrrr)rravg_tokens_savedavg_turns_removeds r&rzAggregateMetrics.to_dicts&& '' (3t/F/F+G G,/  %% && '#d.D.D*E E+,  && '' (3t/F/F+G G,- '+&=&=+/+G+G595[5[151S1S'+'?'?$)$*F*FTMdMdfgIh*hjk$l !% 8 8#66#66-243J3JSQUQiQiklMm3mop-q !% 7 7#55!%!9!9 *//Da)H389I13M49:KQ4O $(#A#A $ ? ? %a4+J+JSQUQoQoqrMs+s&tvw x #88 44$)$*J*JA$N=# # rsN)#rhrirjrkrrormrrrrrrrrrrrrr listrrrprrrrlrrrurrrrrrrsr&rrs4#$S$-.%s.)*!3*    s  %&s&&''',D&AU A#(#>tCy>$)$$?S ?!#3#!!),,4.?401 c3h1 rsrc eZdZdZdefdZdZdZdZde fdZ d e de fd Z d e ee e fde fd Zd e ee e fde e fd Zd e ee e fdeee e ffdZed e ee e fde defdZed e ee e fde de de de f dZd e ee e fde de de fdZedede fdZede de fdZde dede fdZde dede fdZd e ee e fdee ee e feffdZ d e ee e fdee ee e feffdZ!d ee efdeee efeffd!Z"d ee efdeee efeffd"Z#d#e$d$e$fd%Z%d#e$d$e$fd&Z&d'Z'y())TrajectoryCompressora Compresses agent trajectories to fit within a target token budget. Compression strategy: 1. Keep protected head turns (system, human, first gpt+tool) 2. Keep protected tail turns (last N turns) 3. From the compressible middle region, compress only as much as needed 4. Replace compressed turns with a single human summary message 5. Keep remaining middle turns intact (model continues with tools) rfc||_t|_|j|j t j t|_y)zInitialize the compressor.N) rfraggregate_metrics_init_tokenizer_init_summarizerlogging getLoggerrhlogger)rrfs r&__init__zTrajectoryCompressor.__init__XsB !1!3  ''1 rscB ddlm}|j|jj|jj |_td|jjy#t$r*}td|jjd|d}~wwxYw)z4Initialize HuggingFace tokenizer for token counting.r) AutoTokenizer)r+u✅ Loaded tokenizer: zFailed to load tokenizer 'z': N) transformersrfrom_pretrainedrfr*r+rOprintr$ RuntimeError)rres r&rz$TrajectoryCompressor._init_tokenizeres ` 2*:: **"&++"?"?;DN *4;;+E+E*FG H `!;DKK?@@DK $D "'D ii 7 78G"+DKK,C,C+DE,-.. & B *=dkk>R>R*SUDK !%D )0D & 3DKK4S4S3TUV ,T[[-P-P,QRSrscddlm}ddlm}||j||j j |_|jS)a Return an AsyncOpenAI client bound to the current event loop. Created lazily so that each ``asyncio.run()`` call in ``process_directory()`` gets a client tied to its own loop, avoiding "Event loop is closed" errors on repeated calls. r) AsyncOpenAIrr)rrr#rrrfrr)rrrs r&_get_async_clientz&TrajectoryCompressor._get_async_clients@ '>'..()=)=>    rsrc`|jjxsd}t|dryt|dryt|dk(rd|j vryt|d ry t|d st|d s t|d ryt|dryt|dryt|dryy)z6Detect the provider name from the configured base_url.rz openrouter.ai openrouterznousresearch.comnousz chatgpt.comz/backend-api/codexcodexzz.aizaiz moonshot.aiz moonshot.cnz api.kimi.comz kimi-codingzarcee.aiarceez minimaxi.comz minimax-cnz minimax.iominimax)rfrr r lower)rurls r&rz%TrajectoryCompressor._detect_providerskk""(b o 6 &8 9 c "m 3$ 3 f - !#} 5$S-8$S.9 j 1 n 5 l 3rstextc|sy t|jj|S#t$rt|dzcYSwxYw)z4Count tokens in text using the configured tokenizer.rr2)rrOencoder$)rrs r& count_tokensz!TrajectoryCompressor.count_tokenssD "t~~,,T23 3 "t9> ! "s#)AA trajectoryc,tfd|DS)z#Count total tokens in a trajectory.c3`K|]%}j|jdd'yw)valuerNrrb).0turnrs r& z?TrajectoryCompressor.count_trajectory_tokens..s'S4$$TXXgr%:;Ss+.)r)rrs` r&count_trajectory_tokensz,TrajectoryCompressor.count_trajectory_tokenssS SSSrsc j|Dcgc]#}|j|jdd%c}Scc}w)z+Count tokens for each turn in a trajectory.rrr)rrrs r&count_turn_tokensz&TrajectoryCompressor.count_turn_tokenss,EOPT!!$((7B"78PPPs(0cft|}t}dx}x}x}}t|D]@\}} | jdd} | dk(r||}"| dk(r||},| dk(r||}6| dk(s<|?|}B|jj r||j ||jjr||j ||jjr||j ||jjr||j |ttd||jjz |D]}|j ||Dcgc] }||d zks |} }|Dcgc] }||d zk\s |} }| rt| d znd} | r t| n|}|| |fScc}wcc}w) z Find indices of protected turns. Returns: Tuple of (protected_set, compressible_start, compressible_end) Nfromrsystemhumangpttoolrr:r)rset enumeraterbrfr.addr/r0r1rangerr3min)rrn protectedrSrTrUrVirrolehead_protectedtail_protectedcompressible_startcompressible_ends r&_find_protected_indicesz,TrajectoryCompressor._find_protected_indicess  OE ?CB B{BY , GAt88FB'DxL$8 [%8 9#4 J$6   ;; + + 0H MM, ' ;; * *{/F MM+ & ;; ( (Y-B MM) $ ;; ) )j.D MM* %s1a$++"B"BBCQG A MM!   &/=!a1f*!==%.>!qAv+!>>8FS014A2@3~.a,.>>> >>s F)%F)/ F.=F.idxcP|t|k\xs||jddk7S)aReturn True if a region boundary at ``idx`` does not split a turn pair. In the from/value trajectory format a ``tool`` turn (carrying ```` markers) is always emitted immediately after the ``gpt`` turn whose ```` it answers. A compression boundary that lands *on* a ``tool`` turn therefore cuts between a tool call and its response. A boundary is only clean when it sits at the very end of the trajectory or on a non-``tool`` turn. rr)rrb)rrs r&_is_boundary_cleanz'TrajectoryCompressor._is_boundary_clean s,c*o%NC)<)(>z7(S qLG(>(>z7(S  ! !*g 6N )?)? H)U MH )?)? H)Ursstartendc $g}t||D]o}||}|jdd}|jdd}t|dkDr|dddz|d dz}|jd |d |j d |qd j |S)a- Extract content from turns to be summarized. Args: trajectory: Full trajectory start: Start index (inclusive) end: End index (exclusive) Returns: Formatted string of turn contents for summarization runknownrri Niz ...[truncated]... i z[Turn z - z]: z )r rbrrupperjoin) rrrr partsrrrrs r&!_extract_turn_content_for_summaryz6TrajectoryCompressor._extract_turn_content_for_summary4suc" CAa=D88FI.DHHWb)E5zD et '>>tuM LL6!C ~T%A B C{{5!!rscontentc`t|ts|r t|nd}|jS)z0Normalize summary-model output to a safe string.r) isinstancerlstrip)r's r&_coerce_summary_contentz,TrajectoryCompressor._coerce_summary_contentNs''3'&-c'l2G}}rsrcb|xsdj}|jdr|S|sdSd|S)zCNormalize summary text to include the expected prefix exactly once.rz[CONTEXT SUMMARY]:z[CONTEXT SUMMARY]: )r* startswith)rrs r&_ensure_summary_prefixz+TrajectoryCompressor._ensure_summary_prefixUs> 2$$& ??/ 0K+/#Q7J4&5QQrsr[c d|jjd|d}t|jjD]k} |xjdz c_t |jj |jj|jj}t|ddrMddl m }||j|jj d |d g||jjd z }nl|jj d |d g|jjd zd }|||d<|jjjj di|}|j#|j$dj&j(} |j+| cSy#t,$r} |xj.dz c_|j0j3d|dzd| ||jjdz kr9t5j6t9|dz|jj:dnYd} ~ yYd} ~ d} ~ wwxYw)z Generate a summary of the compressed turns using OpenRouter. Args: content: The content to summarize metrics: Metrics object to update Returns: Summary string Summarize the following agent conversation turns concisely. This summary will replace these turns in the conversation history. Write the summary from a neutral perspective describing what the assistant did and learned. Include: 1. What actions the assistant took (tool calls, searches, file operations) 2. Key information or results obtained 3. Any important decisions or findings 4. Relevant data, file names, values, or outputs Keep the summary factual and informative. Target approximately " tokens. --- TURNS TO SUMMARIZE: H --- Write only the summary, starting with "[CONTEXT SUMMARY]:" prefix.rrFr)call_llmuserrr'r:rrmessagesr7 max_tokensrr7r8Nr7Summarization attempt failed: >@ base_delay max_delay[CONTEXT SUMMARY]: [Summary generation failed - previous turns contained tool calls and responses that have been compressed to save context space.]rr)rfr-r r9rr'r4r7rgetattrr#r3rrchat completionscreater+choicesmessager'r.r$rrwarningtimesleeprr;) rr'r[promptattemptsummary_temperaturer3response_create_kwargsrrs r&_generate_summaryz&TrajectoryCompressor._generate_summary]s(@AE @a@a?bc  CF"T[[445' qG& q//14/&FKK33KK++KK(('# 4%8?'!%!3!3"kk==+1f"E!F$7#';;#D#Dq#H  H"&!@!@.4%H$I&*kk&G&G!&K&N +68K}5Bt{{//;;BBT^TH66x7G7G7J7R7R7Z7Z[227;;;' q> q,,1, ##&G?A8G7G8AG>I;GG I8BI3 I#!I3' I;3I88I;ct}t||_|j|}t |}||_||j jkr)d|_||_ t||_ d|_ ||fS|j|\}}}|j||||}||k\r9||_ t||_ ||j jkD|_||fS||j jz }||j jz} d} |} t!||D]} | || z } | dz} | | k\sn| | kr| |kr|} t |||} |j|| ||} | |kr9||_ t||_ ||j jkD|_||fS||_| |_| |z |_|j)||| } |j+| |}g}t!|D]o} || j-}|j/ddk(r5|j j0r|d|j j2z|d<|j5|q|j5d|d t!| t|D]$} |j5|| j-&t||_ |j7||_ |j|jz |_|j |jz |_|jt=|j dz |_ d|_|j|j jkD|_||fS) a| Compress a single trajectory to fit within target token budget. Algorithm: 1. Count total tokens 2. If under target, skip 3. Find compressible region (between protected head and tail) 4. Calculate how many tokens need to be saved 5. Accumulate turns from start of compressible region until savings met 6. Replace accumulated turns with single human summary 7. Keep remaining turns intact Args: trajectory: List of conversation turns Returns: Tuple of (compressed_trajectory, metrics) Tryrrrrrrrr) rurr{rrrvrfr,rrwr|rzrrrr-r rrrr&rOcopyrbr<r=rrr}rxrrrrr[ turn_tokens total_tokensrcompress_start compress_endtokens_to_savetarget_tokens_to_compressaccumulated_tokenscompress_untilrcontent_to_summarizer compressedrs r&compress_trajectoryz(TrajectoryCompressor.compress_trajectorys,$%!$Z,,Z8 ;' ". 4;;88 8+/G ((4G %'*:G $(+G %w& &372N2Nz2Z/ ><,,ZYef \ )(4G %'*:G $'3dkk6S6S'SG $w& && (E(EE %3T[[5V5V$V!'~|4 A +a. 0 UN"%>>    9 9n|>[)N!$[ %M!N  ,,ZYef ^ +(4G %'*:G $'3dkk6S6S'SG $w& &.<*+9(-;n-L* $EE   (()=wG ~& $Aa=%%'Dxx8+ 0N0N $W  0O0O OW   d #  $   ~s:7 4A   jm002 3 4$'z? $($@$@$L! ' 6 69Q9Q Q&669R9RR$+$=$=GD[D[]^@_$_!!%#*#<#<,,ZYef \ )(4G %'*:G $'3dkk6S6S'SG $w& && (E(EE$2T[[5V5V$V!'~|4 A +a. 0 UN!%>>    9 9n|>[)N!$[ %M!N  ,,ZYef ^ +(4G %'*:G $'3dkk6S6S'SG $w& &.<*+9(-;n-L* $EE   445I7SS ~& $Aa=%%'Dxx8+ 0N0N $W  0O0O OW   d #  $   ~s:7 4A   jm002 3 4$'z? $($@$@$L! ' 6 69Q9Q Q&669R9RR$+$=$=GD[D[]^@_$_!!%#*#<#NB/N0N1FNentrycKd|vrt}||fS|d}|j|d{\}}|j}||d<|jjr|j r|j |d<||fS7Uw)z? Process a single JSONL entry (async version). conversationsNcompression_metrics)rurdrVrfrGrrrrer[rcompressed_trajectoryresults r&process_entry_asyncz(TrajectoryCompressor.process_entry_asyncs % '')G'> !?+ /3/M/Mj/Y)Y&w"7 ;; - -'2H2H,3OO,=F( )w*Zs,BBABcd|vrt}||fS|d}|j|\}}|j}||d<|jjr|j r|j |d<||fS)z Process a single JSONL entry. Args: entry: JSONL entry containing 'conversations' field Returns: Tuple of (processed_entry, metrics) rgrh)rurbrVrfrGrrris r& process_entryz"TrajectoryCompressor.process_entrys % '')G'> !?+ )-)A)A*)M&w"7 ;; - -'2H2H,3OO,=F( )wrs input_dir output_dircNtj|j||y)z Process all JSONL files in a directory using async parallel processing. Args: input_dir: Input directory containing JSONL files output_dir: Output directory for compressed files N)rRrun_process_directory_async)rrorps r&process_directoryz&TrajectoryCompressor.process_directorys  D11)ZHIrscP  Kt}tjjj_t j }t|jd}|sjjd|y|jdg}|D]j}t|dd5}t|D]B\} } | j} | s tj | } |j#|| | fD dddlt'|} |jd d |jd ||jd||jdt'||jd| d|jdj(j*d|jdj(j,|jdj(j.|jd d t1j2j(j.t1j4dddd|Dcic]}|ic}d dt6dt8dt:f f d }t=t?tAdtCtEtAdtGtAdtI|d 5}|jKd| dd| }|jKd!d }|D cgc]\}}} |||| |||}}}} t1jL|d{|jO|ddd|jd"|jQd#d#$|D]}||jRz }|}t|jUDcgc]}||||d}}t|d%d5}|D]+} |jWtjX| d&'d z- dddtjjj_-t j |z j_.j_j(j`ru|j(jbz }t|d%d5}tjdjjg|d()ddd|jd*|yy#tj$$r/} jjd|d | d | Yd} ~ d} ~ wwxYw#1swYxYwcc}wcc} }}w7'#1swYxYwcc}w#1swYxYw#1swYxYww)+zW Async implementation of directory processing with parallel API calls. *.jsonlzNo JSONL files found in Nz" [dim]Loading all entries...[/dim]rKrLrMzSkipping invalid JSON at ::  <============================================================u 📂 Input: u 📂 Output: u📄 Files to process: u📊 Total trajectories: ,u🎯 Target max tokens: u📝 Summary target tokens: u⚡ Max concurrent API calls: r file_path entry_idxrec> K4d{ 4d{ dz dddd{ tjj|jjd{\}}||f ||< 4d{j j ||jr dz |jz |jrdz  dz |j||j|d ddd d d dddd{dddd{y7&77 #1d{7swYxYw7778#1d{7swYHxYw#tj$rjjd |d |d jjd 4d{7j xjdz c_dz  dz |j||j|d ddd d d dddd{7n#1d{7swYnxYwd ||<Y2t $r}jj#d|d |d| 4d{7j xjdz c_ dz |j|dddd{7n#1d{7swYnxYw|t%f ||<Yd}~d}~wwxYw7#1d{7swYyxYww)z@Process a single entry with semaphore rate limiting and timeout.Nr)timeoutu [dim]✅ u compressed | ⏭️ u skipped | ⏱️ u timeout | 🔄 u API calls | ⚡ z in-flight[/dim]) descriptionzTimeout processing entry from rwz (>zs)zError processing entry from rx)rRwait_forrlrfrErrrrradvanceupdate TimeoutErrorrrGrr$errorru)r|r}reprogress main_task status_taskprocessed_entryr[r api_callscompressed_count in_flight progress_lockresultsr semaphore skipped_count timeout_counts r&process_singlezETrajectoryCompressor._process_directory_async..process_singleXs !: Q: Q(##NI##5Q5<5E5E007 $ B B60,OW6Eg4NGI&y1 -..EEgN#11,1,%)H)HHI"77)Q.M!Q !((3 '*34D3EEZ[hZii{}J|KK[\e[ffwxAwBBR)S(: Q: Q: Q##### 0&++9KK''*H STU^T__bcgcncndFdFcGGI)JK,..BBaGB%* !Q  ((3 '*34D3EEZ[hZii{}J|KK[\e[ffwxAwBBR)S( 59GI&y1 QKK%%(DYKqQZP[[]^_]`&ab,44..BBaGB!Q  ((344444 6; K= K  K=K" KK" K=7L=LLLLL LLz([progress.description]{task.description}u• )consolerefresh_per_secondz[cyan]Compressing trajectories)totalz[dim]Starting...[/dim]z# [dim]Writing output files...[/dim]Tparentsexist_okwF ensure_asciir:)indentu 💾 Metrics saved to )4rr now isoformatrrrHsortedglobrrGrr_r r*jsonloadsrJSONDecodeErrorrrfr,r-rBrR SemaphoreLockrrorrrrrrrradd_taskgather remove_taskmkdirrPkeyswritedumpsrr_print_summaryrFrHdumpr)!rrorprr jsonl_files all_entriesr|rdline_numlinerer total_entriesrrrrr}tasks output_path file_resultsrsorted_entries metrics_pathrrrrrrrrs!` @@@@@@@@r&rsz-TrajectoryCompressor._process_directory_asyncs)8@||~7O7O7Q4YY[ Y^^I67  KK  ":9+ F G   ;< $ iIiw7 i1&/liNHd::\\^5M5M5O2=AYY[:=U:  ;; & &% (G(GGLlC': Ia $0088:AaH I MM4\NC D 'A $33i KK//2KI;VWX`Waacdecf0ghhi i iB/@ )?" ." .Z  J J I IsB1V&<$U")T UEV&' U,1A=V&.2U; U1 8U;U8U;'A V&VV&+1V B.V& 1V<V&U -$U UU UU) $ V&1U;;V V& V V&V#V&c  |jj}|dd}|dd}|dd}|dd}|dd}|dd}|dd }|dd } |t|d z d z} |t|d z d z} |t|d z d z} td tdddtddddtdddtddddddtdddtdddd|dddtdddd|dd | d!d"d#d tdddd$|d%d | d!d"d#d tdddd&|dd | d!d"d#d tdddd'|dddtdddtdddd(d)dtdddtdddd*|d+d,d-dtdddd.|d+d,d-dtdddd/| d+d,d-dtdddd0|dd1d2d3d|d4kDr| |z d z} tdddd5| d6d7d3dtdddtdddd8d9dtdddtdddd*|d:dd+d;d<dtdddd.|d:d d+d;d<dtdddd=|d:d>d+d;d<dtdddtdddd?d@dtddd|d4kDrUtddddA|dBdCd2d3dtddddD|dBdEdFd3dtddddG|dBdHd6d3dntddddIdJdtdddtddddKdLdtdddtddddM|dNdOd+d@dtddddP|dNdQd+d@dtddddR|dNdSd2d3dtddd|dTdU}|dVkDr |dVz dWdX}n|dWdY}|t|dZz }tdddd[d\dtdddtdddd]|d^d<dtdddd_|d`dad#dtddddb|dTdcddded^d<dtddddf|dTdgddded^d<dtdhddi|jjr|jj}|jj }tdjtdkt |dldmt|dldnt|t|dozdltdpt |dqdmt|dqdnt|t|dozdqydyd)rz3Print comprehensive compression summary statistics.rrrrrrrrrrrdryu╔u══════════════════════════════════════════════════════════════════════u╗u║zTRAJECTORY COMPRESSION REPORTz^70u╠u╣r2u📁 TRAJECTORIESz6 u──────────────────────────────────────────────────────────────────────4zTotal Processed: z>10,z u├─ Compressed: z (z>5.1fz%)z u├─ Skipped (under limit):z>9,u├─ Still over limit: u└─ Failed: u 🔢 TOKENSz< zBefore Compression: z>15,z tokensz zAfter Compression: zTotal Saved: zOverall Compression: rz>14.1%z rzSpace Savings: z>14.1f%u💬 CONVERSATION TURNSz0 rz turnsz zTotal Removed: ru,📈 AVERAGES (Compressed Trajectories Only)z zAvg Compression Ratio: rrzAvg Tokens Saved: rz>14,.0fzAvg Turns Removed: rzNo trajectories were compressedz& u🤖 SUMMARIZATION APIz1 zAPI Calls Made: rXrzErrors: rzSuccess Rate: rrZr<z.1fz minutesz secondsgMbP?u⏱️ PROCESSING TIMEz3 zDuration: z>20zThroughput: z>15.1fz traj/seczStarted: rNzFinished: ru╚u╝u 📊 Distribution Summary:z Compression ratios: min=z.2%z, max=z , median=r:z Tokens saved: min=r{) rrrrrrr rr)rmrraskipped over_limitfailed tokens_before tokens_afterrxcompressed_pct skipped_pctover_limit_pct savings_pctdurationtime_str throughputratiosrs r&rz#TrajectoryCompressor._print_summarys  " " * * ,) 12y\";< I,BCy\"AB 934( N3 {=1 {=1 %s5!}4;UA.#5 $s5!}4; d  H:S!" 3C8<= H:S!" Bq6*6(#67 H:S!" Bq61%fXSIJ Bq65j5Fc.Y^I__abhaiilmn Bq66wsm3{SXFYY[\b[ccfgh Bq65j5Fc.Y^I__abhaiilmn Bq65fT]6(#NO H:S!" Bq6VHC01 H:S!" Bq61-1EWVHTWXY Bq61,t1DGF8SVWX Bq61,t1DGF8SVWX Bq61!H+>Y2Z[a1bcibjjmno 1 '-73>K C1v5k&5I6(RUV W H:S!" Bq60<= H:S!" Bq61!G*^2LT1RRXY_X``cde Bq61!G*]2KD1QQWX^W__bcd Bq61!G*_2Md1SSYZ`Yaadef H:S!" Bq6EfXSQR H:S!" > C1v5a mD[6\]c5dekdllop q C1v5a mDe6fgn5opvowwz{ | C1v5a mDf6ghn5opvowwz{ | C1v>F $ 6 6 H H  0 1 /F C/@s6{SVFWW`aghnaopstzp{}~p~aADaEF G /4E0Fq/IPSTePfghOiirsy{LtMNQRcNdfgNgthijskl m 5rsN)(rhrirjrkr)rrrrrlrrorrrrrrrr staticmethodrnrrqrr&rr+r.rurOrSrbrdrlrnrrtrsrrrrsr&rrLs!  20 2 `+TZ! #:"""T$tCH~2FT3TQDc3h,@QT#YQ.?$tCH~2F.?5QTVY[^Q^K_.?` OtDcN'; O# O$ O Oc3h(    2"Dc3heDcNL]<]6^6 J4 JT JAEAE$AEFjmrsrinputrYrfr,rOsample_percentseeddry_runc ddl}ddl} ddl} tdtdt |} | j r$td|t j|} ntd|dt } |r|| _|r|| _ |+|dks|dkDrtd |ytd |d |d t |} | j std |y| j}|r td|r t |}n)| j| j| jzdzz }g}t| dd5}t|dD]=\}}|j!}|s |j#t%j&|? dddt+|}td|dd| j,|^|j/|t1dt3||zdz }|j5||}tdt+|dd|d|dd |r3tdtdt+|ddtd|y| j75}t |d z }t |d!z }|j9|d"z }t|d#d5}|D]+}|j;t%j<|d$%d&z- dddt?| }|jA|||jj9d'd'(t|d#d5}tC|jEd)D]2}t|dd5}|D]}|j;| ddd4 ddd|| jFz } | j r<|j|jd*zz }!| jI| |!td+|!dddtd,td-|ytd.|r t |}n&| j| j,| jzz }|td/|d0| j75}t |d z }|j9|j/|d}"d}#tC| jEd)D]}g}t|dd5}|D]:}|j!}|s |j#t%j&|< ddd|"t+|z }"t1dt3t+||zdz }|j5|tK|t+|}$|#t+|$z }#||j,z }%t|%d#d5}|$D]+}|j;t%j<|d$%d&z- dddtd|#dd1|"dd2|r1tdtd3|td4| dddyt?| }|jA||dddnG|r(tdtd3| td4|yt?| }|jA| |td,y#t$j($r}td|d|Yd}~bd}~wwxYw#1swY+xYw#1swYxYw#1swYxYw#1swYkxYw#1swYxYw#t$j($rYNwxYw#1swYxYw#1swYxYw#1swYxYw)5a Compress agent trajectories to fit within a target token budget. Supports both single JSONL files and directories containing multiple JSONL files. Optionally sample a percentage of trajectories before compression. Args: input: Path to JSONL file or directory containing JSONL files output: Output path (file for file input, directory for dir input) Default: adds "_compressed" suffix to input name config: Path to YAML configuration file target_max_tokens: Override target token count from config tokenizer: Override tokenizer name from config sample_percent: Sample this percentage of trajectories (1-100) before compression seed: Random seed for sampling reproducibility (default: 42) dry_run: Analyze without compressing (just show what would happen) Examples: # Compress a directory (original behavior) python trajectory_compressor.py --input=data/my_run # Compress a single file python trajectory_compressor.py --input=data/trajectories.jsonl # Compress 15% sample of a file python trajectory_compressor.py --input=data/trajectories.jsonl --sample_percent=15 # Compress 10% sample with custom output python trajectory_compressor.py --input=data/trajectories.jsonl --sample_percent=10 --output=data/sampled_compressed.jsonl rNu🗜️ Trajectory Compressorrzu📋 Loading config from u⚠️ Config not found at z, using defaultsru2❌ sample_percent must be between 1 and 100, got u🎲 Will sample z% of trajectories (seed=)u❌ Input not found: u"📄 Input mode: Single JSONL filez.jsonlrKrLrMru&⚠️ Skipping invalid JSON at line rxz Loaded r{z trajectories from z Sampled z trajectories (z% of u. 🔍 DRY RUN MODE - analyzing without writingu📄 Would process: ru📄 Would output to: rrYztrajectories.jsonlrFrryTrrvz _metrics.jsonu💾 Metrics saved to u ✅ Compression complete!u 📄 Output: u)📁 Input mode: Directory of JSONL filesu. ⚠️ Sampling from directory: will sample z% from each filez from z total trajectoriesu📁 Would process: u📁 Would output to: )&randomtempfileshutilrrexistsr)rgr,r*is_fileparentstemr?r_r r*rrrrrrPrrrosampleTemporaryDirectoryrrrrrtrrrHrVr )&rrYrfr,rOrrrrrr config_pathcompression_config input_path is_file_inputrentriesrdrrrr sample_sizetemp_dirtemp_input_dirtemp_output_dirtemp_input_filere compressorout_f jsonl_filein_f metrics_filemetrics_outputtotal_original total_sampledsampled_entries temp_files& r&mainrLsP *+ (Ov,K )&23.88@ ,VH4DEF.0/@,,5)! Q .3"6 F~FVW X  !.!11I$qQReJ     %eW-.&&(M 23 v,K$++zASAaAa/adl/lmK *cG 4 X"+Aq/ X$zz|Xtzz$'78  X XG  =++>z>OPQ  % KK a]^%Cc%I!JKKmmG[9G KG Q/~>NeTabcSddef g  C D (Wa(8 F G *;-8 9  ( ( * Ah!(^g5N"8nx7O  "-/CCOosW= J$JEGGDJJu5ADHIJ J ..@AJ  ( ( I    $ $TD $ Ak39 .U"()=)=i)H"I.Jj#@.D$(.D!KK-.... .+-?-S-SSL""$!,!3!3{7G7G/7Y!Z L.9.~.>?@9 A< +, k]+, 9: v,K$++zASAaAa/abK  % CNCSScd e,,.) J(!%h'!9$$& D!!" ! #))C"DRJ Gj#@)A$%)D#'::!:LL_`aKL00@AB2;-@AK) J) JP22DE ,,^[IS) J) JVGH,ZL9:.{m<=..@AJ  ( ([ A +,q //X FxjPRSTRUVWWX  X XB J J.. . .# A A@(,';';!)$(!) )) RR7) J) Js %[0$Z[+A[4+1[ A[4-,['[ 3 ['>A#[4$A"\5\ $\\B \51\(A \5\5Z=!Z82[8Z==[[  [ [4[$ [''[1 ,[44[>\ \\ \\% \5(\2 -\55\>__main__)N)Nz#configs/trajectory_compression.yamlNNN*F)8rkrrrHr`rrRpathlibrtypingrrrrr dataclassesr r r utilsr r fire rich.progressrrrrrrr rich.consolerhermes_constantsrragent.retry_utilsrhermes_cli.env_loaderr _hermes_home__file__r _project_envrlrpr'r)rurrrornrrhFirerrrsr&rs> 33(: EEE A.5 H~$$v- |F #! ! !sm!e_ !. `` `F ) )  ) X e e  e P}m}mD 7! V- V- V- V- V-  V-  V- V-V-r z DIIdOrs