Sj! ddlmZmZmZmZddlmZmZmZm Z m Z m Z ddl m Z ddlmZddlmZddlmZddlmZdd lZd eeed eefd Zd eeefd eeeffdZd ddeedeed eeeffdZde deed e fdZ d de dedee d e fdZd dde dedeed e fdZ d de dedeedee d eef dZde ded e fdZ!de ded eeeffdZ"y )!)OptionalListDictAny) ScrapeOptions WebhookConfigDocumentBatchScrapeResponseBatchScrapeJobPaginationConfig)AsyncHttpClient)prepare_scrape_options)handle_response_error)normalize_document_input)validate_batch_urlsN data_listreturncg}|xsgD]8}t|tst|}|jt di|:|S)N) isinstancedictrappendr )r documentsdoc normalizeds Q/root/.hermes/venv/lib/python3.12/site-packages/firecrawl/v2/methods/aio/batch.py_parse_batch_scrape_documentsr sL "IB5 c4 1#6J   X3 3 45 bodyc `|jdst|jdd|jd|jdd|jdd|jd|jd |jd t|jd gxsgd S) NsuccesserrorUnknown error occurredstatus completedrtotal creditsUsed expiresAtnextdatar%r&r' credits_used expires_atr*r+)get Exceptionr)r s r#_parse_batch_scrape_status_responser1s 88I *BCDD((8$XXk1-'1%/hh{+ -dhhvr.B.HbI r)optionsurlsr2c ~|s tdt|Dcgc]$}t|tr|j n|&c}}d|i}|rt |}|r|j ||jdx}'t|tr|n|jd|d<|jdx}||d<|jdx}||d <|jd x}||d <|jd x}||d <|jdx} t|j } | r| |d<|Scc}w)NzURLs list cannot be emptyr3webhookT) exclude_none append_to_id appendToIdignore_invalid_urlsignoreInvalidURLsmax_concurrencymaxConcurrencyzero_data_retentionzeroDataRetention integration) ValueErrorrrstrstriprupdater/ model_dump) r3r2kwargsuvalidated_urlspayloadoptswvtrimmed_integrations r_preparerM!sQ 455(Z^)_UVz!S7I!'')q*P)_`N%~6G%g.  NN4 ZZ ""/",Q"4Q!,,TX,:Y  ZZ ''4 !  ZZ- ..;'(#$ ZZ) **7$% ! ZZ- ..;'(#$ ZZ &&3!!flln %8GM " N)*`s)D:clientc~Kt|fi|}|jd|d{}|jdk\r t|d|j }|j dst |j ddt|j d|j d|j d  S7w) Nz/v2/batch/scrapezstart batch scraper"r#r$idurl invalidURLs)rQrR invalid_urls)rMpost status_coderjsonr/r0r )rNr3rErHresponser s rstart_batch_scraperY<st&v&G[[!3W==Hs"h(<= ==?D 88I *BCDD $((4.dhhuoTXT\T\]jTk ll >s"B=B;BB=job_idpagination_configc |K|jd|d{}|jdk\r t|d|j}t |}|d}|r |j nd}|r|drt ||d||d{}t|d|d |d |d |d |s |d| Sd| S774w)al Get the status of a batch scrape job. Args: client: Async HTTP client instance job_id: ID of the batch scrape job pagination_config: Optional configuration for pagination behavior Returns: BatchScrapeJob containing job status and data Raises: Exception: If the status check fails /v2/batch/scrape/NrPzget batch scrape statusr+Tr*r%r&r'r-r.r,)r/rVrrWr1 auto_paginate_fetch_all_batch_pages_asyncr )rNrZr[rXr rHdocsr^s rget_batch_scrape_statusraGs&ZZ"3F8 <==Hs"h(AB ==?D1$7G 6?D8I%33dM1  FO      x +&g^,<($1WV_   8<  #> s"B<B8A*B<B:3B<:B<)request_timeoutnext_urlrbc K|j||d{}|jdk\r t|d|j}t |}t |d|d|d|d|d |d |d  S7aw) a Fetch a single page of batch scrape results using the provided next URL. Args: client: Async HTTP client instance next_url: Opaque next URL from a prior batch scrape status response request_timeout: Timeout (in seconds) for the HTTP request Returns: BatchScrapeJob with the page data and next URL (if any) Raises: Exception: If the request fails or returns an error response )timeoutNrPzget batch scrape status pager%r&r'r-r.r*r+r,)r/rVrrWr1r )rNrcrbrXr rHs rget_batch_scrape_status_pagerfvs(ZZ/ZBBHs"h(FG ==?D1$7G x +&g^,<( V_ V_  CsA=A;A"A=initial_documentscK|j}|}d}|r |jnd}|r |jnd}|r |jnd} t j } |r|||k\r |S| t j | z | kDr |S|j |d{} | jdk\r6ddl} | jd} | jd| j |S| j} t|}|dD]%}|t||k\rn|j|'|t||k\r |S|d}|dz }|r|S7#t$rY|SwxYww) ag Fetch all pages of batch scrape results asynchronously. Args: client: Async HTTP client instance next_url: URL for the next page initial_documents: Documents from the first page pagination_config: Optional configuration for pagination limits Returns: List of all documents from all pages rNrP firecrawlzFailed to fetch next page: r+r*)copy max_pages max_results max_wait_timetime monotonicr/rVlogging getLoggerwarningrWr1r0lenr)rNrcrgr[r current_url page_countrlrmrn start_timerXrqlogger page_data page_payloaddocuments rr_r_s$"&&(IKJ0A!++dI3D#//$K7H%33dM!J   ! i(? H E  %DNN,yIL %V, 'H'c)n .K   X &  '  ##i.K*G  #6* a K N =1  " % s>BE!EAE!5 EA E! E! EE!EE!cK|jd|d{}|jdk\r t|d|j}|j ddk(S7Cw)Nr]rPzcancel batch scraper% cancelled)deleterVrrWr/rNrZrXr s rcancel_batch_scrapers]]]%6vh#?@@Hs"h(=> ==?D 88H  ,, AsA AAA cK|jd|dd{}|jdk\r t|d|j}|jdst |jdd|S7]w)Nr]z/errorsrPzget batch scrape errorsr"r#r$)r/rVrrWr0rs rget_batch_scrape_errorsrssZZ"3F87 CDDHs"h(AB ==?D 88I *BCDD K EsA;A9AA;)N)#typingrrrrtypesrr r r r r utils.http_client_asyncrutils.validationrutils.error_handlerrutils.normalizer methods.batchrrorrAr1rMrYrafloatrfr_boolrrrrrrs,,rr66870 Xd3i-@T(^ d38n c3h EI49(=*AW[\_ad\dWe6m_mDImTgm59, , , 01, ,f(, ! !!e_ !  !P59 D DDH~D 01 D  (^ DN-o-s-t-/34PSUXPX>r