Sj@dZddlZddlmZmZmZmZmZmZddl m Z m Z m Z m Z mZmZmZddlmZmZmZmZddlmZddl mZd eeed eefd Zd eeefd eeeffd Zddddddddddedeedee deeeefdeedeedeedeedeedeed e fdZ d0dededeed e fdZdddededee d e fd Z! d0deded!eedeed eef d"Z"deded efd#Z# d1deded$ed%eed e f d&Z$ddddddddddd' dedeedee deeeefdeedeedeedeedeedeed$ed%eed e fd(Z%deed eefd)Z&dddddddd*deedee deeeefdeedeedeedeedeed e'fd+Z(d2deed,ed eeefd-Z) d3dedeedee d,ed$ed%eed eefd.Z*deded efd/Z+y)4z4 Batch scraping functionality for Firecrawl v2 API. N)OptionalListCallableDictAnyUnion)BatchScrapeRequestBatchScrapeResponseBatchScrapeJob ScrapeOptionsDocument WebhookConfigPaginationConfig) HttpClienthandle_response_errorvalidate_scrape_optionsprepare_scrape_options)normalize_document_input)CrawlErrorsResponse data_listreturncg}|xsgD]8}t|tst|}|jt di|:|S)N) isinstancedictrappendr)r documentsdoc normalizeds M/root/.hermes/venv/lib/python3.12/site-packages/firecrawl/v2/methods/batch.py_parse_batch_scrape_documentsr"sL "IB5 c4 1#6J   X3 3 45 bodyc `|jdst|jdd|jd|jdd|jdd|jd|jd |jd t|jd gxsgd S) NsuccesserrorUnknown error occurredstatus completedrtotal creditsUsed expiresAtnextdatar)r*r+ credits_used expires_atr.r/)get Exceptionr")r$s r!#_parse_batch_scrape_status_responser5s 88I *BCDD((8$XXk1-'1%/hh{+ -dhhvr.B.HbI r#optionswebhook append_to_idignore_invalid_urlsmax_concurrencyzero_data_retention integrationidempotency_keyclienturlsr7r8r9r:r;r<r=r>c t||||||||} |j| } |jd| | } | js t | d| j } | j dst| j ddt| j d| j d | j d xsd S) aW Start a batch scrape job for multiple URLs. Args: client: HTTP client instance urls: List of URLs to scrape options: Scraping options Returns: BatchScrapeResponse containing job information Raises: FirecrawlError: If the batch scrape operation fails to start r7r8r9r:r;r<r=z/v2/batch/scrape)headerszstart batch scraper&r'r(idurl invalidURLsN)rDrE invalid_urls) prepare_batch_scrape_request_prepare_headerspostokrjsonr3r4r )r?r@r7r8r9r:r;r<r=r> request_datarCresponser$s r!start_batch_scraperO-s80 !/'/ L%%o6G{{-|W{MH ;;h(<= ==?D 88I *BCDD  88D> HHUOXXm,4 r#job_idpagination_configc F|jd|}|js t|d|j}t |}|d}|r |j nd}|r|drt ||d||}t|d|d|d|d |d |s |d| Sd | S) ak Get the status of a batch scrape job. Args: client: HTTP client instance job_id: ID of the batch scrape job pagination_config: Optional configuration for pagination behavior Returns: BatchScrapeJob containing job status and data Raises: FirecrawlError: If the status check fails /v2/batch/scrape/zget batch scrape statusr/Tr.r)r*r+r1r2Nr0)r3rKrrLr5 auto_paginate_fetch_all_batch_pagesr )r?rPrQrNr$payloadrrTs r!get_batch_scrape_statusrWgs(zz-fX67H ;;h(AB ==?D1$7GI8I%33dM*  FO     x +&g^,<($1WV_   8<  r#)request_timeoutnext_urlrXc |j||}|js t|d|j}t |}t |d|d|d|d|d|d|d  S) a Fetch a single page of batch scrape results using the provided next URL. Args: client: HTTP client instance next_url: Opaque next URL from a prior batch scrape status response request_timeout: Timeout (in seconds) for the HTTP request Returns: BatchScrapeJob with the page data and next URL (if any) Raises: Exception: If the request fails or returns an error response )timeoutzget batch scrape status pager)r*r+r1r2r.r/r0)r3rKrrLr5r )r?rYrXrNr$rVs r!get_batch_scrape_status_pager\s(zz(Oz H E  %DNN,cN d . +MMO  >yIL %V, 'H&3y>[+H   X &  '  "s9~'D  #6* a K N %  " % s) E EEc|jd|}|js t|d|j}|j ddk(S)a( Cancel a running batch scrape job. Args: client: HTTP client instance job_id: ID of the batch scrape job to cancel Returns: BatchScrapeStatusResponse with updated status Raises: FirecrawlError: If the cancellation fails rSzcancel batch scraper) cancelled)deleterKrrLr3)r?rPrNr$s r!cancel_batch_scraperw sO$}}09:H ;;h(=> ==?D 88H  ,,r# poll_intervalr[ctj} t||}|jdvr|S|r,tj|z |kDrt d|d|dtj |`)a Wait for a batch scrape job to complete, polling for status updates. Args: client: HTTP client instance job_id: ID of the batch scrape job poll_interval: Seconds between status checks timeout: Maximum seconds to wait (None for no timeout) Returns: BatchScrapeStatusResponse when job completes Raises: FirecrawlError: If the job fails or timeout is reached TimeoutError: If timeout is reached )r*failedruzBatch scrape job z did not complete within z seconds)rgrhrWr) TimeoutErrorsleep)r?rPrxr[ro status_jobs r!wait_for_batch_completionr~&s,!J ,VV<     D D  (:5@!26(:ST[S\\def f =! r#) r7r8r9r:r;r<r=r>rxr[c `t||||||||||  } | j} t|| | | S)a Start a batch scrape job and wait for it to complete. Args: client: HTTP client instance urls: List of URLs to scrape options: Scraping options poll_interval: Seconds between status checks timeout: Maximum seconds to wait (None for no timeout) Returns: BatchScrapeStatusResponse when job completes Raises: FirecrawlError: If the batch scrape fails to start or complete TimeoutError: If timeout is reached r6)rOrDr~)r?r@r7r8r9r:r;r<r=r>rxr[startrPs r! batch_scraperMsPB  !/'/' EXXF % w r#c|s tdg}|D]q}|rt|tstd||jds|jdstd||j |j s|S)z Validate and normalize a list of URLs for batch scraping. Args: urls: List of URLs to validate Returns: Validated list of URLs Raises: ValueError: If URLs are invalid zURLs list cannot be emptyz Invalid URL: zhttp://zhttps://z)URL must start with http:// or https://: ) ValueErrorrstr startswithrstrip)r@validated_urlsrEs r!validate_batch_urlsrs 455N+*S#.}SE23 3y)S^^J-GHNO Ociik*+ r#rBc2t|}d|i} |rt|} | r| j| |+t|tr|| d<n|j d| d<||| d<||| d<||| d<||| d<|t |j | d <| S) z Prepare a batch scrape request payload. Args: urls: List of URLs to scrape options: Scraping options Returns: Request payload dictionary r@r8T) exclude_none appendToIdignoreInvalidURLsmaxConcurrencyzeroDataRetentionr=)rrupdaterr model_dumpr) r@r7r8r9r:r;r<r=rrM scrape_datas r!rHrHs*).N$*N#;L,W5     , gs #&-L #&-&8&8d&8&KL #%1 \"&,? ()")8 %&&,? ()&)+&6&<&<&> ]# r# chunk_sizecng}tdt||D]}|j||||z|S)z Split a large list of URLs into smaller chunks for batch processing. Args: urls: List of URLs to chunk chunk_size: Maximum size of each chunk Returns: List of URL chunks r)rangerlr)r@rchunksis r! chunk_urlsrsAF 1c$i ,. d1Q^,-. Mr#ct||}g}d}|D]>} t|| |||} | jr|j| j|dz }@|S)a Process a large batch of URLs by splitting into smaller chunks. Args: client: HTTP client instance urls: List of URLs to scrape options: Scraping options chunk_size: Size of each batch chunk poll_interval: Seconds between status checks timeout: Maximum seconds to wait per chunk Returns: List of all scraped documents Raises: FirecrawlError: If any chunk fails r)r7rxr[rb)rrr/extend) r?r@r7rrxr[ url_chunks all_documentscompleted_chunkschunkresults r!process_large_batchrsq2D*-JM  '   ;;   -A r#c |jd|d}|js t|d|j}|jd|}|jdg|jd|jdgd}t d i|S) z Get errors for a batch scrape job. Args: client: HTTP client instance job_id: ID of the batch scrape job Returns: CrawlErrorsResponse with errors and robots-blocked URLs rSz/errorszget batch scrape errorsr/errors robotsBlockedrobots_blocked)rrr)r3rKrrLr)r?rPrNr$rVr s r!get_batch_scrape_errorsrszz-fXW=>H ;;h(AB ==?Dhhvt$G++h+!++ow{{CSUW7XYJ  , ,,r#)N)r N)d)Nrr N),__doc__rgtypingrrrrrrtypesr r r r rrrutilsrrrrutils.normalizerrr"rr5boolintrOrWfloatr\rUrwr~rrrrHrrrrr#r!rs ==gf6'Xd3i-@T(^ d38n c3h &(,37"&*.%)*.!%%)7 7 s)7m $ 7 eC./ 0 7 3- 7"$7c]7"$7#7c]77z591 1 1 011 1p(, $ $$e_ $  $V59 D DDH~D 01 D  (^ DN- - - -@! $" $" $"$"c] $"  $"V(,37"&*.%)*.!%%)!3 3 s)3m $ 3 eC./ 0 3 3- 3"$3c]3"$3#3c]33c]33ld3iDIB(,37"&*.%)*.!%/ s)/m $/eC./ 0 / 3- / "$ /c]/"$/#/ /dT#YC$tCy/((,! - - s)-m $- -  - c] - (^-`-J--@S-r#