SjF6ddlmZmZmZmZddlmZmZmZm Z m Z m Z m Z m Z mZmZmZddlmZddlmZddlmZddlmZddlZd ed efd Zd eeed ee fd Zdeeefd eeeffdZded ed efdZ d dddededeedee d ef dZ!dddededee d efdZ" d dddededee deedee d ee f dZ#deded e$fdZ%ded e d e fdZ&deded e fdZ'ded efdZ(y)!)OptionalDictAnyList) CrawlRequestCrawlJob CrawlResponseDocumentCrawlParamsRequestCrawlParamsData WebhookConfigCrawlErrorsResponseActiveCrawlsResponse ActiveCrawlPaginationConfig)handle_response_error)prepare_scrape_options)AsyncHttpClient)normalize_document_inputNrequestreturncP|jr|jjs tdd|ji}|jr|j|d<|jt |j}|r||d<|j It|j tr|j |d<n|j jd|d<|jdd}|jdd|jdd|jd dd d d d dddddddddd }|jD]\}}||vs |j|||< |j|t|dd&tt|dj|d<|S)NURL cannot be emptyurlprompt scrapeOptionswebhookT) exclude_none)r exclude_unsetscrape_options includePaths excludePathsmaxDiscoveryDepthsitemapignoreQueryParametersdeduplicateSimilarURLscrawlEntireDomainallowExternalLinksallowSubdomainsdelaymaxConcurrencyregexOnFullURLzeroDataRetention) include_paths exclude_pathsmax_discovery_depthr%ignore_query_parametersdeduplicate_similar_urlscrawl_entire_domainallow_external_linksallow_subdomainsr+max_concurrencyregex_on_full_urlzero_data_retention integration)rstrip ValueErrorrr!rr isinstancestr model_dumppopitemsupdategetattr)rdataopts request_datafield_mappingssnakecamels Q/root/.hermes/venv/lib/python3.12/site-packages/firecrawl/v2/methods/aio/crawl.py_prepare_crawl_requestrKs ;;gkk//1.// 7;; D~~ X)%g&<&<= $(D !" goos +%ooDO%oo88d8KDO%%4t%LLUD!Xt$%t,''2#:$<2 4-+-2N',,.2 u L &**51DK2 KK w t,8!''="ABHHJ] K data_listcg}|xsgD]8}t|tst|}|jt di|:|S)N)r=dictrappendr )rM documentsdoc_data normalizeds rJ_parse_crawl_documentsrUBsL "IO5 h %1(;J   X3 3 45 rLbodyc Z|jdst|jdd|jd|jdd|jdd|jdd|jd |jd t|jd gd S) NsuccesserrorUnknown error occurredstatus completedrtotal creditsUsed expiresAtnextrDr[r\r] credits_used expires_atr`rD)get ExceptionrU)rVs rJ_parse_crawl_status_responserfKs 88I *BCDD((8$XXk1-'1%2hh{+ &txx';< rLclientc\Kt|}|jd|d{}|jdk\r t|d|j }|j dr+t |j d|j dSt|j d d 7w) aV Start a crawl job for a website. Args: client: Async HTTP client instance request: CrawlRequest containing URL and options Returns: CrawlResponse with job information Raises: ValueError: If request is invalid Exception: If the crawl operation fails to start z /v2/crawlNz start crawlrXidr)rjrrYrZ)rKpost status_coderjsonrdr re)rgrpayloadresponserVs rJ start_crawlrpZs%W-G[[g66Hs"h 6 ==?D xx DHHUODD DHHW&>? @@ 7s!B,B*BB,request_timeoutjob_idpagination_configrrc K|jd||d{}|jdk\r t|d|j}t |}|d}|r |j nd}|r |drt ||d||| d{}t|d |d |d |d |d|s |d|Sd|S774w)a[ Get the status of a crawl job. Args: client: Async HTTP client instance job_id: ID of the crawl job pagination_config: Optional configuration for pagination limits request_timeout: Timeout (in seconds) for each individual HTTP request. When auto-pagination is enabled (default) and there are multiple pages of results, this timeout applies to each page request separately, not to the entire operation Returns: CrawlJob with job information Raises: Exception: If the status check fails /v2/crawl/timeoutNrizget crawl statusrDTr`rqr[r\r]rbrcra)rdrlrrmrf auto_paginate_fetch_all_pages_asyncr ) rgrsrtrrrorVrnrRrys rJget_crawl_statusr{ss0ZZ*VH 5ZOOHs"h(:; ==?D*40GI8I%33dM0  FO  +    x +&g^,<($1WV_   8<  'P s"CB<A,C B> 3C>Cnext_urlc K|j||d{}|jdk\r t|d|j}t |}t |d|d|d|d|d |d |d  S7aw) a Fetch a single page of crawl results using the provided next URL. Args: client: Async HTTP client instance next_url: Opaque next URL from a prior crawl status response request_timeout: Timeout (in seconds) for the HTTP request Returns: CrawlJob with the page data and next URL (if any) Raises: Exception: If the request fails or returns an error response rwNrizget crawl status pager[r\r]rbrcr`rDra)rdrlrrmrfr )rgr|rrrorVrns rJget_crawl_status_pager~s(ZZ/ZBBHs"h(?@ ==?D*40G x +&g^,<( V_ V_  CsA=A;A"A=initial_documentscK|j}|}d}|r |jnd}|r |jnd} |r |jnd} t j } |r|||k\r |S| t j | z | kDr |S|j ||d{} | jdk\r7ddl} | jd}|jdd| ji |S| j} t|}|d D]%}| t|| k\rn|j|'| t|| k\r |S|d }|d z }|r|S7#t$rY|SwxYww) a Fetch all pages of crawl results asynchronously. Args: client: Async HTTP client instance next_url: URL for the next page initial_documents: Documents from the first page pagination_config: Optional configuration for pagination limits request_timeout: Optional timeout (in seconds) for the underlying HTTP request Returns: List of all documents from all pages rNrwri firecrawlzFailed to fetch next pagerl)extrarDr`)copy max_pages max_results max_wait_timetime monotonicrdrllogging getLoggerwarningrmrfrelenrQ)rgr|rrtrrrR current_url page_countrrr start_timerorlogger page_data page_payloaddocuments rJrzrzs*"&&(IKJ0A!++dI3D#//$K7H%33dM!J   !zY'> H E  %DNN,cN d . +MMO  7 BL %V, 'H'c)n .K   X &  '  ##i.K*G  #6* a K N =J  " % s>BE$EAE$8 EA E$E$ E!E$ E!!E$cK|jd|d{}|jdk\r t|d|j}|j ddk(S7Cw)a  Cancel a crawl job. Args: client: Async HTTP client instance job_id: ID of the crawl job Returns: True if cancellation was successful Raises: Exception: If the cancellation operation fails rvNriz cancel crawlr[ cancelled)deleterlrrmrd)rgrsrorVs rJ cancel_crawlrs\]]Zx#899Hs"h7 ==?D 88H  ,, :sA AAA c K|jr|jjs td|jr|jjs td|j|jd}|j d|d{}|j dk\r t |d|j}|jdst|jd d |jd i}i}d d ddddddddddd }|jD]\}} ||vs |||| <d|vr |d} | |d<d|vr|d|d<tdi|S7ȭw)au Preview crawl parameters before starting a crawl job. Args: client: Async HTTP client instance request: CrawlParamsRequest containing URL and prompt Returns: CrawlParamsData containing crawl configuration Raises: ValueError: If request is invalid Exception: If the parameter preview fails rzPrompt cannot be empty)rrz/v2/crawl/params-previewNrizcrawl params previewrXrYrZrDr/r0r1r%r2r3r4r5r6r7r!r9) r"r#r$r%r&r'r(r)r*r,rr.rrrO) rr;r<rrkrlrrmrdrerAr ) rgrrnrorV params_data convertedmappingrIrHwks rJcrawl_params_previewr,ss ;;gkk//1.// >>!5!5!7122kkW^^? ==?D 88I *BCDD((62&K "I''2!:"<24-+)2 G 2 u K *51Ie 2K  #! )D#I )  'Y ''=FsBEEBE-0Ecrawl_idcBK|jd|dd{}|jdk\r t|d|j}|jd|}|jdg|jd|jd gd }t d i|S7w) a" Get errors from a crawl job. Args: client: Async HTTP client instance crawl_id: ID of the crawl job Returns: CrawlErrorsResponse with errors and robots blocked Raises: Exception: If the error check operation fails rvz/errorsNrizcheck crawl errorsrDerrors robotsBlockedrobots_blocked)rrrO)rdrlrrmr)rgrrorVrnrTs rJget_crawl_errorsrasZZ*XJg >??Hs"h(<= ==?Dhhvt$G++h+!++ow{{CSUW7XYJ  , ,,@sBBBBc ^K|jdd{}|jdk\r t|d|j}|jdst |jdd|jdg}g}|D]u}t |t s|j|jd |jd |jd |jd |jd dwtd|Dcgc] }tdi|c}S7cc}ww)z Get active crawl jobs. Args: client: Async HTTP client instance Returns: ActiveCrawlsResponse with active crawl jobs Raises: Exception: If the active crawl jobs operation fails z/v2/crawl/activeNrizget active crawlsrXrYrZcrawlsrjteamIdteam_idroptions)rjrrrT)rXrrO) rdrlrrmrer=rPrQrr)rgrorV crawls_inrTcncs rJget_active_crawlsr{s ZZ 233Hs"h(;< ==?D 88I *BCDD2&IJ  a    eeDk55155+;<uuU|55+   R\5]Bk6GB6G5] ^^!4 6^s)D-D%BD-A.D- D(  D-(D-)N))typingrrrrtypesrr r r r r rrrrrutils.error_handlerrutils.validationrutils.http_client_asyncrutils.normalizerrrPrKrUr>rfrpfloatr{r~rzboolrrrrrOrLrJrs,,    9667 )L)T)XhtCy&9d8n tCH~ $sCx. AoA AA8593 (, 3 3 3 013 e_ 3  3t(, ! !!e_ !  !P59 G (, G GGH~G 01 G e_ G (^GT----*2(2(AS2(Xg2(j-?-c-FY-4_O_8L_rL