SjjSRdZddlZddlmZmZmZmZddlmZm Z m Z m Z m Z m Z mZmZmZmZmZmZddlmZmZmZmZddlmZded dfd Zded efd Zd eeed ee fd Zdeeefd eeeffdZ deded e fdZ! d%dddededeedee"d e f dZ#dddededee"d e fdZ$ d%dddededee deedee"d ee f dZ%deded e&fdZ' d&dddedede(dee(dee"d e f dZ) d&dddedede(dee(dee"d e f dZ*dede d efd Z+d!ed"ed efd#Z,ded efd$Z-y)'z. Crawling functionality for Firecrawl v2 API. N)OptionalDictAnyList) CrawlRequestCrawlJob CrawlResponseDocumentCrawlParamsRequestCrawlParamsResponseCrawlParamsData WebhookConfigCrawlErrorsResponseActiveCrawlsResponse ActiveCrawlPaginationConfig) HttpClienthandle_response_errorvalidate_scrape_optionsprepare_scrape_options)normalize_document_inputrequestreturnc|jr|jjs td|j|jdkr td|jt |jyy)z Validate crawl request parameters. Args: request: CrawlRequest to validate Raises: ValueError: If request is invalid URL cannot be emptyNrzLimit must be positive)urlstrip ValueErrorlimitscrape_optionsr)rs M/root/.hermes/venv/lib/python3.12/site-packages/firecrawl/v2/methods/crawl.py_validate_crawl_requestr#sh ;;gkk//1.//}} W]]a%7122) 6 67*ct|d|ji}|jr|j|d<|jt |j}|r||d<|j dd}|j dd|j dd|j dd|jIt|jtr|j|d<n|jj d |d<d d d d dddddddddd }|jD]\}}||vs |j |||< |j|d|vr)t|dtr|dj|d<|S)z Prepare crawl request for API submission. Args: request: CrawlRequest to prepare Returns: Dictionary ready for API submission rpromptN scrapeOptionsT) exclude_none exclude_unsetr!webhook)r( includePaths excludePathsmaxDiscoveryDepthsitemapignoreQueryParametersdeduplicateSimilarURLscrawlEntireDomainallowExternalLinksallowSubdomainsdelaymaxConcurrencyregexOnFullURLzeroDataRetention) include_paths exclude_pathsmax_discovery_depthr.ignore_query_parametersdeduplicate_similar_urlscrawl_entire_domainallow_external_linksallow_subdomainsr4max_concurrencyregex_on_full_urlzero_data_retention integration) r#rr&r!r model_dumppopr* isinstancestritemsupdater)rdata scrape_data request_datafield_mappings snake_case camel_cases r"_prepare_crawl_requestrP&sG$ 7;; D~~ X),W-C-CD $/D !%%4t%LLUD!Xt$%t," goos +%ooDO&oo88d8KDO('2#:$<2 4-+-2N"#1"6"6"8< J  %+// ;D <  KK D,?!E"=1779] Kr$ data_listc g}|xsgD]6}t|ts|jtdit |8|S)N)rFdictappendr r)rQ documentsdoc_datas r"_parse_crawl_documentsrXosJ "IOM h %   XK(@(JK LM r$ response_datac Z|jdst|jdd|jd|jdd|jdd|jdd|jd |jd t|jd gd S) NsuccesserrorUnknown error occurredstatus completedrtotal creditsUsed expiresAtnextrJr^r_r` credits_used expires_atrcrJ)get ExceptionrX)rYs r"_parse_crawl_status_responseriws   Y ' ))'3KLMM ##H-"&&{A6""7A.%))-;#'' 4!!&)&}'8'8'DE r$clientc@t|}|jd|}|js t|d|j }|j dr.|j d|j dd}t d i|St|j dd) aP Start a crawl job for a website. Args: client: HTTP client instance request: CrawlRequest containing URL and options Returns: CrawlResponse with job information Raises: ValueError: If request is invalid Exception: If the crawl operation fails to start z /v2/crawlz start crawlr[idr)rlrr\r]rS)rPpostokrjsonrgr rh)rjrrLresponserYjob_datas r" start_crawlrrs*'2L{{; 5H ;;h 6MMOM###D) $$U+  (x(( ))'3KLMMr$request_timeoutjob_idpagination_configrtc |jd||}|js t|d|j}t |}|d}|r |j nd}|r>|dr9|r$|j t||j k\st||d|||}t|d |d |d |d |d |s |d|Sd|S)aK Get the status of a crawl job. Args: client: HTTP client instance job_id: ID of the crawl job pagination_config: Optional configuration for pagination behavior request_timeout: Timeout (in seconds) for each individual HTTP request. When auto-pagination is enabled (default) and there are multiple pages of results, this timeout applies to each page request separately, not to the entire operation Returns: CrawlJob with current status and data Raises: Exception: If the status check fails /v2/crawl/timeoutzget crawl statusrJTrcNrsr^r_r`rerfrd) rgrnrrori auto_paginate max_resultslen_fetch_all_pagesr ) rjrurvrtrprYpayloadrVr{s r"get_crawl_statusrs2zzJvh/zIH ;;h(:;MMOM*=9GI8I%33dM  ) ) 5  N/;; ;$  FO  +   x +&g^,<($1WV_   8<  r$next_urlc |j||}|js t|d|j}t |}t |d|d|d|d|d|d|d  S) a Fetch a single page of crawl results using the provided next URL. Args: client: HTTP client instance next_url: Opaque next URL from a prior crawl status response request_timeout: Timeout (in seconds) for the HTTP request Returns: CrawlJob with the page data and next URL (if any) Raises: Exception: If the request fails or returns an error response ryzget crawl status pager^r_r`rerfrcrJrd)rgrnrrorir )rjrrtrprYrs r"get_crawl_status_pagers(zz(Oz J G  %DNN,cN d 0 -MMO  7 BL %V, 'H&3y>[+H   X &  '  "s9~'D  #6* a M P %  " % s+ E EEc|jd|}|js t|d|j}|j ddk(S)a Cancel a running crawl job. Args: client: HTTP client instance job_id: ID of the crawl job to cancel Returns: bool: True if the crawl was cancelled, False otherwise Raises: Exception: If the cancellation fails rxz cancel crawlr^ cancelled)deleternrrorg)rjrurprYs r" cancel_crawlr\sK}}z&23H ;;h7MMOM   X &+ 55r$ poll_intervalrzctj} t|||}|jdvr|S|,tj|z |kDrt d|d|dtj |b)a Wait for a crawl job to complete, polling for status updates. Args: client: HTTP client instance job_id: ID of the crawl job poll_interval: Seconds between status checks timeout: Maximum seconds to wait (None for no timeout) request_timeout: Optional timeout (in seconds) for each status request Returns: CrawlJob when job completes Raises: Exception: If the job fails TimeoutError: If timeout is reached rs)r_failedrz Crawl job z did not complete within z seconds)rrrr^ TimeoutErrorsleep)rjrurrzrtr crawl_jobs r"wait_for_crawl_completionrss2!J $  +    C C   DNN$4z$AW#LF83LWIU]^_ _ =!! r$c^t||}|j}||n|}t|||||S)a Start a crawl job and wait for it to complete. Args: client: HTTP client instance request: CrawlRequest containing URL and options poll_interval: Seconds between status checks timeout: Maximum seconds to wait for the entire crawl job to complete (None for no timeout) request_timeout: Timeout (in seconds) for each individual HTTP request, including pagination requests when fetching results. If there are multiple pages, each page request gets this timeout Returns: CrawlJob when job completes Raises: ValueError: If request is invalid Exception: If the crawl fails to start or complete TimeoutError: If timeout is reached rs)rrrlr)rjrrrzrtrrueffective_request_timeouts r"crawlrsH8FG,I \\F4C3NT[ %1  r$c ^|jr|jjs td|jr|jjs td|j|jd}|j d|}|j s t |d|j}|jdrL|jdi}i}dd d d d d ddddddd }d|vr)|d}t|trtd%i||d<n||d<|jD]\} } | |vs | dk(r|| || } i} ddddddd} | jD]\}}|| vs | || |<d| vr-| d}t|trdd lm}||!| d<n|| d<| jD]\}}|| vs |dk7s|| |<| || <|| || <|jD]\}}||vs |||<d"|vr|d"|d"<t!d%i|St#|jd#d$)&al Get crawl parameters from LLM based on URL and prompt. Args: client: HTTP client instance request: CrawlParamsRequest containing URL and prompt Returns: CrawlParamsData containing suggested crawl options Raises: ValueError: If request is invalid Exception: If the operation fails rzPrompt cannot be empty)rr&z/v2/crawl/params-previewzcrawl params previewr[rJr8r9r:r.r;r<r=r>r?r@r!rB) r+r,r-r.r/r0r1r2r3r5r'r7r*r' include_tags exclude_tagsonly_main_contentwait_forskip_tls_verificationremove_base64_images) includeTags excludeTagsonlyMainContentwaitForskipTlsVerificationremoveBase64Imagesformatsr) ScrapeFormats)rrr\r]rS)rrrr&rmrnrrorgrFrTrrHlisttypesrrrh)rjrrLrprY params_dataconverted_paramsrM webhook_datarOrNscrape_opts_dataconverted_scrape_optsscrape_field_mappings scrape_camel scrape_snake formats_datarkeyvalues r"crawl_params_previewrs ;;gkk//1.// >>!5!5!7122{{..L {{5|DH ;;h(>?MMOM##''3 ++!6 %>&@!6"81/-!6    #&y1L,-.;.Kl.K +.: +&4&:&:&<$ K "J [(0[5L5X'2:'>$,.)'5'5+>#-/F.D -)7L6Q6Q6Sa2 l'+;;BRS_B`1,?a !$44'7 'B %lD9=?LUa?b1)? U&;;y@P9>1#6?4I$Z03>z3J$Z0I$ KN&++- .JC.((- % .  %*7 *B Y '2!122 ))'3KLMMr$ http_clientcrawl_idc`|jd|d}|js t|d |j}|jd|}|jdg|jd|jdgd}t d i|S#t $r}t d |d }~wwxYw) a- Get errors from a crawl job. Args: http_client: HTTP client for making requests crawl_id: The ID of the crawl job Returns: CrawlErrorsResponse containing errors and robots blocked URLs Raises: Exception: If the request fails rxz/errorszcheck crawl errorsrJerrors robotsBlockedrobots_blocked)rrz'Failed to parse crawl errors response: NrS)rgrnrrorrh)rrrpbodyr normalizedes r"get_crawl_errorsrCsH:W=>H ;;h(<= G}}((64(kk(B/%kk/7;;GWY[;\] #0Z00 GA!EFFGsA!B B-B((B-c :|jd}|js t|d|j}|jdst |jdd|jdg}g}|D]u}t |t s|j|jd|jd|jd |jd |jd d wtd |Dcgc] }tdi|c}Scc}w)z Get a list of currently active crawl jobs. Args: client: HTTP client instance Returns: ActiveCrawlsResponse containing a list of active crawl jobs Raises: Exception: If the request fails z/v2/crawl/activezget active crawlsr[r\r]crawlsrlteamIdteam_idroptions)rlrrrT)r[rrS) rgrnrrorhrFrTrUrr)rjrpr crawls_innormalized_crawlscncs r"get_active_crawlsrcszz,-H ;;h(;< ==?D 88I *BCDD2&I  a   $ $eeDk55155+;<uuU|55+ &  Rc5dBk6GB6G5d ee5ds>D )N)rN).__doc__rtypingrrrrrrr r r r r rrrrrrutilsrrrrutils.normalizerr#rTrPrXrGrirrfloatrrr~boolrintrrrrrrSr$r"rs ,, gf68\8d8*FLFTFRhtCy&9d8n S#X 4S>  N N\ Nm NL59> (, > > > 01> e_ >  >J(, $ $$e_ $  $V59 H (, H HHH~H 01 H e_ H (^HV66S6T64! +" (, +" +" +"+"c] +" e_ +"+"b! ) (, ) ) ))c] ) e_ ))XsNsN6HsN_sNlG*GG@SG@ fj f-A fr$