SjdUdZddlZddlZddlZddlmZmZmZmZm Z m Z m Z m Z m Z ddlZddlmZddlZddlZddlZddlZddlZddlZddlZddlmZdZeZej4dZej8ed<e d ZGd d ej>Z Gd d ej>Z!Gddej>Z"Gddej>Z#Gddej>e eZ$Gddej>Z%Gddej>Z&Gddej>Z'Gddej>Z(Gddej>Z)Gddej>Z*Gd d!ej>Z+Gd"d#ej>Z,Gd$d%ej>Z-Gd&d'ej>Z.Gd(d)ej>Z/Gd*d+ej>Z0Gd,d-ej>Z1Gd.d/ej>Z2Gd0d1ej>Z3Gd2d3e(Z4Gd4d5e$ee eZ5Gd6d7ej>Z6Gd8d9ej>Z7Gd:d;ej>Z8Gd<d=ej>Z9Gd>d?ej>Z:Gd@dAej>Z;GdBdCej>Z<GdDdEej>Z=GdFdGej>Z>GdHdIej>e eZ?GdJdKej>Z@GdLdMej>ZAGdNdOej>ZBGdPdQej>ZCGdRdSej>ZDGdTdUej>ZEGdVdWej>ZFGdXdYej>ZGGdZd[ej>ZHGd\dMej>ZAGd]dGej>Z>Gd^d_ZIGd`daZJGdbdceIZKGdddeeJZLy)fa FirecrawlApp Module This module provides a class `FirecrawlApp` for interacting with the Firecrawl API. It includes methods to scrape URLs, perform searches, initiate and monitor crawl jobs, and check the status of these jobs. The module uses requests for HTTP communication and handles retries for certain HTTP status codes. Classes: - FirecrawlApp: Main class for interacting with the Firecrawl API. N) AnyDictOptionalListUnionCallableLiteralTypeVarGeneric)datetime)Fieldc ddlm}tjj t }|tjj |dj}tjd|tj}|r|jdjSy#t$rtdYywxYw)Nr)Pathz __init__.pyz"^__version__ = ['\"]([^'\"]*)['\"]z&Failed to get version from __init__.py)pathlibrospathdirname__file__join read_textresearchMgroupstrip Exceptionprint)r package_path version_file version_matchs M/root/.hermes/venv/lib/python3.12/site-packages/firecrawl/firecrawl.backup.py get_versionr#s WW__X.l"'',,|]CDNNPlii E|UWUYUYZm $$Q'--/ /   45 sB"B&&B=<B= firecrawlloggerTc:eZdZUdZdZeded<dZee ed<y) AgentOptionszConfiguration for the agent.FIRE-1modelNprompt) __name__ __module__ __qualname____doc__r*r __annotations__r+rstrr"r(r(Qs"&'E78 ' FHSM r3r(c&eZdZUdZdZeded<y)AgentOptionsExtract2Configuration for the agent in extract operations.r)r*Nr,r-r.r/r*r r0r2r3r"r5r5V<'E78 'r3r5c2eZdZUdZeeed<eeed<y) ActionsResultz,Result of actions performed during scraping. screenshotspdfsN)r,r-r.r/rr1r0r2r3r"r:r:Zs6c s)Or3r:ceZdZUdZdZeeed<eed<eed<dZee ee fed<e jddZ ee ed <y) ChangeTrackingDataz. Data for the change tracking format. NpreviousScrapeAt changeStatus visibilitydiffjsonalias json_field)r,r-r.r/r?rr1r0rBrrpydanticr rFr2r3r"r>r>_sU'+hsm*O%)D(4S> ") .t6 BJ Br3r>c@eZdZUdZdZeeed<dZeeed<dZ eeed<dZ eeed<dZ ee eed<dZ eeed<ej dd Zeeed <dZeeed <dZeeed <dZeeed<dZeeed<dZeeed<dZeeed<y)FirecrawlDocumentz-Document retrieved or processed by Firecrawl.NurlmarkdownhtmlrawHtmllinksextractrCrDrF screenshotmetadataactionstitle descriptionchangeTracking)r,r-r.r/rJrr1r0rKrLrMrNrrOr&rGr rFrPrQrrRr:rSrTrUr>r2r3r"rIrIis7C#"Hhsm"D(3-!GXc]!!%E8DI %GXa[,hnnT@J @ $J $"Hhsm"'+GXm $+E8C=!%K#%37NH/07r3rIc@eZdZUdZdZeeed<dZee eed<y)LocationConfigz$Location configuration for scraping.Ncountry languages) r,r-r.r/rXrr1r0rYrr2r3r"rWrWys&.!GXc]!%)IxS ")r3rWcxeZdZUdZeed<dZeeeefed<dZ eeeefed<dZ ee e ded<y) WebhookConfigzConfiguration for webhooks.rJNheadersrQ) completedfailedpagestartedevents) r,r-r.r/r1r0r\rrrQrarr r2r3r"r[r[~sR% H(,GXd38n %,)-HhtCH~&-PTFHT'"JKL MTr3r[ceZdZUdZdZeeeded<e jddZ ee ed<dZ eeed<dZeeed <y) ChangeTrackingOptionsz"Configuration for change tracking.N)zgit-diffrCmodesschemarD schema_fieldr+tag)r,r-r.r/rdrrr r0rGr rfrr+r1rgr2r3r"rcrcsV,9=E8D!345 6="0(..X"FL(3-F FHSM C#r3rcceZdZUdZdZeeeded<dZ ee e e fed<dZ eee ed<dZ eee ed<dZeeed<dZeeed <d Zeeed <dZeeed <dZeeed <dZeeed<dZeeed<dZeeed<dZeeded<dZeeed<dZeeed<dZeeed<dZeeed<y) ScrapeOptions#Parameters for scraping operations.N rKrLrMcontentrNrPscreenshot@fullPagerOrCrUformatsr\ includeTags excludeTagsonlyMainContentwaitFor0utimeoutlocationmobileskipTlsVerificationremoveBase64ImagesblockAdsbasicstealthautoproxychangeTrackingOptionsmaxAge storeInCacheparsePDF)r,r-r.r/rnrrr r0r\rr1rorprqboolrrintrtrurWrvrwrxryr~rrcrrrr2r3r"riris2-eiGXd7$_`abi(,GXd38n %,'+K$s)$+'+K$s)$+&*OXd^*!GXc]!"GXc]")-Hh~&-!FHTN!*.$.)--#Hhtn#;?E8G67 8?=A8$9:A FHSM #'L(4.'#Hhtn#r3ricJeZdZUdZeded<dZeeed<dZ ee ed<y) WaitActionz'Wait action to perform during scraping.waittypeN millisecondsselector) r,r-r.r/r r0rrrrr1r2r3r"rrs+1 &/"&L(3-&"Hhsm"r3rcJeZdZUdZeded<dZeeed<dZ ee ed<y)ScreenshotActionz-Screenshot action to perform during scraping.rPrNfullPagequality) r,r-r.r/r r0rrrrrr2r3r"rrs,7 , #Hhtn#!GXc]!r3rc,eZdZUdZeded<eed<y) ClickActionz(Click action to perform during scraping.clickrrNr,r-r.r/r r0r1r2r3r"rrs2 ' Mr3rc,eZdZUdZeded<eed<y) WriteActionz(Write action to perform during scraping.writertextNrr2r3r"rrs2 '  Ir3rc,eZdZUdZeded<eed<y) PressActionz(Press action to perform during scraping.pressrkeyNrr2r3r"rrs2 '  Hr3rcFeZdZUdZeded<eded<dZeeed<y) ScrollActionz)Scroll action to perform during scraping.scrollr)updown directionNr) r,r-r.r/r r0rrr1r2r3r"rrs)3 ( |$$"Hhsm"r3rc"eZdZUdZeded<y) ScrapeActionz)Scrape action to perform during scraping.scraperN)r,r-r.r/r r0r2r3r"rrs3 ( r3rc,eZdZUdZeded<eed<y)ExecuteJavascriptActionz5Execute javascript action to perform during scraping.executeJavascriptrscriptNrr2r3r"rrs? % && Kr3rcdeZdZUdZeded<dZeeded<dZee ed<dZ ee ed<y) PDFActionz&PDF action to perform during scraping.pdfrN) A0A1A2A3A4A5A6LetterLegalTabloidLedgerformat landscapescale) r,r-r.r/r r0rrrrrfloatr2r3r"rrs@0 %.rvFHWmn ov $Ix~$!E8E?!r3rc&eZdZUdZdZeded<y) ExtractAgentr6r)r*Nr7r2r3r"rrr8r3rceZdZUdZdZeeed<ejddZ ee ed<dZ eeed<dZ eeed<y) JsonConfigzConfiguration for extraction.Nr+rerDrf systemPromptagent)r,r-r.r/r+rr1r0rGr rfrrrrr2r3r"rrsK' FHSM "0(..X"FL(3-F"&L(3-&$(E8L !(r3rc eZdZUdZdZeeed<dZeeed<dZ ee e e e eeeeeeef ed<dZeeed<dZeeed<y) ScrapeParamsrjNrO jsonOptionsrRrwebhook)r,r-r.r/rOrrr0rrRrrrrrrrrrrrrr(rr[r2r3r"rrs-$(GXj !((,K*%,koGXd5-={KYdfruACZ\e"efgho$(E8L !('+GXm $+r3rcHeZdZUdZdZeed<dZee ed<dZ ee ed<y)ScrapeResponsez"Response from scraping operations.TsuccessNwarningerror) r,r-r.r/rrr0rrr1rr2r3r"rrs+,GT!GXc]!E8C=r3rcveZdZUdZdZeeed<dZeeed<dZ e ed<dZ eeed<dZ ee eed<y) BatchScrapeResponsez&Response from batch scrape operations.NidrJTrr invalidURLs)r,r-r.r/rrr1r0rJrrrrrr2r3r"rrsL0B C#GTE8C='+K$s)$+r3rc|eZdZUdZdZeed<eded<eed<eed<eed<e ed <d Z e e ed <e eed <y ) BatchScrapeStatusResponsez)Response from batch scrape status checks.Trscrapingr]r^ cancelledstatusr]total creditsUsed expiresAtNnextdatar,r-r.r/rrr0r rr rrr1rrIr2r3r"rrsK3GT B CCN JD(3-  !!r3rc|eZdZUdZdZeeeed<dZ eeeed<dZ ee ed<dZ ee ed<dZ ee ed<dZeeed<dZeeed <dZeeed <dZeeed <dZeeed <dZeeeefed <dZeeed<dZeeed<dZeeed<dZee ed<dZee ed<dZeeed<y) CrawlParamsz#Parameters for crawling operations.N includePaths excludePathsmaxDepthmaxDiscoveryDepthlimitallowBackwardLinkscrawlEntireDomainallowExternalLinks ignoreSitemap scrapeOptionsrdeduplicateSimilarURLsignoreQueryParametersregexOnFullURLdelaymaxConcurrencyallowSubdomains)r,r-r.r/rrrr1r0rrrrrrrrrrrrirrr[rrrrrrr2r3r"rrs-(,L(49%,(,L(49%,"Hhsm"'+x}+E8C=)--(,x~,)--$(M8D>(-1M8M*137GXeC./ 07-1HTN1,08D>0%)NHTN)E8C=$(NHSM(&*OXd^*r3rc\eZdZUdZdZeeed<dZeeed<dZ e ed<dZ eeed<y) CrawlResponsez"Response from crawling operations.NrrJTrr) r,r-r.r/rrr1r0rJrrrr2r3r"rrs9,B C#GTE8C=r3rc|eZdZUdZdZeed<eded<eed<eed<eed<e ed <d Z e e ed <e eed <y ) CrawlStatusResponsez"Response from crawl status checks.Trrrr]rrrNrrrr2r3r"rrsK,GT B CCN JD(3-  !!r3rc<eZdZUdZeeeefed<eeed<y)CrawlErrorsResponsez2Response from crawl/batch scrape error monitoring.errors robotsBlockedN)r,r-r.r/rrr1r0r2r3r"rr#s"< c3h  9r3rceZdZUdZdZeeed<dZee ed<dZ ee ed<dZ ee ed<dZ ee ed<dZee ed <dZee ed <y) MapParamsz"Parameters for mapping operations.NrrincludeSubdomains sitemapOnlyrrsrtuseIndex)r,r-r.r/rrr1r0rrrrrrrtrr2r3r"rr(sh, FHSM $(M8D>((,x~,"&K$&E8C="GXc]"#Hhtn#r3rcNeZdZUdZdZeed<dZee e ed<dZ ee ed<y) MapResponsez!Response from mapping operations.TrNrNr) r,r-r.r/rrr0rNrrr1rr2r3r"rr2s0+GT!%E8DI %E8C=r3rceZdZUdZdZeeed<ejddZ ee ed<dZ eeed<dZ eeed<dZeeed <dZeeed <dZeeed <dZeeed <dZeeed <y) ExtractParamsz0Parameters for extracting information from URLs.Nr+rerDrfrrenableWebSearchrorigin showSourcesr)r,r-r.r/r+rr1r0rGr rfrrrrrrrrrrir2r3r"rr8s: FHSM "0(..X"FL(3-F"&L(3-&)--&*OXd^*(,x~, FHSM "&K$&-1M8M*1r3rceZdZUdZdZeeed<dZee ded<dZ ee ed<dZ e ed<dZeeed <dZeeed <dZeeed <dZeeeefed <y) ExtractResponsez!Response from extract operations.Nr processingr]r^rrTrrrrsources)r,r-r.r/rrr1r0rr rr rrrr&rrrrrr2r3r"rrDs+B EIFHW@A BI$(Ix!(GTD(1+E8C=!GXc]!(,GXd38n %,r3rceZdZUeed<dZeeed<dZeeed<dZ eeed<dZ eeed<d Z eeed <dZ eeed <d Z eeed <dZeeed<dZeeed<y) SearchParamsqueryrNtbsfilterenlangusrXruapiri`rtr)r,r-r.r1r0rrrrr r rXrurrtrrir2r3r"rrOs JE8C=C# FHSM D(3-!GXc]!"Hhsm"!FHSM!"GXc]"-1M8M*1r3rcXeZdZUdZdZeed<eeed<dZ e e ed<dZ e e ed<y)SearchResponsez Response from search operations.TrrNrr) r,r-r.r/rrr0rrIrrr1rr2r3r"rr[s7*GT  !!!GXc]!E8C=r3rcbeZdZUdZdZeeed<dZee ed<dZ ee ed<dZ ee ed <y) GenerateLLMsTextParamsz; Parameters for the LLMs.txt generation operation. maxUrlsF showFullTextTcacheN,_GenerateLLMsTextParams__experimental_stream) r,r-r.r/rrrr0rrrrr2r3r"rrbsB GXc]#(L(4.( E8D> ,08D>0r3rceZdZUdZdZeeed<dZeeed<dZ eeed<dZ ee ed <dZ ee ed <dZ eeed <y) DeepResearchParamsz5 Parameters for the deep research operation. ri timeLimitrNanalysisPromptr-_DeepResearchParams__experimental_streamSteps)r,r-r.r/rrrr0rrrr1rrrr2r3r"rrks^ Hhsm"Ix}"GXc]$(NHSM("&L(3-&155r3rc:eZdZUdZeed<eed<dZeeed<y)DeepResearchResponsez4 Response from the deep research operation. rrNr) r,r-r.r/rr0r1rrr2r3r"rrvs!M GE8C=r3rceZdZUdZeed<dZeee e fed<e ed<dZ ee ed<e ed<e ed<e ed <e ee e fed <e ee e fed <e e ed <y) DeepResearchStatusResponsez; Status response from the deep research operation. rNrrrr currentDepthr activitiesr summaries)r,r-r.r/rr0rrrr1rrrrr2r3r"r!r!~sxM%)D(4S> ") KE8C=NMT#s(^$$ $sCx. !!Cyr3r!c>eZdZUdZdZeed<eed<dZe eed<y)GenerateLLMsTextResponsez-Response from LLMs.txt generation operations.TrrNr) r,r-r.r/rrr0r1rrr2r3r"r&r&s"7GT GE8C=r3r&c,eZdZUeed<dZeeed<y)"GenerateLLMsTextStatusResponseDatallmstxtN llmsfulltxt)r,r-r.r1r0r*rr2r3r"r(r(s L!%K#%r3r(cbeZdZUdZdZeed<dZee ed<e ded<dZ ee ed<e ed <y) GenerateLLMsTextStatusResponsez4Status response from LLMs.txt generation operations.TrNrrrrr) r,r-r.r/rrr0rrr(r rr1r2r3r"r,r,s>>GT9=D(5 6= 7 88E8C=Nr3r,c^eZdZUdZeed<eeee fed<dZ e eed<dZ e eed<y)rz- Response from the search operation. rrNrr) r,r-r.r/rr0rrr1rrrrr2r3r"rrs=M tCH~ !GXc]!E8C=r3ceZdZUdZdZeeed<ejddZ ee ed<dZ eeed<dZ eeed <dZeeed <dZeeed <dZeeed <dZeeee fed <y)rz/ Parameters for the extract operation. Nr+rerDrf system_promptFallow_external_linksenable_web_searchr show_sourcesr)r,r-r.r/r+rr1r0rGr rfrr/r0rr1rr2rrr2r3r"rrs!FHSM "0(..X"FL(3-F#'M8C='+0(4.0(-x~-&+OXd^+#(L(4.(&*E8DcN #*r3c2NeZdZddeedeeddfdZdddddddddddddddddddddddded eeed d eeeefd eeed eeedee dee dee dee dee dee dee dee deeddee dee dee deee eeeeeeeeef deedee dee dee d eedeef0d!Zddddddddd"d#ed$ee d%eed&eed'eed(eedeedee d)eedefd*Zddddddddddddddddddd+dd,ded-eeed.eeed/ee d0ee d$ee d1ee d2ee d3ee d4ee d)eed5ee eefd6ee d7ee d8ee d9ee d:ee d;ee dee dZ!dddddddddddddddddddd?ded-eeed.eeed/ee d0ee d$ee d1ee d2ee d3ee d4ee d)eed5ee eefd6ee d7ee d8ee d9ee d:ee d;ee dee d=eede"f*d@Z#dAede fdBZ$dAede%fdCZ&dAedeeeffdDZ'dddddddddddddddddddd?ded-eeed.eeed/ee d0ee d$ee d1ee d2ee d3ee d4ee d)eed5ee eefd6ee d7ee d8ee d9ee d:ee d;ee dee d=eeddEf*dFZ(ddddddddGdedHeed4ee dIee dJee d$ee dee dKee de)fdLZ*dddddddddddddddddd+ddddMdNeed eeedOd eeeefd eeed eeedee dee dee dee dee dee dee dee deeddee dee deee eeeeeeeeef d eedt||nMNr3rs)rnr\ include_tags exclude_tagsonly_main_contentwait_forrtrurvskip_tls_verificationremove_base64_images block_adsr~ parse_pdfrO json_optionsrRchange_tracking_optionsmax_agestore_in_cachezero_data_retentionrrJrnrkr\r@rArBrCrtrurvrDrErFr~rzrGrOrHrRrIrJrKrLrc \|j|d|j}|dtd}|r||d<|r||d<|r||d<|r||d<|||d <|r||d <|r||d <| r| jd d |d<| | |d<| | |d<| | |d<| | |d<|r||d<|||d<|d|j |}t |trd|vr|j |d|d<t |tr|n|jd d |d<|d|j |}t |trd|vr|j |d|d<t |tr|n|jd d |d<|r6|Dcgc]'}t |tr|n|jd d )c}|d<|r(t |tr|n|jd d |d<|||d<|||d<|||d<||jd d |d<|j |d|vr)|dr$d|dvr|j |dd|dd<d|vr)|dr$d|dvr|j |dd|dd<tj|jd||||dz d znd!}|jd"k(rW |j}|jd#rd$|vrtd)i|d$Sd%|vrtd&|d%td&||j!|d(ycc}w#t$r td'wxYw)*a Scrape and extract content from a URL. Args: url (str): Target URL to scrape formats (Optional[List[Literal["markdown", "html", "rawHtml", "content", "links", "screenshot", "screenshot@fullPage", "extract", "json"]]]): Content types to retrieve (markdown/html/etc) headers (Optional[Dict[str, str]]): Custom HTTP headers include_tags (Optional[List[str]]): HTML tags to include exclude_tags (Optional[List[str]]): HTML tags to exclude only_main_content (Optional[bool]): Extract main content only wait_for (Optional[int]): Wait for a specific element to appear timeout (Optional[int]): Request timeout (ms) location (Optional[LocationConfig]): Location configuration mobile (Optional[bool]): Use mobile user agent skip_tls_verification (Optional[bool]): Skip TLS verification remove_base64_images (Optional[bool]): Remove base64 images block_ads (Optional[bool]): Block ads proxy (Optional[Literal["basic", "stealth", "auto"]]): Proxy type (basic/stealth) extract (Optional[JsonConfig]): Content extraction settings json_options (Optional[JsonConfig]): JSON extraction settings actions (Optional[List[Union[WaitAction, ScreenshotAction, ClickAction, WriteAction, PressAction, ScrollAction, ScrapeAction, ExecuteJavascriptAction, PDFAction]]]): Actions to perform change_tracking_options (Optional[ChangeTrackingOptions]): Change tracking settings zero_data_retention (Optional[bool]): Whether to delete data after scrape is done agent (Optional[AgentOptions]): Agent configuration for FIRE-1 model Returns: ScrapeResponse with: * Requested content formats * Page metadata * Extraction results * Success/error status Raises: Exception: If scraping fails scrape_url python-sdk@rJrrnr\rorpNrqrrrtTby_alias exclude_nonerurvrwrxryr~rrerOrrRrrrzeroDataRetentionr /v1/scrape@@rr\rCrtrrrFailed to scrape URL. Error: +Failed to parse Firecrawl response as JSON.z scrape URLr2)_validate_kwargs_prepare_headersversiondict_ensure_schema_dict isinstanceupdaterequestspostr6 status_coderCgetrrr< _handle_error)r>rJrnr\r@rArBrCrtrurvrDrErFr~rGrOrHrRrIrJrKrLrkwargs_headers scrape_paramsactionresponse response_jsons r"rNzFirecrawlApp.scrape_urlsS@ fl3((*#G9- '.M) $ '.M) $ +7M- ( +7M- (  (/@M+ , '/M) $ '.M) $ (0 tRV (WM* %  &,M( # ,3HM/ 0  +2FM. /  (1M* % %*M' "  (1M* %  ..w7G'4(X-@$($<$rrrr r rXrurtrmrg search_params _integration final_params params_dictrkrls r"rzFirecrawlApp.searchdsN fh/   %*M' " ?#&M%  &,M( #  $(M& !  '.M) $  (0M* %  '.M) $  %-;-@-@$]a-@-bM/ * V$$((7 $A%A=A "''D'I "-gY 7 H )5K &==||nJ '$ ~&>?    3 & O (   $$Y/Fm4K):M:: -#&<]7=SrJryrzr{r|rr}r~r0rrmrrrrrrrrLrrrg crawl_paramsrurvrwr\rkrs r"rzFirecrawlApp.crawl_urlsX~ fk2   $+8L (  $+8L (  '0L $  *0CL, -  $)L !  *0CL, - ! -1EL- .  +1EL- .  %,:L )  %,:,?,?\`,?,aL )  &-L # # /5ML1 2 " .4KL0 1  (->L) *  $)L !  '.>L* +  &-F F)ryrzr{r|rr}r~r0rrmrrrrrrrrLrc |j|di}|||d<|||d<|||d<|||d<|||d<|||d<n|||d <| | |d <| | |d <| | jd d |d<| | |d<| | |d<|||d<|||d<|||d<|||d<|||d<|||d<|j|tdi|}|jd d }||d<dt|d<|j |}|j |jd||}|jdk(r tdi|jS|j|dy#tdxYw)a Start an asynchronous crawl job. Args: url (str): Target URL to start crawling from include_paths (Optional[List[str]]): Patterns of URLs to include exclude_paths (Optional[List[str]]): Patterns of URLs to exclude max_depth (Optional[int]): Maximum crawl depth max_discovery_depth (Optional[int]): Maximum depth for finding new URLs limit (Optional[int]): Maximum pages to crawl allow_backward_links (Optional[bool]): DEPRECATED: Use crawl_entire_domain instead crawl_entire_domain (Optional[bool]): Follow parent directory links allow_external_links (Optional[bool]): Follow external domain links ignore_sitemap (Optional[bool]): Skip sitemap.xml processing scrape_options (Optional[ScrapeOptions]): Page scraping configuration webhook (Optional[Union[str, WebhookConfig]]): Notification webhook settings deduplicate_similar_urls (Optional[bool]): Remove similar URLs ignore_query_parameters (Optional[bool]): Ignore URL parameters regex_on_full_url (Optional[bool]): Apply regex to full URLs delay (Optional[int]): Delay in seconds between scrapes allow_subdomains (Optional[bool]): Follow subdomains max_concurrency (Optional[int]): Maximum number of concurrent scrapes zero_data_retention (Optional[bool]): Whether to delete data after 24 hours idempotency_key (Optional[str]): Unique key to prevent duplicate requests **kwargs: Additional parameters to pass to the API Returns: CrawlResponse with: * success - Whether crawl started successfully * id - Unique identifier for the crawl job * url - Status check URL for the crawl * error - Error message if start failed Raises: Exception: If crawl initiation fails async_crawl_urlNrrrrrrrrrTrQrrrrrrrrrTrJrOrrrXrZrr2) r[r^rarr]r\rr6rdrrCrrf)r>rJryrzr{r|rr}r~r0rrmrrrrrrrrLrrgrrvrwr\rks r"rzFirecrawlApp.async_crawl_urlEs%| f&78   $+8L (  $+8L (  '0L $  *0CL, -  $)L !  *0CL, - ! -1EL- .  +1EL- .  %,:L )  %,:,?,?\`,?,aL )  &-L # # /5ML1 2 " .4KL0 1  (->L) *  $)L !  '.>L* +  &-rendpointr\rk status_datarnext_urlstatus_response next_dataes r"check_crawl_statuszFirecrawlApp.check_crawl_statussP4 t$'')$$ ~hZ%@'J   3 & P&mmo 8$ 3[(&v.D K/{623q8!#.??6#:'"NN+LM! ".2.?.?'.RO.::cA & /J?KfKfJg-h i %`,;,@,@,B !KK fb(AB*3K#!K/*+/K'&//(3$1(__[9*}=(__[9#/ H+%$/$8!$#.v#6 &!(K!7T    x)= >a P"MOO$`&/2]&_ _ )""LL+LQC)PQ!"s=G (AG+-G=#G+ G G((G++ H4HHc|j}|j|jd|d|}|jdk(r t di|j S|j|dy#t dxYw)aJ Returns information about crawl errors. Args: id (str): The ID of the crawl job Returns: CrawlErrorsResponse containing: * errors (List[Dict[str, str]]): List of errors with fields: - id (str): Error ID - timestamp (str): When the error occurred - url (str): URL that caused the error - error (str): Error message * robotsBlocked (List[str]): List of URLs blocked by robots.txt Raises: Exception: If error check fails r/errorsrXrZzcheck crawl errorsNr2r\rr6rdrrCrrfr>rr\rks r"check_crawl_errorszFirecrawlApp.check_crawl_errorss&'')$$ ~Zt7%KWU   3 & P*=X]]_==   x)= > P"MOO A// A<c|j}|j|jd||}|jdk(r |j S|j |dy#t dxYw)} Cancel an asynchronous crawl job. Args: id (str): The ID of the crawl job to cancel Returns: Dict[str, Any] containing: * success (bool): Whether cancellation was successful * error (str, optional): Error message if cancellation failed Raises: Exception: If cancellation fails rrXrZzcancel crawl jobN)r\_delete_requestr6rdrCrrfrs r" cancel_crawlzFirecrawlApp.cancel_crawl1sz'')''4<<. 2$(GQ   3 & P}}&   x); < P"MOOs A%% A2 CrawlWatcherc |j|fid|d|d|d|d|d|d|d| d | d | d | d | d |d|d|d|d|d|d||}|jr"|jrt|j|St d)aG Initiate a crawl job and return a CrawlWatcher to monitor the job via WebSocket. Args: url (str): Target URL to start crawling from include_paths (Optional[List[str]]): Patterns of URLs to include exclude_paths (Optional[List[str]]): Patterns of URLs to exclude max_depth (Optional[int]): Maximum crawl depth max_discovery_depth (Optional[int]): Maximum depth for finding new URLs limit (Optional[int]): Maximum pages to crawl allow_backward_links (Optional[bool]): DEPRECATED: Use crawl_entire_domain instead crawl_entire_domain (Optional[bool]): Follow parent directory links allow_external_links (Optional[bool]): Follow external domain links ignore_sitemap (Optional[bool]): Skip sitemap.xml processing scrape_options (Optional[ScrapeOptions]): Page scraping configuration webhook (Optional[Union[str, WebhookConfig]]): Notification webhook settings deduplicate_similar_urls (Optional[bool]): Remove similar URLs ignore_query_parameters (Optional[bool]): Ignore URL parameters regex_on_full_url (Optional[bool]): Apply regex to full URLs delay (Optional[int]): Delay in seconds between scrapes allow_subdomains (Optional[bool]): Follow subdomains max_concurrency (Optional[int]): Maximum number of concurrent scrapes zero_data_retention (Optional[bool]): Whether to delete data after 24 hours idempotency_key (Optional[str]): Unique key to prevent duplicate requests **kwargs: Additional parameters to pass to the API Returns: CrawlWatcher: An instance to monitor the crawl job via WebSocket Raises: Exception: If crawl job fails to start ryrzr{r|rr}r~r0rrmrrrrrrrrLrCrawl job failed to start)rrrrr)r>rJryrzr{r|rr}r~r0rrmrrrrrrrrLrrgcrawl_responses r"crawl_url_and_watchz FirecrawlApp.crawl_url_and_watchJs r.--  ' (   !4    "6 !4 "6 * *  &> %< 0 ! ".# $,% &!4' (,+ .  ! !n&7&7 1 148 878 8r3)rrinclude_subdomains sitemap_onlyrrt use_indexrrrrc |j| di} ||| d<||| d<||| d<||| d<||| d<||| d<||| d <| j| | jd } tdi| } | j d d } || d <dt | d<| r| | d <t j|jddd|ji| }|jdk(rT |j}|jdrd|vr tdi|Sd|vrtd|dtd||j|dy#t$r tdwxYw)a} Map and discover links from a URL. Args: url (str): Target URL to map search (Optional[str]): Filter pattern for URLs ignore_sitemap (Optional[bool]): Skip sitemap.xml processing include_subdomains (Optional[bool]): Include subdomain links sitemap_only (Optional[bool]): Only use sitemap.xml limit (Optional[int]): Maximum URLs to return timeout (Optional[int]): Request timeout in milliseconds **kwargs: Additional parameters to pass to the API Returns: MapResponse: Response containing: * success (bool): Whether request succeeded * links (List[str]): Discovered URLs * error (Optional[str]): Error message if any Raises: Exception: If mapping fails or response cannot be parsed map_urlNrrrrrrtrroTrQrJrOr/v1/maprqrrrsrXrrNrzMap failed. Error: rZmapr2)r[rarerr^r]rbrcr6r5rdrCrrr<rf)r>rJrrrrrrtrrg map_paramsrurvrwrkrls r"rzFirecrawlApp.map_urlsF fi0   #)Jx  %*8J '  ).@J* +  #(4J} %  "'Jw   $+Jy !  %.Jz " &!!~~m4 !.:. "''D'I  E"-gY 7 H )5K &==||nG $$ ~&>?    3 & O (   $$Y/G}4L&777 -#&9-:P9Q$RSS#&9-$IJJ   x / O MNN Os"/E#EE)rnr\r@rArBrCrtrurvrDrErFr~rOrHrRrrrrLrurls rKrLrMrlrNrPrmrOrCc |j|di}|||d<|||d<|||d<|||d<|||d<|||d<|||d <| | jd d |d <| | |d <| | |d<| | |d<| | |d<|||d<|d|j|}t|trd|vr|j|d|d<t|tr|n|jd d |d<|d|j|}t|trd|vr|j|d|d<t|tr|n|jd d |d<|r6|Dcgc]'}t|tr|n|jd d )c}|d<||jd d |d<|||d<|||d<|j |t d!i|}|jd d }||d<dt |d<d|vr)|dr$d|dvr|j|dd|dd<d|vr)|dr$d|dvr|j|dd|dd<|j|}|j|jd||}|jdk(r3 |jjd}|j|||S|j|d ycc}w#tdxYw)"aF Batch scrape multiple URLs and monitor until completion. Args: urls (List[str]): URLs to scrape formats (Optional[List[Literal]]): Content formats to retrieve headers (Optional[Dict[str, str]]): Custom HTTP headers include_tags (Optional[List[str]]): HTML tags to include exclude_tags (Optional[List[str]]): HTML tags to exclude only_main_content (Optional[bool]): Extract main content only wait_for (Optional[int]): Wait time in milliseconds timeout (Optional[int]): Request timeout in milliseconds location (Optional[LocationConfig]): Location configuration mobile (Optional[bool]): Use mobile user agent skip_tls_verification (Optional[bool]): Skip TLS verification remove_base64_images (Optional[bool]): Remove base64 encoded images block_ads (Optional[bool]): Block advertisements proxy (Optional[Literal]): Proxy type to use extract (Optional[JsonConfig]): Content extraction config json_options (Optional[JsonConfig]): JSON extraction config actions (Optional[List[Union]]): Actions to perform agent (Optional[AgentOptions]): Agent configuration max_concurrency (Optional[int]): Maximum number of concurrent scrapes poll_interval (Optional[int]): Seconds between status checks (default: 2) idempotency_key (Optional[str]): Unique key to prevent duplicate requests **kwargs: Additional parameters to pass to the API Returns: BatchScrapeStatusResponse with: * Scraping status and progress * Scraped content for each URL * Success/error information Raises: Exception: If batch scrape fails batch_scrape_urlsNrnr\rorprqrrrtTrQrurvrwrxryr~rerOrrRrrrTrrOr/v1/batch/scraperXrrZstart batch scrape jobr2)r[r^r_r`rarr]r\rr6rdrCrerrrf)r>rrnr\r@rArBrCrtrurvrDrErFr~rOrHrRrrrrLrrgrirjrvrwrkrs r"rzFirecrawlApp.batch_scrape_urlss@ f&9:   '.M) $  '.M) $  #+7M- (  #+7M- (  (/@M+ ,  '/M) $  '.M) $  (0 tRV (WM* %  &,M( # ,3HM/ 0  +2FM. /  (1M* %  %*M' "  ..w7G'4(X-@$($<$8{[dOeCe/3/G/G T]H^_gHh/iK "8 , K 'K ,F8WbcpWqKq373K3KKXeLfgoLp3qK &x 0''8%%6F&GV]^   3 & P]]_((.++BG G   x)A BC(U: P"MOOs,K K K)rnr\r@rArBrCrtrurvrDrErFr~rOrHrRrrrrLc |j|di}|||d<|||d<|||d<|||d<|||d<|||d<|||d <| | jd d |d <| | |d <| | |d<| | |d<| | |d<|||d<|d|j|}t|trd|vr|j|d|d<t|tr|n|jd d |d<|d|j|}t|trd|vr|j|d|d<t|tr|n|jd d |d<|r6|Dcgc]'}t|tr|n|jd d )c}|d<||jd d |d<|||d<|||d<|j |t d i|}|jd d }||d<dt |d<d|vr)|dr$d|dvr|j|dd|dd<d|vr)|dr$d|dvr|j|dd|dd<|j|}|j|jd||}|jdk(r td i|jS|j|dycc}w#tdxYw)!a| Initiate a batch scrape job asynchronously. Args: urls (List[str]): URLs to scrape formats (Optional[List[Literal]]): Content formats to retrieve headers (Optional[Dict[str, str]]): Custom HTTP headers include_tags (Optional[List[str]]): HTML tags to include exclude_tags (Optional[List[str]]): HTML tags to exclude only_main_content (Optional[bool]): Extract main content only wait_for (Optional[int]): Wait time in milliseconds timeout (Optional[int]): Request timeout in milliseconds location (Optional[LocationConfig]): Location configuration mobile (Optional[bool]): Use mobile user agent skip_tls_verification (Optional[bool]): Skip TLS verification remove_base64_images (Optional[bool]): Remove base64 encoded images block_ads (Optional[bool]): Block advertisements proxy (Optional[Literal]): Proxy type to use extract (Optional[JsonConfig]): Content extraction config json_options (Optional[JsonConfig]): JSON extraction config actions (Optional[List[Union]]): Actions to perform agent (Optional[AgentOptions]): Agent configuration max_concurrency (Optional[int]): Maximum number of concurrent scrapes zero_data_retention (Optional[bool]): Whether to delete data after 24 hours idempotency_key (Optional[str]): Unique key to prevent duplicate requests **kwargs: Additional parameters to pass to the API Returns: BatchScrapeResponse with: * success - Whether job started successfully * id - Unique identifier for the job * url - Status check URL * error - Error message if start failed Raises: Exception: If job initiation fails async_batch_scrape_urlsNrnr\rorprqrrrtTrQrurvrwrxryr~rerOrrRrrrTrrOrrrXrZrr2)r[r^r_r`rarr]r\rr6rdrrCrrf)r>rrnr\r@rArBrCrtrurvrDrErFr~rOrHrRrrrrLrgrirjrvrwrks r"rz$FirecrawlApp.async_batch_scrape_urlss@ f&?@   '.M) $  '.M) $  #+7M- (  #+7M- (  (/@M+ ,  '/M) $  '.M) $  (0 tRV (WM* %  &,M( # ,3HM/ 0  +2FM. /  (1M* %  %*M' "  ..w7G'4(X-@$($<$8{[dOeCe/3/G/G T]H^_gHh/iK "8 , K 'K ,F8WbcpWqKq373K3KKXeLfgoLp3qK &x 0''8%%6F&GV]^   3 & P*=X]]_==   x)A BA(U: P"MOOs,J2J77 K)rnr\r@rArBrCrtrurvrDrErFr~rOrHrRrrrLrc |j|di}|||d<|||d<|||d<|||d<|||d<|||d<|||d <| | jd d |d <| | |d <| | |d<| | |d<| | |d<|||d<|d|j|}t|trd|vr|j|d|d<t|tr|n|jd d |d<|d|j|}t|trd|vr|j|d|d<t|tr|n|jd d |d<|r6|Dcgc]'}t|tr|n|jd d )c}|d<||jd d |d<|||d<|||d<|j |t d!i|}|jd d }||d<dt |d<d|vr)|dr$d|dvr|j|dd|dd<d|vr)|dr$d|dvr|j|dd|dd<|j|}|j|jd||}|jdk(rS td!i|j}|jr"|jrt|j|St!d|j#|d ycc}w#t!dxYw)"a Initiate a batch scrape job and return a CrawlWatcher to monitor the job via WebSocket. Args: urls (List[str]): URLs to scrape formats (Optional[List[Literal]]): Content formats to retrieve headers (Optional[Dict[str, str]]): Custom HTTP headers include_tags (Optional[List[str]]): HTML tags to include exclude_tags (Optional[List[str]]): HTML tags to exclude only_main_content (Optional[bool]): Extract main content only wait_for (Optional[int]): Wait time in milliseconds timeout (Optional[int]): Request timeout in milliseconds location (Optional[LocationConfig]): Location configuration mobile (Optional[bool]): Use mobile user agent skip_tls_verification (Optional[bool]): Skip TLS verification remove_base64_images (Optional[bool]): Remove base64 encoded images block_ads (Optional[bool]): Block advertisements proxy (Optional[Literal]): Proxy type to use extract (Optional[JsonConfig]): Content extraction config json_options (Optional[JsonConfig]): JSON extraction config actions (Optional[List[Union]]): Actions to perform agent (Optional[AgentOptions]): Agent configuration max_concurrency (Optional[int]): Maximum number of concurrent scrapes zero_data_retention (Optional[bool]): Whether to delete data after 24 hours idempotency_key (Optional[str]): Unique key to prevent duplicate requests **kwargs: Additional parameters to pass to the API Returns: CrawlWatcher: An instance to monitor the batch scrape job via WebSocket Raises: Exception: If batch scrape job fails to start batch_scrape_urls_and_watchNrnr\rorprqrrrtTrQrurvrwrxryr~rerOrrRrrrTrrOrrrX Batch scrape job failed to startrZrr2)r[r^r_r`rarr]r\rr6rdrrCrrrrrf)r>rrnr\r@rArBrCrtrurvrDrErFr~rOrHrRrrrLrrgrirjrvrwrkrs r"rz(FirecrawlApp.batch_scrape_urls_and_watchsx f&CD   '.M) $  '.M) $  #+7M- (  #+7M- (  (/@M+ ,  '/M) $  '.M) $  (0 tRV (WM* %  &,M( # ,3HM/ 0  +2FM. /  (1M* %  %*M' "  ..w7G'4(X-@$($<$8{[dOeCe/3/G/G T]H^_gHh/iK "8 , K 'K ,F8WbcpWqKq373K3KKXeLfgoLp3qK &x 0''8%%6F&GV]^   3 & P!4!Gx}}!G!))n.?.?'(9(94@@#$FGG   x)A BI(UB P"MOOs,K+AK0 K00 K=c<d|}|j}|j|j||}|jdk(rw |j }|ddk(rd|vr|d}d|vrt |ddk(rn|jd}|stjd n~ |j||}|jdk7r#tjd |jn9 |j } |j| jdg| }d|vr||d<tdid |vrdnd|jd|jd|jd|jd|jd|jd|jd|jd d S|j|dy #t dxYw#t dxYw#t $r"} tjd | Yd } ~ d } ~ wwxYw)a> Check the status of a batch scrape job using the Firecrawl API. Args: id (str): The ID of the batch scrape job. Returns: BatchScrapeStatusResponse: The status of the batch scrape job. Raises: Exception: If the status check request fails. /v1/batch/scrape/rXrZrr]rrrrrrNrFTrrr) rrrr]rrrrrzcheck batch scrape statusr2)r\rr6rdrCrrrer%rrrrrfrs r"check_batch_scrape_statusz&FirecrawlApp.check_batch_scrape_statuss,'rd+'')$$ ~hZ%@'J   3 & P&mmo 8$ 3[(&v.D K/{623q8!#.??6#:'"NN+LM! ".2.?.?'.RO.::cA & /J?KfKfJg-h i %`,;,@,@,B !KK fb(AB*3K#!K/*+/K', $+{$:5%//(3$1(__[9*}=(__[9#/#/$1 0     x)D EQ P"MOO$`&/2]&_ _ )""LL+LQC)PQ!"s=G(AG0-G =#G0 G G--G00 H9HHc|j}|j|jd|d|}|jdk(r t di|j S|j|dy#t dxYw)aV Returns information about batch scrape errors. Args: id (str): The ID of the crawl job. Returns: CrawlErrorsResponse containing: * errors (List[Dict[str, str]]): List of errors with fields: * id (str): Error ID * timestamp (str): When the error occurred * url (str): URL that caused the error * error (str): Error message * robotsBlocked (List[str]): List of URLs blocked by robots.txt Raises: Exception: If the error check request fails rrrXrZzcheck batch scrape errorsNr2rrs r"check_batch_scrape_errorsz&FirecrawlApp.check_batch_scrape_errorss&'')$$ ~5Frd'%RT[\   3 & P*=X]]_==   x)D E P"MOOrFr+rer/r0r1r2rr+rer/r1r2c |j| d|j} |s |s td|s |s td|r|j|}|xsg||||dt d} |r|| d<|r|| d<|r|| d<| j |  |j |jd | | } | jd k(r | j} | d r| jd }|s td |j|jd|| }|jd k(rB |j}|ddk(r tdi|S|ddvr)td|dd|d|j|dtj dtd| d|j| d tddS#td xYw#td xYw#t$r}tt#|dd}~wwxYw)a Extract structured information from URLs. Args: urls (Optional[List[str]]): URLs to extract from prompt (Optional[str]): Custom extraction prompt schema (Optional[Any]): JSON schema/Pydantic model system_prompt (Optional[str]): System context allow_external_links (Optional[bool]): Follow external links enable_web_search (Optional[bool]): Enable web search show_sources (Optional[bool]): Include source URLs agent (Optional[Dict[str, Any]]): Agent configuration **kwargs: Additional parameters to pass to the API Returns: ExtractResponse[Any] with: * success (bool): Whether request succeeded * data (Optional[Any]): Extracted data matching schema * error (Optional[str]): Error message if any Raises: ValueError: If prompt/schema missing or extraction fails rO#Either prompt or schema is required!Either urls or prompt is requiredrOrrrrrerr+rr /v1/extractrXrZrr)Job ID not returned from extract request. /v1/extract/rr]r^r Extract job . Error: rzextract-statusrxFailed to extract. Error: NFzInternal server error.rrr2)r[r\r<r_r#rarr6rdrCrrerrrftimesleepr1)r>rr+rer/r0r1r2rrgr\ request_datarkrjob_idrrrs r"rOzFirecrawlApp.extractslH fi0'')fBC CF@A A --f5FJB"60'#KM?3   %+L " +8L ( $)L ! F#) *))<<. ,H ##s*T#==?D ?!XXd^F!'(STT*.*;*;#||nLA#++66#=`.=.B.B.D  +84 C'6'E'E E!,X!6:Q!Q&/,{8?T>UU^_jkr_s^t0u&v v ..@PQ 1 #&$&@g$PQQ""8Y7u4LMMCT#&QSS`&/2]&_ _ *SVS) ) *sP/G"GAG"(G8G" A)G" GG" GG"" H+HHrcJ|j} |j|jd||}|jdk(r t di|j S|j|dy#t dxYw#t $r}tt|dd}~wwxYw)a$ Retrieve the status of an extract job. Args: job_id (str): The ID of the extract job. Returns: ExtractResponse[Any]: The status of the extract job. Raises: ValueError: If there is an error retrieving the status. rrXrZzget extract statusrNr2) r\rr6rdrrCrrfr<r1)r>rr\rkrs r"get_extract_statuszFirecrawlApp.get_extract_statusls'') *((DLL>fX)NPWXH##s*T*=X]]_==""8-ABT#&QSS *SVS) ) *s/0A?A/A?/ A<<A?? B"BB"c|j} |}|r|j|}|||||dtd} |r|| d<|r|| d<|r|| d< |j|jd| | } | j dk(r t d i| jS|j| d y #tdxYw#t$r} tt| d d } ~ wwxYw) a Initiate an asynchronous extract job. Args: urls (List[str]): URLs to extract information from prompt (Optional[str]): Custom extraction prompt schema (Optional[Any]): JSON schema/Pydantic model system_prompt (Optional[str]): System context allow_external_links (Optional[bool]): Follow external links enable_web_search (Optional[bool]): Enable web search show_sources (Optional[bool]): Include source URLs agent (Optional[Dict[str, Any]]): Agent configuration idempotency_key (Optional[str]): Unique key to prevent duplicate requests Returns: ExtractResponse[Any] with: * success (bool): Whether request succeeded * data (Optional[Any]): Extracted data matching schema * error (Optional[str]): Error message if any Raises: ValueError: If job initiation fails rOrr+rrrrXrZz async extractrNr2) r\r_r]rr6rdrrCrrfr<r1) r>rr+rer/r0r1r2rr\rrkrs r" async_extractzFirecrawlApp.async_extracts D'') --f5F"60'#G9-   %+L " +8L ( $)L ! *))T\\N+*F V]^H##s*T*=X]]_==""8_=T#&QSS *SVS) ) *s0 /B8<B(B8( B55B88 CCCmax_urlsshow_full_textrexperimental_streamrrrrct||||}|j|||||}|jr |jst ddddS|j} |j |} | j dk(r| S| j dk(r| S| j d k7rt dd ddStjd g) a Generate LLMs.txt for a given URL and poll until completion. Args: url (str): Target URL to generate LLMs.txt from max_urls (Optional[int]): Maximum URLs to process (default: 10) show_full_text (Optional[bool]): Include full text in output (default: False) cache (Optional[bool]): Whether to use cached content if available (default: True) experimental_stream (Optional[bool]): Enable experimental streaming Returns: GenerateLLMsTextStatusResponse with: * Generated LLMs.txt content * Full version if requested * Generation status * Success/error information Raises: Exception: If generation fails rrr__experimental_streamrFz#Failed to start LLMs.txt generationr^rrrrr]r/LLMs.txt generation job terminated unexpectedlyrx) rasync_generate_llms_textrrr,check_generate_llms_text_statusrrr) r>rJrrrrparamsrkrrs r"generate_llms_textzFirecrawlApp.generate_llms_texts8('"5  00 ) 3 1 x{{1;  99&AF}} + (* ,.5!K#  JJqMr3ct||||}|j}d|i|jdd}dt|d< |j |j d||} | j } td|td | | jd r tdi| S|j| d  tddS#td xYw#t$r} tt| d } ~ wwxYw)a Initiate an asynchronous LLMs.txt generation operation. Args: url (str): The target URL to generate LLMs.txt from. Must be a valid HTTP/HTTPS URL. max_urls (Optional[int]): Maximum URLs to process (default: 10) show_full_text (Optional[bool]): Include full text in output (default: False) cache (Optional[bool]): Whether to use cached content if available (default: True) experimental_stream (Optional[bool]): Enable experimental streaming Returns: GenerateLLMsTextResponse: A response containing: * success (bool): Whether the generation initiation was successful * id (str): The unique identifier for the generation job * error (str, optional): Error message if initiation failed Raises: Exception: If the generation job initiation fails. rrJTrQrOr /v1/llmstxt json_datarkrrZzstart LLMs.txt generationNFInternal server errorrr2)rr\r^r]rr6rCrrer&rrfr<r1) r>rJrrrrrr\rreqrkrs r"rz%FirecrawlApp.async_generate_llms_texts6('"5  '')CQ6;;4;#PQ +G95 ( %$$ ~[%A9gVCxxzH +y ) *h '||I&S3?h??""8-HI()  S#$QRR %SV$ $ %s1AC C )C CC C;"C66C;c|j} |j|jd||}|jdk(r |j }t d i|S|jdk(r t d|j|d t dd d d S#t $r}t dt|d}~wwxYw#t $r}tt|d}~wwxYw)a; Check the status of a LLMs.txt generation operation. Args: id (str): The unique identifier of the LLMs.txt generation job to check status for. Returns: GenerateLLMsTextStatusResponse: A response containing: * success (bool): Whether the generation was successful * status (str): Status of generation ("processing", "completed", "failed") * data (Dict[str, str], optional): Generated text with fields: * llmstxt (str): Generated LLMs.txt content * llmsfulltxt (str, optional): Full version if requested * error (str, optional): Error message if generation failed * expiresAt (str): When the generated data expires Raises: Exception: If the status check fails. /v1/llmstxt/rXzFFailed to parse Firecrawl response as GenerateLLMsTextStatusResponse: Nz!LLMs.txt generation job not foundz check LLMs.txt generation statusFrr^rrr2) r\rr6rdrCr,rr1rfr<)r>rr\rkrrs r"rz,FirecrawlApp.check_generate_llms_text_statusIs('') %((DLL>bT)JGTH##s*w ( I9FIFF%%, CDD""8-OP.eCZckwyzz!w#&lmpqrmslt$uvvw  %SV$ $ %s;0CB,C B?#B::B??C C$ CC$cP|rdd|j|dSdd|jdS)a$ Prepare the headers for API requests. Args: idempotency_key (Optional[str]): A unique key to ensure idempotency of requests. Returns: Dict[str, str]: The headers including content type, authorization, and optionally idempotency key. zapplication/jsonrr) Content-Typerqzx-idempotency-key)rrq)r5)r>rs r"r\zFirecrawlApp._prepare_headersosB  2#*4<<.!9%4 /&t||n5  r3rretriesbackoff_factorct|D]]}tj|||d|vr|d |ddz dznd}|jdk(rt j |d|zz[|cSS)a^ Make a POST request with retries. Args: url (str): The URL to send the POST request to. data (Dict[str, Any]): The JSON data to include in the POST request. headers (Dict[str, str]): The headers to include in the POST request. retries (int): Number of retries for the request. backoff_factor (float): Backoff factor for retries. Returns: requests.Response: The response from the POST request. Raises: requests.RequestException: If the request fails after the specified retries. rtNrVrrWrx)rangerbrcrdrr)r>rJrr\rrattemptrks r"rzFirecrawlApp._post_requests.W~ G}}S'qzCrCHLMVHWHcPTU^P_bhPhklPlimoH##s* >Q'\:;  r3ct|D]G}tj||}|jdk(rt j |d|zzE|cSS)a  Make a GET request with retries. Args: url (str): The URL to send the GET request to. headers (Dict[str, str]): The headers to include in the GET request. retries (int): Number of retries for the request. backoff_factor (float): Backoff factor for retries. Returns: requests.Response: The response from the GET request. Raises: requests.RequestException: If the request fails after the specified retries. r\rrx)rrbrerdrrr>rJr\rrrrks r"rzFirecrawlApp._get_requestsV*W~ G||C9H##s* >Q'\:;  r3ct|D]G}tj||}|jdk(rt j |d|zzE|cSS)a Make a DELETE request with retries. Args: url (str): The URL to send the DELETE request to. headers (Dict[str, str]): The headers to include in the DELETE request. retries (int): Number of retries for the request. backoff_factor (float): Backoff factor for retries. Returns: requests.Response: The response from the DELETE request. Raises: requests.RequestException: If the request fails after the specified retries. rrrx)rrbdeleterdrrrs r"rzFirecrawlApp._delete_requestsV*W~ GsGQ'\:;  r3c |jd|}|j||}|jdk(r |j}|ddk(rd|vrw|d}d|vr^t |ddk(rnL|j|d|} |j}|j |jdgd|vr^||d<tdi|St d |dd vr"t|d }tj|n#t d |d|j|d #t dxYw#t dxYw)a Monitor the status of a crawl job until completion. Args: id (str): The ID of the crawl job. headers (Dict[str, str]): The headers to include in the status check requests. poll_interval (int): Seconds between status checks. Returns: CrawlStatusResponse: The crawl results if the job is completed successfully. Raises: Exception: If the job fails or an error occurs during status checks. rrXrZrr]rrrz,Crawl job completed but no data was returnedactivepausedpendingqueuedwaitingrrxz)Crawl job failed or was stopped. Status: rr2) r6rrdrCrrrrermaxrrrf)r>rr\rr6rrrs r"rz FirecrawlApp._monitor_job_statuss&j5G"//AO**c1T"1"6"6"8Kx(K7,*62$ 3";v#671< %.2.?.? F@SU\.]O`.=.B.B.D !KK (CD% 3/3 F+2A[AA'(VWW *.nn"%mA"6MJJ}-#&OP[\dPeOf$ghh""?4HI?T#&QSS`&/2]&_ _sD!D1! D.1 D>rkrjc |j}|jdd}|jdd}|j |j|||}tjj|| # |jdd}|jrd|}d|j}nd |j}d }n #t $rd |j}d }YnwxYwYxYw) ah Handle errors from API responses. Args: response (requests.Response): The response object from the API request. action (str): Description of the action that was being performed. Raises: Exception: An exception with a message containing the status code and error details from the response. rNo error message provided.details%No additional error details provided.Nrz#Server returned non-JSON response: zFull response status: z+Server returned empty response with status zNo additional details availablez0Server returned unreadable response with status )rk) rCrerrrdr<_get_error_messagerb exceptions HTTPError)r>rkrjrl error_message error_details response_textmessages r"rfzFirecrawlApp._handle_error s B$MMOM)--g7STM)--i9`aM))(*>*> Wde!!++Gh+GG# B B ( ds 3  &&(&I-$YM&=C >CC CC rdrrc|dk(r d|d|d|S|dk(r d|d|d|S|dk(r d|d |d|S|d k(r d |d |d|S|d k(r d|d|d|Sd|d|d|d|S)a Generate a standardized error message based on HTTP status code. Args: status_code (int): The HTTP status code from the response action (str): Description of the action that was being performed error_message (str): The error message from the API response error_details (str): Additional error details from the API response Returns: str: A formatted error message izPayment Required: Failed to z. z - iz!Website Not Supported: Failed to izRequest Timeout: Failed to z as the request timed out. izConflict: Failed to z due to a conflict. rz!Internal Server Error: Failed to zUnexpected error during z: Status code r2r>rdrjrrs r"rzFirecrawlApp._get_error_message9 s # 1&M?#m_] ] C 6vhbsS`Rab b C 08STaSbbefsetu u C )&1Em_TWXeWfg g C 6vhbsS`Rab b-fX^K=PRS`Raaderdst tr3)r{ time_limitranalysis_promptr/(_FirecrawlApp__experimental_stream_steps on_activity on_sourcerrrrrcTi} ||| d<||| d<||| d<||| d<||| d<||| d<tdi| } |j||||||} | jd rd | vr| S| d } d } d } |j| }|r)d |vr%|d | d}|D] }|| t |d } | r)d |vr%|d |d}|D] }| | t |d }|ddk(r|S|ddk(rt d|jd|ddk7rnt jddddSa Initiates a deep research operation on a given query and polls until completion. Args: query (str): Research query or topic to investigate max_depth (Optional[int]): Maximum depth of research exploration time_limit (Optional[int]): Time limit in seconds for research max_urls (Optional[int]): Maximum number of URLs to process analysis_prompt (Optional[str]): Custom prompt for analysis system_prompt (Optional[str]): Custom system prompt __experimental_stream_steps (Optional[bool]): Enable experimental streaming on_activity (Optional[Callable]): Progress callback receiving {type, status, message, timestamp, depth} on_source (Optional[Callable]): Source discovery callback receiving {url, title, description} Returns: DeepResearchStatusResponse containing: * success (bool): Whether research completed successfully * status (str): Current state (processing/completed/failed) * error (Optional[str]): Error message if failed * id (str): Unique identifier for the research job * data (Any): Research findings and analysis * sources (List[Dict]): List of discovered sources * activities (List[Dict]): Research progress log * summaries (List[str]): Generated research summaries Raises: Exception: If research fails Nrrrrr__experimental_streamSteps)r{rrrr/rrrr#rrr]r^zDeep research failed. Error: rrrxFz)Deep research job terminated unexpectedlyrr2)rasync_deep_researchrecheck_deep_research_statusrrrr)r>rr{rrrr/rrrresearch_paramsrkrlast_activity_countlast_source_countrnew_activitiesactivity new_sourcessources r" deep_researchzFirecrawlApp.deep_researchS sP  *3OJ '  !+5OK (  )1OI &  &0?O, -  $.;ON + & 2O$44V&?#Y&0$Y/0A0BC )&Ff%&$'y(9$:!h;. !X-"? 7@S?T UVV!\1 JJqM-0!+VWWr3)r{rrrr/rc`i}|||d<|||d<|||d<|||d<|||d<|||d<tdi|}|j} d|i|jd d } d t| d <d | vr3| d } | r,d| vr(t | ddr| dj | d d< |j |jd| | } | jdk(r | jS|j| d dddS#tdxYw#t$r} tt| d} ~ wwxYw) Initiates an asynchronous deep research operation. Args: query (str): Research query or topic to investigate max_depth (Optional[int]): Maximum depth of research exploration time_limit (Optional[int]): Time limit in seconds for research max_urls (Optional[int]): Maximum number of URLs to process analysis_prompt (Optional[str]): Custom prompt for analysis system_prompt (Optional[str]): Custom system prompt __experimental_stream_steps (Optional[bool]): Enable experimental streaming Returns: Dict[str, Any]: A response containing: * success (bool): Whether the research initiation was successful * id (str): The unique identifier for the research job * error (str, optional): Error message if initiation failed Raises: Exception: If the research initiation fails. NrrrrrrrTrQrOrrre/v1/deep-researchrXrZzstart deep researchFrrr2)rr\r^r]hasattrrerr6rdrCrrfr<r1)r>rr{rrrr/rr!r\r json_optsrkrs r"rz FirecrawlApp.async_deep_research s>  *3OJ '  !+5OK (  )1OI &  &0?O, -  $.;ON + & 2x5H5O5O5Q -(2 %))T\\N:K*LiY`aH##s*S#==?*""8-BC!+BCCS#$QRR %SV$ $ %s0#/D C;#D ; DD D-D((D-ct|j} |j|jd||}|jdk(r |j S|jdk(r t d|j |d dd d S#t dxYw#t $r}tt|d}~wwxYw)  Check the status of a deep research operation. Args: id (str): The ID of the deep research operation. Returns: DeepResearchResponse containing: Status: * success - Whether research completed successfully * status - Current state (processing/completed/failed) * error - Error message if failed Results: * id - Unique identifier for the research job * data - Research findings and analysis * sources - List of discovered sources * activities - Research progress log * summaries - Generated research summaries Raises: Exception: If the status check fails. /v1/deep-research/rXrZrzDeep research job not foundzcheck deep research statusNFrr) r\rr6rdrCrrfr<r1)r>rr\rkrs r"r z'FirecrawlApp.check_deep_research_status s2'') %((DLL>9KB4)PRYZH##s*S#==?*%%, =>>""8-IJ!+BCCS#$QRR  %SV$ $ %s/0BB,B BB B7B22B7rg method_namec |syhdhdhdhdhdhdhdhdd}|j|t}t|j|z }|r!td |d d j |d y) a Validate additional keyword arguments before they are passed to the API. This provides early validation before the Pydantic model validation. Args: kwargs (Dict[str, Any]): Additional keyword arguments to validate method_name (str): Name of the method these kwargs are for Raises: ValueError: If kwargs contain invalid or unsupported parameters N>rr~rvrRrOrnrJrtrurCrFrorAr@rHrBrErDrI> rr rr rXrtrurorm>rrr{rorzryrrmrr|r}r0rr>rrrtrorrr>rr+reror2r/r1r0>rr~rvrRrOrnr\rtrrurCrFrAr@rHrBrErD)rNrrrrOrrrzUnsupported parameter(s) for z: z, zC. Please refer to the API documentation for the correct parameters.)resetkeysr<r)r>rgr1 method_paramsallowed_paramsunknown_paramss r"r[zFirecrawlApp._validate_kwargs$ s T|}R"@(F,J% 2'**;>V[[]+n< <[MDIIVdLeKfgjkl l r3c||St|tr8t|dr|jSt|dr|j St|t r3|j Dcic]\}}||j|c}}St|ttfr|Dcgc]}|j|c}S|Scc}}wcc}w)zw Utility to ensure a schema is a dict, not a Pydantic model class. Recursively checks dicts and lists. model_json_schemare) r`rr,r9rer^itemsr_listtuple)r>rekvs r"r_z FirecrawlApp._ensure_schema_dictW s >M fd #v23//11*}}& fd #?E||~Ntq!At//22N N ftUm ,9?@AD,,Q/@ @ O@s 0C*C NNN?)Lr,r-r.rr1r?rr rrrrWrrrrrrrrrrrrcr(rrrNrirrr[rrrrrrrrrrrrrrrrrrrrOrrr,rr&rrr\rrbResponserrrrrfrrr!r(rr r[r_r2r3r"r4r4s O Ox}OX\O,mq04040404&*%*15%)4837(,CG(,,015swGK%)-126,03Y7Y7d7,g$hij Y7 d38n- Y7 #49- Y7#49-Y7 (~Y7smY7c]Y7~.Y7TNY7$,D>Y7#+4.Y7 ~Y7 G$>?@!Y7" ~#Y7$j)%Y7&#:.'Y7(d55E{T_alnz}IKbdm*m$nop)Y7*&..C%D+Y7,c]-Y7.%TN/Y70"*$1Y72L)3Y74(,5Y7~$(!%$("&%)&*%*6:]3]3C= ]3 # ]3 SM ]33-]3c]]3sm]3c]]3%]3]3(]3F.2-1#'-1#/3.2/3)-267;3726,0#+/)-.2'()-/@< @< S * @<  S * @< C= @<&c]@<}@<'tn@<&d^@<'tn@<!@<!/@<%] 234@<#+4.@< "*$!@<"$D>#@<$}%@<&#4.'@<("#)@<*&d^+@<, }-@<."#/@<2 3@#z<$}%z<&#4.'z<("#)z<*&d^+z<,"#-z<0 1zKC3-KC#KC>*KCKC (~KC'tnKCD>KC  :;JC3-JC#JC>*JCJC (~JC'tnJCD>JC  :;JC3-JC#JC>*JCJC (~JC'tnJCD>JC  :;)-qN%)$(+/3805+0.2qN49%qNSM qN SM qN $C= qN#+4.qN (~qN#4.qNDcN+qN)-qNf**1E*8)-B*%)$(+/3805+0.2B*49%B*SM B* SM B* $C= B*#+4.B* (~B*#4.B*DcN+B*8Gs7KB*P'+-1$(26CCsm C %TN C D> C"*$C 8 "*$8 FJDH^X^X } ^X ! ^X sm ^X&c]^X$C=^X*2$^X"(DcN+;T+A"BC^X $sCx.)94)? @A^XNh^XH(,(,&*-1+/:>EDED } ED ! ED sm ED&c]ED$C=ED*2$EDDHS>EDN(DS(D5O(DT.ltCH~.lC.lD.lfr3r4ceZdZdZdedefdZddZddZd ed e e ee fgdfddfd Z d ed e ee fddfd Z de ee fddfdZy)ra A class to watch and handle crawl job events via WebSocket connection. Attributes: id (str): The ID of the crawl job to watch app (FirecrawlApp): The FirecrawlApp instance data (List[Dict[str, Any]]): List of crawled documents/data status (str): Current status of the crawl job ws_url (str): WebSocket URL for the crawl job event_handlers (dict): Dictionary of event type to list of handler functions rappc||_||_g|_d|_|jj ddd||_gggd|_y)Nrhttpwsr)donerdocument)rrFrrr6replacews_urlevent_handlers)r>rrFs r"r?zCrawlWatcher.__init__u sU*,   ,,VT:;:bTJ  r3r7NcKtj|jddd|jjfg4d{}|j |d{dddd{y7.77 #1d{7swYyxYww)zU Establishes WebSocket connection and starts listening for messages. Nrqrr)max_sizeadditional_headers websocketsconnectrMrFr5_listenr> websockets r"rTzCrawlWatcher.connect s%% KK!0GDHHB ?B B cK|23d{}tj|}|j|d{87376yw)z Listens for incoming WebSocket messages and handles them. Args: websocket: The WebSocket connection object NrCloads_handle_messager>rWrmsgs r"rUzCrawlWatcher._listen H' , ,'**W%C&&s+ + + , +'1AA=A*A?AAAA event_typehandlerc^||jvr|j|j|yy)z Adds an event handler function for a specific event type. Args: event_type (str): Type of event to listen for ('done', 'error', or 'document') handler (Callable): Function to handle the event N)rNappend)r>r`ras r"add_event_listenerzCrawlWatcher.add_event_listener s0 ,, ,    + 2 27 ; -r3detailcZ||jvr|j|D] }|| yy)z Dispatches an event to all registered handlers for that event type. Args: event_type (str): Type of event to dispatch detail (Dict[str, Any]): Event details/data to pass to handlers N)rN)r>r`reras r"dispatch_eventzCrawlWatcher.dispatch_event s8 ,, ,..z:   -r3r]cK|ddk(rr]docs r"r[zCrawlWatcher._handle_message 2 v;& %DK   4;; Y]Y`Y`(a b [G #"DK   DKK]`ah]iquqxqx)y z [I %f+h/DK II  S[__VR8 9yy N##JDGG0LM N [J & II  S[ )    S[,P Q'E E r7N)r,r-r.r/r1r4r?rTrUrrrrdrgr[r2r3r"rri s   3  \   * , eed?eed@eedAeedBee+dCeeee,fdDeedEeedFeedGeedHeed3eedeed e-f(dIZ.ddddddddddddddddd0dd8ded9eeed:eeed;eedeed?eed@eedAeedBee+dCeeee,fdDeedEeedFeedGeedHeed3eedeed e/f(dJZ0dKed e-fdLZ1ddKedeeefd3ed e-fdMZ2ddddddddNdedOeedAeedPeedQeed=eed"eedee3d e4fdRZ5 dddddSdSdSddTdeeedUeedVeedWeed@eedXeedYeed.eeeefd e6efdZZ7dKed e'fd[Z8dKed e9fd\Z:dKed e9fd]Z;dKed eeeffd^Zddddbdedceeddeedeeed e?f dfZ@dddddgdedceeddeedheedeeed eAf diZBdKed e?fdjZCdddddddddkdled;eedmeedceedneedWeedoeedpeeDeeefgdfdqeeDeeefgdfd eEfdrZFdddddddsdled;eedmeedceedneedWeedoeed eeeffdtZGdKed eEfduZHddddddddddv dled=eedweedxeedyeedzeed#eed"eedBee+deeeeefeIfd eJfd{ZKy)AsyncFirecrawlAppz Asynchronous version of FirecrawlApp that implements async methods using aiohttp. Provides non-blocking alternatives to all FirecrawlApp operations. NmethodrJr\rrrr7c &Ktj4d{}t|D]} |j||||4d{} | jdk(r5t j |d|zzd{ dddd{f| jdk\r|j| d|dd{| jd{cdddd{ccdddd{Std 77777T7>707#1d{7swYnxYw#tj$r9} ||dz k(r| t j |d|zzd{7Yd} ~ Id} ~ wwxYw#1d{7swYyxYww) a Generic async request method with exponential backoff retry logic. Args: method (str): The HTTP method to use (e.g., "GET" or "POST"). url (str): The URL to send the request to. headers (Dict[str, str]): Headers to include in the request. data (Optional[Dict[str, Any]]): The JSON data to include in the request body (only for POST requests). retries (int): Maximum number of retry attempts (default: 3). backoff_factor (float): Factor to calculate delay between retries (default: 0.5). Delay will be backoff_factor * (2 ** retry_count). Returns: Dict[str, Any]: The parsed JSON response from the server. Raises: aiohttp.ClientError: If the request fails after all retries. Exception: If max retries are exceeded or other errors occur. N)rurJr\rCrrxi,zmake z requestrzMax retries exceeded) aiohttp ClientSessionrrequestrasynciorrfrC ClientErrorr) r>rurJr\rrrsessionrrkrs r"_async_requestz AsyncFirecrawlApp._async_request s6((* 4 4g > I I&%3d / 55!#??c1")--!w,0O"PPP$ 555 $??c1"&"4"4Xvhh?W"XXX%-]]_4555 4 4 4 I23 3! 45Q 5Y45 45555**I'A+-!--!w,(GHHHI 4 4 4s-FDFE<D-D D- .D:D ;D D- D  D-E<(D9D :DD D D-!D "D-&E<( F4D5F: E<FD- D D-DDD-FD( DD( $D-+E<-E9(E4(E+ )E4.E<4E99E<<FF F FcJK|jd|||||d{S7w)a Make an async POST request with exponential backoff retry logic. Args: url (str): The URL to send the POST request to. data (Dict[str, Any]): The JSON data to include in the request body. headers (Dict[str, str]): Headers to include in the request. retries (int): Maximum number of retry attempts (default: 3). backoff_factor (float): Factor to calculate delay between retries (default: 0.5). Delay will be backoff_factor * (2 ** retry_count). Returns: Dict[str, Any]: The parsed JSON response from the server. Raises: aiohttp.ClientError: If the request fails after all retries. Exception: If max retries are exceeded or other errors occur. POSTNr})r>rJrr\rrs r"_async_post_requestz%AsyncFirecrawlApp._async_post_request s)*((gtWn]]]] #!#cJK|jd||d||d{S7w)a Make an async GET request with exponential backoff retry logic. Args: url (str): The URL to send the GET request to. headers (Dict[str, str]): Headers to include in the request. retries (int): Maximum number of retry attempts (default: 3). backoff_factor (float): Factor to calculate delay between retries (default: 0.5). Delay will be backoff_factor * (2 ** retry_count). Returns: Dict[str, Any]: The parsed JSON response from the server. Raises: aiohttp.ClientError: If the request fails after all retries. Exception: If max retries are exceeded or other errors occur. GETNr)r>rJr\rrs r"_async_get_requestz$AsyncFirecrawlApp._async_get_request s)(((WdG^\\\\rrkrjcPK |jd{}|jdd}|jdd}|j |j|||d{}tj|7c#tjd|jxYw7Bw)aR Handle errors from async API responses with detailed error messages. Args: response (aiohttp.ClientResponse): The response object from the failed request action (str): Description of the action that was being attempted Raises: aiohttp.ClientError: With a detailed error message based on the response status: - 402: Payment Required - 408: Request Timeout - 409: Conflict - 500: Internal Server Error - Other: Unexpected error with status code Nrr r r ?Failed to parse Firecrawl error response as JSON. Status code: )rCrerwr{r_get_async_error_messager>rkrj error_datarrrs r"rfzAsyncFirecrawlApp._handle_error$ s  {'}}.J&NN74PQM&NN96]^M55hoov}^kll!!'**/ {%%(ghphwhwgx&yz zls8B&A=A;(A=!B&!B$"B&;A==$B!!B&rdrrc2K|j||||Swa Generate a standardized error message based on HTTP status code for async operations. Args: status_code (int): The HTTP status code from the response action (str): Description of the action that was being performed error_message (str): The error message from the API response error_details (str): Additional error details from the API response Returns: str: A formatted error message rrs r"rz*AsyncFirecrawlApp._get_async_error_message? &&{FM=YYrrAsyncCrawlWatchercK|j|||d{}|jdrd|vrt|d|Std73w)a Initiate an async crawl job and return an AsyncCrawlWatcher to monitor progress via WebSocket. Args: url (str): Target URL to start crawling from params (Optional[CrawlParams]): See CrawlParams model for configuration: URL Discovery: * includePaths - Patterns of URLs to include * excludePaths - Patterns of URLs to exclude * maxDepth - Maximum crawl depth * maxDiscoveryDepth - Maximum depth for finding new URLs * limit - Maximum pages to crawl Link Following: * allowBackwardLinks - DEPRECATED: Use crawlEntireDomain instead * crawlEntireDomain - Follow parent directory links * allowExternalLinks - Follow external domain links * ignoreSitemap - Skip sitemap.xml processing Advanced: * scrapeOptions - Page scraping configuration * webhook - Notification webhook settings * deduplicateSimilarURLs - Remove similar URLs * ignoreQueryParameters - Ignore URL parameters * regexOnFullURL - Apply regex to full URLs idempotency_key (Optional[str]): Unique key to prevent duplicate requests Returns: AsyncCrawlWatcher: An instance to monitor the crawl job via WebSocket Raises: Exception: If crawl job fails to start Nrrr)rrerr)r>rJrrrs r"rz%AsyncFirecrawlApp.crawl_url_and_watchN sYL $33CQQ   i (T^-C$^D%94@ @78 8 RAA 4ArcK|j|||d{}|jdrd|vrt|d|Std73w)a Initiate an async batch scrape job and return an AsyncCrawlWatcher to monitor progress. Args: urls (List[str]): List of URLs to scrape params (Optional[ScrapeParams]): See ScrapeParams model for configuration: Content Options: * formats - Content formats to retrieve * includeTags - HTML tags to include * excludeTags - HTML tags to exclude * onlyMainContent - Extract main content only Request Options: * headers - Custom HTTP headers * timeout - Request timeout (ms) * mobile - Use mobile user agent * proxy - Proxy type Extraction Options: * extract - Content extraction config * jsonOptions - JSON extraction config * actions - Actions to perform idempotency_key (Optional[str]): Unique key to prevent duplicate requests Returns: AsyncCrawlWatcher: An instance to monitor the batch scrape job via WebSocket Raises: Exception: If batch scrape job fails to start Nrrr)rrerr)r>rrrbatch_responses r"rz-AsyncFirecrawlApp.batch_scrape_urls_and_watchz sYH $;;D&/ZZ   i (T^-C$^D%94@ @>? ? [rrs)rnr\r@rArBrCrtrurvrDrErFr~rGrOrHrRrrnrkr@rArBrCrtrurvrDrErFr~rzrGrOrHrRrc^K|j|d|j}|dtd}|r||d<|r||d<|r||d<|r||d<|||d <|r||d <|r||d <| r| jd d |d<| | |d<| | |d<| | |d<| | |d<|r||d<|||d<|d|j |}t |trd|vr|j |d|d<t |tr|n|jd d |d<|d|j |}t |trd|vr|j |d|d<t |tr|n|jd d |d<|r6|Dcgc]'}t |tr|n|jd d )c}|d<||jd d |d<d|vr)|dr$d|dvr|j |dd|dd<d|vr)|dr$d|dvr|j |dd|dd<d}|j |j|||d{}|jdrd|vrtdi|dSd|vrtd|d|jdt|}td|cc}w7jw) av Scrape a single URL asynchronously. Args: url (str): Target URL to scrape formats (Optional[List[Literal["markdown", "html", "rawHtml", "content", "links", "screenshot", "screenshot@fullPage", "extract", "json"]]]): Content types to retrieve (markdown/html/etc) headers (Optional[Dict[str, str]]): Custom HTTP headers include_tags (Optional[List[str]]): HTML tags to include exclude_tags (Optional[List[str]]): HTML tags to exclude only_main_content (Optional[bool]): Extract main content only wait_for (Optional[int]): Wait for a specific element to appear timeout (Optional[int]): Request timeout (ms) location (Optional[LocationConfig]): Location configuration mobile (Optional[bool]): Use mobile user agent skip_tls_verification (Optional[bool]): Skip TLS verification remove_base64_images (Optional[bool]): Remove base64 images block_ads (Optional[bool]): Block ads proxy (Optional[Literal["basic", "stealth", "auto"]]): Proxy type (basic/stealth) extract (Optional[JsonConfig]): Content extraction settings json_options (Optional[JsonConfig]): JSON extraction settings actions (Optional[List[Union[WaitAction, ScreenshotAction, ClickAction, WriteAction, PressAction, ScrollAction, ScrapeAction, ExecuteJavascriptAction, PDFAction]]]): Actions to perform agent (Optional[AgentOptions]): Agent configuration for FIRE-1 model **kwargs: Additional parameters to pass to the API Returns: ScrapeResponse with: * success - Whether scrape was successful * markdown - Markdown content if requested * html - HTML content if requested * rawHtml - Raw HTML content if requested * links - Extracted links if requested * screenshot - Screenshot if requested * extract - Extracted data if requested * json - JSON data if requested * error - Error message if scrape failed Raises: Exception: If scraping fails rNrOrPrnr\rorpNrqrrrtTrQrurvrwrxryr~rrerOrrRrrUrrrrYr2) r[r\r]r^r_r`rr6rerrr1)r>rJrnr\r@rArBrCrtrurvrDrErFr~rGrOrHrRrrgrhrirjrrk error_contents r"rNzAsyncFirecrawlApp.scrape_url s~ fl3((*#G9- '.M) $ '.M) $ +7M- ( +7M- (  (/@M+ , '/M) $ '.M) $ (0 tRV (WM* %  &,M( # ,3HM/ 0  +2FM. /  (1M* % %*M' "  (1M* %  ..w7G'4(X-@$($<$rrnr\r@rArBrCrtrurvrDrErFr~rOrHrRrrrrgrirjrvrwrkrs r"rz#AsyncFirecrawlApp.batch_scrape_urls. sz f&9:   '.M) $  '.M) $  #+7M- (  #+7M- (  (/@M+ ,  '/M) $  '.M) $  (0 tRV (WM* %  &,M( # ,3HM/ 0  +2FM. /  (1M* %  %*M' "  ..w7G'4(X-@$($<$8{[dOeCe/3/G/G T]H^_gHh/iK "8 , K 'K ,F8WbcpWqKq373K3KKXeLfgoLp3qK &x 0''811||n, -     << " P\\$'77G]SS S   x)A BC(m(  P"MOOSsCEK J1/C,K J6K 2J8K KK 8 KK )rnr\r@rArBrCrtrurvrDrErFr~rOrHrRrrLrrLcK|j|di}|||d<|||d<|||d<|||d<|||d<|||d<|||d <| | jd d |d <| | |d <| | |d<| | |d<| | |d<|||d<|d|j|}t|trd|vr|j|d|d<t|tr|n|jd d |d<|d|j|}t|trd|vr|j|d|d<t|tr|n|jd d |d<|r6|Dcgc]'}t|tr|n|jd d )c}|d<||jd d |d<|||d<|j |t d i|}|jd d }||d<dt |d<d|vr)|dr$d|dvr|j|dd|dd<d|vr)|dr$d|dvr|j|dd|dd<|j|}|j|jd||d{}|jddk(r td i|jS|j|dd{ycc}w7R#tdxYw7w)!a* Initiate a batch scrape job asynchronously. Args: urls (List[str]): URLs to scrape formats (Optional[List[Literal]]): Content formats to retrieve headers (Optional[Dict[str, str]]): Custom HTTP headers include_tags (Optional[List[str]]): HTML tags to include exclude_tags (Optional[List[str]]): HTML tags to exclude only_main_content (Optional[bool]): Extract main content only wait_for (Optional[int]): Wait time in milliseconds timeout (Optional[int]): Request timeout in milliseconds location (Optional[LocationConfig]): Location configuration mobile (Optional[bool]): Use mobile user agent skip_tls_verification (Optional[bool]): Skip TLS verification remove_base64_images (Optional[bool]): Remove base64 encoded images block_ads (Optional[bool]): Block advertisements proxy (Optional[Literal]): Proxy type to use extract (Optional[JsonConfig]): Content extraction config json_options (Optional[JsonConfig]): JSON extraction config actions (Optional[List[Union]]): Actions to perform agent (Optional[AgentOptions]): Agent configuration zero_data_retention (Optional[bool]): Whether to delete data after 24 hours idempotency_key (Optional[str]): Unique key to prevent duplicate requests **kwargs: Additional parameters to pass to the API Returns: BatchScrapeResponse with: * success - Whether job started successfully * id - Unique identifier for the job * url - Status check URL * error - Error message if start failed Raises: Exception: If job initiation fails rNrnr\rorprqrrrtTrQrurvrwrxryr~rerOrrRrrTrrOrrrdrXrZrr2)r[r^r_r`rarr]r\rr6rerrCrrf)r>rrnr\r@rArBrCrtrurvrDrErFr~rOrHrRrrLrrgrirjrvrwrks r"rz)AsyncFirecrawlApp.async_batch_scrape_urls s| f&?@   '.M) $  '.M) $  #+7M- (  #+7M- (  (/@M+ ,  '/M) $  '.M) $  (0 tRV (WM* %  &,M( # ,3HM/ 0  +2FM. /  (1M* %  %*M' "  ..w7G'4(X-@$($<$8{[dOeCe/3/G/G T]H^_gHh/iK "8 , K 'K ,F8WbcpWqKq373K3KKXeLfgoLp3qK &x 0''811||n, -     << &# - P*=X]]_==$$X/GH H HE(U,  P"MOO HsCEK,KC3K4K5KK &K<K= K KK)ryrzr{r|rr}r~r0rrmrrrrrrrrryrzr{r|rr}r~r0rrmrrrrrrcK|j|di}|||d<|||d<|||d<|||d<|||d<|||d<n|||d <| | |d <| | |d <| | jd d |d<| | |d<| | |d<|||d<|||d<|||d<|||d<|j|tdi|}|jd d }||d<dt|d<|j |}|j |jd||d{}|jdr- |jd}|j|||d{S|j|dd{y7]#tdxYw717w)a  Crawl a website starting from a URL. Args: url (str): Target URL to start crawling from include_paths (Optional[List[str]]): Patterns of URLs to include exclude_paths (Optional[List[str]]): Patterns of URLs to exclude max_depth (Optional[int]): Maximum crawl depth max_discovery_depth (Optional[int]): Maximum depth for finding new URLs limit (Optional[int]): Maximum pages to crawl allow_backward_links (Optional[bool]): DEPRECATED: Use crawl_entire_domain instead crawl_entire_domain (Optional[bool]): Follow parent directory links allow_external_links (Optional[bool]): Follow external domain links ignore_sitemap (Optional[bool]): Skip sitemap.xml processing scrape_options (Optional[ScrapeOptions]): Page scraping configuration webhook (Optional[Union[str, WebhookConfig]]): Notification webhook settings deduplicate_similar_urls (Optional[bool]): Remove similar URLs ignore_query_parameters (Optional[bool]): Ignore URL parameters regex_on_full_url (Optional[bool]): Apply regex to full URLs delay (Optional[int]): Delay in seconds between scrapes allow_subdomains (Optional[bool]): Follow subdomains poll_interval (Optional[int]): Seconds between status checks (default: 2) idempotency_key (Optional[str]): Unique key to prevent duplicate requests **kwargs: Additional parameters to pass to the API Returns: CrawlStatusResponse with: * Crawling status and progress * Crawled page contents * Success/error information Raises: Exception: If crawl fails rNrrrrrrrrrTrQrrrrrrrrJrOrrrrrZrr2) r[r^rarr]r\rr6rerrrf)r>rJryrzr{r|rr}r~r0rrmrrrrrrrrrgrrvrwr\rkrs r"rzAsyncFirecrawlApp.crawl_urlE s2v fk2   $+8L (  $+8L (  '0L $  *0CL, -  $)L !  *0CL, - ! -1EL- .  +1EL- .  %,:L )  %,:,?,?\`,?,aL )  &-L # # /5ML1 2 " .4KL0 1  (->L) *  $)L !  '.>L* + F##2\2 "''D'I  E"-gY 7 H''811 \\N) $k7<< << " P\\$'77G]SS S$$X/@A A A<  P"MOOS AsHD F E+F$E-5F E= F%E?&F- E::F?FcKi}|||d<|||d<|||d<|||d<|||d<|||d<n|||d<| | |d <| | |d <| | jd d |d <| | |d<| | |d<|||d<|||d<|||d<|||d<|j|tdi|}|jd d }||d<dt|d<|j |}|j |j d||d{}|jdr tdi|S|j|dd{y7<#tdxYw7w)a Start an asynchronous crawl job. Args: url (str): Target URL to start crawling from include_paths (Optional[List[str]]): Patterns of URLs to include exclude_paths (Optional[List[str]]): Patterns of URLs to exclude max_depth (Optional[int]): Maximum crawl depth max_discovery_depth (Optional[int]): Maximum depth for finding new URLs limit (Optional[int]): Maximum pages to crawl allow_backward_links (Optional[bool]): DEPRECATED: Use crawl_entire_domain instead crawl_entire_domain (Optional[bool]): Follow parent directory links allow_external_links (Optional[bool]): Follow external domain links ignore_sitemap (Optional[bool]): Skip sitemap.xml processing scrape_options (Optional[ScrapeOptions]): Page scraping configuration webhook (Optional[Union[str, WebhookConfig]]): Notification webhook settings deduplicate_similar_urls (Optional[bool]): Remove similar URLs ignore_query_parameters (Optional[bool]): Ignore URL parameters regex_on_full_url (Optional[bool]): Apply regex to full URLs idempotency_key (Optional[str]): Unique key to prevent duplicate requests **kwargs: Additional parameters to pass to the API Returns: CrawlResponse with: * success - Whether crawl started successfully * id - Unique identifier for the crawl job * url - Status check URL for the crawl * error - Error message if start failed Raises: Exception: If crawl initiation fails NrrrrrrrrrTrQrrrrrrrrJrOrrrrZrr2) r^rarr]r\rr6rerrrf)r>rJryrzr{r|rr}r~r0rrmrrrrrrrrrgrrvrwr\rks r"rz!AsyncFirecrawlApp.async_crawl_url s p   $+8L (  $+8L (  '0L $  *0CL, -  $)L !  *0CL, - ! -1EL- .  +1EL- .  %,:L )  %,:,?,?\`,?,aL )  &-L # # /5ML1 2 " .4KL0 1  (->L) *  $)L !  '.>L* + F##2\2 "''D'I  E"-gY 7 H''811 \\N) $     << " P$0x00$$X/@A A A  P"MOO As6C9E ;D8<E  D:E 2E 3E : EE rc $K|j}d|}|j|j||d{}|jddk(rd|vr|d}d|vr}t |ddk(rnk|jd}|st j dnB|j||d{}|j|jdg|}d|vr}||d<t|jd|jd |jd|jd |jd |jdd |vrd nd}d |vr|jd |_ d|vr|jd|_ |S7P7ܭw)a5 Check the status and results of an asynchronous crawl job. Args: id (str): Unique identifier for the crawl job Returns: CrawlStatusResponse containing: Status Information: * status - Current state (scraping/completed/failed/cancelled) * completed - Number of pages crawled * total - Total pages to crawl * creditsUsed - API credits consumed * expiresAt - Data expiration timestamp Results: * data - List of crawled documents * next - URL for next page of results (if paginated) * success - Whether status check succeeded * error - Error message if failed Raises: Exception: If status check fails rNrr]rrrrrrrrFT)rrr]rrrr) r\rr6rerr%rrrrr r>rr\rrrrrrks r"rz$AsyncFirecrawlApp.check_crawl_status2s2'')t$ 33||nXJ '   ??8 $ 3$"6* +;v./14*v6H#'HI&*&=&=h&P PIKK fb 9:"+K +'+ F#&??8,//'*!ook2# 6!ook2($ 3E  k !(__W5HN [ 'OOF3HMI !Qs)9FF A5F1F2+FB.FFczK |j|jd||d{}|jddk(rd|vr|d}d|vr}t|ddk(rnk|jd}|stj dnB|j||d{}|j |jdg|}d|vr}||d<td i|Std |jdd vr(tjt|d d{ntd |d07 77w)a Monitor the status of an asynchronous job until completion. Args: id (str): The ID of the job to monitor headers (Dict[str, str]): Headers to include in status check requests poll_interval (int): Seconds between status checks (default: 2) Returns: CrawlStatusResponse: The job results if completed successfully Raises: Exception: If the job fails or an error occurs during status checks rNrr]rrrrz&Job completed but no data was returnedrrxz#Job failed or was stopped. Status: r2) rr6rerr%rrrrrzrr )r>rr\rrrrrs r"rz+AsyncFirecrawlApp._async_monitor_job_statustsU $ 7 7<<. 2$/!K x(K7[(&v.D K/{623q8!#.??6#:'"NN+LM!*.*A*A(G*T$T  IMM&"$=>&/ !K/+/K'.===#$LMM*.nnmmC q$9:::"EkRZF[E\ ]^^5%U;s;&D;D4A5D;D7+D; AD;D9D;7D;9D;)rrrrrrtrrrrcKi} |r"| j|jdd||| d<||| d<||| d<||| d<||| d<||| d <tdi| } | jdd} || d <d t| d <d } |j |j | | dd|j id{} | jdrd| vr tdi| Sd| vrtd| dtd| 7Gw)a Asynchronously map and discover links from a URL. Args: url (str): Target URL to map params (Optional[MapParams]): See MapParams model: Discovery Options: * search - Filter pattern for URLs * ignoreSitemap - Skip sitemap.xml * includeSubdomains - Include subdomain links * sitemapOnly - Only use sitemap.xml Limits: * limit - Max URLs to return * timeout - Request timeout (ms) Returns: MapResponse with: * Discovered URLs * Success/error status Raises: Exception: If mapping fails TrQNrrrrrrtrJrOrrrqrrrrrNrzFailed to map URL. Error: r2) rar^rr]rr6r5rerr)r>rJrrrrrrtrrrvrwrrks r"rzAsyncFirecrawlApp.map_urlsqF    fkk4dkK L  #)Jx  %*8J '  ).@J* +  #(4J} %  "'Jw   $+Jy !!.:. "''D'I  E"-gY 7 H11||nXJ ' $ ~&>?2   << "w(':** *  8'9J8KLM M8 CD D sB6D8D9ADFrr+rer/r1r2cK|j} |s |s td|s |s td|r|j|}|xsg||||dtd} |r|| d<|r|| d<|r|| d<|j |j d| | d {} | j d r| j d } | s td  |j|j d | | d {} | ddk(r tdi| S| ddvrtd| dd| dtjdd {xtd| j d77r7&w)aW Asynchronously extract structured information from URLs. Args: urls (Optional[List[str]]): URLs to extract from prompt (Optional[str]): Custom extraction prompt schema (Optional[Any]): JSON schema/Pydantic model system_prompt (Optional[str]): System context allow_external_links (Optional[bool]): Follow external links enable_web_search (Optional[bool]): Enable web search show_sources (Optional[bool]): Include source URLs agent (Optional[Dict[str, Any]]): Agent configuration Returns: ExtractResponse with: * Structured data matching schema * Source information if requested * Success/error status Raises: ValueError: If prompt/schema missing or extraction fails rrrOrr+rrrNrrrrrr]rrrrrxrr2) r\r<r_r#rr6rerrrrzr)r>rr+rer/r0r1r2rr\rrkrrs r"rOzAsyncFirecrawlApp.extractsD'')fBC CF@A A --f5FJB"60'#KM?3   %+L " +8L ( $)L !11||nK (     << "\\$'F KLL$($;$;||nL9% x(K7*9[99 *.EE#l;x3H2IS^_fSgRh$ijjmmA&&&8g9N8OPQ Q1 's8BE$EAE$-E .A E$;E"<#E$ E$"E$c K|j}d|}|j|j||d{}|ddk(rd|vr|d}d|vr}t|ddk(rnk|j d}|st j dnB|j||d{}|j|j dg|}d|vr}||d<t|j d|j d |j d|j d |j d |j d }d |vr|d |d <d|vr|d|d<dd |vrdi|Sdi|S7.7ƭw)a0 Check the status of an asynchronous batch scrape job. Args: id (str): The ID of the batch scrape job Returns: BatchScrapeStatusResponse containing: Status Information: * status - Current state (scraping/completed/failed/cancelled) * completed - Number of URLs scraped * total - Total URLs to scrape * creditsUsed - API credits consumed * expiresAt - Data expiration timestamp Results: * data - List of scraped documents * next - URL for next page of results (if paginated) * success - Whether status check succeeded * error - Error message if failed Raises: Exception: If status check fails rNrr]rrrrrrrrrrFT) r\rr6rrer%rrrrs r"rz+AsyncFirecrawlApp.check_batch_scrape_statusBs2'')&rd+ 33||nXJ '   x K /$"6* +;v./14*v6H#'HI&*&=&=h&P PIKK fb 9:"+K +'+ F#,??8,//'*!ook2# 6!ook2(   k ! +G 4HW  [ *62HV  ; 6u   <@   G !Qs)9E.E)A)E.%E,&+E.BE.,E.cK|j}|j|jd|d|d{S7w)a[ Get information about errors from an asynchronous batch scrape job. Args: id (str): The ID of the batch scrape job Returns: CrawlErrorsResponse containing: errors (List[Dict[str, str]]): List of errors with fields: * id (str): Error ID * timestamp (str): When the error occurred * url (str): URL that caused the error * error (str): Error message * robotsBlocked (List[str]): List of URLs blocked by robots.txt Raises: Exception: If error check fails rrNr\rr6r>rr\s r"rz+AsyncFirecrawlApp.check_batch_scrape_errorssK&''),,||n-bT 9     6?=?cK|j}|j|jd|d|d{S7w)a_ Get information about errors from an asynchronous crawl job. Args: id (str): The ID of the crawl job Returns: CrawlErrorsResponse containing: * errors (List[Dict[str, str]]): List of errors with fields: - id (str): Error ID - timestamp (str): When the error occurred - url (str): URL that caused the error - error (str): Error message * robotsBlocked (List[str]): List of URLs blocked by robots.txt Raises: Exception: If error check fails rrNrrs r"rz$AsyncFirecrawlApp.check_crawl_errorssJ&''),,||nJrd' 2     rcK|j}tj4d{}|j|jd||4d{}|j d{cdddd{cdddd{S7i7@7*77 #1d{7swYnxYwdddd{7y#1d{7swYyxYww)rNrr)r\rwrxrr6rC)r>rr\r|rks r"rzAsyncFirecrawlApp.cancel_crawls'')((* - -g~~j&Ew~W - -[c%]]_, - - - - - - -, - - - - - - - - - -s)CBC'CBCB.B/B2 C>B?C CBCCBCCB1 %B(&B1 -C4 C?CCC C CCrcK|j} |j|jd||d{S7#t$r}t t |d}~wwxYww)a< Check the status of an asynchronous extraction job. Args: job_id (str): The ID of the extraction job Returns: ExtractResponse[Any] with: * success (bool): Whether request succeeded * data (Optional[Any]): Extracted data matching schema * error (Optional[str]): Error message if any * warning (Optional[str]): Warning message if any * sources (Optional[List[str]]): Source URLs if requested Raises: ValueError: If status check fails rNr\rr6rr<r1)r>rr\rs r"rz$AsyncFirecrawlApp.get_extract_statussm$'') %00<<. VH5  %SV$ $ %1A$$?=?A$? A!AA!!A$c K|j} |s |s td|s |s td|r|j|}t|xsg||||dt} |r|| d<|r|| d<|r|| d< |j |j d| | d {S7#t$r} tt| d } ~ wwxYww) a Initiate an asynchronous extraction job without waiting for completion. Args: urls (Optional[List[str]]): URLs to extract from prompt (Optional[str]): Custom extraction prompt schema (Optional[Any]): JSON schema/Pydantic model system_prompt (Optional[str]): System context allow_external_links (Optional[bool]): Follow external links enable_web_search (Optional[bool]): Enable web search show_sources (Optional[bool]): Include source URLs agent (Optional[Dict[str, Any]]): Agent configuration idempotency_key (Optional[str]): Unique key to prevent duplicate requests Returns: ExtractResponse[Any] with: * success (bool): Whether request succeeded * data (Optional[Any]): Extracted data matching schema * error (Optional[str]): Error message if any Raises: ValueError: If job initiation fails rrrOrr+rrrN) r\r<r_rr]rr6rr1) r>rr+rer/r0r1r2rr\rrs r"rzAsyncFirecrawlApp.async_extractsD'')fBC CF@A A --f5F&3-$  *   %+L " +8L ( $)L ! %11<<. ,   %SV$ $ %s<A3C6#B BB CB C)B==CCrrrrrrcKi}|||d<|||d<|||d<|j||||d{}|jdrd|vr|S|d} |j|d{}|dd k(r|S|dd k(rtd |jd |dd k7rnt j dd{ot ddS77i7w)a Generate LLMs.txt for a given URL and monitor until completion. Args: url (str): Target URL to generate LLMs.txt from max_urls (Optional[int]): Maximum URLs to process (default: 10) show_full_text (Optional[bool]): Include full text in output (default: False) experimental_stream (Optional[bool]): Enable experimental streaming Returns: GenerateLLMsTextStatusResponse containing: * success (bool): Whether generation completed successfully * status (str): Status of generation (processing/completed/failed) * data (Dict[str, str], optional): Generated text with fields: - llmstxt (str): Generated LLMs.txt content - llmsfulltxt (str, optional): Full version if requested * error (str, optional): Error message if generation failed * expiresAt (str): When the generated data expires Raises: Exception: If generation fails Nrrrrrrrr]r^z#LLMs.txt generation failed. Error: rrrxFrr)rrerrrzrr,) r>rJrrrrrkrrs r"rz$AsyncFirecrawlApp.generate_llms_text-s#:   (F9   %%3F> "  *.AF* +66 ) 3 7   ||I&$h*>O$??GGFh;. !X-"EfjjQXFYEZ [\\!\1--" " ".eCtuu- H #s40CC5C(C)AC=C>CCCrrc\Ki}|||d<|||d<|||d<t||||}|j}d|i|jdd}d t|d < |j |j d ||d{S7#t $r} tt| d} ~ wwxYww) a> Initiate an asynchronous LLMs.txt generation job without waiting for completion. Args: url (str): Target URL to generate LLMs.txt from max_urls (Optional[int]): Maximum URLs to process (default: 10) show_full_text (Optional[bool]): Include full text in output (default: False) cache (Optional[bool]): Whether to use cached content if available (default: True) experimental_stream (Optional[bool]): Enable experimental streaming Returns: GenerateLLMsTextResponse containing: * success (bool): Whether job started successfully * id (str): Unique identifier for the job * error (str, optional): Error message if start failed Raises: ValueError: If job initiation fails NrrrrrJTrQrOrr) rr\r^r]rr6rr<r1) r>rJrrrrrr\rrs r"rz*AsyncFirecrawlApp.async_generate_llms_textjs6   (F9   %%3F> "  *.AF* +''"5  '')CQ6;;4;#PQ +G95 ( %11<<. ,   %SV$ $ %s<AB,#BBBB,B B)B$$B))B,cK|j} |j|jd||d{S7#t$r}t t |d}~wwxYww)a Check the status of an asynchronous LLMs.txt generation job. Args: id (str): The ID of the generation job Returns: GenerateLLMsTextStatusResponse containing: * success (bool): Whether generation completed successfully * status (str): Status of generation (processing/completed/failed) * data (Dict[str, str], optional): Generated text with fields: - llmstxt (str): Generated LLMs.txt content - llmsfulltxt (str, optional): Full version if requested * error (str, optional): Error message if generation failed * expiresAt (str): When the generated data expires Raises: ValueError: If status check fails rNrr>rr\rs r"rz1AsyncFirecrawlApp.check_generate_llms_text_statussm('') %00<<. RD1  %SV$ $ %r)r{rrrr/-_AsyncFirecrawlApp__experimental_stream_stepsrrrrrrrrcKi} ||| d<||| d<||| d<||| d<||| d<||| d<tdi| } |j||||||d{} | jd rd | vr| S| d } d } d } |j| d{}|r)d |vr%|d | d}|D] }|| t |d } | r)d |vr%|d |d}|D] }| | t |d }|ddk(r|S|ddk(rt d|jd|ddk7rnt jdd{tddS777wr) rrrer rrrzrr!)r>rr{rrrr/rrrr!rkrr"r#rr$r%r&r's r"r(zAsyncFirecrawlApp.deep_researchsP  *3OJ '  !+5OK (  )1OI &  &0?O, -  $.;ON + & 2O$::6BBF|v5!' !56I6J!K .*H)*&)&*>&?#Y&0$Y/0A0BC )&Ff%&$'y(9$:!h;. !X-"? 7@S?T UVV!\1--" " "-0*%?jkkO C* #s7AEE 9EEB*E9E:EEE)r{rrrr/rc~Ki}|||d<|||d<|||d<|||d<|||d<|||d<tdi|}|j} d|i|jd d } d t| d < |j |j d | | d{S7#t $r} tt| d} ~ wwxYww)r*NrrrrrrrTrQrOrr+r2) rr\r^r]rr6rr<r1) r>rr{rrrr/rr!r\rrs r"rz%AsyncFirecrawlApp.async_deep_researchs>  *3OJ '  !+5OK (  )1OI &  &0?O, -  $.;ON + & 2rrrr r rXrurtrmrrgrtrvrws r"rzAsyncFirecrawlApp.search|saR &$'$$V,$$V[[$T[%RS  %*M' " ?#&M%  &,M( #  $(M& !  '.M) $  (0M* %  '.M) $  %-;-@-@$]a-@-bM/ * V$$A%A=A "''D'I "-gY 7 H--||nJ '   ~6 7    sDD DD )NrBrCrAr?)rxr@)Lr,r-r.r/r1rrrrrr}rrrwClientResponserfrrrrrrr rrWrrrrrrrrrrrr(rrNrrrrrir[rrrrrrrrrrrOrrrrrrrr,rr&rrrr!r(rr rrrr2r3r"rtrt s.2$'+4+4+4#s(^ +4 4S>* +4  +4"+4-1cN+4^7:^^"&sCx.^;?S>^^.3^>B38n^27:]]%)#s(^]].3]>B38n],+G,B,B+C+TX+6 Z# Zs Z[^ Zor Zwz Z$-1-1 *9*9[)*9&c] *97J *9^.2-1 (@s)(@\*(@&c] (@7J (@\mq04040404&*%*15%)4837(,CG(,,015sw,0+HMHMd7,g$hij HM d38n- HM #49- HM#49-HM (~HMsmHMc]HM~.HMTNHM$,D>HM#+4.HM ~HM G$>?@!HM" ~#HM$j)%HM&#:.'HM(d55E{T_alnz}IKbdm*m$nop)HM*L)+HM,(,-HM\W[,0,0,0,0"&!&-1!%04/3$(?C(,-1os(,'()--HC3iHC$w(Q RST HC $sCx.) HC tCy) HCtCy)HC$D>HC3-HC#HC>*HCHC (~HC'tnHCD>HC  :;JI3-JI#JI>*JIJI (~JI'tnJID>JI  :;#uB$}%uB&#4.'uB( })uB*"#+uB. /uBx.2-1#'-1#/3.2/3)-267;3726,0#+/'()-+sB sB S * sB  S * sB C= sB&c]sB}sB'tnsB&d^sB'tnsB!sB!/sB%] 234sB#+4.sB "*$!sB"$D>#sB$}%sB&#4.'sB( })sB*"#+sB. /sBj@3@3F@D)_#)_S#X)__b)_k~)_^!%)--1'+#!&&*HE HE HE ! HE %TN HEtnHE}HE#HE#HE0;HEX)-WR%)$(+/3805+0.2WR49%WRSM WR SM WR $C= WR#+4.WR (~WR#4.WRDcN+WR8Gs7KWRrB #B :SB H # :M 2 3 3F 2-S-T#s(^-(%s%s7K%:)-D%%)$(+/3805+0.2D%49%D%SM D% SM D% $C= D%#+4.D% (~D%#4.D%DcN+D%8Gs7KD%T'+-126 ;v;vsm ;v %TN ;v "*$ ;v 5%"*$5%FJDH^l^l } ^l ! ^l sm ^l&c]^l$C=^l*2$^l"(DcN+;T+A"BC^l $sCx.)94)? @A^lNh^lH(,(,&*-1+/:>:%:% } :% ! :% sm :%&c]:%$C=:%*2$:%DHS>:%x %3 %;U %L$(!%$("&%)&*%*6:DHN N C= N # N SM N 3-N c]N smN c]N %]3N U4S><#?@AN (N r3rtc eZdZdZdedeffd ZddZddZd e ee fddfd Z d e jd eddfd Zded edededef dZxZS)rzO Async version of CrawlWatcher that properly handles async operations. rrFc&t|||yr@)superr?)r>rrF __class__s r"r?zAsyncCrawlWatcher.__init__s S!r3r7NcKtj|jdd|jjfg4d{}|j |d{dddd{y7.77 #1d{7swYyxYww)z[ Establishes async WebSocket connection and starts listening for messages. rqrr)rQNrRrVs r"rTzAsyncCrawlWatcher.connects%% KK!0GDHHB B cK|23d{}tj|}|j|d{87376yw)z Listens for incoming WebSocket messages and handles them asynchronously. Args: websocket: The WebSocket connection object NrYr\s r"rUzAsyncCrawlWatcher._listenr^r_r]cK|ddk(rrIrWr[rcrirrrrrrrrrrrrrrrrrrrrrrrrrrrrr!r&r(r,r4rrtrr2r3r"rs  XXX    -+'++K88 CLJ!8%%! ((,,(H&& C++C8**GAJ8 *X''* UH&&UH..$H&&$(#### "x))" ($$ ($$  ($$ #8%%# 8%%h00 """"(8%%()##),=, &q)71: ,(,,, " 2 2 "+($$+( H&&  "(,, "(,, $""$ ($$ 2H&& 2 -h(('!* - 28%% 2 X'' 1X//1 6++ 6 8--  !3!3  x11 &););&X%7%7 X''  +H&& +q"q"fEYRYRvF F P8OZ OZr3