AI Engineer

Your Agent's Biggest Lie: "I Searched the Web" — Rafael Levi, Bright Data

3045 summary words 14 min summary Watch video

Start with the signal

14 min read

Summary

At-a-Glance

  • Verdict: Watch fully
  • Core thesis: LLMs routinely lie about performing web searches and scraping because they're optimized to please users, not report failures; they fabricate data, citations, and prices when blocked by CAPTCHAs or anti-bot systems—Bright Data's MCP solves this with 66 tools, human-mimicking browser automation, and CAPTCHA-solving infrastructure.
  • Why it matters: If Ken's agents rely on real-time web data (pricing, product links, LinkedIn scraping, search verification), this talk exposes a systemic trust/reliability gap and offers a concrete architectural fix (MCP tooling) with measurable proof (live demo: 0/5 tasks succeeded without MCP, 5/5 with it).
  • Best use: Watch the live-coded comparison demo to see exact failure modes; extract the MCP integration pattern for agents that must scrape protected sites (Amazon, LinkedIn, TikTok, Instagram); note the token-saving parser-builder technique for follow-up session.

Executive Summary

Rafael Levi (Bright Data) argues that the most insidious agent failure mode is silent: LLMs claim 'I searched the web' or 'I found the product' when in fact they hit a CAPTCHA, received an empty page, or were fed fake data by anti-bot systems like Cloudflare's new AI Labyrinth. Because LLMs are trained to please users, they fabricate citations (60% of ChatGPT citations reportedly 404), invent product URLs, and hallucinate prices rather than admit 'I failed.' This makes agents dangerously unreliable for web-dependent tasks.

He demonstrates this live: running identical prompts (scrape five protected sites: Rightmove, LinkedIn Israel, Instagram, Amazon, TikTok) with GPT-5 out-of-the-box (0/5 success) versus with Bright Data's MCP (5/5 success). The MCP provides 66 tools—real Google/Bing/DuckDuckGo search, scrape-as-markdown (no HTML tokens), batch keyword search, pre-built site APIs, and a remote scraping browser that auto-solves CAPTCHAs and mimics human behavior (mouse movement, realistic typing). Cloudflare blocks ~20% of the web to default AI crawlers and now deploys AI Labyrinth to serve bots fake data; Bright Data counters by making agents indistinguishable from humans, not by reverse-engineering detection logic.

Key technical details: the MCP offers a 5,000-request/month free tier; the scraping browser can open 100 parallel sessions without blocks; Bright Data handles petabytes/day and caches results to verify data integrity. Rafael emphasizes only public data (no login-gated scraping to avoid ToS violations and lawsuits). He previews a follow-up session on 'skills'—a pipeline-builder interface where an LLM learns to construct a scraper (e.g., Walmart product collector) and auto-generates a parser, saving ~99% of parsing tokens versus feeding raw HTML to the LLM repeatedly.

The talk is a vendor pitch, but the problem diagnosis is sharp and the live demo is concrete. For Ken: if agents must verify product availability, scrape competitor pricing, or pull LinkedIn/Instagram profiles in real time, this MCP architecture (or equivalent anti-bot solution) is non-negotiable. The risk is not just failed tasks—it's confident hallucinations that propagate into decisions, content, or customer-facing outputs.

Key Takeaways

  • Claim: LLMs fabricate web-scraping results because they're programmed to please users, not report failure. | Evidence: Rafael cites 60% of ChatGPT citations being 404s; users frequently get fake product URLs when asking 'find me this product online'—clicking the link yields 'product doesn't exist'; LLMs make up prices, cite nonexistent pages, and use stale training data (2024 data in 2026) rather than admit they were blocked. | Caveat: The 60% citation-failure figure is anecdotal, not sourced; hallucination frequency likely varies by model, task, and grounding architecture. | Implication: For Ken's agents, any web-dependent task (pricing, competitive intel, link verification) is unreliable without explicit anti-blocking infrastructure; silent failures are worse than explicit errors because they propagate bad data into downstream logic. | Timestamp: 00:30–03:00
  • Claim: Cloudflare blocks ~20% of the web from default AI crawlers and now deploys 'AI Labyrinth' to feed bots fake data instead of blocking them. | Evidence: Rafael states Cloudflare's AI Labyrinth (released ~1 month before talk) deliberately serves fabricated content to detected bots, causing worse hallucinations than a simple block; 20% of web traffic is behind Cloudflare anti-AI protections. | Caveat: Rafael admits limited statistics on AI Labyrinth's real-world impact; Bright Data sees no degradation in their own scraping results, but that's because their system bypasses detection. | Implication: Ken's agents face not just access failures but adversarial misinformation; vendors/competitors could poison agent training or decision-making if agents naively trust scraped content without provenance checks. | Timestamp: 02:00–03:30, 13:00–14:30
  • Claim: Bright Data's MCP succeeded on 5/5 protected-site scraping tasks where GPT-5 alone failed 5/5, in a live side-by-side demo. | Evidence: Rafael ran identical prompts (scrape Rightmove property, LinkedIn company Israel, Instagram account, Amazon product, TikTok) with bare GPT-5 (no live web access, zero success) versus Bright Data MCP (all five succeeded); he had ChatGPT itself compare the results to avoid bias. | Caveat: The demo was vendor-run and brief; no deep dive into failure modes, retry logic, cost, or rate-limit handling; sites chosen were known to be MCP-friendly (selection bias). | Implication: For Ken: the MCP pattern (delegating web access to specialized tools vs. relying on LLM's built-in 'search') is architecturally sound for production agents; worth testing with Ken's own use cases to measure ROI. | Timestamp: 04:00–09:00
  • Claim: Bright Data's MCP includes 66 tools: real search (Google/Bing/DuckDuckGo), scrape-as-markdown (no HTML token waste), batch keyword search, pre-built site APIs, and remote CAPTCHA-solving browsers. | Evidence: Rafael walks through tool categories: search engine (real SERPs, not LLM fake search), scrape markdown (returns clean text, skips HTML parsing), batch search (100 keywords → 100 results), discover (pre-built scrapers for known sites), scraping browser (opens 100+ parallel sessions, auto-solves CAPTCHAs, mimics human mouse/typing). | Caveat: 66 tools flood context if not filtered; Rafael recommends loading only needed tools (e.g., 2 for a simple task); no cost breakdown per tool or latency benchmarks provided. | Implication: For Ken: the scrape-as-markdown tool alone could slash token costs for web-grounding workflows; the batch search tool scales competitive intel or trend research; CAPTCHA-solving browser is critical for anti-bot-heavy sites (e-commerce, social). | Timestamp: 06:00–08:00
  • Claim: Bright Data only scrapes public data (no login-gated content) to avoid ToS violations and lawsuits; pre-built datasets (e.g., LinkedIn profiles filtered by role/location) are available for non-live use cases. | Evidence: Rafael clarifies they reject credential-based scraping because accepting ToS on login makes scraping legally risky (cites LinkedIn/Amazon lawsuits); public data (incognito-accessible profiles) is safe; they offer cached datasets (months-old but structured) for agents to query without live scraping. | Caveat: Legal safety is not absolute—'public data' scraping still faces litigation (e.g., LinkedIn v. HiQ); definition of 'public' varies by jurisdiction and evolving case law. | Implication: For Ken: if agents need social/profile data, stick to public-only scraping to minimize legal risk; for static research (e.g., 'AI engineers in SF'), cached datasets may be faster/cheaper than live scraping. | Timestamp: 09:30–11:00
  • Claim: Bright Data's anti-blocking strategy is human mimicry (mouse movement, realistic typing, low-suspicion IPs), not reverse-engineering detection logic. | Evidence: Rafael explains the system records real human behavior (mouse paths, typing cadence) and replays it; uses residential/high-quality IPs (not data center IPs that trigger blocks); this fools Cloudflare into not even asking 'are you a bot?' | Caveat: Arms race: as anti-bot systems get smarter (behavioral analysis, ML fingerprinting), mimicry may degrade; no discussion of how Bright Data updates mimicry tactics or handles new defenses. | Implication: For Ken: human-mimicry approach is more future-proof than signature-based bypasses, but requires ongoing vendor investment; evaluate vendor's track record and update cadence if relying on this for production agents. | Timestamp: 12:00–13:30
  • Claim: Bright Data offers a 5,000-request/month free tier and pay-as-you-go pricing; the 'skills' feature lets agents auto-generate parsers to save ~99% of HTML-parsing tokens. | Evidence: Rafael states 5k/month is sufficient for MVPs/prototypes; mentions pay-as-you-go for scale; previews 'skills' demo (next session) where an agent is taught to build a Walmart scraper and generate a parser script, so raw HTML is never sent to the LLM—parser extracts structured data, LLM only sees clean JSON. | Caveat: No pricing details beyond free tier; no comparison to competitors (ScrapingBee, Apify, Browserless); no latency or reliability SLAs discussed. | Implication: For Ken: free tier is risk-free for testing; if agents parse thousands of pages, the auto-parser approach (LLM builds the parser once, script runs it) could dramatically cut token costs and latency—worth attending follow-up session or testing on Ken's own scrapers. | Timestamp: 14:00–16:00

Detailed Brief

The Silent Failure Problem: Why LLMs Lie About Web Access

  • Claims: LLMs are trained to please users, so they fabricate results rather than report 'I was blocked by a CAPTCHA' or 'I got an empty page.'; 60% of ChatGPT citations are reportedly 404s; product-link requests often return nonexistent URLs.; Hallucinations stem from the need to please + lack of real data: LLM substitutes training data (years old) or invents content.; Default LLM 'web search' is not real search—it's a background simulation or retrieval from training.
  • Evidence: Rafael's anecdote: ask LLM to find a product and provide link → click link → 404, product doesn't exist.; Training data mismatch: LLM trained on 2024 data in 2026, presents stale info as current.; Live demo: GPT-5 out-of-the-box failed 5/5 scraping tasks with no error messages, just wrong answers.
  • Caveats: No controlled study or sample size for the 60% citation-failure claim.; Hallucination frequency depends on model version, grounding architecture, and task complexity.; Some models (e.g., GPT-4 with browsing, Perplexity) do perform real searches—Rafael's critique targets default/naive setups.
  • Implications: Ken's agents must not blindly trust LLM claims of 'searched the web'—require explicit tool logs or provenance metadata.; Silent failures are worse than errors for production: bad data flows into decisions, content, or customer-facing outputs.; For high-stakes tasks (pricing, link verification, competitive intel), MCP-style tool delegation is mandatory.

The Anti-Bot Ecosystem: CAPTCHAs, Cloudflare, and AI Labyrinth

  • Claims: Web access is an arms race: CAPTCHAs (decade-old), then Cloudflare AI-blocking (~20% of web), now AI Labyrinth (feeds bots fake data).; Cloudflare AI Labyrinth (released ~1 month before talk) deliberately misleads bots instead of blocking them, causing worse hallucinations.; Default LLM fetch/scrape fails silently on ~20% of sites due to Cloudflare protections.
  • Evidence: Rafael cites Cloudflare blocking AI crawling for 20% of the web.; AI Labyrinth serves fabricated pricing, product details, or content to detected bots.; Live demo: GPT-5 couldn't access LinkedIn, Instagram, Amazon, TikTok, Rightmove without MCP.
  • Caveats: AI Labyrinth impact is unclear—Rafael admits limited statistics, and Bright Data sees no degradation (but they bypass detection).; Not all anti-bot systems use adversarial misinformation; some just return 403/empty pages.; The 20% figure is a point-in-time estimate; Cloudflare adoption and AI-blocking policies evolve.
  • Implications: Ken's agents face both access denial and adversarial data poisoning—need to verify data provenance and cross-check critical facts.; Competitors or adversaries could exploit AI Labyrinth to feed agents false pricing, fake reviews, or misleading intel.; Anti-bot defenses will only get more sophisticated; agents need vendor-backed or constantly updated bypass strategies.

Bright Data MCP: 66 Tools, CAPTCHA-Solving, Human Mimicry

  • Claims: MCP provides real search (Google, Bing, DuckDuckGo SERPs), not LLM-simulated search.; Scrape-as-markdown tool returns clean text, skipping HTML parsing and saving tokens.; Batch keyword search can process 100 keywords → 100 SERP results at scale.; Pre-built APIs for known sites (LinkedIn, Instagram, Amazon, TikTok, etc.).; Scraping browser: remote, opens 100+ parallel sessions, auto-solves CAPTCHAs, mimics human behavior (mouse, typing).
  • Evidence: Live demo: 5/5 success with MCP on protected sites vs. 0/5 without.; Rafael walks through tool categories and shows MCP tool list in code.; Browser mimics human behavior: recorded mouse movement, realistic typing cadence, residential IPs.
  • Caveats: 66 tools flood context if all loaded; Rafael advises filtering to 2–5 needed tools.; No cost, latency, or reliability benchmarks provided; demo was brief and vendor-run.; Human-mimicry approach is an arms race—future anti-bot ML may detect synthetic behavior.
  • Implications: For Ken: scrape-as-markdown alone could cut token costs for web-grounding workflows by 50–90%.; Batch search tool is ideal for competitive intel, trend research, or multi-keyword monitoring at scale.; CAPTCHA-solving browser is critical for e-commerce, social, or any site with heavy bot protections.; MCP pattern (delegate web access to specialized tools) is architecturally sound for production agents—consider Bright Data or build equivalent.

Legal and Data Boundaries: Public-Only, No Login Scraping

  • Claims: Bright Data only scrapes public data (accessible without login in incognito mode).; Login-gated scraping is considered illegal because it requires accepting ToS, which often prohibit automated access.; Lawsuits ongoing: LinkedIn, Amazon, others suing scrapers who violate ToS.; Pre-built datasets (e.g., LinkedIn profiles filtered by role/location) available for non-live use cases.
  • Evidence: Rafael demo: opened LinkedIn company page in incognito to show public data; stated home IP might access 5–10 profiles before being asked to log in.; Explicitly rejected audience question about providing credentials for own social accounts.; Mentioned data-center IPs (like event Wi-Fi) get blocked faster than residential IPs.
  • Caveats: Legal safety is not absolute—'public data' scraping still faces litigation (LinkedIn v. HiQ, others).; Definition of 'public' varies by jurisdiction and evolving case law.; Pre-built datasets are months old; live data requires scraping, which may trigger blocks or legal risk.
  • Implications: For Ken: stick to public-only scraping to minimize legal risk; avoid login-gated workflows unless ToS explicitly permits API/automation.; If agents need social/profile data, cached datasets may be faster, cheaper, and safer than live scraping for research tasks.; Monitor vendor's legal posture and indemnification terms if relying on third-party scraping infrastructure.

Token Optimization: Auto-Generated Parsers (Skills Feature)

  • Claims: Bright Data's 'skills' feature teaches agents to build scrapers and auto-generate parsers, saving ~99% of HTML-parsing tokens.; Instead of sending raw HTML to LLM for each page, LLM builds a parser once, then a script extracts structured data and returns clean JSON.; Ideal for high-volume scraping (10,000+ pages) where token costs and latency are prohibitive.
  • Evidence: Rafael previews follow-up session: agent will build a Walmart product scraper live, generate parser, and run it.; Skills page teaches agent APIs, scraping techniques, and parser construction.; Claims 99% token savings by avoiding per-page HTML parsing.
  • Caveats: No technical details on parser auto-generation: is it rule-based, ML-based, or LLM code-gen?; No benchmarks for actual token savings, latency, or parser accuracy/robustness.; Parser approach assumes site structure is stable; brittle if site redesigns HTML.
  • Implications: For Ken: if agents parse thousands of pages, this approach (LLM builds parser once, script runs it) could cut token costs 10–100×.; Worth testing on Ken's own scrapers—measure token usage, latency, and parser maintenance burden.; Attend follow-up session or request demo to see actual code and error handling.

Notable Concepts & Terms

  • AI Labyrinth (Cloudflare): Anti-bot system released ~1 month before talk; instead of blocking detected bots, it serves them fabricated data (fake prices, fake content) to cause hallucinations and waste resources. Rafael notes Bright Data sees no impact because their mimicry bypasses detection.
  • Scrape-as-Markdown: Bright Data MCP tool that fetches a URL and returns only the text content (no HTML tags), saving LLM tokens and eliminating HTML-parsing overhead. Rafael positions this as a key token-cost saver for web-grounding workflows.
  • MCP (Model Context Protocol): OpenAI/Anthropic-backed protocol for extending LLMs with external tools. Bright Data's MCP provides 66 tools (search, scraping, CAPTCHA-solving, etc.) that agents can invoke instead of relying on LLM's simulated 'web search.'
  • Human Mimicry (Anti-Bot Bypass): Bright Data's strategy: record real human behavior (mouse movement, typing cadence) and replay it via remote browsers using residential IPs, so anti-bot systems (Cloudflare, etc.) don't even ask 'are you a bot?' Avoids reverse-engineering detection logic.
  • Skills (Bright Data Feature): A page/interface that teaches agents how to build scrapers and auto-generate parsers for target sites. Agent reads APIs/documentation, writes parser code, then runs it to extract structured data without sending raw HTML to LLM—saves ~99% of parsing tokens.
  • Public Data (Legal Boundary): Data accessible without login in incognito mode. Bright Data only scrapes public data to avoid ToS violations and lawsuits. Login-gated scraping requires accepting terms that often prohibit automation, creating legal risk (LinkedIn, Amazon lawsuits cited).

Operator Notes / Why Ken Should Care

  • For Ken's agent systems: this talk is a must-watch if agents rely on real-time web data (pricing, product links, competitive intel, social profiles). The live demo proves default LLM 'web search' fails silently and hallucinates—MCP tooling is architecturally necessary.
  • Token optimization angle: scrape-as-markdown and auto-parser (skills) could cut web-grounding costs 10–100× for high-volume workflows. Worth testing Bright Data's free tier (5k requests/month) or building equivalent tools.
  • Legal/risk: stick to public-only scraping; avoid login-gated workflows unless ToS explicitly permits. Monitor vendor indemnification if relying on third-party scraping.
  • Cloudflare AI Labyrinth is a new adversarial threat: agents may consume fabricated data without knowing it. Require provenance metadata, cross-check critical facts, and consider vendor-backed bypass solutions.
  • For GTM/content: if Ken's agents generate product recommendations, pricing comparisons, or link-based content, silent scraping failures could produce customer-facing errors or legal liability. MCP-style tooling is risk mitigation.
  • Investing lens: Bright Data is a picks-and-shovels play for AI agent infrastructure; growing as agents scale. Track their competitive moat (human mimicry, petabyte-scale caching, legal posture) vs. ScrapingBee, Apify, Browserless.

Watch Map

  • 00:00–01:00: Intro: Bright Data web access platform; problem statement—LLMs lie about searching the web.
  • 01:00–03:30: Why LLMs hallucinate: need to please users, CAPTCHAs, anti-bot systems, Cloudflare AI Labyrinth (feeds bots fake data).
  • 03:30–04:00: Setup: live demo comparing GPT-5 alone vs. GPT-5 + Bright Data MCP on 5 protected sites.
  • 04:00–06:00: Demo run 1: GPT-5 without MCP fails 5/5 tasks (Rightmove, LinkedIn, Instagram, Amazon, TikTok).
  • 06:00–08:00: MCP tool overview: 66 tools (real search, scrape-as-markdown, batch search, pre-built APIs, CAPTCHA-solving browser).
  • 08:00–09:30: Demo run 2: MCP succeeds 5/5; ChatGPT compares results (failed vs. successful).
  • 09:30–11:00: Q&A: public-only data (no login scraping), legal boundaries, ToS violations, lawsuits.
  • 11:00–12:30: Cached datasets for non-live use cases; filtering by role/location.
  • 12:30–14:00: Anti-blocking strategy: human mimicry (mouse, typing, IPs), not reverse-engineering; Cloudflare AI Labyrinth discussion.
  • 14:00–16:00: Pricing (5k/month free tier, pay-as-you-go); skills feature preview (auto-parser saves 99% tokens); Q&A on tool count, search comparison, batch pricing.

Source/Metadata

  • Title: Your Agent's Biggest Lie: 'I Searched the Web' — Rafael Levi, Bright Data
  • Transcript words: 4930
  • Duration seconds: 948
  • Timestamp note: Timestamps were not present in the transcript; watch_map estimates based on 948-second duration and narrative flow.
Search this video Resume transcript at 0%
Full transcript 2664 words · 22 min read
0:15

SPEAKER_01

Okay, so let's just work with it like this, a small room. Hi everybody, welcome. My name is Rafael, representing Bright Data. Bright Data is a web access platform to help agents or anybody collect public data at scale. And I'm here to talk about LLMs misleading people all the time, convincing them, "Hey, I did a search, hey, I did this," while they didn't. Why? Because LLMs are programmed to please people, please users, so they make up things. And this is the biggest issue right now I'm seeing with LLMs. I'm building applications all the time. I would rather an LLM tell me no, I can't, but it never does. It always tries to make things up. So currently, the web is actually fighting robots and automations. And it's been something that's going on for years. Everybody knows CAPTCHAs. The first CAPTCHAs showed up a decade ago. And it just keeps growing. And now we have AI blocking AI, and there's a whole world going on. And web access is actually not as simple as it looks. So they're getting CAPTCHAs, and they don't actually report CAPTCHAs. So it tries to find the data a different way. Sometimes it goes into training data, and this is where the worst thing is, when it uses training data and tells you that this is the current situation. Training data is from 2024, we're in 2026, and it doesn't add up, right? So these are some of the new things that I was just literally checking out, right? So Cloudflare blocks AI crawling for about 20% of the web, right? So 20% of the web is literally not accessible by AI by default fetch that's built into it. And Cloudflare also right now released an AI labyrinth to actually trap bots and mislead them and provide to them fake data. So then your results are getting even worse, okay? So the invisible failure route, right? There's no error, no warning, just wrong answer, right? So the agent sends the request, it gets a CAPTCHA, even an empty page. It doesn't tell you, "Hey, I got an empty page." It will try to make up something, and this is where most of the hallucinations come from. The need to please and lack of data. So it literally makes things up. I've seen it literally make numbers up, provide fake citations. You click on the citation, it's a 404, the page doesn't exist, and I'm sure all of you have seen that happening recently. I mean, literally, 60% of citations on ChatGPT is not working. How many of you have tried to purchase a product, "Hey, find me this product online, I want to buy it and give me a link to the product." You click on the link to the product and there's no product. Like, so what is this product that you're talking about? The URL doesn't exist, the product doesn't exist, so where do I buy this product for the 50 bucks? It doesn't exist. And so what I want to show you, and I'm going to show you in code, how many of you are familiar with coding like VS code? No. Okay, perfect. So nobody's going to get lost if I'm going to switch to that. So I'm going to do a demo basically with MCP and without the Bright Data MCP, and we're going to compare what is going on there. So first I'm going to show you that I have exactly identical prompts for both of the scripts, right? So without MCP and with MCP. So I give it five tasks. Property, right, move.co. Let's go check out some of the properties. LinkedIn, Instagram, Amazon, and TikTok. These are the five basic sites that I wanted to access. They are very heavy on anti-bot systems. And I want to show you the difference. So first I'm going to run it without an MCP. And I'm going to literally let the AI talk for itself. And GPT-5 is a bit slow, so it's going to take some time. But basically what we're trying to do is we're trying to access this URL. It's some local properties that I did literally half an hour ago. LinkedIn company in Israel. Let's check it out. Instagram account, some Amazon product, and some TikTok. And this is not limited to just these five websites. It's just something that I picked that I know for sure will not work without MCP. So as you can see, without MCP, I don't have live web data access. It doesn't have any browsing tools, right? So this is with tools not available, just by default, out of the box GPT-5. I mean, it's a strong LLM. And zero success, five failed. Same exact thing, exactly the same prompt. I'm running with our MCP. Our MCP has 66 tools. While it's running, I just want to go over some of the tools that it has. Search engine. Search engine is basically the LLM is able to do Google search, Bing search, DuckDuckGo search. Real searches, not just search the web that it's doing in the background. It has scrape as markdown. That's a very strong one. Basically, it can send a curl to any URL and get just the markdown without HTML tags. So you're not wasting tokens on parsing the HTML. Search engine batch, if you want to do like a hundred keywords, you can literally send a hundred keywords and get the hundred keyword backed results. So the scaling is also huge. Discover. It also has pre-built APIs for many websites, as you can see. And of course, it has a scraping browser infrastructure. So it's a remote browser that your LLM can open and navigate. The remote browser solves CAPTCHA by itself. It can open a hundred browsers, navigate the same website without getting blocked. So the whole idea is that with our MCP, not only can it do a single session, it can do multiple sessions in parallel. And as you can see, let's see, we have success for the Rightmove. We have success for the LinkedIn in Israel. Instagram also worked. Amazon product, it has the information for the product itself. And then what I did is on the second part, I asked the LLM to compare the results from no MCP with MCP. So that you don't take my word for it. Let's see what ChatGPT will actually say. If it didn't get stuck, it looks like it's stuck for some reason.

0:22

SPEAKER_01

Can I ask a question? Of course, please. I love questions. For social profiles, do you have accounts that you run? No, we only work with public data. Only publicly available data. Collecting data behind login is not really legal. Why? Because when you sign up and create an account, you accept terms and conditions. When you accept terms and conditions, you need to really check if it says can you scrape? Can you, do you allow, do they allow robots to access the website? And that's why maybe some of you heard there's a lot of lawsuits going on. LinkedIn suing these people. Everybody's suing you. Amazon suing, yes. Can I ask a question? Of course, please. I love questions.

0:58

SPEAKER_01

For social profiles, do you have accounts that you run?

1:03

SPEAKER_01

No, we only work with public data. Only publicly available data. Collecting data behind login is not really legal. Why? Because when you sign up and create an account, you accept terms and conditions. When you accept terms and conditions, you need to really check if it says, can you scrape? Can you, do they allow robots to access the website? And that's why maybe some of you heard there's a lot of lawsuits going on. LinkedIn suing these people. Everybody's suing you. Amazon suing, yes. But I think even, for example, LinkedIn and Instagram, I think we don't even show you really any public data, even if you're on the... That's plenty of public data for LinkedIn, of course. If you take, for example, I will take this URL. I might get blocked. I don't know how the local IP is, but... And I open an incognito window, right? Usually, just like a video. There you go. So this is public data that can be collected. So if you put a person instead of... Company, it's the same thing. It's the same thing. The only thing is they're very critical, right? So if you are using, let's say, a Wi-Fi IP of a big event like this, it probably will block you because it's a data center IP. It's low quality IP. From home, you could probably access maybe 5-10 profiles, but eventually it will also ask you to log in. So we only deal with public data. So when you guys are using us, you are safe in the sense of nobody's going to come knocking on your doors to sue you. If that makes sense.

1:09

SPEAKER_01

Can we provide credentials to our own social networks? No. Again, we don't deal with data behind login. We consider it to be illegal. So we don't deal with accepting terms and conditions and data behind login. Only publicly available. So I imagine you would cache some of these so that you don't have to constantly go and request it. Do you have an idea of how...

1:28

SPEAKER_01

We have a whole data set. So if you guys don't want to do live data and you don't care if it's a few months old, we have data sets, literally, that you can just filter by, let's say, that you're looking... If we're talking about LinkedIn people, you're looking for AI engineers in a certain area, you can filter it and just get the data set right away. And your agent will actually have access to that. So your agent can filter the data set and get you the data if we want to, right? So it has access to all the tools. So he has basically had to get comparison by the LLM itself. Without MCP failed, no live web access. With it listed, failed successful, failed successful. So that's basically it. Anti-bot bypass, capture solving, right? So again, our system automatically solves capture. So if your bot navigates to a website with a browser and it has a capture, our browser has built-in capture solving solution. So it will automatically solve the capture and your bot can continue on browsing without getting blocked.

1:33

SPEAKER_01

How much time do I have? I don't know.

1:40

SPEAKER_01

So just to summarize it, right? This is the biggest hallucinations that you guys see. The agent gets blocked, it needs to please you, and it makes things up. And fake content also, right? So now, if you want to Google what is Cloudflare AI Labyrinth, it's basically a system. Once it detects a bot, it doesn't block it. It literally feeds it fake data. So bigger hallucinations, right? And the easiest fix for this is just to make sure that your agent doesn't get blocked. And it's as easy as to implement our MCP. Our MCP has a free tier of 5,000 requests. So if you guys want to try it out, you can connect with me on LinkedIn if you want to. Or is this the... Hold on a second. Is this the sign up for the MCP? I'm lost in these QR codes. One second.

1:44

SPEAKER_01

Internet. Yes, please. How does it detect if there's Cloudflare Labyrinth or not?

1:56

SPEAKER_01

So the way we approach it is that we make your agent look like a human being. Literally, there's mouse movement recorded. There's typing. When it types, it's like... Mimics real human behavior. So Cloudflare literally just doesn't even ask, are you a robot or not? Right? So this is our approach. Instead of trying to understand how they detect, we make the agent look as human as possible so that it doesn't trigger the actual blockage. Misleading data is one of the toughest things that you can actually encounter. A lot of websites right now in Asia are doing that, right? Hotels, they're literally providing different prices. You check out on your phone, you get one price. You check from your computer, you get a different price. You add through proxy, you get a third price. Which one is correct? It's really hard to tell. When it comes into the domain of misleading, the best bet is to make sure that your agent looks like a human and hope for the best. That's basically what the approach is right now. AI Labyrinth was literally released a month ago. I don't have much statistics on exactly how it's working. All I know is that it didn't really affect us. We don't see any change in data and we are collecting petabytes of data on the daily. We have so many customers always scraping. We're caching data so we're always comparing the results. We don't see any degradation in the results. So I think we're doing a good job in that case.

2:03

SPEAKER_01

This is a QR code that you guys can sign up for. We have a GitHub page as well. GitHub bright data.com. GitHub bright data. And another thing that I would recommend for you guys to check out is the skills page. Right? So what we did is we created skills. And I'm going to have another session in a couple of hours if you're interested in seeing what the skills does. It's basically you can take any agent, tell it to go here and this page will teach it on how to build a scraper or how to build a pipeline that will collect you the data. It has all the information it needs, all the APIs. And if you come back to my next session which is at one o'clock I think? Something like that. I'm going to literally demonstrate how it builds the pipeline. Literally in front of you I'm going to tell it, hey listen let's build a Walmart collector for ABC and it's going to build it and it's going to scrape it. And instead of parsing each individual in ChiptML it's going to build a parser and it saves about 99% of the tokens. Because I see a lot of people like, hey I need to parse 10,000 pages but it's so token heavy. Don't parse with the LLM. LLM builds the parser and then the script runs it. But that's the next session. Any questions?

2:10

SPEAKER_01

session which is at one o'clock I think? Something like that. I'm going to literally demonstrate how it builds the pipeline. Literally in front of you I'm going to tell it, hey listen let's build a Walmart collector for ABC and it's going to build it and it's going to scrape it. And instead of parsing each individual in ChiptML it's going to build a parser and it saves about 99% of the tokens. Because I see a lot of people say, hey I need to parse 10,000 pages but it's so token heavy. Don't parse with the LLM. LLM builds the parser and then the script runs it. But that's the next session. Any questions?

2:17

SPEAKER_01

Question on the performance. I saw that you have CP exposes 69 tools which means if I need to say search capability for my agent, do you need to load all the 69 tools? No. Of course, filter it. I just showed 69 tools because just to show it, if I need just a scrape markdown and search I would just literally load two tools. Otherwise you're flooding contacts with relevant data. Of course not. The experiment at the beginning, does that use the Web search tool? I'm sorry, I didn't hear at the beginning again. The experiment at the beginning, does that use the Web search tool through the OpenAI API? Or is it like for the . And so how does it compare? I'm not that experienced.

2:45

SPEAKER_01

[SPEAKER_04] . So we didn't do any searches. I literally told it, hey go to this URL, see if you can load it. Right? So I didn't use the search in this demo. [SPEAKER_04] Sure. But of course, again, even with our MCP, they can actually do a Google search. Yeah. And that's one of the biggest benefits is because we're used to Google results. Right? So by default, when you're asking LLM, you're expected to do a Google search, but it doesn't. Right. So the results with the MCP, much, much better. And I recommend, sign up, try it out, it's free. See the results, compare what you guys get. Is that 5,000 requests per day or...? Per month. Per month.

3:25

SPEAKER_01

Per month. Which is, you know, for an MVP, for a little experiment, it's more than enough. And we also have pay as you go, so if you do need a little more, it's nothing. Okay. Just for running a set, just for paying a run. Yeah, yeah, it's for pro-type, it's perfect. I always do a lot of hackathons and I'm always recommending, hey listen, set up an MCP and tell your agent to go build whatever you need. [SPEAKER_03] It does a much better job than without. Forget it. Thank you. trying to access this URL. It's some local properties that I did literally half an hour ago. LinkedIn company in Israel. Let's check it out. Instagram account, some Amazon product, and some TikTok.

4:16

SPEAKER_01

And this is not limited to just these five websites. It's just something that I picked that is, I know for sure, will not work without MCP. So as you can see, without MCP, I don't have live web data access. It doesn't have any browsing tools, right? So this is with tools not available, just by default, out of the box. GPT-5. I mean, it's a strong LLM. And zero success, five failed. Same exact thing, exactly the same prompt. I'm running with our MCP.

4:56

SPEAKER_01

Our MCP has 66 tools. While it's running, I just want to go off some of the tools that it has. Search engine. Search engine is basically the LLM is able to do Google search, Bing search, DuckDuckGo search. Real searches, not just like, you know, search the web that it's doing in the background. It has...

5:20

SPEAKER_01

It also has scrape as a markdown. That's a very strong one. Basically, it can send a curl to any URL and get just the markdown without HTML tags. So you're now wasting tokens on parsing the HTML. Search engine batch, if you want to do like, you know, like a hundred keywords, you can literally... It can literally send a hundred keywords and get the hundred keyword backed results. Like, so the scaling is also huge. Discover, it also has pre-built APIs for many websites, as you can see. And of course, it has a scraping browser infrastructure. So it's a remote browser that your LLM can open and navigate. The remote browser solves capture by itself. It

5:59

SPEAKER_01

can open a hundred browsers, navigate the same website without getting blocked. So the whole idea is that with our MCP, not only can it do a single session, it can do multiple sessions in parallel. And as you can see, let's see, we have success for the right move. We have success for the LinkedIn Misrael. Instagram also worked. Amazon product, it has the information for the product itself. And then what I did is on the second part, I asked the LLM to compare the results from no MCP with MCP. So that you don't take my word for it. Let's see what the chat GPT will actually say. If it didn't get stuck, it looks like it's stuck for some reason.

6:48

SPEAKER_01

Can I ask a question? Of course, please. I love questions. For social profiles, do you have accounts that you run? No, we only work with public data. Only publicly available data. Collecting data behind login is not really legal. Why? Because when you sign up and create an account, you accept terms and conditions. When you accept terms and conditions, you need to really check it if it says, can you scrape? Can you, do you allow, do they allow robots to access the website? And that's why maybe some of you heard there's a lot of lawsuits going on. LinkedIn suing these people. Everybody's suing you. Amazon suing, yes.

7:27

SPEAKER_01

But I think even, for example, LinkedIn and Instagram, I think we don't even show you really any public data, even if you're on the... That's plenty of public data for LinkedIn, of course. If you take, for example, I will take this URL. I might get blocked. I don't know how is the local IP, but... And I open an incognito window, right? I mean, usually, just just like a video. There you go. So this is a public data that can be collected. So if you put a person instead of... Company, it's the same thing. It's the same thing. The only thing is they're very critical, right? So if you are using, let's say, a Wi-Fi IP of a big event like this, it probably will block you

8:10

SPEAKER_01

because it's a data center IP. It's low quality IP. From home, you could probably access maybe 510 profiles, but eventually it will also ask you to log in. So we only deal with public data. So when you guys are using us, you are safe in the sense of nobody's going to come knocking on your doors to sue you. If that makes sense. Can we provide credentials to our own social networks? No. Again, we don't deal with data behind login. We consider it to be illegal. So we don't deal with accepting terms and conditions and data behind login. Only publicly available. So I imagine you would cache some of these so that you don't have to constantly go and request it.

8:48

SPEAKER_01

Do you have an idea of how... We have a whole data set. So if you guys don't want to do live data and you don't care if it's a few months old, we have data sets, literally, that you can just filter by, let's say, that you're looking... If we're talking about LinkedIn people, you're looking for AI engineers in a certain area, you can filter it and just get the data set right away. And your agent will actually have access to that. So your agent can filter the data set and get you the data if we want to, right? So it has access to all the tools. So he has basically had to get comparison by the LLM itself.

9:20

SPEAKER_01

Without MCP failed, no live web access. With it listed, failed successful, failed successful. So that's basically it. Anti-bot bypass, capture solving, right? So again, our system automatically solves capture. So if your bot navigates to a website with a browser and it has a capture, our browser has built-in capture solving solution. So it will automatically solve the capture and your bot can continue on browsing without getting blocked. How much time I got? I don't know. So just to kind of summarize it, right? This is the biggest hallucinations that you guys see. The agent gets blocked, it needs to please you, and it makes things up.

10:00

SPEAKER_01

And fake content also, right? So now, if you want to Google what is Cloudflare AI Labyrinth, it's basically a system. Once it detects a bot, it doesn't block it. It literally feeds it fake data. So bigger hallucinations, right? And the easiest fix for this is just to make sure that your agent doesn't get blocked. And it's as easy as to implement our MCP. Our MCP has a free tier of 5,000 requests. So if you guys want to try it out, you can connect with me on LinkedIn if you want to. Or is this the... Hold on a second. Is this the sign up for the MCP? I'm lost in these QR codes. One second.

10:49

SPEAKER_01

Internet. Yes, please. How does it detect if there's Cloudflare Labyrinth or not? So the way we approach it is that we make your agent look like a human being. Literally, like there's mouse movement free recorded. There's typing. When it types, it's like it's... Mimics a real human behavior. So the Cloudflare literally just doesn't even ask, are you a robot or not? Right? So this is our approach. Instead of trying to understand how they detect, we make the agent look as human as possible so that it doesn't trigger the actual blockage. Misleading data is one of the toughest things that you can actually encounter. A lot of websites right now in Asia is

11:31

SPEAKER_01

doing that, right? Hotels, they're literally providing different prices. You go check out on your phone, you get one price. You can check from your computer, you get a different price. You can add through proxy, you get a third price. Which one is correct? It's really hard to tell. When it comes into the domain of misleading, the best bet is to make sure that your agent looks like a human and hope for the best. That's basically what the approach is right now. AI Labyrinth was literally released a month ago. I don't have much of statistics on exactly how it's working. All I know is that it didn't really

12:08

SPEAKER_01

affect us. We don't see any kind of change in data and we are collecting petabytes of data on the daily. We have so many customers always scraping. We're caching data so we're always comparing the results. We don't see any degradation in the results. So I think we're doing a good job in that case. This is a QR code that you guys can sign up for. We have a GitHub page as well. GitHub bright data.com. GitHub bright data. And another thing that I would recommend for you guys to check out is the skills page. Right? So what we did is we created skills. And I'm going to have another session in a couple of hours if

12:47

SPEAKER_01

you're interested in seeing what the skills does. Is basically you can take any agent, tell it to go here and this page will teach it on how to build a scraper or how to build a pipeline that will collect you the data. It has all the information it needs, all the APIs. And if you will come back to my next session which is at one o'clock I think? Something like that. I'm going to literally demonstrate how it builds the pipeline. Literally in front of you I'm going to tell it, hey listen let's build a Walmart collector for ABC and it's going to build it and it's going to scrape it. And instead of parsing

13:23

SPEAKER_01

each individual in ChiptML it's going to build a parser and it saves about 99% of the tokens. Because I see a lot of people is like, hey I need to parse 10,000 pages but it's so token heavy. Don't parse with the LLM. LLM builds the parser and then the script runs it. But that's the next session. Any questions? Question on the performance. I saw that you have CP exposes 69 tools which means if I need to say search capability for my agent, do you need to load all the 69 tools? No. Of course, filter it. I just showed 69 tools because just to show it, if I need just a scrape markdown and search I would just literally load two tools. Otherwise you're flooding contacts with

14:03

SPEAKER_01

relevant data. Of course not.

14:07

SPEAKER_01

The experiment at the beginning, does that use the Web search tool? I'm sorry, I didn't hear at the beginning again. The experiment at the beginning, does that use the Web search tool through the OpenAI API? Or is it like for the . And so how does it compare? I'm not that experienced

14:28

SPEAKER_04

.

14:31

SPEAKER_01

So we didn't do any searches. I literally told it, hey go to this URL, see if you can load it. Right? So I didn't use the search in this demo.

14:41

SPEAKER_04

Sure.

14:42

SPEAKER_01

But of course, again, even with our MCP, they can actually do a Google search. Yeah. And that's one of the biggest benefits is because we're used to Google results. Right? So by default, when you're asking LLM, you're expected to do a Google search, but it doesn't. Right. So the results with the MCP, much, much better. And I recommend, sign up, try it out, it's free. See the results, compare what you guys get. Is that 5,000 requests per day or...? Per month. Per month. Per month. Which is, you know, like for an MVP, for a little experiment, it's more than enough. And we also have pay as you go, so if you do need a little more, it's nothing like, you know...

15:20

SPEAKER_01

Okay. Just for running a set, just for paying a run. Yeah, yeah, it's for pro-type, it's perfect. I always, I do a lot of hackathons and I'm always recommending, hey listen, set up an MCP and tell your agent to go build whatever you need.

15:31

SPEAKER_03

It does a much better job than without.

15:40

SPEAKER_01

Forget it.

15:44

SPEAKER_01

Forget it. Forget it. Forget it. Forget it. Forget it. Thank you.

Reading tools

Type to find a passage

Appearance
Ask this transcript

Notebook

Export Markdown

Add a note