Skip to content

Web search and public data extraction

How can your app find a public fact and check it?

Find a candidate page, open the original and confirm that its publisher, date and wording answer the question. Keep anything the page does not establish unconfirmed.

When should your app search the public web?

Search is useful when someone asks for a public fact and you do not know the publisher’s page. If you already have the official link, start there and check whether it answers the exact question.

For a library workshop, look for the current event notice. The library’s hours page might answer when the building opens. You need the workshop notice or another official page to answer who can attend.

What is the difference between a search result and page text?

A search result helps you choose a page to open. Extracted text is a copy of what a service could read from a page you selected.

Either can miss a date, a reservation condition or information shown in a form. Open the original public page. Check who published it and whether it covers the event you were asked about.

Compare tools

Scroll the table sideways to read every column.

Compare how each service finds public pages and what you still need to check.
ToolWhat comes backLinks, previews or text from a known page.What you still checkHow to confirm the answer at the original source.What affects costThe request, page or credit you pay for.What to check before reuseRules for saving results and using the text.
Brave Search APIWeb result links and short previews.Open the publisher’s page for the date and conditions.$5 per 1,000 Search requests, with $5 in monthly credits.Standard terms restrict storing results. Check your agreement before building a saved-results feature.
ExaSearch for public pages or retrieve text from a known URL.Text helps shortlist sources. Use the original to check missing context.$7 per 1,000 Auto Search requests for up to ten results. Other modes use different rates. Contents is separately priced per page and output type.Check query handling in your agreement and publisher rights before reusing text.
TavilyFind public links with excerpts or retrieve a known page.Check the official page if an excerpt omits a condition or extraction fails.Researcher offers 1,000 monthly credits. Basic and Advanced Search cost one and two credits.Output can generally be kept under Tavily’s terms. Check publisher rights and cross-customer use.
FirecrawlReadable text or another selected format from approved pages.Check the exact original section and table headings.Basic conversion counts pages. Some output and processing options use extra credits.Keep original links and confirm permitted retention or reuse.
Jina AIReader converts a known address. Search finds pages and can return their text.Check effective dates and table context on the original page.Reader and Search count tokens differently. Check the selected service and limits.Send approved public material only and verify the rights for saved text.
ApifyStructured rows from a hosted Actor, with exports such as CSV and JSON.Check what the Actor collects and compare a few rows with the original pages.Compare Actor-specific pricing with compute, storage and transfer as applicable.Keep approved source addresses and export needed results within the dataset retention policy.
Crawl4AIReadable text or structured results from approved pages.Check headings, collapsed content and tables against the original.Budget for your runtime and upkeep or check the separate hosted allowance.Keep extraction settings and review whether you may retain the text.
ScrapySelected fields from approved pages, exported in a format such as CSV or JSON.Check rows, duplicates and blank fields against the actual pages.The BSD-licensed framework runs on infrastructure you provide.Keep extraction rules and approved sample pages for future checks.
BrowserbaseA hosted browser session for your authorized check of a rendered page.Confirm the page shows the expected content after loading.Compare browser hours, session limits and the plan’s allowances.Check retention and who can view logs or recordings before using the session.
TrafilaturaMain text and available metadata from an approved HTML copy.Verify the heading and dates against the original.Apache 2.0 software. Budget for local compute and storage.Keep the approved original reference and decide which text you may save.

What should you count before paying for search?

Count calls per reader question, then count any separate pages or text formats you ask the service to retrieve.

If your app searches once and reads two pages, estimate all three operations using each provider’s rules. A returned link, a request and a retrieved page are different units.

What can you do if a source is unavailable or restricted?

Mark the fact as unconfirmed or look for another official public page. Leave sign-in pages and sites you do not have permission to collect from out of the workflow.

Keep queries free of customer records and private links. Before storing excerpts, check the provider’s output terms and the publisher’s permissions. Your own notes can record a source link and what you verified without keeping a copy of the page.

How can you practise verifying a public fact?

Use a made-up library and three fictional search results. The question is whether this Sunday’s workshop accepts walk-ins.

  1. Sort the sources
    Draw a recent official hours page, a past workshop announcement that accepted walk-ins, and the current event page.
  2. Write the rule only when the page says it
    If the current page says registration is required, record that rule, the event date and the link. If you cannot read the current notice, say the walk-in rule is unconfirmed.