search⌘KGet started

Extract text from a public web page

Already public, so there is no credential to mint and no header to send. Here's how a document actually gets from a public web page to a txtfetch response.

The problem

A public web page needs no login and no token. Point ?url= at it, and txtfetch fetches the page on the server, on your behalf. A server-side fetch is not the same as a click in a browser, though. It can reach addresses a browser tab never sees, so txtfetch checks and blocks every hop before it connects.

?url=, and why

How The open web fits the decision table, cited and dated
Pre-authenticated URL?Yes — nothing to authenticate, the page is already public
txtfetch path?url=
Link lifetimeAs long as the page stays up
The traptxtfetch reads what the server sends, not what a browser renders.

A public page needs no authentication at all. ?url= reaches it with a plain GET, the same request a browser would make.

How it works

txtfetch fetches the page over http or https, on port 80 or 443. It rejects a URL that carries a username or password, and it re-checks the address behind every redirect. txtfetch blocks a private, loopback, or link-local address. It checks the DNS answer again at connect time, so a rebind cannot slip through. A reply that is not a 2xx status comes back as a typed fetch_failed error. The whole fetch runs under a fixed time and size budget. What comes back is the text the server sent, chrome included. txtfetch draws no line between an article and its navigation or footer.

the script

One call to txtfetch, with the page's URL in the ?url= parameter. There is nothing to fetch first, and no vendor SDK on either side.

Python
import os

import requests

page_url = "https://example.com/blog/2024-annual-report"

r = requests.post(
    "https://api.txtfetch.com/v1/extract",
    headers={"Authorization": f"Bearer {os.environ['TXTFETCH_KEY']}"},
    params={"url": page_url},
)

if r.status_code != 200 or r.json().get("status") == "fetch_failed":
    # A bot block, a cookie wall, or a dead link all land here.
    raise RuntimeError(f"fetch_failed: {r.text}")

print(r.json()["extracted_text"])
JavaScript
const pageUrl = "https://example.com/blog/2024-annual-report";

const endpoint = new URL("https://api.txtfetch.com/v1/extract");
endpoint.searchParams.set("url", pageUrl);

const res = await fetch(endpoint, {
  method: "POST",
  headers: { Authorization: `Bearer ${process.env.TXTFETCH_KEY}` },
});

const body = await res.json();
if (!res.ok || body.status === "fetch_failed") {
  // A bot block, a cookie wall, or a dead link all land here.
  throw new Error(`fetch_failed: ${JSON.stringify(body)}`);
}

console.log(body.extracted_text);

txtfetch ships no connector, plugin, or client for The open web. It needs none. A public page needs no credential, so the URL is the whole request.

frequently asked questions

Does txtfetch run a browser to fetch the page?
No. txtfetch runs no headless browser and no JavaScript. It fetches the raw HTML a server sends and parses that. A page whose text loads only after a script runs returns just the shell the server sent.
Does txtfetch check robots.txt before fetching a page?
No. txtfetch checks neither robots.txt nor a site's terms of service. Following those rules for any page you fetch is the caller's responsibility.
What happens if the URL points at a PDF instead of a web page?
txtfetch reads it as a PDF, not as HTML. The Content-Type and Content-Disposition headers name the file, and txtfetch checks the bytes too, so a mislabeled file still parses correctly.
Why did my fetch fail against a page that loads fine in my browser?
A bot block, a cookie wall, and a login wall all return a non-2xx status, and txtfetch turns that into a typed fetch_failed error. A private, loopback, or link-local address fails the same way, on purpose.

sources

Related

other sources

Point it at a URL.

Pass the page's address and the text comes back. There is no credential to mint and no connector to install.

Get an API key →

See the URL docs →