Website Content Extraction API: Clean Text from Any Page

What a website content extraction API does

Download any business homepage and look at the source. A page that reads as four paragraphs in a browser is hundreds of kilobytes of HTML, most of it menus, tracking scripts, inline styles, image markup and repeated footer text. The words you wanted are in there, but finding them by hand means writing selectors for each site, and selectors break the next time the site changes its theme.

A website content extraction API does that job once, for every site. It fetches the page, works out which part of the document is the main content, discards the rest, and returns the readable text with the metadata that describes the page. The Locomint extract endpoint returns, for each page, a status, title, description, language, a headings list, the content itself, and a word_count.

Because the endpoint was built for business data, it also pulls the contact points from the same fetch: emails, socials, whatsapp, contact_form_url and tech_stack, at no extra credit.

Why raw HTML is the wrong input for a language model

Cost first. Language models are priced by token, and a raw HTML page spends most of its tokens on markup that carries no meaning. The extracted text of the same page is a fraction of the size, so more pages fit in a context window and the model is not reading cookie notices and menu labels. Set max_chars when the downstream step needs only the opening of each page; the default is 50,000 characters.

Then quality. Asked to summarise a page, classify a business or answer a question from a document, a model does better with paragraphs under headings than with a soup of tags. Markdown keeps the structure that matters, which text is a heading and which is a list item, and drops everything else. That is why markdown is the default output.

Then repeatability. A pipeline that feeds pages into a model needs the same page to produce the same text next week. Server-side extraction with one fixed rule set gives you that; a per-site scraper does not.

Your first extraction request

The endpoint is POST /v1/websites/extract. The body carries a list of full URLs, up to 20 per call, and an optional output of markdown or text. Each page fetched costs one credit; a page the API refuses to fetch (status refused) costs nothing. Authenticate with the X-API-Key header; a free key from the signup page gives you 200 credits a month with no card.

curl -X POST "https://api.locomint.io/v1/websites/extract" \
  -H "X-API-Key: $LOCOMINT_KEY" -H "Content-Type: application/json" \
  -d '{"urls": ["https://www.rfc-editor.org/rfc/rfc2142"], "output": "markdown", "max_chars": 20000}'

Send the full URL with its scheme. A bare domain such as example.com is not a URL, and one bare domain anywhere in the list fails the whole call with a 400 and code invalid_request; validate the list before you post it.

The same request in Python, as a complete script:

import os

import httpx

resp = httpx.post(
    "https://api.locomint.io/v1/websites/extract",
    headers={"X-API-Key": os.environ["LOCOMINT_KEY"]},
    json={"urls": ["https://www.rfc-editor.org/rfc/rfc2142"],
          "output": "markdown", "max_chars": 20000},
    timeout=120,
)
resp.raise_for_status()

for row in resp.json()["data"]:
    print(row["status"], row.get("title"), row.get("word_count"), "words")
    print(row.get("headings"))
    print((row.get("content") or "")[:500])

Set the client timeout above 90 seconds. The server gives a batch 90 seconds; if the fetches do not finish in that time it answers 504 with code timeout and charges nothing, and you should send fewer URLs. A client timeout of 30 or 60 seconds cuts the connection before that answer arrives, which leaves you unable to tell a slow batch from a dead one.

Markdown or plain text: choosing the output

Both outputs come from the same extraction and cost the same, so the choice is about what the next step needs.

• Markdown keeps headings as # lines, lists as bullets, links as [text](url) and emphasis as asterisks. Use it for anything that needs the structure: feeding a language model, building a knowledge base, rendering a preview, or importing documentation from an HTML site.
• Text returns paragraphs separated by blank lines and nothing else. Use it for full-text search indexes, keyword matching, language detection, or any statistic where the markup would only get in the way.

If you are unsure, take markdown. Stripping it down to text later is a one-line regular expression; putting structure back is not possible.

What each status value means

Every row carries a status. A pipeline that only checks whether content is empty loses pages without knowing why; one that branches on the status can retry the right ones and drop the rest. All eight values:

The two statuses that surprise people are thin and blocked. A thin page is not a failed fetch; the server answered, but the words are not in the HTML because a script writes them after load. A blocked page is the site saying no. Locomint reads public pages, keeps its request rate conservative, and stops on any block. It never solves a CAPTCHA, signs in, or routes around an anti-bot system, and a pipeline built on it should treat those rows as final. Do not fetch the same URL through your own proxy afterwards; that is the circumvention the status exists to prevent.

Python: turning a list of URLs into clean documents

The script below reads URLs from a text file, one per line, sends them in batches of 20, and writes each successful page to its own markdown file, plus a CSV index with the status, title and word count of every URL. On a 429 (code rate_limited) it sleeps for the number of seconds in Retry-After. On a 402 (code quota_exceeded, nothing charged) it stops, so a quota problem never looks like a fetch problem.

import csv
import os
import re
import sys
import time
from pathlib import Path

import httpx

API = "https://api.locomint.io/v1"
KEY = os.environ["LOCOMINT_KEY"]
INPUT = "urls.txt"          # one full URL per line
OUT_DIR = Path("pages")
INDEX = "index.csv"
BATCH = 20                  # the endpoint's maximum per call


def extract(client, urls):
    """POST one batch, waiting on rate limits and stopping on quota exhaustion."""
    while True:
        resp = client.post("/websites/extract", json={
            "urls": urls,
            "output": "markdown",
            "max_chars": 40000,
            "include_contacts": True,
        })
        if resp.status_code == 429:
            wait = int(resp.headers.get("Retry-After", "5"))
            print(f"rate limited, waiting {wait}s", file=sys.stderr)
            time.sleep(wait)
            continue
        if resp.status_code == 402:
            sys.exit("monthly quota reached; nothing was charged for this call")
        resp.raise_for_status()
        return resp.json()["data"]


def filename_for(url):
    stem = re.sub(r"[^a-z0-9]+", "-", url.lower()).strip("-")
    return OUT_DIR / f"{stem[:120]}.md"


urls = [line.strip() for line in open(INPUT, encoding="utf-8") if line.strip()]
OUT_DIR.mkdir(exist_ok=True)

with httpx.Client(base_url=API, headers={"X-API-Key": KEY}, timeout=120) as client, \
        open(INDEX, "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerow(["url", "status", "title", "language", "word_count",
                     "emails", "contact_form_url", "file"])
    for start in range(0, len(urls), BATCH):
        batch = urls[start:start + BATCH]
        for url, row in zip(batch, extract(client, batch)):
            path = ""
            if row["status"] == "ok" and row.get("content"):
                path = filename_for(url)
                path.write_text(
                    f"# {row.get('title') or url}\n\n{row['content']}\n",
                    encoding="utf-8")
            writer.writerow([
                url, row["status"], row.get("title") or "",
                row.get("language") or "", row.get("word_count") or "",
                "; ".join(row.get("emails") or []),
                row.get("contact_form_url") or "", str(path),
            ])
        print(f"processed {min(start + BATCH, len(urls))} of {len(urls)}")

Rows come back in the same order as the URLs you sent, which is why the script can zip them. Put 10 URLs in urls.txt for the first run and read the index before you commit a larger batch. Each page fetched is one credit, so a 200-line file spends a full free month; the free GET /v1/usage call shows what is left.

Contacts and links come with the text

With include_contacts left at its default of true, every row also carries emails, socials, whatsapp, contact_form_url and tech_stack, extracted from the same fetch at no extra cost. The email list contains role mailboxes on the site's own domain, such as info@ or hello@; person-named addresses are dropped before the response is built, in line with the policy on our privacy page. If you only want the text and would rather not receive contacts at all, set "include_contacts": false.

Pass "include_links": true to get up to 500 outgoing links per page. That turns the endpoint into a simple crawler seed: extract a homepage, collect the internal links from the response, and send the ones that matter in the next batch. Keep it to the same host and keep it small; the point is the pages you need, not the whole site.

If the pages you are extracting belong to businesses you found through a search, you may not need this endpoint at all. The place details endpoint already reads the business website and returns the same enrichment block as part of the record. Use extraction when you start from URLs; use details when you start from a category and a city. The guide to finding business email addresses walks through both routes.

Where to start this week

Take ten URLs you know well: your own site, a supplier, a few competitors, a documentation page and a news article. Run the single-URL script on each and read the markdown next to the live page. That shows you what the extraction keeps, what it drops, and where a page needs a more specific URL. Then run the batch script on a real list of 50 and read the status column of the index before anything else; the split between ok, thin and unreachable is what your source list is worth.

If the list came from a place search, the Python tutorial on building a prospect list shows how the details endpoint fits in front of this one. Both experiments fit in the free plan's 200 credits with no card, and the Starter, Growth and Scale plans on the pricing page raise the monthly credits without changing a line of your code.

Frequently asked questions

Does the extraction API render JavaScript?

No. It reads the HTML as the server sends it. Pages that build their content in the browser come back with status thin and very little text. For most business websites, blogs, documentation and news pages the server-rendered HTML holds the full article, so this is rarely a problem; single-page apps are the exception.

What is the difference between markdown and text output?

Markdown keeps headings, lists, links and emphasis as lightweight markup, which is useful when a language model or a renderer needs the structure. Text strips all of that and returns paragraphs only, which is smaller and simpler for search indexing or keyword matching. Both come from the same extraction and cost the same one credit per page.

Will the API fetch a page that blocks bots?

No. If a site answers with a block, a challenge page or a refusal, the row comes back with status blocked or refused and the API stops there. Locomint reads public pages with conservative rate limits and does not try to get around CAPTCHAs, logins or anti-bot systems. Treat those statuses as final for that site.