Turn a website into clean text you can search.

crawl follows a site's links, pulls the readable text out of every page, splits it into passages and saves files any knowledge base can import.

Buy crawl for $49

One payment of $49, plus sales tax where it applies, and every future update. Delivered as a private GitHub repository you're invited to.

python -m webcrawl https://dominares.org --out ./kb --max-pages 8

Run summary (stdout)

{
  "ok": true,
  "pages": 7,
  "chunks": 84,
  "errors": 0,
  "out_dir": "./kb"
}

kb/chunks.jsonl: one passage from each of the first 3 pages (84 in total)

{"id": "89954e29b1085301-65c02f1af10ca592-0",
 "url": "https://dominares.org",
 "title": "crawl: turn a website into clean, searchable text | Dominares Tools",
 "text": "Turn a website into clean text you can search."}{"id": "4e7d1b845266d582-701995221c94124f-0",
 "url": "https://dominares.org/access",
 "title": "Get access | Dominares Tools",
 "text": "Already bought crawl? Enter the GitHub username you bought it for. If that account has a purchase, it gets access or a f…"}{"id": "e703e66fd6c21c8c-1c96f757d43322ed-0",
 "url": "https://dominares.org/license",
 "title": "License | Dominares Tools",
 "text": "This is the license included in the crawl repository as LICENSE."}

Real output from crawl 1.0.1, run against dominares.org on 13 September 2026. Passages shortened to fit.

What it does

Crawls politely: Follows links within a site, obeys robots.txt and its crawl delay, can space out requests to each host, and can start from sitemap.xml.
Keeps the words: Pulls each page's title and readable text out of the HTML. PDFs and JavaScript-rendered pages work with optional extras.
Re-reads only changes: --resume skips pages whose content hasn't changed and asks servers whether a page is newer before downloading it.
Splits for retrieval: Breaks text into passages sized for search and AI retrieval, each with an id built from its page and content.
Searches two ways: Keyword ranking is built in. For search by meaning, use OpenAI embeddings or plug in any embedding model. An offline word-matching index is included for testing without a key.
Sends it anywhere: Portable JSON files, a local JSON or SQLite vector store, pgvector, Pinecone, or your own webhook.
Runs as a service: An HTTP job API with API-key auth and per-client rate limits that refuses private, loopback and cloud-metadata addresses.

What you get

crawl 1.0.1: the Python library, the webcrawl command and the HTTP API, with 220 automated tests. Python 3.10 or newer; the base install needs only httpx.

A commercial license. Use it and change it in your own projects and your clients' projects. Don't share or resell the source. Read the license.

Every update. New versions land in the same repository; git pull to get them.

A bonus pack of 11 small tools, in the same repository:

Tool	What it does
`ad-gallery-generator`	Build one sectioned gallery.html from a folder of PNG images (the page links to the images next to it).
`apple-notes-exporter`	Export a macOS Apple Notes folder to normalized JSON.
`bare-catch-linter`	Find empty JavaScript catch {} blocks that silently hide failures.
`branding-leak-guard`	Fail a build when forbidden brand or identity words appear in what you ship.
`email-groups-copier`	Split a big email list into copyable groups, offline in the browser.
`env-drift-check`	Detect Python environment drift before tests and gates run.
`gate-runner`	Run a repo's gate commands in order and stop at the first failure.
`git-secret-scan`	Scan a git repository's history, staged files or working tree for committed secrets.
`html-to-png-renderer`	Screenshot every HTML file in a folder to a crisp 2x PNG.
`live-homepage-snapshot`	Save the exact HTML a public URL returns right now to a timestamped file.
`update-bundle-kit`	Build and apply self-contained software update bundles with rollback.

How buying works

Enter your GitHub username. You'll see the account before you pay, so you can check it's yours.
Pay $49, plus any sales tax shown at checkout, on Stripe's checkout page.
Accept the invitation GitHub emails you. Signed in as that account, you'll also find it at github.com/dominares-tools/crawl/invitations. Invitations expire after 7 days; you can get a new one any time.

Clone and install:

git clone https://github.com/dominares-tools/crawl.git
pip install ./crawl

Questions

Do I need GitHub?: Yes. Access is a read-only invitation to a private repository, sent to the GitHub account you choose.
Can I get a refund?: Yes, within 14 days, no questions asked. Access ends when the refund goes through. Refund policy.
Who takes the payment?: Stripe. Your purchase is sold through Link, Stripe's checkout service, which also works out sales tax and sends your receipt.
Something wrong?: Email josephlinares02@gmail.com. Every buyer shares the same repository, so support happens by email rather than in GitHub issues.