About 
turns a web page into clean, readable text: the article, without the clutter. It’s built on goose-ts, an open-source TypeScript port of python-goose, and the whole thing — library, command line tool and this site — is open source.
Open the tool Request full access · I have an access code
What’s limited about this demo, and why
The public demo is deliberately smaller than the tool. Fetching other people’s pages on behalf of strangers is something I want to do carefully, and this runs on one small server paid for by one person. Here is what’s restricted and the reasoning behind each limit.
- Only a short list of sites can be fetched by URL. The demo fetches from sites that publish openly licensed or public-domain material and where I’ve checked extraction works:
wikipedia.org,wikisource.org,wikibooks.org,wikinews.org,gutenberg.org,nasa.gov,nps.gov,loc.gov,nih.gov,epa.gov,plos.org,theconversation.com. Why: a tool that fetches any address for anonymous visitors can be pointed at sites that don’t want to be scraped, turned into a way to bypass paywalls or used as a relay to attack other servers. Restricting the demo to sites that welcome reuse removes that risk entirely and keeps me on the right side of other people’s terms. - Pasting HTML works for any page. If you’ve already got a page’s source, paste it. Why: nothing is fetched from anywhere, so there’s nobody to impose on.
- The demo doesn’t fetch or score images. Why: every image is another request to someone else’s server, and image scoring is the slowest, most abusable part of an extraction. Signed-in members get image detection.
- Usage is rate limited: 10 extractions per 10 minutes per visitor, and 300 a day across everyone in the public demo. Members get 30 per 10 minutes. Why: it keeps the service available to everyone, stops runaway automation and puts a ceiling on what it costs to run. When a limit is reached, the page tells you and you can try again later.
- Size limits: pages up to 3 MB (5 MB for members) and pasted HTML up to 1 MB. Why: the server is small, and a very large page could take it down for everyone.
- It obeys robots.txt. If a site’s robots.txt asks automated tools to stay away from a page (or can’t be read),
won’t fetch it and says why. News sites often do this. Why: it’s the established way for a site to say “no”, and respecting it is the right default.
- It doesn’t run JavaScript. Pages that build their content in the browser can come back empty or thin. Why: that needs a full headless browser, which is far heavier and costlier than this project is meant to be.
- It doesn’t get around paywalls or logins. It sees what an anonymous visitor would see. Why: that’s the line between a reading tool and a circumvention tool.
- Extraction is heuristic. It’s an algorithm that guesses which part of a page is the article, so some pages (live blogs, index pages, unusual layouts) come out incomplete.
- Nothing is saved. There’s no history, library or bookmarking. Why: not storing what people read is a privacy feature, and it keeps the project simple.
- The server naps when idle, so the first visit after a quiet spell can take a few seconds to wake up.
Full access
Full access lifts the URL list (any publicly reachable page), turns on image detection and raises the limits. It’s by invitation: tell me a bit about what you’d like to use it for and I’ll get back to you by email with an access code. There are no passwords — you enter the code once, and afterwards you can return with an emailed sign-in link.
How it behaves toward other sites
- It identifies itself. Requests carry the User-Agent
regoose/0.2.0 (+https://regoose.com/about), which points to this page. - It respects robots.txt, as above.
- One page per request. It doesn’t crawl, follow links or run in the background.
- It refuses private and internal network addresses, on every redirect and for every image.
Want
to stay away from your site?
Add a rule for regoose to your robots.txt (User-agent: regoose then Disallow: /) and it stops immediately. Or contact me at sam@bapt-base.com and I’ll block it by hand.
See also the Terms of Service, the Privacy Policy and the author page.