← Back to Blog
pullmdweb-scrapingclaude-codeextractionApril 19, 2026 · 9 min read
StandardSimply explained

PullMD Part 3: Beyond Reddit, Beyond Readability

URL → ROUTING GATE → EXTRACTION PATHS
Editorial diagram: URL input feeds a central routing gate that fans out into four extraction-path cards (Cloudflare MD, Reddit JSON, Readability, Trafilatura), each connecting to a Quality Score panel on the right; a small log panel bottom-left tracks health, signal and routed status.Editorial diagram: URL input feeds a central routing gate that fans out into four extraction-path cards (Cloudflare MD, Reddit JSON, Readability, Trafilatura), each connecting to a Quality Score panel on the right; a small log panel bottom-left tracks health, signal and routed status.

This article tells three acts. In the first act, on the morning of March 20, a single sentence blew open the boundaries of a tool I had built for exactly one use case. In the second act, through April, the service matures in increments — driven by an internal briefing I kept for myself. And in the third act, on April 25, I find the bug that had been living under everything for weeks without my noticing — hidden in a default parameter that silently triggered the wrong code branch. No crash and no error message — just a wrong answer that looked good enough to slip past me. Anyone who read Part 2 knows: for three weeks the service ran quietly, Reddit-only, on the phone, only for me. What happened next started with a typo in a question.

Act 1: The Pivot (March 20)

Three weeks after the first deploy, the service ran exactly as I had built it: Reddit URL in, markdown out. The Web Share Target API worked reliably, the PWA was installed on the home screen, and when I shared from the Reddit app, the post landed as clean markdown on screen. For that one workflow the system was done.

Then came March 20 — and through pure accidental use, the service received a URL that wasn’t a Reddit link. Without thinking much about it, I had entered a different URL into the PWA. What came back was an empty result: the service simply had no idea what to do with it. The web extraction logic didn’t exist yet. There was only the Reddit path.

That morning, at 09:33, I wrote into the Claude Code session:

„die URL soll auch bei normalen seiten im markdown erscheinen… das war doch bisher nicht so oder?“ — [the URL should also appear as markdown for normal pages… that wasn’t the case before, was it?]

The answer was clear: no, it wasn’t. The cage had grown from the spec — Reddit first — but the actual need was broader. Any URL. Not just Reddit.

The first extension was Mozilla Readability as an extraction layer: a library Mozilla built for Firefox Reader Mode that pulls the main content from arbitrary HTML — article text, headings, images — without sidebar, navigation, or cookie banners. Turndown came along as the HTML-to-markdown converter. Together they formed a second path: for everything that isn’t Reddit, run through Readability, convert the result with Turndown, deliver markdown.

Shortly after, a third source joined that I had already known from Part 1: Cloudflare offers a native markdown path for sites that support it. When a site has it enabled, Cloudflare delivers clean markdown directly — faster and more reliable than any custom extraction logic, because no HTML sanitization is needed. The service now checks this as the first step: Cloudflare markdown available? Take that. Reddit? Take the ?format=json path. Everything else? Readability plus node-html-markdown.

Three paths where there had been one. The cage was open.

Act 2: The Maturing Phase (April 11–25)

At some point in April I started keeping an internal document — not a formal spec, more a collection of questions I was asking myself while using the tool. The central question: what’s missing when an agent consumes this output? Not a person who reads the text and fills in the gaps — but an agent that needs structured information to work with.

From that collection came three improvements I called P1.1, P1.2, and P1.3 — the first round of Extraction Pipeline v0.2.

P1.1: Turndown → node-html-markdown. Turndown is the classic HTML-to-markdown converter, but it doesn’t support GitHub Flavored Markdown. Tables weren’t output as GFM tables — they were flattened or dropped entirely. Fenced code blocks with a language hint — ```typescript or ```python — were lost: the code block came out without a language tag, and downstream syntax highlighting stopped working. node-html-markdown handles GFM natively: tables stay tables, task lists stay task lists, code fences keep their language hint. One line swap in the pipeline, but a clear difference in the output.

P1.2: Rich Metadata. Up to now the service only delivered markdown — no context about what the text actually was. lib/metadata.js with cheerio now extracts titles from <title> and <meta> tags (og:title, twitter:title), plus description, author, published time. The result: every markdown output gets a YAML frontmatter with these metadata fields. For API clients there’s also a ?format=json variant that delivers metadata and markdown text as a structured JSON object — directly consumable for pipelines that don’t want to parse frontmatter first.

P1.3: Readability Fallback. Readability fails on some pages — paywalls, pure JavaScript apps, sites with no semantic article content in the DOM. The result is outputs shorter than 200 characters, obviously unusable. Rather than delivering nothing in that case, a fallback now kicks in: cleaned body HTML, without Readability’s aggressive sanitization. The source marker in the frontmatter reads readability-fallback, so you can see in the logs how often this path is taken and for which domains.

Up to this point everything was incremental. Then came April 25 — and with it the bug that put everything in question.

Act 3: The Bug That Lived for Weeks (April 25)

The question was actually a different one: is Readability really the best extraction path? I wanted to evaluate that before building further on top of it. So I set up a comparison eval: 15 URLs from the production cache, run through three engines. Readability, @extractus/article-extractor, and Trafilatura — a Python library originally developed for web archiving projects, known for extracting substantial text even from difficult pages.

While reviewing the results, something stood out. The Readability outputs looked different from what I expected — shorter, somehow rougher, without the cleanly structured paragraphs I knew from Firefox Reader Mode. I looked in lib/web.js and traced the code path back.

server.js was passing wantComments=true to extractWeb for all URLs. That was the default — it made sense for Reddit, because Reddit comments should be extracted too. But wantComments=true triggered a separate branch in the convertWithReadability function: when comments are wanted, skip Readability and deliver cleaned body HTML. That branch was designed for Reddit. It was never designed for web URLs. But the default value true applied to all URLs — and therefore to every web request.

That meant: production had been running for weeks on “cleaned body-HTML” output, not on Readability. What I knew as “Readability output” was in reality cleaned raw HTML. My comparison had been measuring the wrong engine against Trafilatura for weeks.

The fix was one line: for web URLs, server.js always passes comments: false. After that, the eval suddenly made sense.

Trafilatura was complementarily strong exactly where Readability actually failed. On a personal blog page — internally code-named nano-banana2 — Readability delivered 107 characters of output after the fix; Trafilatura delivered 14,755 characters, a factor of ~138. On a tech news site: 471 characters with Readability, 1,667 characters with Trafilatura. On another blog: Trafilatura +80% more content. @extractus was byte-identical to Readability for most URLs — plus three hard failures on 15 URLs. No uplift, no further argument.

Readability won where tables needed to be preserved — Wikipedia, API documentation, structured pages. Trafilatura extracts more aggressively and sometimes loses structure in the process. The result wasn’t an either-or decision: Trafilatura as a Python sidecar container, reachable internally over a Docker network. Both engines run in parallel — a quality score picks the better result per request.

The Quality Score (5 Points)

The heuristic in lib/scoring.js is simple and deliberately kept that way — five factors that roughly measure whether an extraction output is substantial:

  • +0.3 if the markdown output is longer than 500 characters
  • +0.2 if the ratio of extracted text to original HTML size is above 2%
  • +0.2 if <article> or <main> is present in the original HTML (a semantic signal that the page has article structure)
  • +0.15 if there is at least one heading in the output
  • +0.15 if there are at least three paragraphs with 40+ characters

Reddit JSON and Cloudflare markdown receive +0.4 instead of the two web-only bonuses — both are “clean by construction,” meaning no junk HTML needs filtering. The score lands in the X-Quality header of every response and in metadata.quality in the JSON output, so as a client you can see how reliable the output was for a given URL. When running two engines in parallel, take the result with the higher score.

When Your Own Production Has Been Running Wrong for Weeks

What does it say when a bug lives for weeks in a self-hosted service without anyone noticing?

For one thing, that the output was “good enough.” Cleaned body HTML for most pages contains the majority of the relevant text — even without Readability. That’s the uncomfortable insight: when the difference between correct and incorrect behavior is hard to see, you don’t see it. Not because you’re not paying attention, but because there’s no visible signal.

That led directly to P3.1: an extraction_log SQLite table that stores source, quality score, and latency for every request. Auto-prune after 30 days. Plus a GET /api/stats?window=-7+days endpoint that aggregates the data — how many requests went through Readability, how many through Trafilatura, how many through the fallback path, which domains consistently produce low quality scores. The next time a parameter default triggers the wrong code branch, the source distribution in the stats endpoint shifts. It’s not a guarantee — but it’s a signal.

For every self-hosted tool: either you build observability in, or you fly blind. The observability doesn’t have to be elaborate. A SQLite table and a JSON endpoint are enough as a first step.

What Came Next

Three extraction paths, a quality score, an MCP server interface, and a PWA frontend — but all of it was still running under a codename I’d have to drop before the public release, for legal reasons. → PullMD Part 4: From My Tool to an OSS Project

Part 3 of 9 in the PullMD series.

← PullMD Part 2: From Spec to Running Container | PullMD Part 4: From My Tool to an OSS Project →