PullMD Part 9: Wiring in Microsoft's MarkItDown — and ripping three parts back out
PullMD is my little self-hosted tool that turns web pages into clean, readable text. This part of the series does not start with a plan. It starts with an itch.
I had a few HTML files sitting on my computer that I wanted to hand to an AI assistant. The tool already handles web pages. So why not local files too? That became a small new feature: drag a file in, get clean text back out.
The small itch
Most of the guardrails came from Claude: no saving uploaded files to the cache, no browser engine for uploads, and the filename goes in a hidden header instead of the web address so it never lands in the server’s visitor log. All sensible. My own reasoning was simpler: why would you ever cache a file you convert once by drag-and-drop? You are not going to fetch it again later.
The whole thing works by drag-and-drop. While testing, the obvious follow-up hit me:
“but on mobile you could turn it into ‘or open file’, right? a downloaded html file is something you might want to convert on mobile too.”
That was my prompt, translated from German. So the drop area became tappable: desktop and mobile, same function behind it, just a different door in front.
A Microsoft library enters the picture
Two days later I stumbled across Microsoft’s freshly released MarkItDown. It is a free library, released under the MIT license (named after the university where it was created), written in Python. It handles a whole zoo of document formats: PDF, Word, PowerPoint, Excel, EPUB, and more. Exactly the kind of tedious parsing work you sensibly hand off rather than maintain yourself.
I gave Claude the repository and asked: can we wire this into PullMD in a sensible way? Claude analyzed the library and came back with a plan. MarkItDown handles only the non-HTML formats. For HTML it would be a step backward, because the tool already does that job. So it becomes a sidecar: a small helper service sitting next to the main one, called only when a document format comes in that the existing system cannot handle.
So far, so clean. Then I looked at the plan in detail. And “just wire it in” turned into “wire it in, but not three parts of it.”
Three parts back out
Reading through Claude’s integration plan, two things caught my eye: the YouTube converter was missing without comment, and for audio it listed “privacy” as the rationale while the plan still called an external service. I pushed back:
“why did you leave out YT transcript? … and what does ‘privacy’ mean if you use openai anyway?”
Claude’s answer walked through the reasons along the code. That became the actual work of this release: not the wiring in, but the deliberate taking back out.
Audio. MarkItDown’s default sends audio files quietly to Google’s speech recognition. For a self-hosted tool, that decision belongs to the operator, not to a silent default. So: a dedicated path where the operator picks the processing engine. Local processing for those who want it, cloud services only for those who deliberately choose them.
YouTube. The YouTube handling went out entirely. Claude’s look into its code found a fragile spot: if a video has no transcript or is blocked, the whole conversion crashes. You lose even the title and metadata, even though they were right there in the HTML. In its place came a dedicated handler with graceful fallback, meaning it still gives you what it can even when the ideal path fails.
Image captioning. Instead of using MarkItDown’s AI layer, the vision calls run through PullMD’s own system. Zero coupling to MarkItDown for the AI part: a MarkItDown update cannot break any of it.
The principle went into the project memory as one sentence: MarkItDown only for file types; fragile and third-party paths we build ourselves.
Keeping the helper small
The next morning, that sentence grew into a bigger architecture question. I put it on the table:
“are we building our own markitdown replacement? would that be bad design?”
Claude’s answer was no, and it was the right one. Do not rebuild the whole Office, EPUB, and spreadsheet parser zoo. Nobody wants to maintain that. But the AI layer belonged outside. Vision and speech-to-text moved entirely out of the Python sidecar and into the main server. My reasoning: if someone does not need documents at all, they should not have to run the helper service alongside everything.
The sidecar shrank to what it does well: documents and YouTube.
At that same spot, a fourth piece almost wandered in: Docling, IBM’s table engine. It gives the best results locally, but needs several gigabytes of storage and memory, with cold starts of 60 to 140 seconds. I left it out. My objection was not technical, it was about the experience:
“if there’s some endless warmup like that, I get frustrated and pin it on ‘what a bad project, this pullmd’.”
The guiding line: a light door for everyone, a heavy self-host door only for those who deliberately open it. Same logic for advanced PDF table extraction. Mistral OCR (OCR stands for Optical Character Recognition, the technology that reads text from images and scans) is available as an option, but opt-in only, so it never quietly costs money.
In the end, one real breaking change remained. The output format changed slightly: source and fetch date now live only in the YAML frontmatter (YAML is a small structured-data section at the top of the file), no longer duplicated in the text body. AI assistants read fewer unnecessary words now. Anyone who needs the old format gets it back via a setting. That turned the release into v3.0.0.
The new model as bouncer
That evening, with v3 finished, Fable 5 appeared, Anthropic’s new flagship AI model. A brand-new model the evening before a release that had just gained a public upload path: new attack surface that nobody had yet looked at with hostile eyes. The most obvious first task could only be one thing.
First use, straight on the real code changes:
“please do a code review and find bugs and security issues.”
Seven real findings, one false positive. The two that gave me the most pause were security issues.
A YAML injection via a special character. The function that puts quotes around text in the frontmatter escaped one type of line break but not another. YAML (the structured-data format) treats a bare carriage return as a line break: an attacker-controlled page title could have smuggled in its own metadata line. The fix was a single extra character in the pattern. A one-character hole. Claude had written the code, and it had slipped through every review before this one.
A zip bomb, or DoS (Denial of Service, a way to overwhelm a server until it stops responding), on the public upload path. Anyone could have uploaded a malicious document that eats all the server’s memory. The fix: conversion runs in a throwaway subprocess with a time limit and a memory cap. A public upload path without a resource ceiling is not something you defer to later.
Then the bugs. A missing safety net meant an error in vision or speech-to-text would have crashed the page instead of failing gracefully. A cache shortcut lost some data when a certain setting was active. An MCP (Model Context Protocol, the channel through which AI assistants talk to tools) path had drifted from the main API (the interface other programs use to talk to PullMD), so one feature was not reachable through it at all. The one false positive turned out to be harmless: the affected data gets thrown away before it ever reaches anyone.
Reading the review stays mandatory. A finding is a hypothesis, not a verdict.
The next morning I spotted a single broken image on a share page. I pulled on it, and a whole sweep came with it. Image URLs had not been converted from relative to absolute against the source domain. An OAuth secret had not been passed through in any compose file since an earlier version. And the opt-out flag for the breaking change was reachable via no path at all. That last gap was the most important one, because a breaking change with no reachable opt-out is just a break.
After all of that: 721 Node tests plus 4 sidecar-guard tests were green, up from 644 when the branch was opened.
Out the door
Squash-merge, tag, release. Minutes later the community post. No triumphant tone: the thing is out, the latest version now points to v3, done. Before the push, a privacy and secrets scan ran across the whole branch. That mattered more to me than clean commit aesthetics, for obvious reasons that have already been their own topic in this series.
The division of labor was the same across the whole arc: I made the decisions, Claude analyzed and built. What sets this part apart from the previous ones is that the most important decisions were not the wiring in. They were the deliberate taking back out. The boring half I handed off. The delicate half I kept.
A third-party library spares you the parser zoo. But every path that either breaks easily or quietly ships data outward is one you had better not adopt unchecked: that one you build yourself, or you at least turn it into a deliberate choice instead of a silent default.
Part 9 of 9 in the PullMD series.