case study · 01

mdingest

live

Articles in, clean Markdown out. Built for the readers that aren't human.

An ingestion service that turns blog posts and newsletters into LLM-ready Markdown — exposed as a versioned HTTP API, a CLI, and an MCP server, all backed by one core.

live service source ↗← all work

the problem

Feeds aren't machine-legible

My own workflow needed it first: I keep a learning vault and feed articles to LLMs constantly. Medium sits behind a paywall wall, Dev.to and Substack wrap content in markup soup, and copy-paste loses title, author, and date.

Existing scrapers try to be universal — one readability pass for the whole web. The output shows it. The insight: three platforms cover most of what engineers actually read, and each has a structured path to the content. Build for those properly instead of everything badly.

the shape

One core, four doors

Everything resolves to a single ingest(url) that detects the provider and delegates. The surfaces are thin.

GET /v1/{provider}?url=

versioned HTTP API — the public surface

mdingest <url>

CLI for humans and scripts

MCP over stdio

local agents: Claude, Cursor, Devin

MCP over HTTP

remote agents hitting the deployed service

url in → detect provider → provider module fetches source → clean → markdown + yaml frontmatter out

the calls

Decisions, and what they beat

01

One ingestion core, four entry points

rejected · Separate implementations per surface

HTTP API, CLI, and MCP (stdio + HTTP) all call the same `ingest()` router. A fix in the core fixes every surface; a new provider ships everywhere at once.

02

A provider module per source

rejected · Generic HTML scraping for everything

Dev.to has a real JSON API, Substack exposes /api/v1/posts, Medium needs a mirror. Structured sources beat readability heuristics — each provider owns its quirks behind one interface.

03

NestJS inside a Cloudflare Container

rejected · A plain Worker with hand-rolled routing

The provider matrix grows. DI, modules, and validation pipes are worth the container — the Worker is just a Durable Object proxy to it.

04

Markdown + YAML frontmatter as the output

rejected · JSON with embedded html

The consumers are LLMs and agents. Frontmatter carries title, author, date, reading time, and source_url — metadata they can use without parsing the body.

05

Validation errors with trace IDs

rejected · Generic 400s

Every failure returns a code (VALIDATION.FAILED, DEVTO.UNAVAILABLE) plus a traceId. When a user reports a bug, the ID is the whole conversation.

118

tests, including per-provider fixtures

4

entry points on one core: API · CLI · MCP stdio · MCP http

3

providers: medium · devto · substack

~0.4s

observed response on a warm container

proof

A real response

Captured from the live API while building this page: one Dev.to article in, 380ms later, publication-ready Markdown out. Frontmatter carries the metadata an LLM actually needs.

try it live
live capture · GET /v1/devto?url=…200 · 380ms
---
title: "We All Have a \"Serious Work\" AI and a \"Just Vibing\" AI. When Did That Happen?"
subtitle: "Hi Guys!!! As you know I wasn't good for well, a week and Now.....   Let's Dive In!!!  I never..."
author: "Dhruv Jani"
date: "2026-09-22T15:27:32Z"
published: "2026-09-22T15:27:32Z"
reading_time: "4 min read"
free: true
source_url: "https://dev.to/dj29/we-all-have-a-serious-work-ai-and-a-just-vibing-ai-when-did-that-happen-5fl2"
provider: "devto"
tags:
  - "discuss"
  - "ai"
  - "gemini"
  - "antigravity"
---

Hi Guys!!! As you know I wasn't good for well, a week and Now.....
![Friends-gif-where-joey-gest-his-role-back-at-DOOL](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/ngfjxrae0sk3rswb1ych.gif)

Let's Dive In!!!

I never consciously decided which AI gets which job. Somehow, I just ended up with a ranking in my head. It just happened — one small decision at a time, until one day you noticed you have an entire unconscious hierarchy, and you can't fully explain how it got built.

Here's mine, laid bare.

## The IDE has its own pecking order

I use Antigravity as my AI IDE now — GitHub Copilot went pay-to-use, so that door closed. Inside Antigravity, I don't use one model. I use two, and which one shows up depends on what's actually at stake.

honestly

What I'd do differently

  • Caching should have been day one, not later. Same article requested twice pays the upstream cost twice — a KV layer with content-hash keys is the obvious next layer.
  • CORS stayed closed longer than it should have. The API is server-callable only today; opening it with rate limits would let people build on it directly.
  • The provider interface earned its keep — Medium's quirks would have leaked everywhere without it. The pattern I underused: shared retry/backoff middleware instead of per-provider handling.

More case studies as the lab ships.

← back to selected work