The JavaScript → TypeScriptmigration benchmark
We migrated five real open-source codebases with Refactyl, with airbnb/ts-migrate, and with a raw LLM, then let the TypeScript compiler judge every output. No cherry-picking: these are the numbers.
This benchmark measures the JavaScript to TypeScript migration. Run yours free, up to 100 files, no credit card.
ts-migrate: 0 of 5 repos
The 600K-downloads/month incumbent failed to convert a single file on any repo. It crashes on modern Node, abandoned in practice, not just on paper.
Raw LLMs hide errors behind “any”
The same model, prompted naively, floods code with any (up to 40/KLOC), silencing the compiler instead of typing the code. Error count alone is a gameable metric.
Refactyl wins both metrics at once
Fewer compiler errors AND 3–30× lower any-density on all five repos. The compiler-feedback repair loop fixes errors with real types, not escapes.
Results
Judge: npm install + tsc --noEmit on each tool's delivered output. errors = remaining TypeScript compiler errors (lower is better). any/KLOC = explicit any per 1,000 lines of produced TypeScript (lower is better; it measures whether the code is actually typed).
These are single-pass counts (one transform, then measure) so you can see the raw model output before our repair loop touches it. The shipped product runs Verify & Repair until tsc is clean or the file is honestly flagged.
| Repository | Refactyl errors · any/KLOC | Raw gpt-4.1-mini errors · any/KLOC | ts-migrate |
|---|---|---|---|
| expressjs/multer @ v1.4.5-lts.1 | 7 · 10.3 | 40 · 25.3also broke npm install | did not run |
| winstonjs/winston @ v3.13.0 | 42 · 4.9 | 93 · 39.7 | did not run |
| debug-js/debug @ 4.3.4 | 22 · 0.9 | 44 · 28.2 | did not run |
| websockets/ws @ 8.17.0 | 90 · 8.9 | 133 · 18.9 | did not run |
| expressjs/serve-static @ v1.15.0 | 1 · 0.0 | 10 · 3.8 | did not run |
Run of July 3, 2026. Repos cloned at the pinned tags shown; test/bench directories intentionally remain JavaScript in phase one (standard migration practice) for every tool equally. “did not run” = ts-migrate-full crashed before converting any file.
Why we publish two numbers
Any tool can reach “zero errors” by writing const user: any everywhere. That was ts-migrate's documented approach (any + @ts-expect-error by design), and it's what raw LLM prompting drifts toward. You end up with files that compile but aren't typed: no autocomplete, no refactoring safety, no caught bugs.
So we hold every tool to both standards at once: how many compiler errors remain, and how much of the “typing” is actually escape hatches. Refactyl's pipeline is built for that combination: deterministic transforms for the known-in-advance changes, model conversion for the judgement calls, and a compiler-feedback loop that repairs each file against its real tsc errors with precise types (never @ts-ignore, never any as a fix).
The same loop runs inside every Refactyl migration you start. The benchmark measures the product, not a demo build.
Methodology (reproducible)
- Five unmodified open-source repos, cloned at the pinned tags in the table.
- All three tools received the identical file set to convert and the identical judge:
npm install --ignore-scriptsfollowed bytsc --noEmit(same pinned TypeScript for all). - Raw-LLM baseline uses the same model Refactyl uses (gpt-4.1-mini), prompted per file (“convert this to TypeScript”), plus a prompted tsconfig and package.json, a faithful stand-in for pasting files into a chatbot.
- ts-migrate ran via its official
ts-migrate-fullpipeline on a fresh git copy of each repo. any-density counts explicitanyannotations (: any,as any,<any>) per 1,000 lines of produced TypeScript.
Run it on your codebase
Upload a repo, get compiler-verified TypeScript back: typed signatures and real interfaces, not any-everywhere. Your first migration is free.
The Vue 2 → Vue 3migration benchmark
We migrated five real open-source Vue 2 codebases with Refactyl, with vue-codemod, and with a raw LLM, then let @vue/compiler-sfc judge every output. Same dual-metric philosophy as the JS→TS bench: correctness and hidden-migration density.
See the full Vue 2 to Vue 3 migration page. Run yours free, up to 100 files, no credit card.
vue-codemod: incomplete coverage
The official Vue 3 codemod handles basic cases but misses template-compile failures. Like ts-migrate, it produces outputs that pass lint but fail the real template compiler gate.
Raw LLM hides Vue 2 behind v2 residue
The naive LLM outputs SFCs that compile, but leaves beforeDestroy, this.$on(), and this.$set() in place. The v2 residue metric catches this; SFC errors alone do not.
Refactyl: compiler-clean + low residue
Deterministic breaking-change rules cover the API surface; the SFC gate validates every file; the CSS transition rename post-pass catches what AI skips. Both metrics move together.
Results
Judge: npm install --ignore-scripts + @vue/compiler-sfc parse + compileTemplate() on each tool's delivered output. SFC errors = files that fail template compilation (lower is better). v2 residue/KLOC = count of Vue 2 API patterns still present per 1,000 lines of .vue output (lower is better; measures how much Vue 2 surface the tool quietly left in place).
Benchmark run in progress. Numbers will appear here once published. cd backend && npx tsx bench-vue.ts all
Why two metrics for Vue too
The Vue 3 template compiler catches broken syntax, but it doesn't catch Vue 2 API calls that were silently removed. A component that calls this.$on() or uses beforeDestroy will compile cleanly with @vue/compiler-sfc, then fail at runtime when Vue 3's event bus is gone and the lifecycle hook is never called.
So we hold every tool to both: does the SFC compile, and how much Vue 2 API surface did it leave in place. A tool that converts the template syntax but keeps this.$set() throughout the script looks clean on SFC errors and is a latent bug on v2 residue.
Refactyl's Vue 2→3 engine applies breaking-change rules as deterministic transforms first (not LLM guesses), then validates every file against the real compiler. The CSS transition-class renames (v-enter → v-enter-from) run as a deterministic regex post-pass because models reliably skip <style> blocks.
Migrate your Vue 2 project
Every converted .vue file validated by @vue/compiler-sfc before handoff. Options API preserved. Your first migration is free.
The Next.js Pages Router → App Routermigration benchmark
We migrated real Next.js repos with Refactyl and with a raw LLM, judged by a real next build. Dual metric: build errors and Pages-era API residue (getServerSideProps, useRouter from next/router, etc.) per KLOC.
See the full Next.js Pages to App Router migration page. Run yours free, up to 100 files, no credit card.
Raw LLM: Pages API residue
The naive LLM produces App Router files, but leaves getServerSideProps, useRouter from next/router, and next/head imports in place. The build error count looks better than the actual migration completeness.
Refactyl: deterministic path mapping + lint gate
pages/ → app/ path mapping is deterministic. A dedicated App Router lint (no pages-era exports, no next/router, correct export shapes) backs the next build gate, catching what the compiler alone doesn't.
The @next/codemod gap
Vercel's official codemod handles Next.js version bumps but not the full Pages → App Router migration. Layout merging, server/client component splits, and data-fetching rewrites are out of scope for the codemod.
Results
Judge: next build on each tool's delivered output. Build errors = pages that fail next build (lower is better). Pages-API residue/KLOC = count of Pages Router API patterns (getServerSideProps, useRouter from next/router, next/head, next/document) remaining per 1,000 lines of output (lower is better; measures how much the tool actually migrated vs patched the surface).
Benchmark run in progress. Numbers will appear here once published. cd backend && npx tsx bench-next.ts all
Why two metrics for Next too
next build only catches structural errors: missing layout files, invalid page exports, missing 'use client' directives. It doesn't catch a file that still calls getServerSideProps (ignored silently in the App Router) or imports from next/router instead of next/navigation (runtime error, not a build error).
So we hold every tool to both: does the build succeed, and how much Pages-era API surface did it leave in place. A raw LLM that moves files to app/ but keeps useRouter from next/router throughout looks clean on build errors and is a latent runtime failure on residue.
Refactyl's Next.js engine includes a deterministic App Router lint that checks for these patterns before handoff, the same lint that's already enforced in the production product.
Migrate your Next.js app to App Router
Deterministic path mapping, layout merge, and a real next build gate. Your first migration is free.