TL;DR: We built and open-sourced a 1,140-sample, 6-language benchmark for the one thing general OCR benchmarks don't test: burned-in video subtitles sitting next to a watermark or credits line. Every engine we tested — PaddleOCR (all three PP-OCRv6 size tiers), EasyOCR, Tesseract, and GeekLink's own bundled models — loses roughly 2.6 to 2.9x accuracy (character error rate) the moment a watermark-style text region overlaps the subtitle, regardless of how good the engine otherwise is. We also found PP-OCRv6 small and medium are statistically tied on accuracy at 3.7x the speed difference, and that GeekLink's real Mac backend (CoreML) is 5.8x faster than the plain-CPU number most benchmarks would report, for no real accuracy cost. Then we checked the whole ranking against 1,494 real (private, unpublished) production correction samples — and it flipped: PP-OCRv6, which led on our clean synthetic data, drops behind GeekLink's older bundled models once the video gets messy. Full dataset, eval script, and every prediction file are open on GitHub.
Correction (2026-08-21): an earlier version of this dataset had 437 samples (38%) where the ground-truth text was wider than the video frame, so the rendered subtitle got clipped off-screen. All affected samples have been re-rendered with proper line wrapping, and the dataset extended from 600 to 1,140 samples in the same pass. Every number on this page reflects the corrected data.
Why Existing OCR Benchmarks Don't Test This
We build GeekLink, a tool that extracts burned-in subtitles from video. Every OCR benchmark we could find tests document scans, receipts, or street signs. None of them test the specific, narrow, constantly-annoying problem we deal with in production every day: a subtitle line sitting a few pixels away from a channel watermark, a "restored by" credit, or a streaming platform's logo — and an OCR engine that can't reliably tell which text is the actual subtitle.
Document-OCR benchmarks (FUNSD, SROIE, ICDAR) test flat, single-purpose images — a scanned invoice has one thing on it, an invoice. Scene-text benchmarks get closer, testing text against cluttered backgrounds, but they're built from arbitrary photos, not the specific composition of a burned-in subtitle: consistently bottom-anchored, consistently short, and — in a meaningful share of real video — sharing screen space with whatever the person who encoded the file stamped on top of it. So instead of arguing about whether that gap matters, we built a benchmark that isolates it.
How We Built the Test Data
We didn't use real user video for the public dataset. We started to, and reversed course. An earlier pass built on real (anonymized) OCR-correction data from GeekLink users turned up enough privacy and copyright edge cases that we scrapped it: a portrait-orientation video where a subtitle-position heuristic broke down and nearly surfaced a bystander's face in a crop; adult content mixed into the same raw correction pool; a personal documentary with real names in it. None of that is a reasonable thing to publish, however anonymized the pipeline is supposed to be.
So the public 1,140-sample set is synthetic-but-realistic instead: real footage, real difficulty, zero real people, zero rights ambiguity. The images are frames from three openly-licensed films — The General (1926, Public Domain Mark), Nosferatu (1922, CC0), and a 1955 Cinemascope travelogue of San Francisco (CC BY-SA) — with subtitle lines burned in via the same ffmpeg + libass pipeline GeekLink itself uses for hard-subtitle export. The subtitle text is short, colloquial lines adapted from public-domain literature (Sherlock Holmes, Dracula, Alice in Wonderland, Pride and Prejudice, A Christmas Carol, The Wizard of Oz, Moby-Dick, and others), translated into English, Spanish, Japanese, Korean, Chinese, and Greek — not scraped, and there's exact ground truth for every line because we wrote it ourselves. Lines too wide for the frame at render size are automatically wrapped onto multiple lines rather than clipped.
About 19% of samples (213 of 1,140) have a second line of text rendered near the subtitle — a fake "archive restoration" credit or channel-style watermark — specifically to stress-test whether an engine can isolate the real subtitle from nearby noise instead of just dumping every text region it finds.
What "Raw Output" Means Here
Every engine is scored the same way: concatenate every text region the engine detects, and diff that against the single-line ground truth using character error rate (CER) and word error rate (WER, via Levenshtein edit distance). No cropping to the subtitle region, no manual region selection, no post-processing to drop low-confidence or off-position boxes.
This measures underlying detection+recognition difficulty, not any tool's finished product-level accuracy. A real subtitle-extraction product adds a selection step on top of raw OCR output — positional filtering, frequency-based anchor detection, confidence thresholds — specifically to solve the watermark problem this benchmark surfaces. These numbers argue that selection step is necessary, not that it doesn't exist in real products.
Results: PaddleOCR vs. EasyOCR vs. Tesseract vs. GeekLink
All CPU, no GPU, same Apple Silicon machine, model load time excluded. Speed is averaged over the 101 English samples.
| Engine | Overall CER | Clean CER | Watermark CER | ms/image (CPU) |
|---|---|---|---|---|
| PP-OCRv6 tiny | 0.658 | 0.507 | 1.358 | 169 |
| PP-OCRv6 medium | 0.597 | 0.451 | 1.272 | 1399 |
| PP-OCRv6 small | 0.599 | 0.448 | 1.298 | 381 |
| GeekLink (CoreML) | 0.609 | 0.460 | 1.302 | 97 |
| EasyOCR | 0.622 | 0.483 | 1.286 | 591 |
| Tesseract | 0.946 | 0.888 | 1.216 | 106 |
GeekLink's bundled models are PaddleOCR's own official PP-OCRv5 weights (PP-OCRv4 for Japanese specifically), converted to ONNX/CoreML for local inference — not a custom-trained model. We're still collecting real correction data toward eventually fine-tuning our own detection and recognition models. ("ms/image" is CPU for the PP-OCR tiers and Tesseract; GeekLink's number is its real Mac backend, CoreML — see the Apple Silicon section below for the full breakdown.)
How Much Does a Nearby Watermark Hurt Accuracy?
Every single engine we tested loses roughly 2.6 to 2.9x accuracy (CER) the moment a watermark-style text region overlaps the subtitle — including PP-OCRv6, the newest and most accurate model in the comparison. PP-OCRv6 medium goes from 0.451 CER on clean subtitles to 1.272 CER with a watermark nearby. GeekLink's numbers move almost identically, 0.460 to 1.302. That consistency across five very differently-built engines is the actual headline finding here: raw text recognition quality barely matters once a second text region is competing for the same space in the frame. The bottleneck is detection and selection, not recognition accuracy — which is exactly the step a raw OCR call skips and a real subtitle-extraction tool has to add back in.
Which PP-OCRv6 Size Should You Actually Use?
PaddleOCR's newest release, PP-OCRv6, ships in three sizes — tiny (1.5M parameters), small (7.7M), and medium (34.5M) — and defaults to medium if you don't specify.
PP-OCRv6 medium and small are statistically tied on accuracy (0.597 vs. 0.599 CER — a 0.4% relative difference, well within noise) for 3.7x the latency (1,399ms vs. 381ms). Medium doesn't buy anything meaningful here. Tiny is dramatically faster (169ms) but loses heavily on Japanese specifically — 1.461 CER vs. small's 0.841 — which suggests the tradeoff in PP-OCRv6's "one model for 50 languages" design lands hardest on CJK scripts at the smallest parameter count. If you're picking a PP-OCRv6 tier for a real product rather than a benchmark leaderboard, small gets you the same accuracy as the default for a fraction of the latency.
Does CoreML Actually Make OCR Faster on a Mac?
The results table above is deliberately CPU-only, for a level cross-platform comparison — but it understates what you'd actually get running these tools on a Mac. GeekLink's real production path isn't ONNX Runtime CPU, it's the same model weights through CoreML, and EasyOCR's PyTorch backend has an Apple GPU path (MPS) that isn't its default either. Re-running each engine with its actual best-available backend on Apple Silicon:
| Engine | Backend | ms/image | Overall CER |
|---|---|---|---|
| GeekLink | CoreML (Apple Neural Engine / GPU) | 97 | 0.609 |
| EasyOCR | PyTorch MPS | 142 | 0.622 (unchanged) |
| Tesseract | CPU only — no GPU path exists | 106 | 0.946 |
| PaddleOCR (any tier) | CPU only — no Apple GPU backend | 169–1399 | 0.597–0.658 |
GeekLink's CoreML path is 5.8x faster than its own CPU number (564ms → 97ms) for essentially no accuracy cost (0.6098 → 0.6093 CER) — a difference well within normal floating-point noise between inference backends — and ends up the fastest engine in the entire comparison, faster even than Tesseract, which is doing far less work. EasyOCR gets a genuine 4.2x speedup from enabling MPS. PaddleOCR has no such option on macOS at all: we checked directly (paddle.device.is_compiled_with_mps doesn't exist as an attribute, no custom device types are registered), so its numbers are identical to the CPU table regardless of which tier you pick. If you're benchmarking OCR tools on Apple Silicon and only testing CPU, you could be off by 5-6x on latency without realizing it.
Do These Rankings Hold on Real Production Video?
A synthetic benchmark, however carefully built, is still synthetic. So before publishing any conclusions, we checked the same four engines against 1,494 real GeekLink user OCR-correction samples — genuine burned-in subtitles from real videos, with the human-verified correct text as ground truth. This data isn't published (see above for why we don't publish real user video), but we can share the aggregate result, because it changed our conclusion.
| Engine | Synthetic benchmark CER | Real production data CER |
|---|---|---|
| GeekLink (CoreML) | 0.609 | 0.686 |
| EasyOCR (MPS) | 0.622 | 0.693 |
| PP-OCRv6 small | 0.599 | 0.761 |
| Tesseract | 0.946 | 0.831 |
The ranking flips. On the clean synthetic benchmark, PP-OCRv6 small edges out GeekLink's bundled models. On real production video — theatrical stage lighting, stylized fan-edit fonts, streaming-platform compression artifacts — GeekLink's older, ONNX/CoreML-converted PP-OCRv5 models actually hold up better (0.686 vs. 0.761 CER), and Tesseract's relative position improves too. Our working theory is that PP-OCRv6's newer weights are tuned against benchmark-style clean text more than the kind of messy, inconsistently-lit, compressed footage that makes up real subtitle-extraction traffic — but we don't have a rigorous explanation for that gap yet, just the measurement.
One caveat worth stating plainly: the real-data sample skews toward Spanish (812 of 1,494 samples, from a single TV drama series in our correction pool) rather than the roughly even per-language split in the synthetic set, so this isn't a clean apples-to-apples replication — treat it as a directional validation, not a second controlled benchmark. That's exactly why it lives in this article and not in the benchmark repository itself: it's real-world context for interpreting the numbers, not a reproducible data point anyone else can check.
What This Benchmark Doesn't Measure
This is raw detection+recognition, not a finished product. Tools that ship an actual subtitle-selection step on top — positional filtering, frequency-based anchor detection, confidence thresholds — will score meaningfully better on the watermark subset than these raw numbers suggest, because that selection logic exists specifically to solve the problem this benchmark measures.
We also didn't test cloud-hosted vision-language models used for OCR (DeepSeek-OCR, GLM-OCR, Surya, and others). Those are a different category of tool — general multimodal models rather than dedicated OCR engines — aren't fairly comparable on latency since it's network-dependent rather than local compute time, and cost money per call. We're treating that as a natural follow-up piece rather than folding it into this one.
Several dedicated hardcoded-subtitle extraction tools aren't in this comparison either, because their local OCR engine is PaddleOCR itself — a raw-recognition test would just reproduce the PaddleOCR row above and tell us nothing new. What actually differs between tools in that category isn't recognition accuracy, it's how the subtitle region gets identified in the first place. The common approach is asking the user to manually position a crop box over the subtitle area — sometimes just once, sometimes per video — which sidesteps the watermark problem by construction rather than solving it algorithmically: if you never look outside the box you drew, the watermark never gets a chance to compete with the subtitle for OCR attention.
GeekLink instead scans the full frame and detects where the subtitle sits automatically — no manual box-drawing step per video. That's a real, useful difference for anyone processing more than a handful of clips, but it's a workflow/UX question, not a recognition-accuracy one, so it isn't something a CER table can capture. Manual region selection vs. automatic detection is a genuinely interesting comparison on its own; it's just a different one than the raw-engine numbers above.
FAQ
Does this benchmark measure real-world subtitle-extraction accuracy?
No — it measures raw OCR detection+recognition with no post-filtering, specifically to isolate how much watermark-style interference hurts before any product-level selection logic is applied. A real tool's end-to-end accuracy will generally be higher on the watermark subset than these numbers, because filtering out the watermark is exactly what that selection logic is for.
Which PP-OCRv6 size tier should I use for video subtitle OCR?
Based on this benchmark: small gets the same accuracy as the default medium (0.597 vs. 0.599 CER, within noise) for 3.7x the latency. tiny is worth considering if Japanese isn't one of your target languages — it lost significantly more accuracy there than on the other five languages tested.
Does using CoreML instead of CPU actually make OCR faster on a Mac?
Yes, substantially. In our testing, the same model weights ran 5.8x faster through CoreML (97ms/image) than through ONNX Runtime's CPU execution provider (564ms/image), with essentially no accuracy cost (0.6098 vs. 0.6093 CER) — a difference well within normal floating-point noise between backends. PaddleOCR's official package has no equivalent Apple GPU backend, so its numbers don't improve on macOS regardless of hardware.
Do I need to manually select the subtitle region for OCR to work?
Depends on the tool. Several dedicated hardcoded-subtitle extractors ask you to position a crop box over the subtitle area yourself — which does sidestep watermark interference, since the tool never looks outside the box you drew, but adds a manual step per video. GeekLink scans the full frame and detects the subtitle region automatically instead. That's a workflow difference, not something this benchmark's CER numbers capture, since they measure raw recognition on a fixed region for every engine equally.
Why does watermark-adjacent text hurt OCR accuracy so much?
Because raw OCR engines detect and recognize every text region in a frame with no concept of which one is "the subtitle" — a nearby watermark, credits line, or channel logo gets concatenated into the output alongside the real subtitle text, which tanks character-level accuracy against a single-line ground truth. In our testing this held true for every engine, not just weaker ones — PP-OCRv6, the newest and most accurate model tested, lost accuracy at the same rate as the others.
Do synthetic OCR benchmarks predict real-world performance accurately?
Not perfectly, based on what we found. We validated this benchmark's ranking against 1,494 real production OCR-correction samples, and the ranking changed — an engine that led on clean synthetic data (PP-OCRv6 small) fell behind an older model (GeekLink's bundled PP-OCRv5) once tested against messier real-world video with inconsistent lighting and compression artifacts. Treat synthetic benchmark rankings as directional, and validate against real data before making a production decision if you can.
Is GeekLink's own OCR a custom-trained model?
Not yet. It currently runs PaddleOCR's official PP-OCRv5 weights (PP-OCRv4 for Japanese) converted to ONNX for local inference speed, not a model GeekLink trained itself. We're collecting real correction data toward eventually fine-tuning our own detection and recognition models.
Disclosure: GeekLink is our own product. The public benchmark dataset, evaluation script, and all prediction files are open source (MIT code / CC0 + CC BY-SA data) at github.com/GeekLinkDev/geeklink-ocr-benchmark. The real-production-data validation numbers in this article are aggregate statistics only — the underlying user video and images are never published.