How OCR actually works (and why your scan looks fine but converts badly)

A short history

The first patent for a reading machine for text belongs to the Austrian engineer Gustav Tauschek, filed in Germany in 1929. His device was a mechanical-optical contraption that compared characters against a rotating drum of stencils using photoelectric cells. It worked, sort of, on a narrow set of typefaces.

The modern lineage starts with Ray Kurzweil's reading machine for the blind, which Kurzweil Computer Products launched in 1976. It combined a flatbed scanner, omni-font character recognition software, and a speech synthesizer. The first customer was Stevie Wonder. The software ran on a DEC PDP-11 and cost $50,000.

For most of the next two decades OCR was the domain of commercial products — OmniPage, Readiris, ABBYY FineReader, Adobe Acrobat's built-in engine — with quality slowly creeping upward and prices slowly creeping down. The open-source side was anemic until HP donated its in-house engine to Google in 2005, who released it as Tesseract. Tesseract 3 was competent but clearly behind the commercial state of the art. Tesseract 4, released in October 2018, replaced the character-classifier core with an LSTM neural network, and the accuracy gap closed dramatically.

The pipeline, stage by stage

Modern OCR is not one algorithm but a sequence of them. Each stage hands its output to the next, and an error early in the pipeline propagates — you can't recover a character from a region the line detector failed to find.

1. Binarization

The first thing the engine does is reduce the input image to two colors: black where it thinks the ink is, white where it thinks the paper is. A naive approach picks a single global threshold and applies it to every pixel. This breaks immediately on real-world scans, where lighting is uneven and the paper darkens at the spine of a book. Modern engines use adaptive thresholding, often Sauvola's method (1997) or Otsu's method (1979) computed in tiles across the page, so the threshold varies based on local brightness. A bad binarization step turns "rn" into "m" and "cl" into "d" before any recognition has happened.

2. Deskew and despeckle

Pages put on a flatbed at a slight angle have to be rotated back to level. Engines estimate skew by running Hough transforms on the binarized page or by analyzing horizontal projection profiles — the row of pixels that contains the most text density is a baseline, and the angle of that line is the skew. Once the page is upright, small specks of noise (dust on the scanner glass, sensor noise from a phone camera) get filtered out by morphological opening.

3. Layout analysis

Now the engine has to figure out what's a column, what's a heading, what's a caption, what's a figure. This is harder than it sounds. A two-column journal article needs to be read down column one and then down column two, not left-to-right across both. Tables need to be recognized as tables rather than misread as paragraphs. Page numbers and headers need to be separated from body text. Tesseract uses a page-segmentation mode that you select up front (single column, sparse text, single block, table, etc.); newer engines try to detect layout automatically.

4. Line and word detection

Within each text block, the engine finds individual lines by looking at horizontal projections of black pixel counts. Within each line, it segments words by detecting gaps wider than the typical inter-character spacing. This is where italic and condensed typefaces tend to confuse classical engines — "in" and "m" look very similar when the letters touch.

5. Character recognition

The character classifier is the headline algorithm. In Tesseract 3 and earlier engines, this stage extracted hand-crafted features from each glyph (stroke geometry, topological holes, aspect ratios) and matched against a feature database. In Tesseract 4 and later, the line of pixels is fed directly to an LSTM — a recurrent neural network that reads the line left to right and emits character probabilities at each position. The LSTM doesn't recognize one character at a time; it sees a sliding window of pixels and predicts the most likely character sequence. This is why it handles connected scripts and ligatures better than the older approach.

6. Language modeling

The classifier's raw output is a set of probabilities. "cl" and "d" look almost identical at low resolution; the engine decides which by checking which produces a more plausible word in the target language. Tesseract ships per-language data files (eng.traineddata, fra.traineddata, and so on) containing dictionaries and character-bigram statistics. Pick the wrong language for the page and you'll see the classifier insisting that English text contains "the" but mangling everything around it. Pick the right language and accuracy rises measurably.

Why 300 DPI matters

OCR engines are tuned for the resolution at which classical office scans were produced: 300 dots per inch. At 300 DPI, a 10-point character is about 42 pixels tall — enough that the classifier can see the difference between the bowl of a "b" and the bowl of a "d," the foot of an "f" and the lack of one in "t."

Below about 200 DPI, character details collapse. The cross-stroke of "e" gets indistinguishable from the body of "o." Serif details disappear. The engine starts substituting visually similar letters at random.

Above 600 DPI you don't get more accuracy — you just get larger files and slower processing. A few engines actually do worse at very high DPI because their internal scaling assumes mid-range input. If you have control over the scan, 300 DPI grayscale is the sweet spot. Photographing a page with a phone camera, you want the text to be at least 25-30 pixels tall — angle the camera carefully and crop tight.

Why your scan looks fine but converts badly

A page that looks perfectly readable on screen can still produce terrible OCR. The common causes:

• Low effective resolution. A 1080p photo of a full page is about 130 DPI. Looks great, OCRs badly.
• JPEG compression artifacts around text. Strong JPEG compression introduces ringing artifacts around high-contrast edges. The engine sees those ringing pixels as part of the character and misclassifies.
• Uneven lighting. Phone photos of a textbook held open often have a bright center and dark margins. Adaptive binarization helps but doesn't fix everything.
• Curved pages. A book spine warps lines. Some engines (ABBYY's commercial product, modern Google Cloud OCR) do dewarping; Tesseract doesn't.
• Background patterns. A receipt printed on patterned paper, a form printed on light-blue safety paper — the binarization step has nothing clean to lock onto.
• Mixed fonts and sizes. A page that combines a serif body with a sans-serif caption with monospaced code samples confuses classifiers tuned for any one style.

Handwriting is a different beast

OCR engines tuned for printed text fail almost completely on handwriting. The character shapes are too variable, the spacing is unreliable, and the language-modeling assumptions don't carry over. Handwriting recognition (HTR) is its own field, and the modern approach uses CNN+LSTM or transformer architectures trained on specific datasets — IAM for English block handwriting, Bentham for historical English script, and various script-specific corpora.

Google's online Handwriting API and Apple's PencilKit live-handwriting recognition use approaches very different from page-based OCR. For most converter tools, including this one, handwriting is unsupported — if you feed in a handwritten page, you'll get a handful of recognized words and a lot of garbage.

Output as text vs as a PDF text layer

OCR engines can emit results in two practically different forms. The first is plain text or formatted text — you get a .txt or .docx or Markdown file containing the recognized words. The original image is gone; what's left is the engine's best guess at the words.

The second form is a searchable PDF. Here the original page image is preserved, and the OCR engine adds an invisible text layer positioned exactly over the visible glyphs. To the eye, the PDF looks identical to the scan. To Cmd-F (or Ctrl-F), the text is suddenly there. Copy a sentence out and you'll paste the OCR engine's interpretation, not the image. This is the right output when you want to keep the original document but be able to search it — archived books, legal exhibits, anything where the visual artifact matters.

If you want just the text, the PDF OCR and Scanned PDF to Text converters here output plain text. For the searchable-PDF flavor, Make PDF Searchable preserves the image and adds the invisible layer. For OCRing a single image (a photo of a sign, a screenshot of a recipe), Image to Text is the entry point. All of them run Tesseract in WebAssembly inside the browser, so the page stays on your machine.

Languages and scripts

Tesseract ships trained data for about 100 languages, with quality varying widely. Latin-script European languages are well-covered. Cyrillic is good. Arabic and Hebrew are supported but right-to-left handling adds complexity. CJK (Chinese, Japanese, Korean) requires special data files because the character set is huge — a typical Chinese OCR data file is hundreds of megabytes against the few megabytes for English — and recognition accuracy on these scripts depends heavily on font and clarity. Devanagari, Thai, and other complex scripts have their own engines and tend to be where the modern transformer-based approaches significantly outperform the LSTM.

The single most useful thing you can do to improve OCR quality is correctly identify the language up front. Multilingual pages can be processed by listing multiple languages in the engine config (Tesseract accepts eng+fra), at the cost of slower recognition and a slightly higher rate of language-confusion errors.

Where to go from here

If you're getting bad OCR, look at the input before blaming the engine. Re-scan at 300 DPI in grayscale. If you only have a photo, take it again with the page flat, even lighting, and the camera held parallel. Crop tightly. Save as PNG or TIFF rather than JPEG. Run a quick contrast adjustment if the page is faded. Then re-OCR.

If the input is the best you can get and the output is still rough, accept that OCR is statistical — even excellent engines hit 99% accuracy on clean printed text, which means one wrong character per hundred. For legal or archival use, a human pass over the OCR output is non-negotiable. For "I just want to grep this thing," 99% is often plenty.