Models and benchmarks
ChassidusHub reads scans and hears recordings with models of its own. Here is what each one does, which one is in use, and how it did on a test it never learned from. The models are ChassidusHub’s own and are not released; only their descriptions and scores are published.
Whatever a model writes stays labelled on the site as machine output until a person checks it, however good its score.
The Likkutei Sichos reader (OCR)
Reads each line of a scanned Likkutei Sichos page into text. Letters and Words: how many were read right; Lines perfect: whole lines read with no mistake at all; Body and Notes: the letter score in the main text and in the footnotes; Hard: mix-ups of the look-alikes ב/כ, ת/ח, ז/ן/ו.
- chassidushub-kraken-ls-v1Reads Likkutei Sichos from the scans; reading vols 15-25 nowin use
- chassidushub-kraken-v1Reads the maftechos (indexes); not scored herein use
Headline
4,500 lines of Likkutei Sichos vol 39 that no model learned from. The answer key is the typed text, corrected against the scan, with 35 line-edge dashes and ". . ." added that the typed text lacked and the scan has.
| Model | Letters | Words | Lines perfect |
|---|---|---|---|
| PP-OCRv6, as it came where it started | 96.45% | 86.39% | about 44% |
| An earlier candidate (the best before it) superseded | 99.82% | 99.41% | 96.1% |
| chassidushub-kraken-ls-v1 in use | 99.91% | 99.68% | 97.7% |
| + the word check after reading in use | 99.88%* | 99.60%* | 97.3%* |
| A later candidate (its best) not used | 99.87% | 99.51% | 97.0% |
* Scored on a stricter answer key (without the 35 added marks), so it is not lower than the line above: on the same key the word check only fixes lines. It fixed 15 of the 16 lines with a hard-letter mistake and broke none.
Every candidate on the 4,500 lines
4,500 lines of Likkutei Sichos vol 39 that no model learned from. The answer key is the typed text, corrected against the scan, with 35 line-edge dashes and ". . ." added that the typed text lacked and the scan has.
| Model or candidate | Letters | Words | Lines perfect | Body | Notes | Mistakes | Hard | Dash dropped / added | Dots dropped / added |
|---|---|---|---|---|---|---|---|---|---|
| recover start | 99.65 | 98.38 | 88.9 | 99.70 | 99.33 | 699 | 74 | 15 / 10 | 15 / 2 |
| 00-0.9985 | 99.81 | 99.41 | 96.2 | 99.82 | 99.79 | 372 | 24 | 57 / 1 | 43 / 0 |
| 00-0.9986 | 99.83 | 99.37 | 96.0 | 99.85 | 99.73 | 338 | 19 | 41 / 6 | 26 / 1 |
| 00-0.9984 | 99.83 | 99.36 | 95.9 | 99.84 | 99.74 | 346 | 14 | 33 / 1 | 34 / 2 |
| 00-0.9986-v1 | 99.80 | 99.30 | 95.6 | 99.82 | 99.70 | 399 | 18 | 57 / 0 | 47 / 0 |
| 01-0.9986 | 99.82 | 99.36 | 95.8 | 99.85 | 99.65 | 366 | 20 | 55 / 1 | 27 / 2 |
| 01-0.9986-v1 | 99.83 | 99.40 | 96.0 | 99.85 | 99.68 | 347 | 17 | 49 / 0 | 33 / 0 |
| 01-0.9987 | 99.82 | 99.38 | 96.0 | 99.84 | 99.69 | 361 | 17 | 49 / 0 | 41 / 0 |
| 01-0.9987-v1 superseded | 99.82 | 99.41 | 96.1 | 99.84 | 99.69 | 357 | 18 | 51 / 0 | 42 / 0 |
| chassidushub-kraken-ls-v1 in use | 99.91 | 99.68 | 97.7 | 99.93 | 99.81 | 179 | 18 | 4 / 0 | 6 / 1 |
Without the 35 added marks it still wins: 99.87% letters and 97.0% lines perfect, against 99.84% and 96.3%. It halved the mistakes (357 to 179), almost all of them marks: dropped dashes 51 to 4, dropped dots 42 to 6. The hard letters did not move (18 to 18).
The later candidates
The same 4,500 lines, with a key of the typed text plus 24 line-edge marks; chassidushub-kraken-ls-v1 read again the same way, to compare.
| Model or candidate | Letters | Words | Lines perfect | Body | Notes | Mistakes | Hard | Dash dropped / added | Dots dropped / added |
|---|---|---|---|---|---|---|---|---|---|
| chassidushub-kraken-ls-v1 in use | 99.90 | 99.63 | 97.5 | 99.91 | 99.80 | 206 | 20 | 4 / 1 | 0 / 10 |
| r3-00-0.9982-2347 | 99.83 | 99.38 | 95.8 | 99.84 | 99.74 | 348 | 59 | 38 / 0 | 10 / 3 |
| r3-00-0.9983-0003 | 99.83 | 99.35 | 95.8 | 99.85 | 99.74 | 337 | 41 | 37 / 0 | 6 / 5 |
| r3-00-0.9984-0019 not used | 99.87 | 99.51 | 97.0 | 99.88 | 99.81 | 258 | 22 | 21 / 0 | 10 / 2 |
| r3-00-0.9986-0035 | 99.83 | 99.43 | 96.5 | 99.83 | 99.83 | 338 | 18 | 39 / 1 | 11 / 1 |
| r3-01-0.9984-0050 | 99.83 | 99.47 | 96.5 | 99.84 | 99.79 | 339 | 21 | 51 / 0 | 11 / 1 |
| r3-01-0.9984-v1-0105 | 99.84 | 99.48 | 96.6 | 99.84 | 99.82 | 323 | 21 | 47 / 0 | 11 / 1 |
| r3-01-0.9984-v2-0137 | 99.84 | 99.48 | 96.6 | 99.84 | 99.81 | 327 | 21 | 44 / 0 | 11 / 1 |
| r3-01-0.9985-0120 | 99.84 | 99.49 | 96.8 | 99.85 | 99.81 | 318 | 23 | 42 / 0 | 11 / 2 |
| r3-final-ppm_best | 99.83 | 99.43 | 96.5 | 99.83 | 99.83 | 338 | 18 | 39 / 1 | 11 / 1 |
None beats it; the gap is almost all dropped line-opening dashes.
300 checked lines of vol 39
An earlier, smaller test: 300 lines of vol 39 whose answer key was checked line by line against the scan. The differences between candidates here are 1 to 6 mistakes in 300 lines, within noise, which is why the 4,500 lines decide.
| Model or candidate | Letters | Words | Lines perfect | Body | Notes | Mistakes |
|---|---|---|---|---|---|---|
| PP-OCRv6, as it came where it started | 96.48% | 86.07% | 44.3% | 96.41% | 96.95% | 459 |
| s2-00-0.9953 | 99.82% | 99.24% | 94.7% | 99.8% | 99.94% | 24 |
| s2-00-0.9982 | 99.9% | 99.64% | 97.7% | 99.89% | 100.0% | 13 |
| s2-00-0.9981 | 99.92% | 99.64% | 97.7% | 99.93% | 99.88% | 10 |
| s2-abort | 99.92% | 99.64% | 97.3% | 99.93% | 99.82% | 11 |
| s2-00-0.9986 | 99.95% | 99.68% | 98.0% | 99.94% | 100.0% | 7 |
| s2-00-0.9982-v1 | 99.95% | 99.72% | 98.0% | 99.94% | 100.0% | 7 |
| s2-00-0.9984 | 99.89% | 99.52% | 96.7% | 99.89% | 99.82% | 15 |
| s2-00-0.9980 | 99.9% | 99.56% | 97.0% | 99.92% | 99.77% | 13 |
| s2-01-0.9982 | 99.92% | 99.6% | 97.0% | 99.95% | 99.77% | 10 |
| s2-01-0.9979 | 99.92% | 99.52% | 97.3% | 99.92% | 99.88% | 11 |
| s2-00-0.9985 | 99.92% | 99.68% | 98.0% | 99.91% | 100.0% | 10 |
The reader's doubts on vols 15-17
The first volumes it read (182 sichos, 725k words), mostly Yiddish: 157 sichos against 25 Hebrew. There is no typed text of them to compare with, so these are the reader’s own doubts, an estimate, not a measured score.
| Text | Words | Confidence below 0.9 | Below 0.6 |
|---|---|---|---|
| Hebrew sichos, body | 43,178 | 1.24% | 0.33% |
| Yiddish sichos, body | 402,534 | 2.81% | 0.64% |
| Hebrew sichos, notes | 7,548 | 2.04% | 0.46% |
| Yiddish sichos, notes (mostly Hebrew) | 189,563 | 1.85% | 0.49% |
Weak spots
- Yiddish: on Yiddish body text the reader is about twice as unsure as on Hebrew. That is the biggest gap.
- Look-alike letters: ב/כ, ת/ח, ז/ן (about 18 in the 4,500 lines; the word check fixes most).
- פ and ט read as ס in the small note type (פירוש as סירוש, הבעש"ט as הבעש"ס).
- The point under a Yiddish alef is dropped now and then.
- Bare stars "*" are not read by the reader; they are taken from the ink.
Next: a round for Yiddish, scored again the same way to see the gap close.
The Miram (bold) detector
In the Rebbe's sichos, Miram type marks the stressed words, and ChassidusHub shows them as bold. The detector says, word by word, which words on a page are in Miram.
- chassidushub-facenet-v2Finds which words on a page are in Miramin use
- chassidushub-facenet-v1Finds which words on a page are in Miramsuperseded
Five pages checked by eye
Five regular sicha pages (38:40, 34:42, 31:93, 36:50, 33:70), 1,832 words, 82 of them in Miram, each word checked by eye.
| Model | Words agreeing with the eye |
|---|---|
| Thickness rules, no model superseded | partial words; misses whole Miram notes |
| chassidushub-facenet-v1 superseded | 1,832 / 1,832 |
| chassidushub-facenet-v2 in use | 1,832 / 1,832 |
The first review round (2026-10-02) found 12 Miram misses, all on pages built before chassidushub-facenet-v2 ran. Five pages is a small test; every page checked in the review tool adds to it.
Weak spots
- The test is small: five pages.
The Rebbe's voice to text (Whisper)
Hears a recording of the Rebbe's Yiddish and writes it down, each word timed. Scores are words wrong / letters wrong (WER / CER); lower is better.
- chassidushub-whisper-v3Hears the Rebbe's Yiddish and writes it downin use
- chassidushub-whisper-v2Hears the Rebbe's Yiddish and writes it downsuperseded
- chassidushub-whisper-v1Hears the Rebbe's Yiddish and writes it downsuperseded
Held-out farbrengens
Farbrengens no version learned from, in the booklets' spelling.
| Held out | ivrit.ai Yiddish (where it started) | v1 | v2 | v3 (in use) |
|---|---|---|---|---|
| Three farbrengens of 5742, 395 clips | 11.8% / 5.7% | 10.1% / 5.2% | ||
| 17 Tammuz 5742, 109 clips | 10.8% / 4.6% | 8.2% / 3.8% | ||
| 11 Nissan 5733 (an older era), 344 clips | 13.7% / 5.2% | 13.7% / 5.1% | ||
| 17 Tammuz 5742, the whole 29 minutes | 58% / 28% | 13.0% / 5.9% | 11.9% / 5.3% | 11.5% / 5.5% |
As each version was scored when it was new
v3's 5742 score is in the booklets' spelling and the earlier ones were not, so the table above is the one to compare by.
| Model | 5742 farbrengens | 11 Nissan 5733 | Whole 17 Tammuz 5742 (29 min) |
|---|---|---|---|
| ivrit.ai Yiddish where it started | about 60% | ||
| chassidushub-whisper-v1 superseded | 12% / 6% | 22.5% / 9.0% | 13.0% / 5.9% |
| chassidushub-whisper-v2 superseded | 9.5% / 4.8% | 13.8% / 5.2% | 11.9% / 5.3% |
| chassidushub-whisper-v3 in use | 10.1% / 5.2% | 13.7% / - | 11.5% / 5.5% |
Weak spots
- Rarer words (תנות for תענית).
- The older years (11 Nissan 5733 above).
Next: a v4, scored on the same held-out farbrengens and used only if it scores better.
Model descriptions and benchmarks: CC BY-NC-ND 4.0. The models are not released; all rights reserved.