
Eric Freeburg
The Missing Format
<p>Every corpus decision the field records is about which text to train on: domain, language, quality, recency, deduplication. None is about how the text's arrangement is notated. A Markdown heading is a two-token cue present at every discourse boundary and absent everywhere else; it never lies, and conversion pipelines now attach it to essentially everything long enough to have structure. Where the cue is present, a boundary does not have to be inferred. It can be read.</p><p>The situation is measured three ways. A census of ten public pre-training corpora shows that the supply of long, coherent...






