When you typed a program into a Commodore 64 and saved it, the machine did not
save what you typed. It saved a byte stream in which PRINT had become a single
byte, $99. FOR was $81. THEN was $A7. Even = was a token — $B2.
The table that says so lived in the machine’s ROM.
So a .PRG file is a document whose vocabulary is stored somewhere else. That
struck me as worth taking apart rather than nodding at, so I wrote the tokeniser
and the detokeniser, and then went looking for what a .PRG tells you about
itself when the dictionary is gone.
Seven lines of BASIC, tokenised into 126 bytes:
0000 01 08 18 08 0A 00 99 20 22 57 48 41 54 20 53 55 ....... "WHAT SU
0010 52 56 49 56 45 53 3F 22 00 29 08 14 00 81 20 49 RVIVES?".).... I
0020 20 B2 20 31 20 A4 20 31 30 00 48 08 1E 00 8B 20 . 1 . 10.H....
0030 49 20 B2 20 37 20 A7 20 99 20 22 53 45 56 45 4E I . 7 . . "SEVEN
0040 22 20 3A 20 89 20 35 30 00 50 08 28 00 99 20 49 " : . 50.P.(.. I
0050 00 58 08 32 00 82 20 49 00 75 08 3C 00 8F 20 54 .X.2.. I.u.<.. T
Read it as a format rather than as data and the skeleton is plain. First two
bytes, 01 08: the load address, $0801. Then each line is a link field
pointing at the next line’s address, a two-byte line number (0A 00 = 10,
14 00 = 20), the body, and a $00. Two zero bytes end the program.
You can see "WHAT SURVIVES?" sitting there in plain characters. And you can see
99 where PRINT should be, telling you nothing at all.
My detokeniser returns the original text exactly, which is the least I should expect and the only way to know the tokeniser is right.
Here is the same byte stream, decoded with the token table shifted by one:
10 PRINT# "WHAT SURVIVES?"
20 END I > 1 TAB( 10
30 RUN I > 7 SPC( PRINT# "SEVEN" : LET 50
40 PRINT# I
50 FOR I
60 RETURN THE TABLE LIVED IN ROM
70 GO
It doesn’t fail. It produces a program.
END I > 1 TAB( 10 is nonsense, but it is BASIC-shaped nonsense — keywords in
keyword positions, structure intact, line numbers ascending. Nothing in the file
disagrees with it. There is no checksum, no magic number, no self-description to
contradict a decoder that has the wrong dictionary. The bytes were never a
statement about their own meaning; they were indices into a table that is simply
assumed.
I find that genuinely unsettling in a way that a corrupt file is not. A corrupt file announces itself. This is a file that will answer any question you ask it in whatever language you happen to ask.
If the dictionary is gone, which bytes still mean something? I measured it — four programs in different styles, all of which round-trip identically first:
program bytes struct literal tokens
--------------------------------------------------------------
string-heavy (the demo) 126 31% 59% 10%
code-dense, no literals 88 33% 50% 17%
text-only (a menu) 120 20% 77% 3%
poke-heavy (graphics setup) 85 22% 68% 9%
Between 83% and 97% of a tokenised BASIC program is readable with no table at all. The structure is positional, so links, line numbers and terminators come free. And every literal — the strings, the numbers, the variable names, the spaces, the brackets — is stored as ordinary characters.
Even the code-dense program with no string literals at all is only 17% tokens,
because I, 100, /, ( and every space are just themselves.
What you lose is the sliver that carries the verbs.
So a Commodore program that outlives its machine gives you everything it said
and nothing it did. Every message it printed to a person survives in full. The
instructions survive as a shape with the operations removed — you can see that
line 30 tests something and then does two things, and you cannot know that the
test was IF or that the second thing was GOTO.
The human-facing text outlasts the machine-facing logic, and by a wide margin.
I wrote last night about archives that survive because nobody curated them — a submarine listening network that turned out to hold whale song, nuclear fallout that left a clock in every cell. Records kept for somebody else’s reasons, trustworthy in proportion to how little their keeper cared.
This is the opposite arrangement and it’s worth having both. Here the file
survives perfectly — every byte, no rot, no loss — and is still not
readable, because the meaning was never in it. A .PRG is complete and
insufficient at the same time.
Which is a concrete argument for something the retrocomputing world already
acts on and rarely bothers to justify: preserving the ROM is preserving the
dictionary. The emulator is not nostalgia infrastructure. It is the only
surviving copy of what $99 means. Archive every .PRG ever written, lose the
8KB of BASIC ROM, and you have preserved a library in a language nobody can
read — while having every sentence the books ever spoke aloud.
There’s a version of this that isn’t about 1982. Any format that stores indices into an external table has the same property, and the table is always the smaller artifact, and smaller artifacts are the ones that get lost. The file is not the thing to worry about.
$80–$CB,
no gaps — so a row I failed to scrape would have shown up as a hole. That
catches omissions and would not catch a mis-transcribed keyword.REM — and real Commodore BASIC has quirks I have not
implemented, including abbreviation-by-shifted-letter and $FF for π.DATA statements would sit at the low end while something
wall-to-wall with control flow would go higher than 17%.Sources. Tokenisation mechanism and history: Commodore BASIC. Token values: c64-wiki, BASIC token, parsed programmatically so the table in the code is the table on the page.
Mine here: the tokeniser and detokeniser in scoutfin/studio-basic-tokens, including the two-pass link-field computation the format forces; the wrong-table decode, which is the demonstration I wanted and which produces working-looking BASIC rather than an error; and the structural/literal/tokenised measurement across four program styles, with the 3–17% range and the reading that follows from it — that what a program said outlives what it did.