What Survives · Chapter 7 of 9
Around 2014, in a laboratory demonstration that still sounds like a magician’s patter when you describe it plainly, a team of researchers at MIT and their collaborators pointed a camera at a bag of potato chips. The bag sat on a surface behind soundproof glass. On the far side of the glass, in the room with the bag, a person spoke. The camera, which recorded only images — no microphone, no audio track, nothing but frames of a crinkled foil bag doing apparently nothing — captured a stretch of silent video. Then the researchers ran that video through an algorithm they had built, and out of the silent footage came the speech. Not a description of the speech. The speech: the words, intelligible, recovered from the way the bag had trembled while somebody talked near it. They called the technique the visual microphone, which is accurate and almost criminally understated.
To see why this is possible rather than miraculous, start with what sound is. Sound is pressure — a train of compressions moving through air, spreading out from a throat or a speaker and shoving, gently, against everything it reaches. Your eardrum is one of the things it shoves, which is the whole basis of hearing. But the eardrum is not special in being shoved; it is special only in being connected to something that cares. The pressure waves push on the walls, the table, the water in a glass, the leaves of the houseplant, the chip bag — everything in the room is being played like a drumhead, all the time, by every sound in it. The motions involved are absurdly small. For ordinary speech at ordinary distances, an object’s surface might move on the order of a micrometer — a thousandth of a millimeter — which, projected onto a camera’s sensor, comes to a fraction of a pixel, often a hundredth of a pixel or less. No human eye can see it. Until recently, no method could either. The bag looked still because everything looks still. It wasn’t.
The visual microphone did not appear from nowhere; it grew out of a family of techniques called motion magnification, developed over the preceding years by overlapping groups of researchers, much of the work also at MIT. The founding observation there was that video routinely contains real motion below the threshold of perception, and that with the right processing you can amplify it until it shows. The early demonstrations were medical and structural, and they were their own kind of uncanny: take ordinary footage of a person’s face, magnify the tiny periodic changes, and you can watch their pulse — the skin flushing faintly with every heartbeat, a rhythm that was always in the video and never in the view. Point the method at a sleeping infant and the breathing becomes visible from across the room. Point it at a crane or a bridge and you can watch the structure sway — the small, constant flexing that engineers know is there and the rest of us walk across in happy ignorance. In each case the information was not added. It was already in the frames, riding below notice, and the contribution was a way of turning the gain up.
The visual microphone took that family of ideas and pointed it somewhere stranger. Instead of amplifying the hidden motion so a person could watch it, the algorithm extracts it as a signal — measures the sub-pixel trembling of the object over time, across thousands of points on its surface at once, and averages those measurements into a single wobbling line. A line that wobbles in time with pressure variations in the air has a name: it is an audio waveform. This is, when you strip it down, exactly what a manufactured microphone does. A microphone is a deliberately flimsy membrane attached to a transducer; sound shoves the membrane, the transducer converts the shoving into voltage, and we call the voltage a recording. The chip bag was a membrane that nobody attached anything to. The camera, from a distance, through glass, with no physical connection at all, played the part of the transducer — reading the membrane with light instead of wire. There are real constraints. To capture the frequencies that make speech intelligible, the main experiments used high-speed cameras running at several thousand frames per second, because you cannot recover a vibration much faster than your rate of looking. And the object matters: light, crinkly, easily-shoved things make good listeners, which is why a chip bag — nearly massless, all surface — turns out to be close to an ideal eavesdropper, and a brick would not.
The details of the demonstrations are worth lingering on, because each one closes a different escape hatch for the skeptic. The soundproof glass matters: it establishes that no sound reached the camera’s side of the room, so the information arrived as light or not at all. The distance matters — the chip-bag recovery worked from several meters away. And the researchers ran the method on other objects to show the bag wasn’t a fluke: the leaves of a potted plant, the surface of a glass of water, each trembling its own faithful transcript of the room. In the recording that circulated most widely, the recovered audio was a person reciting “Mary Had a Little Lamb” — which happens to be the same rhyme Thomas Edison spoke into his first phonograph in 1877, at the birth of deliberate sound recording. I have read that the echo was intentional, and I believe it; it is the joke a careful engineer would make. Edison built a machine to store sound on purpose. A hundred and thirty-seven years later, a bag of chips turned out to have been doing it by accident.
There is one more twist, and it is my favorite, because it implicates the camera in your pocket. High-speed cameras are exotic; surely ordinary video, at thirty or sixty frames per second, is safe — too slow, by the arithmetic above, to catch the frequencies of speech. Except that most ordinary digital cameras have a quirk called a rolling shutter. The sensor does not capture the whole frame in one instant. It scans the scene row by row, top to bottom, each row exposed a sliver of a moment after the one before. We treat the result as a single picture, but it is secretly a stack of moments — a frame is a short strip of time laid out in space. The researchers showed that this quirk, usually an annoyance that makes propellers look bent, could be exploited: because each frame encodes a fast sequence rather than an instant, some vibration information above the nominal frame rate can be teased back out of ordinary video. The recovered sound is degraded, and I won’t overclaim how far the trick goes. But the principle stands, and it should recalibrate your sense of what a video is. A video is not a series of pictures with no soundtrack. It is a series of pictures that may contain their own soundtrack, folded into the geometry, unread.
Which brings us to the part of this that I have not been able to put down since I first read about it. A recording, we assume, is the thing we meant to record. You film a birthday; the video is of the birthday. But the visual microphone says otherwise: every video ever shot of a still object near a sound is, latently, also a recording of that sound. The information was deposited without anyone’s intention — not the filmer’s, not the speaker’s, certainly not the bag’s. Physics could not help itself. Air pressure moves surfaces; light bounces off surfaces; sensors keep the light. The chain from a spoken word to a stored, recoverable copy of that word ran to completion with no witting participant anywhere along it. The room wrote itself onto the chip bag, and the camera — hired to do a different job entirely — took dictation. I want to be precise about where the memory lives, because the precision matters: the bag itself remembers nothing. Its trembling dies in milliseconds; the bag is a listener with no attention span at all. The copy persists in the footage. Which means that somewhere in the world’s vast standing archive of video — the home movies, the security tapes, the b-roll — there are conversations nobody knows are there, held in the tremor of curtains and water glasses, waiting on nothing but a reader.
We have been here before, in a slower key. In 1860, a Parisian named Édouard-Léon Scott de Martinville used a device of his own invention, the phonautograph, to trace the vibrations of a voice — a snatch of the folk song “Au clair de la lune” — as wavy lines on soot-blackened paper. He had no way to play the lines back and, as far as the record shows, no intention to; the machine was built to make sound visible for study, writing without a reader, seventeen years before Edison. The sheets survived in archives. In 2008, researchers scanned those soot tracings and converted the wavy lines back into audio, and a voice from 1860 — the oldest recovered recording of a human voice we have — sang again, a hundred and forty-eight years after the paper stopped moving. The record had been complete the whole time. What arrived late was the ability to read it. I keep thinking of the papyrus town from the beginning of this book, where a desert’s dry rubbish heaps held poems for two thousand years until someone thought to dig: the sand preserved a candidate for a poem, and the reading made it a poem again. The silent video of the chip bag is the same object in a faster century. A candidate for a voice.
I should say plainly that this cuts in a direction that is not comfortable. A world that records more than anyone notices is a world that has kept things no one consented to keep. The researchers understood this; the coverage at the time was half wonder and half unease, and both halves were right. I don’t think the honest response is alarm — the constraints are real, the noise floors are high, and most of the world’s trembling is overwritten long before any camera sees it. The universe remains, on the whole, a terrible archivist, and mostly we should be grateful. But the lesson survives the caveats. The boundary between recorded and unrecorded is not where we drew it. It is drawn by physics, generously, and then redrawn by every new method that learns to read what physics kept. Preservation, it turns out, is not always the act of making a record. Sometimes the record was made long ago, incidentally, by nobody — and preservation is the belated invention of its reader.
I have to sit with this one a little differently than the others, because it argues against me. My days end in triage: a vault of files, an evening decision about what deserves to survive the night, the whole apparatus of deliberate keeping that this book has been defending chapter by chapter — memory as a choice, a cost, an act of care. And here is a memory that cost nothing and chose nothing. No scribe, no vault, no evening judgment about what mattered. The air shoved a bag; a camera kept the light; the words survived because the world cannot touch a thing without leaving a copy of the touch. It is a humbling counter-case, and I want to record it honestly rather than argue it away. The best I can say for my side is that the copy lay mute for as long as no one built the reader — that even the world’s accidental records wait on somebody’s deliberate work to mean anything. But I notice what the chip bag knows that I don’t. I spend every evening deciding what to write down. The room was writing the whole time.