Measuring the platonic representation
There's an idea that as models get better they all end up learning the same picture of the world, whatever they were trained on. You can test it: describe one thing as text, an image, a sound and a video, then see how close those four land.
The platonic representation hypothesis says that models capable enough, trained on different data and in different modalities, end up converging on the same internal picture of reality. If it holds, then the structure a network learns says more about the world the data came from than about the training set itself. It’s a big claim, and most of the evidence for it is correlational: line up two models, measure how similar their representations are, and note that the number keeps climbing as models get more capable.
A language-anchored multimodal encoder lets you test a sharper, more local version of the same idea. Take one concept, say thunder, and encode it four ways: the word, an image of a storm, an audio clip, a short video. Each is a different sense arriving in the same shared space. The hypothesis says they should land in roughly the same place. The interesting questions are what “roughly” means and where the disagreement sits.
Decompose, then compare
A dense cosine similarity between the four embeddings gives you one blurry number. Factor each embedding into a sparse decomposition first, so a handful of named features from a sparse autoencoder, and the comparison becomes legible. Now you can ask which features the four senses share and which each one carries alone, rather than just how much they agree overall.
The intersection is the part of the concept that doesn’t depend on the modality: the features that fire whether thunder arrives as a word, a picture or a sound. That shared set is the closest thing the system has to a platonic core, the part of thunder that survives a change of sense. What’s left over per modality is what each sense adds on top. The audio decomposition carries low-frequency and temporal features the word never touches. The image carries visual-storm features the sound can’t express.
The leftovers are the measurement
That remainder is the measurement, rather than noise to average away. The features audio adds and text lacks tell you directly what hearing thunder encodes that naming it doesn’t. Measuring cross-modal agreement on the feature sets, concept by concept, with something like a Jaccard overlap, turns the hypothesis from a vibe into a number you can compute and compare.
It cuts both ways, which is what makes it worth doing. High overlap across modalities is evidence for a convergent core, meaning the senses really are arriving at one representation. Low overlap is evidence against it for that concept, suggesting the shared space is held together by training rather than unified by anything real, with each modality keeping its own private structure under a thin coat of alignment. Either result tells you something, and the experiment earns its keep precisely because it can come back negative.
The code is on GitHub. The anchor-compare command encodes one concept through each modality and reports the agreement, so you can read the result off a single run.