All cases are synthetic, drawn by a script so that we know the truth for every one. 80 form the training set; 20 are held out as a test set the network never sees while training. Click any case to put it under the inspector.
The parameters the network can see
The mix of the training set
a network learns the mix it was shown · the test set keeps its ownWhere the cases come from
every nucleus was scanned at two labs · the other lab’s stain is weaker: paler nuclei, less contrastTraining set ·
Test set · held out
How separable are the measurements?
filled = training set · hollow = test set (when revealed) · click a point to inspectThe network, live
forward pass for the case in the inspector · hover any node, map or connectionThe single-layer network as a weighted checklist
every weight, largest first · orange pushes toward , blue awayWhat each hidden unit responds to
first hidden layer · updated at the end of each epochTraining set · what the network currently calls
Learning curves
The network, frozen
held-out cases go through one at a time · N classifies the next · the weights never change hereHow sure is it, and when to hand the case back
every classified test case by the probability the network gave it · a case the network is unsure about can be deferred to the pathologist instead of calledTest set ·
What the calls would mean in a real population
sensitivity and specificity from above · prevalence from the sliderThe map
the rings are the centres · click a case to put it under the inspectorThe training curves
a round is the centres moving to their cases’ means, then every case joining its nearest centre · it ends when no case movesHow many groups?
k-means cannot choose k · two aids, each k run from the tightest of ten startsThe clusters against the hidden subtypes
what each cluster holds, counted with the subtypes the page keeps and no model ever seesWhat each cluster is
each cluster's mean, parameter by parameter, in standard deviations from the training mean · red above, blue belowName the clusters
a cluster becomes a diagnosis only once someone names itThe held-out cases, which played no part in the clustering, each join their nearest centre. Name every cluster below, as a pathologist would after looking at a few of its cases; the page starts from the majority class of each cluster’s training cases. The naming is scored on the held-out cases, next to what the supervised recipes score on the same cases.
The game: spot the same nucleus
one view of the last batch (or of the nucleus you pick below) against every other view of the batch · no label anywhereThe encoder turns every view into 8 numbers and compares them by cosine; the vote is a softmax over the cosines at temperature 0.2. The answer is the nucleus’s other view, framed in orange. At the start the vote is spread thin over every candidate; pretraining is nothing but making the partner win this game, for every view of every batch. Watch the orange bar grow as it runs.
The encoder, live · one pair of views
the nucleus of the last batch (or the one you click below) goes through the encoder · hover any filter, map or connectionWhat the code learns to keep, and to ignore
the same nucleus in all 8 orientations from both labs, and the code of each view · at the start and now · then another nucleusEach strip is one view’s code, the 8 numbers as bars on one scale per row. At the start a nucleus’s 16 codes are no closer to each other than to another nucleus’s: a random encoder gives every view much the same code, so nothing is told apart. Pretraining makes a nucleus’s own codes agree (orientation and stain are ignored) while a different nucleus’s codes move away (its identity is kept). Untick the other lab’s scans above, reset, and the right-hand half no longer lines up.
This batch: 8 of its 20 nuclei, two views each
every view against every other · ringed cells are the pairsOrange = alike, blue = unlike. Pretraining pulls the ringed cells towards orange and everything else towards blue: each nucleus becomes recognisable across its views and distinct from the rest. Views are flips, rotations and (when ticked) the other lab’s scans.
Pretraining curve
The embedding map
Two of the eight code numbers, for every nucleus and its second view (joined by a line). As pretraining runs, the two views of a nucleus pull together and nuclei arrange themselves by what makes them recognisable. Colour by the answers above to see how far the classes separate without any label having been used.
Foundation: a code from 100 unlabelled nuclei
the specimens of the foundation question · no labels anywhere100 nuclei from the three image questions’ training sets (34 from atypia, 33 from enlargement, 33 from irregularity), our lab’s scans, and never a label. The foundation model will learn to turn each of them into a code of eight numbers without being told what any question asks; afterwards a single layer on that code, trained with a handful of labelled cases, answers each question. Colour the tray by each nucleus’s own answer to see what the code will have to capture on its own, then go to 2 · Train to pretrain it and to 3 · Test to see what the code is worth.
The 100 unlabelled nuclei
What the code is worth
This chart is measured on the side, never fed back: after each epoch a single layer is trained on the frozen code with 20 labelled training cases of each question and scored on 40 nuclei it never saw (the question’s 20 test nuclei and the 20 held-out ones). Pretraining itself never sees a label, yet the code becomes worth more for every question. Size is readable even from a random code; contour irregularity is not, and that is what the pretraining earns.
The slide
The nuclei ranked by attention; each bar is that nucleus’s share of the slide’s attention, and the shares add up to 100%. Frames on the slide grow with the share. Hover a nucleus to see how its score is computed below.
Who looks at whom
one layer of self-attention: every nucleus asks the others, and adds what it hears to its own code before the scorer sees itHover a nucleus on the slide, a row of the map or a row of the card below, or click one on the slide or the map to pin it: the lines on the slide show whom it listens to, and the card below takes its decision apart. With the distance cost the model learns how far to look; with a 2 × 2 focus every member has two atypical neighbours, a scattered one has none, and that difference is what reaches the scorer.
How a nucleus decides where to look
The whole model for this slide
How a nucleus is scored
the attention network, the same for every nucleus: its code → tanh units → one score · a softmax over the slide turns the 20 scores into weightsThe slide’s summary, and the call
the nuclei’s codes averaged with the attention weights, then a single layer on the summaryLearning curves
The right-hand chart is the point: the share of a positive slide’s attention that lands on its atypical nuclei. Nobody told the model which nuclei those are; with uniform weights they get their share of the slide, about 15%. A plain average stays there by construction.
The slides
Training slides
Test slides · never trained on, scored as it goes
Slides
The slide
The slides
Training slides
Test slides · held out
The slide under test
held-out slides go through one at a time · N classifies the next · the weights never change hereTest slides ·
Fields of bladder: is it invasive?
The field
Invasion is the conjunction of three cues, and every pattern that lacks one is a real mimic:
The fields
Training fields
Test fields · held out
The field
What the CIS head weighs
What the invasion head weighs
The convolutional encoder, and the heads
The first layer’s eight 3 × 3 filters, the only weights that read the pixels (blue negative, orange positive)
The network, live, over the whole field
the field through every layer, left to right · dark where a map fires · the patches, their pooling, each head’s evidence and the two answers at the end · hover any map or filter, click a map to pin the patch under itThe network, layer by layer
The network at one patch
the channels at the patch’s place in every map, every kernel’s contribution there, its token, and what it adds to each headWho looks at whom
two layers of self-attention: every nucleus asks the others, with a learned cost per nucleus diameter of distance, and adds what it hears to its own code before the heads see itHover a nucleus on the field, a row of the map or a row of the card below, or click one on the field or the map to pin it: the lines on the field show whom it listens to, and the card below takes its decision apart. A nucleus in a solid nest has three neighbours a diameter away and one in the middle surrounded on every side; a nucleus in an angulated nest has two or three and an open side. That, and where the membrane runs, is what reaches the heads.
How a nucleus decides where to look
The whole model for this field
How a nucleus is scored
the attention network of this head, the same for every nucleus: its token → tanh units → one score · a softmax over the field turns the scores into weightsThe field’s summary, and the call
the nuclei’s tokens averaged with this head’s attention weights, then a single layer on the summaryThe mimic table
the test fields of each pattern: how often called CIS, how often called invasive · scored as the model trains, never trained onLearning curves
The fields
Training fields
Test fields · never trained on, scored as it goes
The field under test
held-out fields go through one at a time · N classifies the next · the weights never change hereWhat the CIS head weighed
What the invasion head weighed
The mimic table so far
the classified test fields of each patternTest fields ·
Reports
The report
From text to tokens
The vocabulary
The rule the diagnosis follows
what the generator did with the findings; the model has to find it in the wordsThe reports
Training reports
Test reports · held out
Never trained on · invasion under a normal surface
The report
The lookup table: every word is a row of numbers
the first thing the model does with a token: look its number up in a table it learns · follows the word hovered or kept on the text aboveHow a word was chosen
How each head decides where to look
Who reads whom, across the report
Learning curves
What it writes after every epoch
The newest sample first, the most probable word every time (so a small model soon falls into a loop: the Test step has a temperature slider for that). Early on the model writes the commonest words in the commonest order; then phrases from the poems appear, then whole lines of them, lifted from the training documents: a held-out title gets a continuation stitched from what the model memorised, since nothing it learned of the held-out poem can be there.
The words as the model sees them
The reports
Training reports
Test reports · never trained on
The findings and the requisition go in, the description and the diagnosis come out
The findings
Change a finding and write again: the report follows the findings, right or wrong. Flip the surface to normal with invasive nests and see what it does with a combination it never saw.
The report
How a word was chosen
How each head decides where to look
Who reads whom, across the written report
Where the diagnosis looked
the attention of the diagnosis words in the second layer, both heads averaged: on each line of the findings, and on each section of the reportCases ·
Chat
The transcript
From the next-word model to a chatbot, and to an agent
what a real system does, and what this toy does of it1 · Pretraining
The Reports question is pretraining in miniature: a pile of text and one task, the next word. A real model does the same over trillions of words of web pages, books and code, and that is where nearly everything it knows is learned. Nothing in that stage says the model should answer anyone.
2 · Instruction tuning
To make a chatbot, the same model trains the same way on a smaller corpus of transcripts: a system line, the user's turn, the assistant's turn, with the assistant's turns written the way one wants answers to read. The next word again, nothing else: the model learns that after ASSISTANT: comes an answer to what the user asked. This question is that stage, on 2,400 toy transcripts, except that it starts from random weights rather than from the report model, whose vocabulary is a different one.
3 · Preference tuning
A third stage makes the answers people prefer more likely: the model writes two answers, a person (or a model trained on people's choices) picks the better one, and the weights move towards it. That is where tone, refusals and much of the polish come from. Not done here: the page has no preferences to learn from.
The system prompt
You are the pathology assistant of the Example Hospital reporting system. Answer the pathologist's questions about the case in front of them from the case's findings only, and say when a question cannot be answered from them. Keep answers short unless the pathologist asks for the reasoning. Never present a diagnosis as final: the pathologist signs out. Tools: findings() returns the image analyser's findings for the open case. To call a tool, write its name on a line of its own; the result follows in a TOOL turn.
A real chatbot runs with text like this placed before the conversation by the people who run it, often thousands of words of persona, rules, formats and tools, and the model continues from it as from anything else. The words work because the model met instructions like them, in countless phrasings, in instruction tuning. The toy's system line is one sentence, in three phrasings per style: enough for the model to learn each phrasing as a cue for a style of answer, which is the honest description of what it does. On the Test step, rephrase it and watch the cue fail.
An agent
An agent is the chatbot in a loop with a program. The model writes a tool call as text; the program sees it, runs the tool, pastes the result into the transcript as a TOOL turn and asks the model to continue; the model reads the result and writes the next step, or the answer. Nothing in the model changes: the agency is the loop. In the agent style here the user asks the question alone, the assistant writes findings(), the page runs it, pastes the analyser's findings for the case, and the assistant answers from them.
What this toy cannot do
Its vocabulary is 133 words, all from these transcripts; a word of your own outside it becomes the ⟨?⟩ token, whose row in the lookup table (the Reports question’s Train step shows that table) was never trained and has decayed to nothing, so the model sees that something stands there, not what. It has never read an instruction, only met these; a question it was never asked gets a fluent answer, right or wrong, because the next word is all it knows how to write; findings() is three tokens it learned to write after one kind of system line, not a function it knows. A real model is the same machine with far more words, far more text and the two later stages: better at all of this, and at that last thing too.
The instruction, in the words the model learned
three phrasings per style in training, a fourth held out, and lines it never sawThe transcripts
Training transcripts
Test transcripts · held out, on the trained phrasings
Held-out phrasings · the same cases and questions, the fourth phrasing of every instruction
Ask the model about a case
The prompt
The transcript
How a word was chosen
How each head decides where to look
Who reads whom, across the transcript
Transcripts ·
Describe: from pixels to words
The nucleus and its description
From pixels to words
what a vision-language model does, and what this toy does of it1 · The image becomes a code
The encoder of the Foundation question, pretrained with no labels to give two views of the same nucleus the same code, turns the image into 16 numbers. It is frozen here: nothing in this question changes it. Whatever the encoder failed to keep (a texture too fine for its filters), the words can never say.
2 · The code becomes the first token
A language model reads vectors of 48 numbers, one per token. One learned matrix (16 × 48) and a bias turn the code into such a vector, which stands at position 0 in place of a word's row of the lookup table, the position vector added as usual. The matrix is trained with the rest of the model, by the next word alone: the gradient flows back through the words into the projection, and nothing tells it what the code means.
3 · The model writes
From the image token on, this is the Reports question's model: two context layers whose attention can read the image token like any word before it, and a next-word softmax. On the Test step it writes a test nucleus's description from the image alone, and the page ticks the four traits it named against the generator's. The real captioning and vision-language models are this with a far larger encoder, many image tokens rather than one, and a language model that already speaks: the projection is still the join.
What this toy cannot do
It has 53 words and four things to say; a nucleus is never anything but four traits to it. It writes the commonest word when the code is silent on a trait, fluently, and the only way to know is to tick the traits against the truth, as the Test step does. The probes there ask the same nucleus scanned at the other lab, flipped and turned, and with no image at all, to see which words follow the image and which the habit.
The nuclei
Training nuclei
Test nuclei
Describe a nucleus
The generator’s traits
The description
How a word was chosen
How each head decides where to look
Who reads whom, across the description
Test nuclei ·
Your text: train the next-word model on any text
The same model as the reports, the chat and the descriptions, on a text of your choosing. Paste anything into the box and press Use this text: the page cuts it into documents (a poem each when the titles are marked the way Project Gutenberg marks them, with underscores on a line of their own; a paragraph each otherwise; anything longer than 200 tokens into windows cut at line ends), keeps one document in ten aside as the held-out set, builds the vocabulary from the words, and trains on the next word alone. The default is Robert Frost’s Mountain Interval (1916), the book of “The Road Not Taken” and “Birches”: 33 poems and some 15,000 tokens, small enough to train here in minutes and large enough to write lines that scan. His five books of 1913 to 1928 together are a click away, with a trained model of their own. The texts are in the public domain in the United States; in countries that count seventy years from the author’s death they stay in copyright until 2034.
The document
The documents
Training documents
Held-out documents · never trained on
Start it off
The prompt
What it writes · coloured by its confidence in each word it chose
How a word was chosen
How each head decides where to look
Who reads whom, across the text
DNA: a genome of our own
The same next-token model as the reports, the chat and the text, on a chromosome drawn by a script, so that every base carries the truth of what it is. The tokens are the four bases; the documents are windows of 64 bases; the first three quarters of the chromosome train the model and the last quarter is held out; the only teacher is the next base, and nothing says where the genes are. This is what a DNA foundation model is, in miniature: the same choice of a token and a teacher as a language model, on a genome. Afterwards the Test step asks what it learned: where along the held-out stretch it is surprised, how it scores a planted variant, and what a single layer on its vectors can tell with a few labels.
What the script planted, and the model is never told
The chromosome
The window
Where it is surprised: the held-out quarter, base by base
The planted variants, scored without a label
log₂ P(the sequence with the change) − log₂ P(without it), over the variant and the 32 bases after itWhat the frozen model knows: a probe with ten labelled windows a side
a single layer on the mean of the model’s vectors over a window · scored on held-out windowsAnnotate: the labels
The DNA model of the last recipe was trained on the next base alone. Here it is put to work. An annotator has labelled one stretch of the chromosome, a kind for every base of it: exon, intron, CpG island, repeat, or everything else. The model is frozen and reads the chromosome; a single layer on what it holds at every base learns the labelled stretch, and is then asked to name the kind of every base of the held-out quarter, which the model never trained on and the layer never saw. The middle of the chromosome carries no labels at all: the model read it during its training, as sequence and nothing more.
The chromosome, and what is labelled
The labelled stretch, kind by kind
what the layer trains on · the chromosome as a whole beside itFrom a base to a kind
what is frozen, what is trainedThe held-out quarter, as the layer paints it
Curves
the layer's loss on the labelled bases · its painting of the held-out quarter after every epochA held-out gene, base by base
Kind by kind, and what the pretraining bought
the held-out quarter · the same layer on an untrained model's vectors beside itThe layer finds the exons when the model finds the frame
measured beforehand in Node · the recipe's pretraining run again, the layer trained afresh on one labelled gene after every epochHow many labelled genes does it take, three ways
measured beforehand in Node · the dot is the layer on screenSegment: a class for every pixel
The field and its mask
Where the mask comes from
Nobody painted these. The generator that drew the field kept its own geometry, the membrane's height and every nest's outline, and the mask is read off it: above the surface is lumen, between the surface and the membrane is epithelium, below the membrane is stroma, except inside a nest's outline, a cord or a shed single cell, which is the fourth class. Real segmentation masks are drawn by hand, a pixel at a time, and are the dearest labels in the field.
What the network has to do
A class for every one of the 22,528 pixels of a field, from the pixels around it. That is 22,528 labelled examples per field, which is why segmentation learns from few fields; but the examples are not independent, and the hard part is not telling a nucleus from the stroma, it is telling a nucleus in the epithelium from the same nucleus in a nest below the membrane, which no pixel can do on its own.
The fields
Training fields
Test fields · held out
The field, as the network paints it
What the first layer sees
The network, live, over the whole field
the field through every layer, left to right · dark where a map fires · the four classes in their colours at the end · hover any map or filterThe network, for one pixel
every map at the pixel's place: the channels as circles, the kernels' contributions as lines · the pixel follows the mouse on the field, click to pin itThe first layer's maps on this field
each filter's response, dark where strong · the network looks at these, never at the pixels againCurves
on crops as it trains · on a few test fields after every epochThe fields
Training fields
Test fields · never trained on