All field notes

AI · Retrospective

CLIP, DALL·E and the labelled data my deep-learning research needed

A stack of photo cards, the top one tied to a blank tag with blue thread, beside a pile of loose tags.
Concept illustration · AI-generated

On 5 January OpenAI announced two image models. DALL·E is a 12-billion-parameter version of GPT-3 trained to generate images from text descriptions. It has been shown through samples and is not open for anyone to use. CLIP is trained to match images with text. The part of CLIP I keep coming back to is not the model but its training data. The paper released with it describes 400 million image and text pairs collected from the internet, instead of a dataset someone labelled for the purpose.

That is the part that speaks to my own work.

Labels were the cost

My Master by Research at Multimedia University, as a TM R&D-sponsored scholar, applied deep learning to predicting network faults. The work was published in IJET in 2018. I left the programme before writing the thesis. Supervised learning of that kind needs examples of the outcome you want to predict, recorded and labelled as such. The model can only be as good as the record of what went wrong.

OpenAI's announcement makes the same point about images. It notes that ImageNet, one of the largest labelling efforts, needed more than 25,000 workers to annotate 14 million images across 22,000 categories. The usual vision model is trained on a fixed list of categories that somebody paid to label.

What changes when the labels come from the web

CLIP drops the fixed list. It trains an image encoder and a text encoder together, so that each image scores higher with its own caption than with the other captions in the same batch. To classify with it, you write the category names out as text, such as "a photo of a dog", and choose the closest match.

The result OpenAI highlights: on ImageNet, without using any of its 1.28 million labelled training examples, CLIP matches the accuracy of the original ResNet-50. The labelling was done by everyone who ever wrote a caption next to an image, without knowing they were doing it.

That changes the cost, and it also changes who the labeller is. Captions carry the habits and gaps of the people who wrote them, and no one can check 400 million labels the way a team checks a few thousand.

Where it would not have helped me

Fault data is not on the web. A network's fault records belong to its operator, they have no captions, and the events worth predicting are rare. There is no scraped collection that pairs it with descriptions, so this recipe does not transfer to it directly.

Fine distinctions remain hard. OpenAI lists the weak spots itself: zero-shot CLIP struggles to tell apart car models, aircraft variants or flower species, and with tasks such as counting objects. In July I wrote down a plan for a durian-grading model. It lists what people use to identify and grade a durian: the colour and shape of the thorns, the gaps between them, the overall shape, the colour of the shell, and sometimes the weight of the fruit. That is a fine-grained judgment, and weight does not appear in a photograph.

What that plan did get right, in a small way, is the lesson CLIP teaches at scale. Its first phase was an app where people record stall prices and tag their own photos with the species and whether they show seed, flesh or husk. The labels would arrive as a by-product of something useful, without a separate labelling project.

Some data labels itself. At Hexon Data we built live bus positions and arrival estimates for the PJ City Bus app. Every trip eventually produces its own answer: the bus arrives, and the time is recorded. When the world supplies the label, the size of the dataset grows with use instead of with a labelling budget.

DALL·E, as far as the samples show

I cannot judge how reliable DALL·E is, because the public has only the samples OpenAI has published. One detail is worth noting: those samples were reranked using CLIP. A generator that produces many candidates, paired with a separate model that scores them against the text, is a useful pattern in its own right. It moves part of the quality problem from generation to judgment, and the judgment model inherits all the limits of its own training data.

Where the labels come from

The question I take from this month is less about architecture than about supply. For CLIP, the labels come from the web. For network faults, they come from the operator's own records. For durians, they would come from a crowd. For buses, they come from the clock. Before choosing a model, I want to know which of these a problem has, and who pays for it.