Echo Nolan's Blog

Can you train a text-to-image model without any text?

Posted on March 9, 2024 Despite Betteridge's law, I contend the answer is "probably". I'd prefer to be writing a post without a question mark at the end of the title, but I haven't actually gotten it to work yet. This post is something of a research log documenting my efforts, though written ex post. This code for the models and data processing is here.

Why should it be possible?

A text-to-image model is one which, well, takes in text and outputs images. DALL-E and Midjourney are the most famous examples, but there are tons. The user types in a prompt ("Central Park during the freak snowstorm of 2024, shot on Hasselblad medium format") and the model generates an image that resembles the prompt.

What's in there?

Tons and tons of screenshots of GTA. Seriously, it's absurd.

How to judge outcomes?

People make various claims about what advantages various programming languages have. Java is good for large enterprise projects, Ruby lets you get something working fast, C lets you write fast programs, Haskell makes it harder to write bugs, etc, etc. In the interest of simplicity I've decided to skip anything more precise than user satisfaction.

How to collect explanatory variables?

We need to collect the languages used in the projects, what components they're used for, confounding variables and any other useful predictors. For open source software this is relatively easy - I can probably write a tool to gather language and component information and maybe even topic; age and contributor count are in source control. For commercial projects we may have to survey people who work there.