What exactly is ‘AI-generated art’? How does it work? Will it replace human visual artists? | Art

Josh, I’ve been hearing a lot about ‘AI-generated art’ and seeing a whole lot of truly insane-looking memes. What’s going on, are the machines picking up paintbrushes now?

Not paintbrushes, no. What you’re seeing are neural networks (algorithms that supposedly mimic how our neurons signal each other) trained to generate images from text. It’s basically a lot of maths.

Neural networks? Generating images from text? So, like, you plug ‘Kermit the Frog in Blade Runner’ into a computer and it spits out pictures of… that?

You aren’t thinking outside the box enough! Sure, you can create all the Kermit images you want. But the reason you’re hearing about AI art is because of the ability to create images from ideas no one has ever expressed before. If you do a Google search for “a kangaroo made of cheese” you won’t really find anything. But here’s nine of them generated by a model.

You mentioned that it’s all a load of maths before, but – putting it as simply as you can – how does it actually work?

I’m no expert, but essentially what they’ve done is get a computer to “look” at millions or billions of pictures of cats and bridges and so on. These are usually scraped from the internet, along with the captions associated with them.

The algorithms identify patterns in the images and captions and eventually can start predicting what captions and images go together. Once a model can predict what an image “should” look like based on a caption, the next step is reversing it – creating entirely novel images from new “captions”.

DALL E mini is an AI model that generates images from any prompt you givehttps://t.co/q8KgTWdYFH pic.twitter.com/BDqBMaO5eH

— Flo (@sketchbookflo) June 5, 2022

n”,”url”:”https://twitter.com/sketchbookflo/status/1533414898541150208?s=20&t=usUO1lq2HoSoeWdKUwTQEg”,”id”:”1533414898541150208″,”hasMedia”:false,”role”:”inline “,”isThirdPartyTracking”:false,”source”:”Twitter”,”elementId”:”fe49bc8e-6b14-449a-8df1-110ad38cafe7″}}”/>

When these programs are making new images, it is finding commonalities – like, all my images tagged ‘kangaroos’ are usually big blocks of shapes like Esteand ‘cheese’ is usually a bunch of pixels that look like Este – and just spinning up variations on that?

It’s a bit more than that. If you look at this blog post from 2018 you can see how much trouble older models had. When given the caption “a herd of giraffes on a ship”, it created a bunch of giraffe-colored blobs standing in water. So the fact we are getting recognizable kangaroos and several kinds of cheese shows how there has been a big leap in the algorithms’ “understanding”.

Dang. So what’s changed so that the stuff it makes doesn’t resemble completely horrible nightmares any more?

There’s been a number of developments in techniques, as well as the datasets that they train on. In 2020 a company named OpenAi released GPT-3 – an algorithm that is able to generate text eerily close to what a human could write. One of the most hyped text-to-image generating algorithms, DALLE, is based on GPT-3; more recently, Google released Imageusing their own text models.

These algorithms are fed massive amounts of data and forced to do thousands of “exercises” to get better at prediction.

‘exercise’? Are there still actual people involvedlike telling the algorithms if what they’re doing is right or wrong?

Actually, this is another big development. When you use one of these models you’re probably only seeing a handful of the images that were actually generated. Similar to how these models were initially trained to predict the best captions for images, they only show you the images that best fit the text you gave them. They are marking themselves.

But there’s still weaknesses in this generation process, right?

I can’t stress enough that this isn’t intelligence. The algorithms don’t “understand” what the words mean or the images in the same way you or I do. It’s kind of like a best guess based on what it’s “seen” before. So there’s quite a few limitations both in what it can do, and what it does that it probably shouldn’t do (such as potentially graphic imagery).

hm pic.twitter.com/VB0RkDUXXE

— Matt Bevan (@MatthewBevan) June 9, 2022

n”,”url”:”https://twitter.com/MatthewBevan/status/1534752593372905473?s=20&t=alX2bHC8hcBnR71SKZ-Yjg”,”id”:”1534752593372905473″,”hasMedia”:false,”role”: “inline”,”isThirdPartyTracking”:false,”source”:”Twitter”,”elementId”:”41f6cf98-1692-48ae-88f3-b92a31b11daf”}}”/>

OK, so if the machines are making pictures on request now, how many artists will this put out of work?

For now, these algorithms are largely restricted or pricey to use. I’m still on the waiting list to try DALLE. But computing power is also getting cheaper, there are many huge image datasets, and even regular people are creating their own models. Like the one we used to create the kangaroo images.

I doubt anyone knows what will happen to artists. But there are still so many edge cases where these models break down that I wouldn’t be relying on them exclusively.

I have a looming feeling AI generated art will devour the economic sustainability of being an illustrator

not because art will be replaced by AI as a whole – but because it’ll be so much cheaper and good enough for most people and corporations

— Freya Holmer (@FreyaHolmer) June 2, 2022

n”,”url”:”https://twitter.com/FreyaHolmer/status/1532261886078631940″,”id”:”1532261886078631940″,”hasMedia”:false,”role”:”inline”,”isThirdPartyTracking”: false,”source”:”Twitter”,”elementId”:”2d878b3f-76a2-4cd9-a98b-b4567bafee61″}}”>

I have a looming feeling AI generated art will devour the economic sustainability of being an illustrator

not because art will be replaced by AI as a whole – but because it’ll be so much cheaper and good enough for most people and corporations

— Freya Holmer (@FreyaHolmer) June 2, 2022

Are there other issues with making images based purely on pattern-matching and then marking themselves on their answers? Any questions of bias, say, or unfortunate associations?

Something you’ll notice in the corporate announcements of these models is they tend to use innocuous examples. Lots of generated images of animals. This speaks to one of the massive issues with using the internet to train a pattern matching algorithm – so much of it is absolutely terrible.

A couple of years ago a dataset of 80m images used to train algorithms was taken down by MIT researchers Because of “derogatory terms as categories and offensive images”. Something we’ve noticed in our experiments is that “businessy” words seem to be associated with generated images of men.

So right now it’s just about good enough for memes, and still makes weird nightmare images (especially of faces), but not as much as it used to. But who knows about the future. Thanks Josh.