ESRGAN At The Movies (image heavy)

ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks) is software that tries to make an image larger by hallucinating additional detail with a neural network.

So one question is whether ESRGAN can be used to upscale movies, for example to make a DVD-quality image look like HD.

To examine this I decided to experiment with some films that I have on both DVD and Blu Ray. But what to choose? I have several BD+DVD combo packs but in that case they’re probably both downscaled from the same source and I can rig that sort of test from any HD image. A harder test would be one where the DVD is clearly from a different/inferior mastering. I’ve got several of those as well, and quickly narrowed it down to only the most worthy masterpieces of modern cinema:

So let’s just take a frame and see what happens. ESRGAN is trained to scale by 4x and plugging a DVD frame into it actually makes something close to a 4K image, so I’ve shrunk the result back down to around the same size as the 1080p BD. Here’s a crop of most of the frame, first the DVD frame scaled up by ESRGAN:

And now the same frame from the Blu-Ray:

So yeah, no contest – even ESRGAN can’t pull enough of a magic trick to make this DVD look like a proper HD image. Here’s some zoom-ins of a few spots. Each strip has the DVD image scaled with a more traditional cubic algorithm on the left, ESRGAN’s version in the middle, and then the actual HD on the right.

Here we can easily see that Ozone’s studdy belts are missing a lot of detail and the metal bits just blur together. ESRGAN gives sharper edges but it also amplifies the compression artifacts at the bottom.

Poor Kelly’s hair and face is still just a rosy-cheeked blur, and she doesn’t even seem to be looking in the same direction.

One thing about movies is that the frames tend to have a lot of motion blur. We can see here that the DVD compression on Ozone’s sweet hand moves was just terrible to begin with, and ESRGAN only made it moreso. Also everybody’s teeth are much better in the HD version.

Um, that guy.

ESRGAN obviously doesn’t do well with noise or glitches – they just cause it to hallucinate even more of them. Note that I’m using the ESRGAN model directly. The software has a second low-noise model available and provides a way to use that to smooth out some of the excess noise that ESRGAN introduces, but where’s the fun in that?

With the DVD you sometimes get a bunch of compression artifacts in a grid pattern, and ESRGAN makes it worse. In a few spots on Ozone’s arm where the compression was particularly rough, ESRGAN added a bunch of waves (the green lines are just here to highlight the grid):

So I guess the answer to the original question is very much “NO”.

⋰
⋱
⋰
⋱
⋰
⋱
⋰
⋱
⋰

With that out of the way, what else to try? Well, ESRGAN was trained with images scaled down from high-quality starting points, so let’s see what it does if you give it something more like that. Since ESRGAN scales by 4x, if we downscale a 1080p (1920x1080) image to 480x270 and then feed it through ESRGAN, we should get back a 1080p image that we can directly compare pixel-for-pixel.

So here we have a totally not chosen on purpose shot of Jean-Claude Van Damme cheering Kelly’s dance moves

NOTE: to save bandwidth, some frame images are scaled down and linked to full resolution versions.

Scale down, ENHANCE, and hey that doesn’t look too bad…

Oh.

Oh dear.

This zoom-in has the HD original, the 1/4 resolution image that was fed into the scaler, what you’d get with a more traditional cubic upscale, and finally what ESRGAN perpetrated.

There is so much going on here, such as what happened to this finger. Uh, exactly what sort of images was ESRGAN trained on again?

Here we see JCVD’s fist get all Cronenberged:

And while this guy is still smiling, now he looks like he just came from a fight:

When subtle details are lost in the downscale it sometimes recovers something similar to the original, but this bike wheel now has crazy spokes and those Adidas socks didn’t survive:

What’s really interesting is that the ESRGAN version sometimes seems to have more detail than the original, even though it started with a much lower-resolution image. Of course this extra detail is all fake but it “fits” with the image pretty well, such as this guy’s hair and the tree behind him:

There are certainly spots in the image where it’s very impressive compared to the blurry cubic scaling, for example even some really fine lines such as brake cables and the band on Ozone’s hat came back. The buildings in the background also got a facelift but they don’t look obviously wrong, just a little different.

So it’s off to the club for a dance battle…

Most of that frame did okay and Ice-T even got spikier, though the text on his necklace got lost not to mention HIS FACE:

At the auditions the panel is still impressed by TKO Crew’s performance:

However it seems that the snooty judges have been replaced by ravenous elder gods:

Over at the hospital the surgeon still notices the lineup of dancing nurses:

But now he looks more like he’s just seen the Ark of the Covenant, and the patient’s heartrate has understandably become irregular:

Ozone’s rooftop dance gets grainier but doesn’t do too bad:

Except that down on the ground it looks like there’s been a hurricane:

Will the zoning board save the community center from the greedy developer?

W̖͉̯̫̰͎͘ͅE̺̥ H̱̤̲̤̲̳A͍̩̮̘̮̖̥͠V͍̱E̺͚͎̫ S̺̥͉̪͔̻͡U̘̼̜̺̦̗̗C̨͈͓̝̦̥Ḫ͢ ̰S̞̣I̧̞͙̱̦̻͙ͅG̛̪̟̟H͍̗͕̯T̴̺͔̳̭̲͖S̶ ̧̫T̛O͉̦̬͚ͅ ̩͡S̬̞͔̱H̜͔̼͍̠͖O̜̮̘̬̙͜ͅW̯̱ ̡̪͍̱̥̼̹YO͍̤̮̳͇͔U͔͙͞

⋰
⋱
⋰
⋱
⋰
⋱
⋰
⋱
⋰

OKAY, BUT CAN WE GO DEEPER? What happens if I cut the resolution to only 1/8 scale (240x135) and give ESRGAN only a thumbnail’s worth of pixels to work with?

Obviously the faces are a lot worse. Even medium-sized faces such as Franco’s don’t fare well:

And in the background, Ozone looks like he’s just stepped into a painting:

It’s still able to add realistic details in some spots, though, such as Turbo’s hair:

And some of the bricks are really just amazing:

How about a dance lineup from the Miracles montage?

Lil’ Wizard’s shirt is still readable but the font has changed, and the Vans have had their checkerboards swapped out for stripes. When ESRGAN finds lines or creases it seems to add more MORE MORE of them, which heavily affects clothing and faces:

At the garden party there’s more face rearranging, and James’s hand gets turned into a rough sketch. As usual it manages to do something reasonable with hair, though:

Over to the club to take down Electro Rock:

I don’t even…

Turbo’s ceiling dance doesn’t look too bad if you can avoid looking at his face:

Monster faces aside, it really is pretty remarkable what it’s able to construct from a few pixels, such as here where it somehow recovers Ozone’s collar bars and round snaps:

Unlike some other scaling algorithms ESRGAN isn’t just looking at a few neighboring pixels. I originally tried scaling some cropped regions on their own, and it produced noticeably different results compared to giving it the entire image to work with.

Before I started this I was wondering if someone had done an ESRGAN plugin for a photo editor and didn’t find anything. Now that I’ve actually used it I can see why – to run it you need a lot of other software installed such as Python 3.x and a pile of libraries. The library that handles the neural network also comes in several variants and you have to pick one specific to your OS and GPU/graphics driver.

Even with all the training done ahead of time, applying the trained model to an image still involves a lot of computations. I originally tried to run it on a machine that doesn’t have a CUDA-capable driver, which meant doing all of the computations on the CPU. With 6 cores pegged at 100% it took over a minute for a really small test image, and trying to give it a DVD frame resulted in several minutes of churning followed by a crash. I switched over to a system with a 1080Ti GPU and installed the latest CUDA driver, and when using the graphics card to do the heavy lifting it only takes about a second to process a DVD frame.

Since it can create “interesting” distortions, that raises the question of what it does if you apply it again? What seems to happen is that the image gets noisier (no surprise there), and after several iterations it starts to acquire a cyan tint.

Here’s what happens when applying it repeatedly with a 4x reduction-enhancement cycle, so: start with 1920x1080, cubic downscale to 480x270, ESRGAN back up to 1920x1080, and so on. After 3 iterations things are already pretty bad, and by 5 the cyan is visible.

NOTE: the full images were 150MB+ so this just shows a cropped slice of them with lossy compression.

Another:

And here’s starting with an 8x reduction from the original and then doing 4x up/down scaling after that, so 1920x1080 cubic down to 240x135, ESRGAN up to 960x640, cubic to 240x135, etc. It’s about the same, but the noise is more obvious because of the lower overall resolution:

I tried running one for a few more iterations but it only changed a little more in each step, and the only obvious difference was that the shading and details from the original image kept getting a bit fainter. I suspect that after a while it just turns into uniform black/white/cyan noise, perhaps with the blobs and lines shifting around each time.

BTW a warning: searching for other sample ESRGAN images can turn out to be very not work-safe. It might not have been trained for porn, but there is apparently no shortage of dudes trying to use it to upscale their collections anyway.

♫ I BELIEVEBLARGLGLGLLL IN THE BEAT! ♫