New method for upscaling pixel art

Hey folks. So I've been fiddling with a new way to upscale pixel art. You might be aware of methods like hq4x etc which provide a limited upscaling ability. There are also methods to convert to vector graphics which are very nice, like this Microsoft paper for example: http://research.microsoft.com/en-us/um/people/kopf/pixelart/. So I thought I'd try to come up with something similar, with a unique algorithm.

This mainly works best with a limited palette, because in order to get joined `areas` turning into shapes there have to be pixels of the same or similar-enough color. Otherwise, e.g. with a 24-bit image, you get a ton of individual dots of all different colors which doesn't work so well. One way to overcome the dots is to blur and/or posterize/quantize the image.

Results so far are below.

Original small image 128-pixels width, upscaled 8x with nearest neighbor:




Image upscaled with best-quality `sinc (lanczos3)` scaling in Gimp:



Image upscaled with gaussian filter:



Image upscaled with my custom method, preserving 1x1 pixel blocks:



Image upscale with my custom method, preserving only 2x2 blocks (less detail, smother edges):


Basically I'm trying to not have any blurring or fuzziness at all, converting everything to smooth curves. I'm not doing anything as advanced as generating splines as in the Microsoft approach, but using a different method to create a somewhat similar result.

On the one hand I like that the image is crisp and doesn't appear fuzzy, and it can upscale to ANY resolution. One part I don't like is how individual pixels - very common in antialiased source images - turn into individual `dots`... in some images there can be a lot of them. This is alleviated with some blurring/smoothing as in the final image, but then loses the very fine details. So I guess it depends how small or detailed your source image is. I think it's pretty good considering it started at 128 pixels and now it's 1024 pixels.

 

A side-by-side of my custom method (left) vs sinc/lanczos (right), 4x upscale:

 

A 32x magnification closeup...

 

interesting, btw. have you seen this one:
(scroll down to) "A New Interpolation Technique"
http://www.world-machine.com/blog/

 

Ahh, looks interesting, sort of similar to hg4x etc in terms of quite straight vertical and horizontal edges. With mine I'm trying to interpret edges which were originally intended to be at a variety of angles and come up with a straight edge at that angle, which is a bit harder to do.

 

Here's a comparison using a yoshi sprite... 12 colors, 8x magnification.

Nearest neighbor on the left, then Microsoft's vector version, then my smoothed version, then my more accurate version.



Clearly some difference in terms of how larger areas are more grouped together in the Microsoft version. A little bit more `bubbly` looking in my version. I do notice though that in some areas mine looks almost identical ... the white and black spot on the nose, the eyes ... overall not quite as smoothly joined but. ... it's ok. I need to find a better way to smooth without losing details.

Meanwhile.. a more antialiased version. Simply a matter of magnifying much larger as `supersampling` then downsampling.

 

Another comparison with a mario/yoshi sprite...

Left to right - nearest neighbor, microsoft's vector version, super2xsal, and my version on the right. When you compare the nearest neighbor original to mine, versus microsoft's, I think sometimes my representation is closer to the original shapes, even though it's not quite as smooth.


Also my method has a curious capability of over-inflating or deflating the size of areas, which can result in something like this:

 

And a bomberman sprite...

Left to right... nearest neighbor, microsoft's vectors, super2xsal, and mine on the right (antialiased). I notice that microsoft's algorithm on this sprite is a little bit funky in terms of shapes... I think the shapes in mine fare a bit closer to the original. But again theirs is smoother.

 

Barbarian sprite comparison...

Nearest neighbor, microsoft's vectors, mine with detail, mine smoothed on the right.

 

Keyboard comparison...

Top left nearest neighbor, top right microsoft, bottom left super2xsal, bottom right mine. You can see the shape distortions in the microsoft version ... though in most the keys are not square at all.

 

Anyone have any interest in this, maybe as an asset store tool which converts textures?

 

This is a rather cool filter. I remember hq4x and that other lq2x one that MAME used... it always produced an interesting and mesmerizing effect, and now yours is even more interesting with the under/over inflation settings.

Perhaps your single-dot islands might be alleviated by applying a heuristic to remove single dots based on their relative proximity to similar colors nearby, or their relative proximity to complementary colors nearby, in which case you might give them more weight?

Just random thoughts... cool stuff man.

Kurt

 

Yes I have this bit of code which figures out which dots are islands and `doubles` their size... it generally works in most cases. Sometimes with pixel art it's hard to decide what to do, like in a 2x2 grid of pixels where the colors are in a checkerboard pattern, how to figure out whether to draw a diagonal line going one way, or the other... you sort of have to understand what the artist intended or what's happening in the nearby neighborhood. I have another approach to smoothing things out without losing the precision, which is to sample positions inbetween pixels and add a slight extra bulge there, to alleviate some of the jagginess.

 

Fascinating stuff here, and it looks like you're getting some good results so far! Out of curiosity, have you seen any work in this vein done with lines instead of curves? Wonder what that would look like.

 

I haven't seen a line-based one... other than that some will use mostly axis-aligned or diagonal shapes, like in hq4x etc. You could take any pixel image and do pretty much any image processing you like with it to come up with a new stylized representation.

 

i don't know when i need this but it looks cool

 

Interesting!

In the examples you've provided I tend to prefer either yours (esp barbarian & keyboard) or Microsoft's (yoshi & mario) and yours second. This is just a subjective evaluation though. In some cases, like bomberman (to a lesser extent the keyboard) I think a mutant hybrid of your version and microsofts would just about nail it.

However in the very first example you lost a lot of detail compared to the sinc/lanczos. This is especially apparent when you compare for example the first red guy (eye missing) and star (major loss of definition).

This looks like it's shaping up to be an interesting effect.

Keep up the good work.