I think I'm missing something. I get the Sketch-to-Photo synthesis that this is based on. It's really weird, neat stuff. But as a layman, I'm having trouble seeing the difference between the result of this anime-to-sketch synthesis and what I'd expect to get out of a simple edge detection. Is the difference that it's more clever about which details to ignore?
I only dabble in graphics - but generallly simple edge detection needs really uniform tonality and no textures in the input to work well. Look for example in the more "sketchy" examples how the linework that "looks right" is extracted from a quite noisy input. Also, in the top example where there are houses, the contrast difference that gets extracted to linework is lower than in the character areas.
So, the flat shaded images with explicit black outlines - yes, it's likely there is much difference with edge detection. But when the image has lots of different contrasts and tonalities this looks much more impressive.
Also not familiar but presumably a temporal aspect weighs in, so whether something is a meaningful edge isn’t strictly dependent only on the content within a specific frame?
My high level guess is, you would do some linear cost minimization in deciding what are edges over time when doing things the traditional way. The neural network can handle nonlinearities for the optimization so you can get better results for some set of inputs (in this case, some class of anime images)
This looks like a great tool to generate some Pokémon and Beyblade coloring pages for my kids. We went through everything in Google image results many moons ago.
Do it iteratively one after the other to see if after a while the results are unrecognizable from the originals. Like those experiments that translated a text between languages to create gibberish.
This feels like a tool with lots of business cases.
Studios may be able to accelerate digitalization and colorization.
The ability to convert stills to a fillable outline or repurchase for labels/marketing/branded coloring books (or apps) could be worth some money to those with a large content library.
Semi off-topic – is there a tool to turn a picture into an drawing? I sometimes see websites where people have created a avatar of their headshot that looks ‘toonish’.
Interesting. I wonder how it fares with 3D renderings? I'm a Blender user and unfortunately, Blender "Toon Shading" capabilities are not very good compared to say Cinema 4D.
Photoshop just detects edges and does thus ends up detecting both sides of a drawn line or change in shading for example. This does not appear to show such artefacts.
I wonder if this can be used for comic book inking. It looks like they have an example of that.
Typically the workflow is pencil drawing -> cleaned up ink drawing (japanese animation uses a similar process too). If this can speed up that process it could save a lot of time.
Does anyone know a similar model that transforms normal images into Western Comic Book style? I've seen it a lot for Anime/Manga, but never for that classic style of 90's comic books.
I'm not super familiar with Deep Learning, but based on the fact this is effectively extracting edges and the ConvTranspose2d layers I'm guessing it's some sort of Convolutional Neural Net?
What could we use this for? The immediate thing that comes to mind is making a coloring book. I’m wondering if I could use it to make something original
So, the flat shaded images with explicit black outlines - yes, it's likely there is much difference with edge detection. But when the image has lots of different contrasts and tonalities this looks much more impressive.
Search for example for "Coherent Line Drawing" by Henry Kang (image from the research: https://d3i71xaburhd42.cloudfront.net/92b2ec5ef4f58c4206fc5c...) or for "XDoG: An eXtended difference-of-Gaussians ..." by Holger Winnemoller, Jan Eric Kyprianidis, ... (https://ars.els-cdn.com/content/image/1-s2.0-S00978493120004...)
The default Image.save quality is very low to a point that the JPEG artifact is more prominent than the line art themselves.
L91 @ data.py: image_pil.save(image_path, format='PNG')
Studios may be able to accelerate digitalization and colorization.
The ability to convert stills to a fillable outline or repurchase for labels/marketing/branded coloring books (or apps) could be worth some money to those with a large content library.
It would be really interesting to see this in OpenToonz or the like.
https://i.imgur.com/6lFYnmi.jpg
BEER NPR (MALT renderer) - https://blendernpr.org/beer/
Lightning Boy shader 2.0 - https://www.youtube.com/watch?v=8fHZcnTFYEI
Procedural hatching and manga shaders - https://gumroad.com/l/tcKOI
Typically the workflow is pencil drawing -> cleaned up ink drawing (japanese animation uses a similar process too). If this can speed up that process it could save a lot of time.
https://youtu.be/7U7BDn-gU18
I think this would be pretty cool if it would support any picture or video.
https://i.imgur.com/LfZI0QD.png