We were about to make a video about Nicholas' findings. After seeing your brilliant explanation, we don't have much to add! Thank you so much for sharing our research!

Really nice to find a Stable Diffusion channel that actually understands how the underlying technology works! Subscribed. So many of the bigger channels just do how-tos and make "breaking news" videos about new features without taking the time to understand *why* these things work. Great explanation about the issue caused by SD's noising function. I understood the issue perfectly despite not having any ML experience (though, I am a software developer, so that helps lol)

wow, in hindsight seems so obvious. It only shows how that we're in early days and there's huge things just waiting to be discovered in the diffusion models. Love your work. Z.

Midjourney Right Now: Nooooooo! My Dineros!!! My Dineros!!!

god I love this channel. What a great, straightforward explanation on the topic! I don't really know anyone else going this in depth on these stable diffusion topics on KZfaq. Greatly appreciated!

Really appreciate all the effort that goes into research and how well you explain it along the visualizations. Keep up the great work!

This dude is singlehandedly wiping the floor with pretty much every other channel about SD. You are a godsent mydude. Lets' watch that subscribe count going to the moon.

You are like teaching how and what paintbrushes are made of, the characteristics of canvases and all the fundamental stuff an artist should know to make good art. Thanks for the insights and your efforts to make clear some obscure subjects.

Your videos were already good to begin with and they're getting better. Great information, good explanations. Appreciate the good work! Cheers! :D

wow I just redid my last dreambooth model using offset noise, and I'm blown away by the difference it makes in general quality!

Got to be the most informative video on SD I’ve watched in months. I actually understand how it works a little better now; Excellent content!

there’s nothing more beautiful then a simple fix

Hoping for another leak soon to learn more secrets like this.

what an amazingly simple description of a fourier transform

.. and you made this video with all that content. That alone is a lot!! Thank you.

Great video. Perfect balance of technical and everyday. Keep going. Subscribed.

Sweet. The other day I was trying to generate a scene set at night, and no matter how much I fine tuned the positive prompt to emphasize dark (and used the negative to omit light-related concepts), I kept getting outputs with really bright skies. Night-associated hues, but day-like intensities. Intentional or not, this video answered a lot of my questions regarding SD's trouble in this specific case. These deeper dive videos are much appreciated. Keep the goods coming!

Thank you for the explanation! I imagine the best hack is to randomize a seperate offset on each image, but the more faithful method I’ve seen is to use cosine noise schedule and fully destroy the image, which for some reason SD, uses a smaller standard deviation of 0.12 instead of 0.2 in previous works

The best SD channel , THKs Koi (Smiling:1.1)

You explained noising so well, thank you for your awesome videos!

Another wonderful video with Amazing explanations!!! So awesome see how far Cacoe has come.

Brilliant analysis. Another awesome vid!

all of your videos are the best video ive seen on that topic. all of them.

They could've used simplex or perlin noise with a number of harmonics for training to properly represent low and high frequency noise.

Great video and explanation! Kudos 🙌

It makes good sense! Well done with the explanation.

The one that Ive noticed and would like to see is these models doing more to improve symmetry and anatomy of odjects. As an artist these are things that we are taught and take years of practice to get right. And perhaps wants they get is down right, the need for artist in this reguard would minimize but you have to remember that this is a tool that is for people who havent mastered these aspect of art yet.

I like this guy! Clearly Midjourney's style is so distinctive, like a single, well-known artist, that you can immediately distinguish which is which. While SD with all its quirks is much more diverse and versatile, like thousands of artists working behind the scenes. And that's a huge advantage of SD (plus you can run it locally).

Misjourney has already met its doom since it started thanks to SD being open-sourced. It was eventually going to meet its match with the growing community of model trainers and people discovering many awesome functions and tricks.

This was a joy to watch :)

midjourney still has the edge, maybe that can change with next version of illuminate

The title and the thumbnail of the video made me think this was an art video, so I started it, then took a long break to read the original post, went down a rabbit hole back to '90s research on power spectra, and finally came back to the video to realise it was not an art video after all, but rather an amazing technical analysis of the work. I do want to point out though it's not really that the noise doesn't affect low frequency features - white noise has a flat power spectrum and affects features of all frequencies equally. What's happening is that "natural" images have a bias that low frequency features have more power, so a constant amount of noise affects the signal to noise ratio of high frequency features more.

Midjourney is far ahead (in part) because they add random tokens that are hidden to prompts based on RLHF (user feedback and ratings) So if you can rate aesthetics of a model based on an reinforced approach to the weights on the prompts then this helps

People always says "this is the midjourney killer/this could replace midjourney" As if stable diffusion and it's custom models aren't already beating it for a quite while now

Awe. Kaapstad in the house. Great explanation bro!

Awesome video 😉

Great explanation! The principal kind of sounds like HDR photography. Combining multiple photos of different exposures to get a high dynamic range. The results can be stunning.

Very cool. I hope stability ai sees this and updates these noising functions for the 3.0 mode training that’s coming up

so how do you use these offset thing in SD?

Sweet!!

good technical and fun staff

thnx for the update

What is happening? I was reading this topic about a hour ago o_O It seems like we starting to enter the singuliarity phase more and more every second.

great video overview thanks ! yea i think there is so much more experimentation we can do with diffusion , like implementing all the augmentations of styleganADA and more . i wonder if we think of and apply as many linear transformations as possible that would influence how the model has to learn about it , for example chromatic aberration , emboss , edge sharpening would likely influence the model to utilize a kind of convolution process to undo that type of augmentation , which could be a useful tool for the model in reconstructing an image as well .

Always Koi, nice vid :)

Is it affecting only training process, or generation of images on pretrained model will also benefit from this solution? Can I copypaste his code into my local Automatic1111 SD code to get better generations?

So simple yet so insane

BEHOLD 1.1 is here: civitai.com/models/11193/illuminati-diffusion-v11 praise be to cacoe!!!

i thought the video is on 2.5x speed when i looked at the webcam.

♥

Thanks! Very interesting! But have to admit, I have been way more worried about missing arms and wonky shapes and details in general :D

Sir, you are the 3blue1brown of AI Art.

Great explanation. There's a lora called epi_noiseoffset which does this.

Question: From the graphs in the video, as I understand it, the model assumes somewhat of a symmetrical normal distribution around the noise with set black and white points (0, 255). These assumptions (black/white points and normal distributions) don't hold true for most images - even more so for images that are extra-ordinary. Is there a way to train / denoise while accounting for these additional factors?

Noice! Surely we want noise controls for the user? Sometimes you want composition, sometimes detail…

man it's been a wild west out there an year later.

14:35 exactly what I was thinking about.

Youre hecking sick aswell fam 😊

Have you tried noising the discrete cosine transform? That way you obscure all frequencies at the same rate.

Super interesting stuff man. I use both SD and MJ, and always wishing SD was on par with MJ, particularly for “art”. SD is incredible for realism, but definitely lagging behind MJ on the artistic front; hopefully this means there’s light at the end of the tunnel for SD and art?

is the 1.1 version not available anywhere?

I'm new to SD. So we need to alter the code of SD then train our own models to approach the quality of MidJourney?

i have a question why the mj images have more like an identical art style like some digital painting with high contrast value, while sd's style seems more diverse? thank you

Sorry if I missed this in the video, but does SD 1.5 have this built in now? Or is it forthcoming? Or is there something I have to turn on?

I Think you put the wrong llink to cacoe`s server in ya description... just links to the example images...

Should try adding noise in fourier transform space instead of pixel space to tackle every frequency overall.

I don't know anything about anything when it comes to this, but how would it change if the noise were added to the color represented as HSV or CMYK or something, rather than RBG?

Very nice! But, when I try to think mathematically about *why* the high frequency components would be washed out first, even though it makes sense intuitively that that would be the case, it isn’t clear to me how to justify that conclusion mathematically? I guess the thing to do would be to describe the distribution over Fourier transforms of Gaussian noise, but at first blush, I see no reason why the variance for higher frequency components would be larger than for lower frequency components. Like, say that we have functions f : (Z/pZ) -> R (Or maybe C , but whatever) for some prime number p, if we do a discrete Fourier transform of that, then, the components/coefficients (other than the frequency zero component) will be the dot products with the different p-th roots of unity, in different orders (possibly multiplied by some constant normalization factor). But, because these have the same terms, just in different orders (this is true because I picked the number of entries to be prime, and multiplication mod p by things other than zero, is invertible.) then, for independent identically distributed random values for the components of the function, these Fourier components should also be identically distributed. Now, maybe this could be just because I picked Z/pZ instead of Z/nZ , but still, I would think that for most natural numbers less than n, for a typical n, they should have no factors in common with- ok actually that’s not true... Half of the natural numbers n are even, and half of the natural numbers less than n (for large n) are even, and so at least half the time, at least half of the numbers less than n will have a factor in common with n and therefore have no inverse... But, would that really be responsible for the low frequency components being influenced less by noise?! Would we really expect that if the training only used images where both the width and height of the image are a prime number of pixels, that this phenomenon would go away?! (Except for the frequency zero component) That doesn’t sound like it should be true. That would be pretty bizarre, I think? Maybe the thing is just that there are many more frequencies (among integer multiples of the base frequency) that we would consider to be “high frequency” than that we would consider “low frequency”, and so most of the variance ends up in what we would consider “high frequency”? If you know the answer to my confusion, even if you are reading this comment multiple years after I wrote it, please reply to let me know the answer (provided that no one else has already sufficiently explained it, but that kinda goes without saying I guess.) .

i wish there was a way to implement Dalle mini text encoder into stable diffusion

Anyone who has been training their own custom models, textual inversions, and LoRAs knows that Midjouney doesn’t hold a candle to A1111 and Kohya

great explain. can you make video about controlnet. thanks

We know this for years in denoising. High frequency will go first

We're seeing this a bit more often now - the Midjourney killers. But these platforms only help Midjourney, competition is good business and will push MJ to improve and not get comfortable. Also, until the actual release and some time being spent by millions of users, we won't really know how great this all might be. Leonardo is something that is getting a lot of action recently, but will it stand the test of time? Everything new will always be hot for a moment - but when the dust settles, that will be the true indicator.

Great channel, bump up your audio levels though! ^_^ X

maybe change the line not to 1*old_noise + 0.1*new_noise but to a*old_noise + (1-a)*new_noise ; that might help not to make the overall noise being too much

I am a little confused. You say that the Illuminate images are better, or at least as good as, MIdjourney but I find them too de-saturated. is this because you are only focusing on the values and expect Illuminate to correct this problem in the future? Okay, I had a look at the images in the Discord and they do appear better, some are quite saturated. However, in low light images the colors still do seem to desaturate. This does mirror what would happen with your eye since color sensitivity peaks in bright light and declines in lower light. In very dim light you are in effect color blind. There is a question here, though, about whether this is desirable. The dim images with higher saturation from Midjourney are, to me, more appealing.

This also kinda works as an anti "stable-diffusion-detector"

Mateeeeeeee

👋

It's wonderful to see how profit stunts technological development once again.

fun fact... this is how photography and our eyes work. around 50-70% of the world around you is... grey. what you see... is for the most part... grey.

Open source all the way!

Pardon a noob: if the models were trained on regenerating source-images that don't have a 0,5 mean necessarily from noise (With a 0,5 mean), why would it learn to target a 0,5 mean?

Open source wins yet again. Technology belongs to the people!

That is very cool, thanks for sharing! Maths wins again lol

The discovery is great but doesn't explain midjourney quality and coherence. Only data does.

If it's Open source, midjourney will just use it

Much improved but doesn’t come close to Midjouney in terms of the aesthetic and creativity

Parody 😂🎉

So did i get it right? You even haven't tested that model, 1.1 right? Great

Those are not even close to parity. MJ still has the better composition.