"If you young kids don't know what an RBF kernel is ... you map it into an infinite space using Guassian Kernels ... yeah ... maybe wikipedia is better at that than I am"

I said it before and I'll say it again. Thank you very much for these videos. You are saving me (and I imagine others in the community) a lot of time having to parse through the details of these papers.

At the first glance this looks like an incredibly complex paper. Thank you for explaining it so simply.

A small note on the why the gradient of the SDF is 1 almost everywhere: The SDF is just the (signed) distance to the closest point, so the gradient will of course point in the opposite direction to that point and and if you move one unit away from that point the SDF will increase by one. Hence the gradient is one. The almost everywhere part is just that there may be multiple points equally far away or that you are exactly at another point. Also not sure if it was mentioned but the sign is just representing if we're inside or outside of an object.

Please explain this paper "Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains ", which is highly related to SIREN

Was reading this paper last night and get confused about "one image is one dataset" as well. So glad what I finally understood is actually true.

Thank you for the simple and intuitive explanation of what at first glance looked like a dense and difficult paper.

Thank you for making this videos!!! It surely saves me a ton of time. Thank you again! Please keep making more!!

I don't know if it would have been possible for me to understand this paper all by myself. Thank you so much, Yannic.

Wow that was quick!! Thanks for this

As a CogSci, I have to recommend you to review the paper "GAIT-prop: A biologically plausible learning rule derived from backpropagation of error" by Ahmad, van Greven and Ambrogini. I feel like a more bio-inspired way of encoding propagation of error signals through processing networks could hold potential for the investigation of functional behavior by for example Drosophila

Thank you for making this paper so understandable

Just saw that you broke this into sections for easier access. You're beautiful. Thank you

this was super helpful, thank you so much!

Wow this helps me a lot. Thank you!

Great summary Yannic. I am a PhD student with focus in CNNs for classification of binary images and presented this at journal club. Your explanation of implicit neural representations was inspiring.

dude...you deserve a new table! Let's chip in so you can buy a new one. Very good job explaining this interesting paper. Thank you!

The almost everywhere comes from measure theory, it means that it should hold except for elements of omega with measure zero (the boundary omega_zero does not have measure zero, but a distance to the boundary of zero). But it seems a bit overkill to state it there..

As Yannic says at 17:30: sin(x) has been tried many times before, vanilla and also in combinations like SinReLU. I currently doubt that (even with special initialization) vanilla sin(x) outperforms ReLU & variants like SELU in many classic applications (like classification networks). From my personal experience I can confirm that sin(x) converges faster. The differences of the 3D scene reconstruction-results that they show in their paper-video are impressive though. Maybe it is worth trying sin(x) for generative models.

Great introduce! And I wonder what tool you use, the highlight and pen line is so comfortable

Nice video! Yes, start monetizing, your videos are priceless :)

I think the broader impact statement is there because this is a NeurIPS submission and they require an impact statement this year.

Great video!

44:04 I think a more intuitive way to formulate this point would be to say that we want the boundary to be a level set for the neural representation, since the signed distance is supposed to be zero everywhere on that boundary

Yannic, are you familiar with Stephane Mallat's work in physically based neural networks? He talks a lot about using wavelet functions to improve the function approximations of neural networks. Sirens' use of sine activation functions reminded me of that.

Thank you very much

After watching the first few minutes of your Gradient Origin Networks video, I am realizing something: SIREN seems alot like a static shader. That is, a shader (ie shadertoy.com) is defined by input coordinates (x,y) and output colors. The mapping is typically non-linear. One major difference is, for shaders, we often use a frame count (time) as an input as well. However, it's perfectly possible to craft static shaders.

Great paper. I’m glad, the Broader Impact reads like a persiflage of broader impact. We usually call such text „bullshit bingo“. Simply drop it, people!

Is approximating the gradient for the real image as simple as (2/imageSize)(a-b), where a and b are the pixel values to the left and right, and the same for above and below (assuming image function maps take inputs in range 0 to 1)? Would be really cool to see a neural SDF used in raymarching for games

Yannic, that function has a similar structure to the mathematics used in physics a function F that depends on coordinates x, function phi, and the gradient of phi w.r.t. x and higher order derivatives. Look at Laplace's equation, the Lagrangian in classical mechanics, and other functionals (oversimplified: functionals map functions to numbers).

The fastest gun in the west. I don't know if it's because I asked for this paper but as always, thank you.

Thank you!

13:51 LOL! I like your sense of humor 😉

what about using the Laplace transform instead of Fourier? It's more generic

In the "Poisson image editing section", I do not understand what the training dataset is. But, you said that the data to fit is always (x,y) -> (r,g,b) (or here (x, y) -> (luminosity) since it's gray scale). But in this case we don't know the composite image since we are trying to generate it. So what does phi(x) produce ?

Was just viewing the video from the authors via a Hinton tweet and Yannic already has a video? :o

Awesome

It seems the initialisation with uniform distribution, also multiply 30 in the sin function are so crucial. If you tried the code, and change the number to i.e (5,6,7 etc), the results just mess up. Does anybody know why the 30 is the good choice, a mysterious?

Few mistakes, around 12:20, its not the Laplacian, since the symbol is the upside down triangle ^2, this is delta, the difference between the ground truth and the result. Also 6:36, subpixel values for images can be done using linear intepolation or other interpolation between the values of the pixels surrounding the subpixel region.

At 33:35 what do you mean by "the more differentiable you make it"? I thought functions were either differentiable or not.

Can anybody explain the initialization? I read the paper, but I'm missing something. I get scaling the later weights by omega_0, but why do we multiply it back into the weights in the forward pass? I built one in Julia as an MNIST classifier, and its learning is incredibly fast and stable *only if I don't multiply by omega_0 in the forward pass*

Great video! Isn't the proposed initialization just Uniform Xavier?

Interesting tool

21:43 what about using an exponential function?

Is it better than a Fourrier transform, though ? Because it seems to me that it does something similar.

i am not sure if i understand correctly how they match the gradient of the network, you compute sobel of the output generated over the entire "dataset" (image as x,y)? or you just compute the true gradient dphi/dx, dphi/dy using autodiff? and if you input that into the loss and run backward does not that mean that you need to compute derivative over derivative?

Can you use this to resample a signal like an image to get many similar representations of the same image?

THis is interesting because if errors can be backpropogated as phase shifts rather than magnitude changes you can have as many layers as you like and the error doesn't decay away

Weird I never understood the math in these papers because I thought it would be way too abstract, but then I realize it's not abstract at all!

How does it perform if we use it for upsampling?

I think this is somehow limited by the amount of sine operations, as using it as an activation function uses it a lot less than let's say projecting every input to the fully connected layer with a different sine. The computational cost of the sine can be offset by letting the GPU handle a lot of sines simultaneously. Also, this requires a specific learning rate to work well. Too little and this converges to a flat surface, too large and this converges to noise, therefore I think it is beneficial to use periodic learning rate schedulers here. AdamW with amsgrad also seem to work better than vanilla Adam here. I've tried an MNIST classifier with this, didn't work that well...

For anyone wondering about RBF kernels...An RBF kernel doesn't really do the infinite dimensional stuff right away. Really, all an RBF (radial basis function; it's a hint!) does is output the distance of two points according to some metric. The normal one people mean when they say RBF is the Gaussian distance. I don't remember the exact form off the top of my head, but it's something like f(x) = [e^ -(x - x_0)] / sigma, where sigma is some scale parameter. This function will output a 1 (if the two points x and x_0 are equal), and will decay to 0 at an exponential rate. In the context of a neural network, the comparison value can be a learned parameter (with the logit obviously being the other input), but the literature I remember reading would normally set these values randomly at init and leave it there. Hope that sates somebody's curiosity! Postscript: Whenever you hear the words "kernel" and "infinite-dimensional" in the same sentence, you're in the land of what's called an RKHS, and the kernel they're refering to is a very specific kind of distance matrix. That sort of stuff is relevant for SVM theory, but kind of goes beyond the scope of this comment. To give a brief sketch, if you do something like linear regression on that kernel matrix, you're implicitly searching through a family of (potentially nonlinear) functions defined by the distance functions. So, nonlinear function approximation but the whole operation is strictly convex and solvable closed-form. People often get mixed up with the "infinite dimensional" function space of the RKHS, and the "project the data to a higher dimension" quote which is also associated with SVMs.

Has anyone tried making a generative model out of this yet?

in the past I fell victim of thinking "sin" could be the holy grail of activation functions but gradient descent would play with that function like a roller coaster, I even fell victim of trying to use quaternions but that is another failed story 😆

pogchamp

I saw a video which was about the fact that you can represent every picture through sine-waves (I forgot how and why). Is this somehow related? (Sorry if this is answered later in this video, I’m writing the comment at minute 17)

a very important nerf paper

Thank you a lot for the content, Yannic. I wish you could discuss through your narrative this 2 papers: arxiv.org/abs/1711.10561 and arxiv.org/abs/1711.10566 , as it would be amazing to have a complex analysis in such application of Deep Learning methods (apart from the fact that modern and didactic narratives about Physics Informed Neural Networks are not easy to find in video content with an insiders approach). Of course there is content, but maybe approaches such as yours could provide more achievable understanding for those who are starting in this methodology. Best regards!

How to use SIREN for image classification?

Without reading the paper..it seems that they are simply using a nn to learn a fourie representation of the image seen a sampled field+gradients..

Kinda reminds me of grid cells in the brain.

Why FCC and not CNN, since gradients are local and not global?

I believe you misspoke at about 33:08 when you said "over the entire image". You should have said "over all the images", right?

2:18 Lol, they must have come from physics. That's the general form of an "action".

The initialisation proposed looks like the default weight initialisation in tensorflow: github.com/tensorflow/tensorflow/blob/6bfbcf31dce9a59acfcad51d905894b082989012/tensorflow/python/ops/init_ops.py#L527

Here as an example of the kind of things that SIRENs allow you to do: github.com/juansensio/nangs

Thank you for clearly explaining this paper. It's one that I wanted to dig into but found the math off-putting. The authors should have done a better job of communicating this simple idea.

Such a shame the authors were probably more worried about seeming clever and professional than making their writing approachable 😔 that introduction says it all! You explain it so well though! Thank you so much!

does any one else speed up his videos cause he speaks so slow?