Hi Josh, Love your content. Has helped me to learn a lot & grow. You are doing an awesome work. Please continue to do so. Wanted to support you but unfortunately your Paypal link seems to be dysfunctional. Please update it.

Awesome!!! I'm glad you're enjoying my videos. :)

@@statquest Enjoying them as well as hoping they'll help me ace my Data Mining exam tomorrow 😂

@@hayagreevansriram326 Good luck tomorrow and let me know how it goes.

what do you think about the tuition colleges charge?

Same 😂

You’re welcome!!! I’m glad you like my videos so much. I have a lot of fun putting them together. :)

ps, I have 3 more videos on logistic regression coming out in July. :)

Thank you very much!!! I'm so glad that my videos have helped you and good luck with your PhD! BAM! :)

Thank you!

:)

bam!

Thank you! :)

Hooray!!! I'm so glad you can watch and learn from my videos. I'm very passionate about helping everyone learn.

Thank you very much! :)

Thank you! :)

Thank you! :)

Thank you!

Thanks, will do!

Thanks!

Glad you enjoy it!

You’re welcome! I’m glad you like the videos! I have 3 more on Logistic Regression coming out in July. :)

Thank you! :)

Thanks! :)

Glad you like them!

Glad you like them!

Awesome, thank you!

Thank you!

Thank you so much 😀

I've got the slides all done for it - so it's ready to go. The bummer is that I'm traveling a lot in the next two weeks so it won't be out for a while... unless I can somehow make it happen this Friday.... I'll see what I can do.

Thank you! :)

Gracias!!! :)

Thank you! :)

bam! :)

Triple bam! :)

Very true.

Thanks! :)

Muchas gracias!

bam!

Here is what I think:There are 5 mice obese and 4 not obese,totally 9 mice.Without considering for weight,the probability of a mouse being obese is 5/9=0.56.If we map the probability(5/9) to the right figure,that is log(0.56 / 1-0.56)=log(5/4)=0.22.

Thank you very much for supporting StatQuest!!! TRIPLE BAM!!! :)

Those are all on the to-do list... I'll get to them one day! I hope that day is soon! :)

bam! :)

Awesome! I'm glad you like my videos! :)

@@statquest Hi Josh, Can you please come up with Image processing algorithms or NN models as well

@@bhargavpotluri5147 I'm working on the NN videos.

@@statquest Wow, Thanks Josh :)

Yes and presumably. At least to me, it seems reasonable that you could have a model with a lot of parameters, where each parameter only contributes a tiny amount to the overall fit - so in the big picture, you have a predictive model, but the individual parameters don't have much of an effect.

You are welcome!!! I'm always so happy to hear how much you like the videos! :)

StatQuest with Josh Starmer this is awesome channel for machine learning... Hope next exercise is in R

I've got one more video, on the saturated model and deviance statistics, and then we put everything together with "Logistic Regression in R".

StatQuest with Josh Starmer woowwww.... We love statquest videos

Hooray!

Hi! I think Josh would give you a much better explanation, but i'll try :) Chi-square distributions come in different degrees of freedom. In the case of logistic regression, the degrees of freedom is 1 (2 parameters in the logistic regression (y-intercept and slope), and 1 parameter for the overall probability (y-intercept, just a horizontal line), thus 2-1=1). Thus, you need to use the Chi-square distribution with 1 degree of freedom. *The p-value is given by the area under the 1-DoF chi-square distribution (integral) from [ 2*(LL(fit) - LL(overall Probability))] to infinity!* In the first example: Since, by definition, the area under a statistical distribution curve is always 1, and [ 2*(LL(fit) - LL(overall Probability))] = 0, the integral is over the entire distribution (chi-square support (domain) is from 0 to +infty), thus 1. Therefore, the p-value = 1. In the second example: [ 2*(LL(fit) - LL(overall Probability))] = 4.82. The integral of the 1-DoF chi-square distribution from 4.82 to +infinity is 0.03. Thus, the p-value = 0.03, which is statistically significant in most situations, since it is less than 0.05. Hope this helps!

The formula to calculate the p-value from the test statistic in logistic regression is based on the principles of hypothesis testing and the properties of the standard normal distribution. Here's a step-by-step explanation of how the formula is derived: 1. **Null Hypothesis and Test Statistic**: In hypothesis testing, you start with a null hypothesis (\(H_0\)) that assumes no effect (e.g., the coefficient is zero). The test statistic \(z\) is calculated to measure how far the estimated coefficient (\(\hat{\beta}\)) is from the null hypothesis value (usually zero). The formula for the test statistic is: \[ z = \frac{\hat{\beta}}{SE(\hat{\beta})} \] 2. **Standard Normal Distribution**: Under the null hypothesis, the test statistic \(z\) follows a standard normal distribution (\(N(0, 1)\)). This is a fundamental property of hypothesis testing. 3. **Two-Tailed Test**: Since you're interested in whether the coefficient is significantly different from zero (two-tailed test), you want to calculate the probability of observing a test statistic as extreme as \(z\) in either tail of the standard normal distribution. 4. **Cumulative Distribution Function (CDF)**: The cumulative distribution function (\(\Phi(z)\)) of the standard normal distribution gives you the probability that a standard normal random variable is less than or equal to \(z\). In mathematical notation: \(\Phi(z) = P(Z \leq z)\). 5. **Probability Calculation**: The p-value is the probability of observing a test statistic as extreme as \(z\) in both tails of the distribution. Since the standard normal distribution is symmetric, you can calculate the probability of observing a test statistic as extreme as \(z\) in one tail and then multiply it by 2 to account for both tails: \[ p = 2 \cdot (1 - \Phi(|z|)) \] Here, \(|z|\) ensures that the value inside the cumulative distribution function is positive. In summary, the formula \(p = 2 \cdot (1 - \Phi(|z|))\) calculates the p-value by determining the probability of observing a test statistic as extreme as \(z\) in both tails of the standard normal distribution. If this probability is small (i.e., the p-value is small), you have evidence to reject the null hypothesis and conclude that the coefficient is statistically significant.

You can also use a confusion matrix and associated metrics (like sensitivity and specificity and ROC). For details, see: kzfaq.info/get/bejne/gcqjo5mk1NelZ6M.html kzfaq.info/get/bejne/rLZgacR-1OCYpmw.html kzfaq.info/get/bejne/atCCdbV1x7DJnYE.html

BAM! :)

I believe that is correct.

Thank you! :)

Ooops. I didn't do a good job rounding! The true value is 0.55555555....repeating, which rounds to 0.56. However, I messed up on the next slide and just put 0.55. Sorry for the confusion.

The mean of the thing we want to predict is thought of as the worst fitting line because that is what we would fit if we had nothing to predict (no x-axis value).

Good luck! :)

@@statquest Thanks! (: and while we're at it, can I ask what program you use to make your graphics?

@@jaegermeistersfriend I draw most things by hand in Keynote. Other graphs are created in R.

For linear regression, R^2 can never go below 0. This is because your model can never be worse than the base line model. However, in other settings it is possible to have your model fit worse than the base line model.

@@statquest thanks alot :) !

You are correct. The reason I organized the videos the way I did was to follow the output that R gives you when you do Logistic Regression. The first thing it prints out are the coefficients, and the last thing it prints out is the R^squared. So I was just going from the top and working my way down the output.

@@statquest Thank you 😊. Best regards from Germany

@@ml6352 Thanks! :)

In this case we are using a Chi-Square distribution to determine a p-value, but we are not performing a standard Chi-Squared test. This is similar to how a z-test is based on the normal distribution, but the normal distribution is used for a lot more things than just the z-test.

Thanks! There is no way to calculate a "normal" correlation for logistic regression because of the infinite distance between the data and the log(odds) linear fit.

Unfortunately, deriving that equation would probably take a whole video.

This is your best yet.

@@statquest Thank you! If you ever want to hear a pun on a particular topic, just let me know.

Thank you!

BAM! That's a good question and I'll keep it in mind for a future video.

That seems correct. I rounded the value to 0.03.

likelihood ratio test converge in distribution to chi-square asymptotically

en.wikipedia.org/wiki/Wilks%27_theorem

To learn more about how we fit lines and squiggles to data in logistic regression, see: kzfaq.info/get/bejne/eMx7lNGdlse3d2Q.html

These are great questions. I have a bunch of videos that talk about R-squared and P-values. Check out: kzfaq.info/get/bejne/a8BpgqyKpp6slWw.html kzfaq.info/get/bejne/aKeBftColprReIE.html kzfaq.info/get/bejne/pNFidrR6udPDlaM.html

StatQuest with Josh Starmer thanks :)

yep

You can have a terrible R-squared value and still have a small p-value if you have a lot of data. However, if the R^2 value is bad, then, even with a significant p-value, your model may not be worth very much.

I really appreciate the time you take to answer questions !! Thanks, I already have it clearer

The degrees of freedom is the difference in the number of parameters between the fitted model and the overall probability (which typically only has 1 parameter). So if the fitted model has 3 parameters, then DF = 3 - 1 = 2. People often use the Akaike information criterion (AIC) to choose the best model. For details, see: en.wikipedia.org/wiki/Akaike_information_criterion

There is a lot in this video, so can you tell me what time point (minute and seconds) is confusing you?

@@statquest Check the video at 5.18(LLfit) , 6.51 (overall prob) and 8.41 (LLfit)

​@@TheRamnath007 OK, so in this video, I use three different datasets to demonstrate how to calculate the R^2 value. For the first dataset weight is correlated with obesity, and I calculate LL(fit) = -3.77 and LL(overall) = -6.18. Then I calculate the R^2 = 0.39 at 7:25 . Thus, the R^2 confirms that weight is correlated with obesity. After that first example, I then create a new dataset that does not have a correlation between weight and obesity. I then calculate LL(fit) and LL(overall) for the new dataset. In this case, both LL(fit) and LL(overall) = -6.18. I then plug this number into the formula for R^2 and get R^2 = 0 (see 9:22 ). So the R^2 confirms that this new dataset is not correlated. After the second example, I then create a new dataset where there is tons of correlation between weight and obesity. I then calculate LL(fit) = 0 and LL(overall) = -6.18 for this new dataset. Lastly, I calculate R^2 and get 1 (see 11:26 ). My guess is that the thing that is confusing is that the number -6.18 keeps coming up in each example. This is because each made up dataset for the three examples has 4 obese mice and 5 mice that are not-obese. This means that the LL(overall) will be -6.18 in all three examples. However, it also means that LL(fit) = -6.18 in the second example because the data are not correlated and the best fit is a horizontal line at the log(odds), just like LL(overall). Does this make sense?

Your question makes me suspect that you skipped watching Part 2 in this series. Part 2 explains the role that maximum likelihood plays in logistic regression. Hint, maximum likelihood does something completely different from Wald's test. For more details, see: kzfaq.info/get/bejne/eMx7lNGdlse3d2Q.html

@@statquest I went back and rewatched the video. Thanks man!

Thanks both! I have the same qn here: 1) does it mean with one x-variable, the p value of the coefficient (part 1) and p value of the model (part 3) are the same? 2) and if there are more than 1 x-variable, p value of the model (part 3) means if the combined effects of the x-variables are stats sig? Thank you!

You would probably compare accuracy or some other metric used for classification.

We can use theory to derive the distribution. This is pretty advanced stuff (I did it once a long time ago), so usually we just look it up when needed rather than derive it from scratch.

Thanks for the reply Josh, Can you give an example of the keywords we may use to lookup the corresponding distribution? Like i know for testing the coefficients of a linear regression model we use the T-test, but in time-series data, we use the ADF test for checking stationarity. Here the value for the T statistic of a coefficient is to see if it is higher than a certain threshold and based on that we reject or fail to reject the hypothesis. The problem is the threshold that is set here is higher than the one you get if you test it with a normal T-test(I don't know the exact distribution but it follows another distribution). So how may i go about finding the distribution for testing the statistic in the above case? @@statquest

@@rishavdhariwal4782 To be honest, I'm not sure I understand your question. However, if you are interested in why these specific statistics have a chi-squared distribution, you can look at how Mcfadden's R-squared is derived.

It depends on what tool you use. In R, we calculate it with: 1 - pchisq(2*(ll.proposed - ll.null), df=10).

I answer your question in this video: kzfaq.info/get/bejne/eMx7lNGdlse3d2Q.html

@@statquest Thank you so much!!! I will look into that :)

The ordering of the red and blue dots in the left figure at 11:39 is based on the ordering that is introduced at 7:44, when weight has no relationship with obesity.

Plug the x-axis coordinate for the data into the equation for the line to find the corresponding y-axis coordinate on the line.

If we only had 2 data points, then we could get the squiggle or line to fit them perfectly, resulting in a high r-squared. However, any too random points will result in a perfect fit (just connect the two points), so the p-value will be terrible. Thus, one thing the p-value can tell us is how much data supports the r-squared value.

Maybe! I don't know off the top of my head. However, the log is often used to avoid underflow errors, so if you don't have too much data, it might work without the log.

The p-value only tells us if the relationship is significantly different from random noise. The r-squared value tells us the strength of the relationship. How "strong" is "strong" depends on the field or area being studied.

So the relationship is significantly different from random noise as the p value is so small. Here, I have one thing to ask, what is random noise? Though, the relationship is significantly different from random noise, the strength of the relationship is not quite good as we obtain only 0.39. Do I interpret correct?

@@hang1445 Random Noise is just "random stuff", things that are not related. And if the p-value small, then you can conclude that your relationship is significantly different from random stuff that is not related (and that suggests it represents a true relationship). As for the R-squared value. Depending on the field, 0.39 may be considered a "weak" relationship, other fields might consider it "strong". It depends on the type of data you are working with.

Well explained! Thanks ：）

Mittlbock and Schemper (1996) “Explained variation in logistic regression.” discuss *12* different R-squared formulas for Logistic Regression: citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.477.3328&rep=rep1&type=pdf

stats.idre.ucla.edu/other/mult-pkg/faq/general/faq-what-are-pseudo-r-squareds/

@@janinajochim1843 Thanks!

* I should have also added that I have multiple IVs in my model and 3-4 of them are significant. I wonder to what extent I can interpret them as important predictors regardless of high R-square

0.03 seems pretty small to me, and thus, despite the significance of the independent variables, they do not give you very much information about what is really going on with what you are trying to model.

:C

@@statquest The promised funny story: Recently overheard two of my fellow students having the following exchange: Student 1: I am not sure what to do over the summer Student 2: Mh ... Student 1: Was thinking about doing some modelling Student 2: Oh cool. What like for magazines? Student 1: What? Student 2: You didn't mean on catwalks, right? Student 1: What? I meant with my mice- data!

@@janinajochim1843 That is great!!! Very funny. I got a big laugh out of that. :)

It's actually possible to use the sum of the squared residuals, but it doesn't always work as well. To learn more see: kzfaq.info/get/bejne/bKeihtykmtescYk.html (NOTE: To understand what is going on, just replace "cross entropy" with "log(odds)")

This is a good question! Talk about this in "Part 1" and "Part 2" of this series: kzfaq.info/get/bejne/rLRllrF_l5Osh3k.html and kzfaq.info/get/bejne/eMx7lNGdlse3d2Q.html

Also, once you understand how parameters are estimated for Logistic Regression, it's easy to see that it works just like like regular multiple regression when you have more variables predicting whatever it is you're predicting.

Thanks! one more (stupid) question. When you convert the probability of obesity to log odds of obesity, the x axis- weight is also converted to log weight? If not then what is the x axis in log odds graph?

Not a stupid question at all. The x-axis stays the same. The parameter (slope) tells you that for every one unit of weight (the x-axis in the original units), you increase (or decrease, depending on the angle of the slope) the log(odds) of obesity (you either go up or down along the y-axis, which is now now in log(odds) units).

Yeah, it's possible to have negative R-squared values. However, typically with Logistic Regression we compare "nested models". In other words, one model is the "simple model" and the other, the "fancy model", contains all of the variables in the "simple model" plus others. When this is the case for Logistic Regression, the fancy model can not do worse than the simple model because otherwise the parameters for the new variables would be zero (or not significantly different from zero), and thus, in the worst case, the simple model = the fancy model, which results in an R^2 = 0. However, when you don't use nested models, or you are working with something other than logistic regression, you can get negative values.

LL(fit) is the log-likelihood of the fitted squiggle. We can use that as input to an algorithm that can maximize the likelihood. To learn more about maximum likelihood, see: kzfaq.info/get/bejne/jsugi9ednr-vp5c.html

I believe it is the log( likelihood of the data given the line)

@@statquest Thank you for the answer. I thought we were trying to find optimum parameters of the linear equation which would yield in the best sigmoid. Thus finding the MLE of the sigmoid (hence parameters) given the data. I'll watch your video on the MLE again then. I am still confused with the difference between the two.

@@thomasamet5853 Regardless of how you phrase it, the likelihoods are the y-axis coordinates on the squiggle for each data point.

That helps a lot. Thank you again for taking the time to answer and for the amazing content :)

No answer 🙁‼️

:)

:)

You are correct. That's a typo. Sorry for the confusion.

Umm... this whole video is intended to answer your question. Is there a specific time point that is confusing?

@@statquest I was reading a article on google and it said that R square in logistic regression is not used to tell the explained variance but rather the improvement in model likelihood over null. I wasn't able to relate with this video. Just wondering what actually it reresents in logistic regression.

@@anshulsaini5401 I guess I don't understand the question since this video, and the article on google, say that the R^2 is the improvement in model likelihood over the null.

I'll keep that in mind.

Not yet.

@@statquest I can not find its content on internet I have been beated by this statistic ... most of academics usually teach about binomial one

@@henri9289 Noted

​@@statquest I have searched for content on both internet and library, I have only found binomial's equations... Iam looking for multinomial in order to write the equations on my dissertation

Again, this is just poor rounding on my behalf. The true value is 0.4452208, which rounds to 0.45.

:)

We use maximum likelihood and gradient descent. For an example, see: kzfaq.info/get/bejne/eMx7lNGdlse3d2Q.html and kzfaq.info/get/bejne/qaqmZ8ll2Ji3cmw.html

Sir ,can't we calculate the slop and intercept to logistic regression without using gradient decent?

@@narendrasompalli5536 There is not an analytical solution, so you have to use some iterative method. Gradient Descent is a popular method, but there are others you could use.

Sir i said that we can calculate the best slop in linear regression by using sum((x-x bar) (y-y bar)) /sum(x-x bar) ^2

Like that can't we calculate in logistic regression!? Sir

In log() space, you want to have a linear response.

@@statquest But this linearity must be checked only if the predictor is continues right? Is there anything to check for categorical variables? Also thanks for responding.