Not sure if I understand you correctly, but your suggestion is basically to only look at one of the triangles of the matrix without the diagonal. Is that correct? In that case you would have to divide by n(n-1) only and not by 2 to get the same value.

@@statsandscience I'll try and elaborate more: The sum of all the squares is 812. Divided by n^2 (because that's the number of squares we're considering), that's 16.571… Divided by 2, that's 8.28571… And finally, with Bessel's correction, it's 9.666… The sum of half of the squares is 406. Divided by n(n-1)/2 (because that's the number of squares we're considering), that's 19.333… So we still have to divide by 2 to get 9.666… I'm not saying the formula is wrong, just the explanation ("each value is in here twice"). In both cases, you have to divide by the number of squares AND by 2. So the division by 2 is not explained by counting the squares twice.

@statsandscience But the number of squares in one of the triangles is 1+2+3+4+5+6+7=21, not 42.

@@cannot-handle-handles Yes, I deleted my comment because I noticed my mistake prior to your answer. It makes sense what you say, I did not do this test before. Do you have a good intuitive explanation for the 2?

Is it because you basically calculate the means of all pairs of points?

Thank you!

That was super helpful, thanks! And extra thanks for providing all the correct time stamps!

Thanks, glad you liked it!

Great, thank you for the comment!

Thank you, I really appreciate it!

It's honestly just powerpoint and I won't recommend it for standard use, I am sure there are better options out there...

I brushed over this a bit because it was not the focus here. Intuitively, it makes sense I think that the variance of the sample mean must be smaller than the population variance and that this depends on n because as I explained, there is no way to get the most extreme observed values as means, and the mean will always become "less extreme" in comparison the higher n is. However, I don't know an intuitive explanation for the exact formula, but the reasoning goes like this: You try to calculate the variance of the sample mean, that is, the sum of the observations divided by n, like so: Var(obs1+obs2+obs3.../n). You can rewrite this to Var((1/n)*obs1 + (1/n)*obs2 + (1/n)*obs3...). A linear combination like this has a variance equal to the sum of whatever the factor is squared (in this case 1/n^2) times the variance of the individual components: (1/n^2)*Var(obs1) + (1/n^2)*Var(obs2)... When you then assume identical variances for the observations, this equals (1/n^2*)n*Var(obs) which is Var(obs)/n. You can find that a bit nicer formatted also here: online.stat.psu.edu/stat414/lesson/24/24.4 Hope this helps, thank you for the comment!

I was wondering too!

I am not sure if I understand correctly but I think there might be something to it framing the mean as containing 1/n parts of the information within the sample... Would you say you were right after watching?

Hey, thank you so much to take the time to give detailed feedback. 1) I am actually using such a microphone, but maybe it wasn't well positioned? I will check that. 2) thanks, I will try! It is not that I want to shorten videos, I am just used to talk fast I guess...

​@@statsandscience good luck with your work and thank you from the bottom of my heart, I really do understand why we divide by n-1 now :D .

To make an even clearer statement of the problem...suppose my sample size is always 499. Now suppose that the actual population is either 500 or 1000. So that's 2 cases in total. According to the n-1 rule, I should apply the same correction (499/(499-1)) to 499 samples in a 500 population as I should apply for 499 samples in a 1000 population. That doesn't seem to be the best we can do if we know the actual population size, since I should not need to correct as hard when sample size is very close to population size. So is the n-1 rule designed only for the case where one does not know the population size? If we do know the population size, can we do better? Using what formula?

Sorry that I did not come around earlier to answer this question. You put a lot of effort into this and I hope you still benefit from an answer! When you take intro stats classes, a quite basic assumption that lurks basically everywhere is that the population you are dealing with is infinite. Of course, this assumption is also basically always wrong. Usually that does not matter though, as populations are usually "big enough", so that wrong estimates of the actual population size do not influence our outcomes to a degree we would care about. The same is true here in this formula: It is not the "average" correction factor for all possible samples, but the one for an infinite population - again, it usually does not matter what the actual size is, except when the sample size comes close to the population size. Now you correctly identified that this can cause problems because in this case you actually know a lot more than what the formula is giving you credit for. What people came up with for this case it the Finite Population Correction (FPC) - I would advise to just google it and look for yourself as the space here is quite limited (of course you can also ask follow-up questions about that here if you like!). However, in a nutshell this correction does what you pointed out - it prevents that you correct "to hard".

@@statsandscience Thank you, it is a very useful answer. I didn't know about that assumption, and so when your examples had a population size of 7, I was extra confused. Thanks for clearing it up. That makes it clearer why the correction should be greater when the sample size is smaller. Maybe mention that assumption in your video description to help others in the same boat as me?

Thanks!

They would still occur, wouldn't they? Because the margins of the table are identical either way, so there would be zeros on the diagonal. Sampling without replacement is also a separate issue, as for instance discussed here: stats.stackexchange.com/questions/70124/unbiased-estimator-of-variance-for-samples-without-replacement

@@statsandscience Thank you for your reply. Zeros occur when we substract each data point from itself and this doesn't happen in case of sampling without replacement.

You will probably find the general concept in any applied statistics textbook. As I said it is a basic step from descriptive statistics where you only draw conclusions about a particular sample to inferential statistics where you use a sample to draw conclusions about a bigger population and that is basically what is always needed and taught in applied statistics. The issue is that those books tend to be shallow in that regard and other books with more detail might only be helpful with a serious understanding of the math behind it. Which is why I made the video to bridge between these two. Let me know if that was what you had in mind!

Well, it is, but you can sort of getting around that in an explanation like this one. If you are interested, feel free to watch my video on degrees of freedom. :)

Thank you! Yes, I do have that and I always wanted to make proper subtitles but just did not get to it yet. KZfaq auto generates subtitles as you probably know but I don't really like them. I will try to look into that soon and let you know.

@@statsandscience Thank you for your reply. I will wait for this precious script.

@@user-ws5sq8fm4k English subtitles are up now! I hope you will find them helpful

The main problem is that you don't know that. Remember that we do all this with samples because we do not have access to the population - and this is a problem that happens because of sampling, but not when you can use the population values. Imagine a student who goes to the school cafeteria every day, and who knows that the staff tends to hand out portions that are too small most days. So they ask for something extra every day (and receive it). This will move the portion size to the optimum most days, but on days where the portion size was correct in the first place or even greater, the request will make it worse. However, this is still better because on most days the size is too small, so the average will be closer to the optimum. Does that help?

@@statsandscience Thank you for your response. It definitely helps but I still have the question of how you would know that the data values from a sample are too small. You cannot infer that it's too large, but why can you infer that it's too small? Shouldn't it go both ways? Maybe naturally, samples tend to gravitate around smaller data values as with the portion size example you gave? If that's the case then it does actually make sense since you'd typically not want to exceed the normal portion size so you don't run out (and this idea of scarcity can be applied to any other examples).

@@se0271 you indeed don't know that for a particular value. It can be too big or too small. It is just more likely that it is smaller. I'm afraid that when I go into more details I would just repeat what I said in the video but when you have specific questions I would be happy to help!

@@statsandscience I see, I appreciate the explanation- thank you!

Thanks, glad you liked it!

Thanks! I will try to improve speaking next time!

Yes, that sounds great! I think you should now be able to just download it after I have added the subtitles.