As a math student who is interested in information theory and neural networks, i discovered this gem of a lecture series when i was looking for videos to fall asleep to! In fact i've finished the lectures when i was sleeping :D Now I decided to start it properly and just finished watching this lecture and taking notes. I would love to send David a mail when i finish the course. Thanks for leaving this behind my man, rest in peace.
@mohamedrabie4663 Жыл бұрын
RIP prof David, you were, and still a great inspiration to us
@eddiehazel12597 ай бұрын
ah man sad to hear 😔
@prajwolgyawali67704 жыл бұрын
Fundamental Problem: Reliable communication over unreliable channel Binary Symmetric Channel: 9:35 Disk Drive Problem: 11:30 Redundancy: 23:00 Repetition Code: 24:35 Decoding is inference Inverse Probability: 31:33 Forward Probability: 40:30 Hamming Code: 47:10 (kzfaq.info/get/bejne/m8-odqqiydKrqIU.html%29) Capacity of channel: 58:10 Shannon Noisy Channel Coding Theorem: 58:45 The Weighing Problem 1:00:50
@wampwamp14582 жыл бұрын
thank you :)
@fjs11112 жыл бұрын
Unreliable channel = Unreliable information
@lesliefontenelle72248 жыл бұрын
I am not involved in information technology but this lecturer is making a difficult subject like information theory look so easy. You really must see this...
@shellingf2 жыл бұрын
kinda boring though
@the_anuragsrivastava3 жыл бұрын
One of the best lecture of " information theory and coding " I have ever seen....love from India 🇮🇳🇮🇳🇮🇳🇮🇳
@linlinzhao90855 жыл бұрын
Dr. Mackay is a great explainer. Anyone interested in machine learning and Bayesian statistics can also read his doctoral thesis.
@iwtwb88 жыл бұрын
I was saddened to read that David MacKay passed away earlier this year.
@JTMoustache8 жыл бұрын
what a great teacher.. he will live on :)
@wunanzeng70517 жыл бұрын
He was a very great teacher! Very Articulate!
@yousify7 жыл бұрын
I was shocked when I read your comment. I'm following his lecture and his book on this course, it is a sad news.
@jimmylovesyouall7 жыл бұрын
what a great teacher.. he will live on
@bayesianlee64476 жыл бұрын
RIP For great teacher of human beings. His passion and endeavor for tech would remain for descendant
@JerryFrenchJr7 жыл бұрын
How am I just now discovering this lecture series??! This is awesome!
@JakobFoerster7 жыл бұрын
better late than never! glad you are finding it useful
@anantk26755 жыл бұрын
i got it now bro, i am latter than ya ; )
@siweiliu99252 жыл бұрын
@@anantk2675 I'm later than you, hhhh
@trueDeen91111 ай бұрын
@@siweiliu9925 i am later than you
@dragonfly31397 жыл бұрын
Thank you for these great lectures your memories will live on ... RIP
@IrfanAli-jl7vb5 жыл бұрын
Thank you, Thank you, Thank you for sharing these excellent video lectures. Dr Mackay is amazing at teaching complicated topics. These lectures are great supplement to his excellent book on information theory which has so many excellent plots and graphs that enables one to visualize information theory. Information theory comes alive in pictures. Thank you for sharing these.
@AvindraGoolcharan4 жыл бұрын
This may be the longest blackboard I've ever seen
@hyperduality28384 жыл бұрын
Repetition (redundancy) is dual to variation -- music. Certainty is dual to uncertainty -- the Heisenberg certainty/uncertainty principle. Syntropy (prediction) is dual to increasing entropy -- the 4th law of thermodynamics. Randomness (entropy) is dual to order (predictability) -- "Always two there are" -- Yoda. Teleological physics (syntropy) is dual to non teleological physics. Duality: two sides of the same coin.
@a_user_from_earth8 ай бұрын
what an amazing lecture and great teacher. May you rest in peace. You will not be forgotten.
@baganatube7 жыл бұрын
I don't mind the slowness. With some decoding, my brain is receiving a cleaner signal with that extra redundancy.
@sadimanesadimane67464 жыл бұрын
Bagana also does he have to write EVERYTHING
@monazy117 жыл бұрын
It was amazing, Finally I learned Shannon noisy channel theorem: there is exist an encoding and decoding system that could reach to the capacity of channel. so error correcting and detecting course is about to learn these encoding and decoding system. wow Amazing lots of thanks to the teacher
@AlexandriaRohn5 жыл бұрын
00:30 Information Theory invented by Claude Shannon to solve communication problems. Fundamental problem: Reliable communication over an unreliable channel. e.g. [Voice->(Air)->Ear], [Antenna->(Vacuum)->Mars Rover], [Self->(magnetized film)->Later Self] 04:00 Received signal is approximately equal to the transmitted signal because of added noise. 05:45 What solutions are there for having received and transmitted signal be the same? Either physical solutions or system solutions. 07:30 Source message -> [Encoder] -> Coded transmission -> [Channel (noise introduced)] -> Received message -> [Decoder] -> Best guess at original source 08:45 Encoder is some system that adds redundancy. Decoder makes use of this known system to try to infer both the source message and n.
@george51205 жыл бұрын
So nice to watch a video like this that does not have music.
@fireflystar53332 жыл бұрын
This is a great lecture. The design of case problem is really helpful here. Thanks for the lecture.
@kikirizki43184 жыл бұрын
Thankyou very much prof David MacKay, Your lecture helps me understand information theory. Btw I live in the country where toilet is not so common
@the_anuragsrivastava3 жыл бұрын
You are from where??
@TheNiro872 жыл бұрын
This is great, thank you! The lecture is far more entertaining than just reading the book.
@HeMan-tm8wl2 ай бұрын
whihc book is it?
@palfers15 жыл бұрын
Genius camera work
@rafaelespericueta7345 жыл бұрын
Indeed so. It's so frustrating and irritating when the camera focuses on the lecturer when you really want to look at the slide with plots and equations.
@oscarbergqvist49925 жыл бұрын
Thank you for a great lecture, looking forward to follow the rest and study the book!
@abhishekpal58718 жыл бұрын
this really helps. I really wanted to learn information theory. this video series is really easy to understand and awesome.
@reyazali27686 жыл бұрын
awesome example of teaching style
@leduran2 жыл бұрын
These lectures are great. Thanks for sharing.
@jedrekwrzosek69182 жыл бұрын
I looove the lectures! Thank you for the upload!
@dr.alaaal-ibadi86443 жыл бұрын
I like this channel. I'm already teaching this topic for student in Iraq. In Arabic.
@qeithwreid77454 жыл бұрын
This is fantastic I love it.
@Handelsbilanzdefizit8 жыл бұрын
I would use SUDOKUS for communication. Only few numbers have to be transmitted correctly, and the other numbers/information can be restored by the decoder :-D
@Hastur8765 жыл бұрын
The problem is that only the sudoku numbers that you transmit count as information: the rest of the numbers are constrained by having to follow the rules of Sudoku, and can't be any numbers you want, thus they can't be information. So you're still having to get 100% data transmission.
@RippleAnt9 жыл бұрын
Ah! thanks, was looking for a good series... this is just the one. A great cliffhanger at the end to be precise... ^_^
@GOODBOY-vt1cf2 жыл бұрын
15:52
@GOODBOY-vt1cf2 жыл бұрын
9:50
@gadepalliabhilash757514 күн бұрын
Could anyone explain how is the flip pobability is 10^-13 and and 1% failure is 10^-15. Video reference is at 22:10
@GOODBOY-vt1cf2 жыл бұрын
32:13
@spring74light9 жыл бұрын
Cheering, in these days of advanced educational gizmolgy, to see Professor MacKay making extensive and effective use of a stick of chalk and a polychromatically absorbent surfaced board. {Tyneside, England.]
@ciceroaraujo51835 жыл бұрын
Thank you professor
@ozgeozcelik89219 жыл бұрын
awesome! thanks for sharing
@papatyavanroode23292 жыл бұрын
3/8 3/8 2/8 cake and siblings That's the most equal way to share In the 3rd cake they are most close to equal 4/82/8 2/8 is far behind equal
@christopheguitton752311 ай бұрын
J'ai compris pourquoi répéter 3 fois la même information de suite à mes enfants n' était pas forcément efficace :-)
@deeplearningpartnership4 жыл бұрын
Thanks for these.
@oyindaowoeye46710 жыл бұрын
I wonder if he was making a joke when he said "Mr. Binomial" or if he actually meant to say "Bernoulli".
@vik24oct19916 жыл бұрын
he is being sarcastic i think , he finds it silly that distributions are named after people who discovered them while it would have been more logical if they were named after what they represent (which binomial distribution does).
@PierrePark4 жыл бұрын
@@vik24oct1991 he doesn't find it silly, he is himself just being silly, because so many things are named after someone he's joking and extending that to binomial
@Omar-th5vv2 жыл бұрын
I'm new to information theory. At 4:24 why is the received signal is "approximately" and not "equal" to the transmitted signal + noise? like what else is there other than the transmitted signal and the noise? Thanks for sharing such helpful lectures.
@caribbeansimmer78942 жыл бұрын
It gets extremely difficult trying to model everything that affects the signal, but it's relatively easy to add noise to the signal, but not just noise but an assumption that the noise is additive (hence the plus sign), white, and gaussian( normally distributed). As you would know from statistics, a normal distribution has nice properties for estimation. This stuff gets really complex but I guess you get the idea. Note, we can make other assumptions about the characteristics of noise
@nishantarya98 Жыл бұрын
Received = Transmitted + Noise makes a *lot* of simplifying assumptions! In the real channel, the noise might be multiplied, not added. The transmitted signal might go through weird transformations, like you might receive log(Transmitted), or it might be modeled as a filter which changes different parts of your Transmitted signal in different ways!
@cupteaUG9 жыл бұрын
Any idea about the final puzzle? My answer is 3. :-)
@deepkushagra4 жыл бұрын
at 46:50, what is rate? i guess it is (1/no of repetitions) but what does it mean in layman terms
@Rockyzach882 жыл бұрын
"People need 20 GB drives nowadays" Laughs in Call of Duty
@artmaknev373810 ай бұрын
After listening to this lecture, my IQ went up 20 points!
@terrythibodeau92652 жыл бұрын
I am curious to know how Shannon would have interpretted the internet as part of his theory... noise perhaps
@motikumar14426 жыл бұрын
Helpful lecture
@forheuristiclifeksh78365 ай бұрын
2:00
@ncckdr9 жыл бұрын
very great theory
@sahhaf1234 Жыл бұрын
towards 56:00 he uses the term "bit error" and "block error" but doesnt define them properly..
@derekcrone467910 жыл бұрын
is dis how u mak a plane
@AtharvSingh-vj1kp7 ай бұрын
I'm a bit curious about when these Lectures were recorded. Was it 2003??
@aritraroygosthipaty36625 жыл бұрын
38:19 I am unsure of the 1/2 in the denominator According to my calculations P(r=011) = (1-f)f^2+f(1-f)^2 This is done by the sum rule. The numerator should have 1/2 due to the P(s=1) term My final answer is P(s=1|r=001) = (1-f)/2 Could anybody help me with my concerns?
@dlisetteb3 жыл бұрын
When you add up the probability of r given s=1 and given s=0, you must include the probability of each of those events. It results as P(r=011) = P(r=011, s=0) + P(r=011, s=1) P(r=011) = P(r=011/s=0) * P(s=0) + P(r=011/s=1) * P(s=1) P(r=011) = (1-f)f^2 * 1/2 + f(1-f)^2 * 1/2
@brandnatkinson598110 жыл бұрын
K
@ridwanwase7444 Жыл бұрын
the link of getting free book is not working,can anybody tell from where i can get that free book? Thanks in advance
@driyagon3 жыл бұрын
can someone explain how to solve the homework problems?
@minglee51644 жыл бұрын
I have read the extraordinary book.
@dharmendrakamble62824 ай бұрын
🎉
@arthurk72707 жыл бұрын
I'm a bit confused. If we're trying to estimate the mean of the Binomial distribution, wouldn't we use the sigma/sqroot(n) formula for the +- bound? In other words, the mean +- the standard deviation of the estimator: 1000 +- 30/sqroot(10000) = 1000 +- 0.3? My statistics is a bit rusty.
@JakobFoerster7 жыл бұрын
Arthur K thanks for the comment. Which part of the lecture are you referring to?
@arthurk72707 жыл бұрын
Hi. I was referring to the discussion at 14:00.
@JakobFoerster7 жыл бұрын
Arthur K I believe you are confusing the standard deviation of the sample mean rate with the standard deviation of the mean total count. The standard deviation of the mean rate does indeed drop as 1 / (N)^0.5, while the standard deviation of the total count increases by (N)^0.5. Since we care about the total number of bits flipped, it's about the total count rather than the rate. You can see that multiplying the standard deviation of the rate, 1 / (N)^0.5, with the total number, N, results in a standard deviation of the total ~(N)^0.5. Please let me know if that clarifies.
@turkiym28 жыл бұрын
Was Mr. Bionomial a joke that fell on dead ears or was it a genuine confusion with Bernoulli?
@MarkChimes8 жыл бұрын
+omniflection I laughed when I heard it. I take it was just a very dry joke.
@yltfy8 жыл бұрын
+Mark Chimes Hahah. Me too. This is when I clicked the pause and check the comments...
8 жыл бұрын
+Mark Chimes Well hello Mr. Chimes :)
@dharmendrakamble62822 жыл бұрын
Information theory
@izzyr95905 жыл бұрын
im learning this at school... yet I'am here watching a lecture on KZfaq ... I dont know why ... I should pay attention in my class I guess.
@javatoday50025 жыл бұрын
you're definitely exercising inference
@keshavmittal10772 жыл бұрын
hey is there any pre-requisite of it
@vedantjhawar75533 жыл бұрын
Hi, I just wanted to ask what was meant when at 24:20 when he says that "1 is the same as 5, and if there was a 4, there would be a 0."
@MN-sc9qs3 жыл бұрын
1 and 5 are both ofd so assigned 1, and 4 is even so assigned 0.
@vedantjhawar75533 жыл бұрын
@@MN-sc9qs Thanks.
@mustafabagasrawala77906 жыл бұрын
I'm fairly new to this. What is a "flip" ?
@aritraroygosthipaty36625 жыл бұрын
a flip is to change the bit from a 1 to 0 or a 0 to a 1.
@carloslopez72044 жыл бұрын
What are the requirements to understand this lecture?
@AndreyAverkiev7 ай бұрын
As it says the only piece of Mathematics is binomial distribution 16:08
@stevealexander64257 жыл бұрын
Good lecture except for the highly distracting camera work.
@pauldacus45907 жыл бұрын
You know he's a jokster cuz at 1:00:13 the slide says his textbook weighs "~35 lbs".
@rarulis7 жыл бұрын
XD, that's the price. 35 british pounds.
@Hussain1Salman7 жыл бұрын
Thanks for the lectures. I am just curious about what happened to lecture 2. It says it was deleted.
@JakobFoerster7 жыл бұрын
Hi Hussain, thanks for catching this. I have reached out to youtube support to find out. Hopefully will be resolved soon.
@JakobFoerster7 жыл бұрын
Hi Hussain, Apparently there was a bug in the youtube system and they deleted it by accident. The video is back online now.
@MDAZHAR1005 жыл бұрын
Binomial is more related to probability topics. Bernoulli is about hydraulics.
@JakobFoerster5 жыл бұрын
en.wikipedia.org/wiki/Bernoulli_distribution
@circlesinthenight31417 жыл бұрын
rip david
@afarro4 жыл бұрын
I was able to achieve f=0.1 for this video with x1.5 speed ...
@GSSIMON18 жыл бұрын
the fact is the received signal is not identical to sent signal due to corruption and distortion in the signal, in a process , so how much of the the original signal is received ,what would be the measurement in what unit ,,,,,thats why i do drugs ,and dont give a damm !
@javatoday50025 жыл бұрын
measurement would be in bits and I am not joking
@user-rl7wg8tp8w Жыл бұрын
channel immigration
@pandasworld41686 жыл бұрын
hehe if you re like me, then you have that exam in one week
@tonewreck13 жыл бұрын
Profesor McKay is certainly extremely competent in the subject but this really is the most un-intuitive way of introducing information theory. We are given the answer right from the start and work our way backwards to see that it is an effective system, instead of trying to understand the problem and find the adequate solution. We are never trying to understand the nature of the problem but instead made to test the effectiveness of the solution. Typical of classical academic philosophy. Let's make knowledge as boring and abstruse as possible so the riff raff is kept out of our little club!
@vedantjhawar75533 жыл бұрын
Do you know of any sources to check out that might teach the content in the method you are talking about? Thank you.
@tonewreck13 жыл бұрын
@@vedantjhawar7553 try this ...kzfaq.info/get/bejne/htqdrcmhu5yndHk.html
@vedantjhawar75533 жыл бұрын
@@tonewreck1 Thank you. I took a look at it and felt that it described the act of measuring information very thoroughly. Are there any videos you recommend for learning about system solutions for transmission errors?
@tonewreck13 жыл бұрын
@@vedantjhawar7553 I am glad you found it helpful. I suggest to see the other videos in this series. Once you get a feel for what information theory is really about, you can go back to MacKay for error correction and all the nitty gritty, he does the details very effectively.
@renduvo2 жыл бұрын
@tonewreck1 For some reason I'm unable to see the reply in which you've suggested the source that @Vedant Jhawar has requested. Could you please share the source again?
@PseudoAccurate8 жыл бұрын
This is painfully slow. I'm sure the information is fantastic but you have to watch him write everything he says on the chalkboard. When I got to him telling the class to discuss how much 10% of 10,000 is I couldn't take it anymore. Can anyone suggest a similar lecture that moves more quickly?
@Meeran8 жыл бұрын
use the youtube speed feature, set it to 2x, done
@PseudoAccurate8 жыл бұрын
Lol, nice, good idea.
@unorthodoxresident75328 жыл бұрын
Lectors goal ultimatly is to share information and for audience to absorb as much as possible. There are different ways how people can do that and for each it's different. Some people are better at listenening others better absorb visual representation (either looking at chalkboard or writing notes themselves) :)
@PseudoAccurate8 жыл бұрын
That's completely understandable - that's what it was like when I went to school. It's just that most lectors now give you the notes themselves so they don't have to take the time to write much and can spend the lecture explaining the material.
@MlokKarel8 жыл бұрын
Now the question wasn't aimed at the 10% of 10k but, rather at the +- part, i.e. std.dev, IMO. Did you get that correctly as well?
@seweetgirlnay3 жыл бұрын
Where in the fock am I?
@jabbatheplutocrat10746 жыл бұрын
There is no information here and when will it be realized?Never!!!
@csaracho20094 жыл бұрын
In the first example, p is equal to f? p should be 1-f = 0.90
@abdullahmertozdemir9437 Жыл бұрын
p is equal to f, and q is equal to 1 - f. Since we assumed f to be 0.1, 0.1 * (1 - 0.1) = 0.09 thus giving us the variance of 900 after multiplying 10.000 by 0.09