I hit subscribe automatically after watching this single video. It's the first time I found this channel.

Great video! I’ve been designing and building software full-time (mostly large UNIX backend systems) for the last 30 years, and I still learn so much from all of your videos. I love your from-the-ground-up approach. It reminds me of things I’d forgotten, and fills in the details of things I never consciously thought about, but have just used without fully understanding them for years and years. Many thanks for helping humanity!

Thanks Scott! I never learned this stuff. It is very useful and you present the material in a very simple way, so it is easily understood. I'll watch every single video you make in this series!

Never gets old this stuff, I’m trying to debug Japanese PDF generation using an ancient open source library. Hours of fun.

Thank you very much. You showed us so clearly how this is used in practice. This is a tech topic that has never been demo'ed adequately. Humbly requesting you to share a video on how Unicode text and its encoding is handled in PDF files and displayed/printed.

I am learning so much, and I love the conversational tone. I did not finish my formal education, but I am pursuing an IT career and this really helps make the task less daunting. Thank you.

I've had your tools page bookmarked for a decade, subbed immediately after seeing this video. Top quality 👌

Sir, you're right, not everyone shares the same class, really appreciate what you're doing. I hope you're channel grows.

Great content! It's amazing the things you realize that you either forget or never knew. I was aware of the various encodings. The BOM , however, I either never knew or completely forgot about, Thanks Scott!

Brought back memories. My high school crush's name had an accented e, so Alt-0223 was committed to memory really early :).

Well i for one have never been taught this before, thank you for sharing your knowledge.

I really enjoyed this video. I was aware of encodings, but this has always been an area of confusion to me so I appreciate it. Keep it up Scott!

Really cool series! The basics are often overlooked although they are essential building blocks.

00:42 Character Encoding 01:26 ASCII 02:19 ASCII: writing an app 05:57 "who decides that this character means A" - understanding what encoding is 07:11 encoding: "another way to think about it" 07:24 notepad: "it looks like crap" 09:11 notepad2: new file & Chinese chars & CMD 11:32 encoding: signature/utf BOM 12:24 unicode website 12:31 windows charmap utility 13:40 encoding: you can potentially loose information 13:58 why BOM is useful 14:50 the app: hard-coding the BOM 16:36 summary This table of content was created using "Smart Bookmarks for KZfaq" chrome extension. You can import and edit them using this extension. You can install it from the official Chrome Store Page (shortened link): smb.page.link/store

Idk who's been in your comments section telling you, you are anything other than awesome but I love your videos and the name because 1)the name feels extremely accurate (I always go to your videos for things I wasn't taught) and 2) As the Bob Ross of coding tutorials you could never deliver a subject poorly.

Entire series "Computer Stuff They Didn't Teach You" is excellent. Please continue and I wish you good luck ! Idea for future video: "Asynchronous/Synchronous Requests"

Thank you for taking the time to share, and enlighten us as well, with your acquired knowledge.

FYI... The Unicode standard says "Use of a BOM is neither required nor recommended for UTF-8". Using a BOM with UTF-8 is almost exclusively a "Windows thing" -- you'll rarely see it on other platforms. Most of the non-Windows world treats text files as UTF-8 by default.

Great stuff, Scott, thanks so much! I'd welcome your insights on Int32/Int64 and single- and double-precision IEEE-754. My students would appreciate it as well!

I was aware of some of this but never learned it. How have I just now found your videos? I am hooked! Thank you!

Good stuff Scott, keep them coming. Thank you for all your work. 🙂

You are the teacher I never got. Thanks for uploading!

I love your video, you're course is more understanding than anyone. Thanks

Just found these list of videos. Really great. Thanks for this Scott

Keep making videos like this. Amazing stuff. I like how productive you are with VS Code.

Great explanation, I was familiar with them still learnt more about tools and background. Please keep up this series

Awesome and very useful series. Looking forward to the next videos. Thanks Scott! :)

Excellent Explaination. Feels like learned something unique today. Thanks

Providing Fantastic Quality Content as always, Thank You Sir.

Love the title and the series. Thanks for helping the community!

I would thumbs up you twice if I could. I don't know where I missed this course but I've been programming for more than 20 years. Just shows you how little that means.

Scott, thank you for share your time and large knowledge with us. For me this tips are amazing.

Such a nice video. This made me subscribe to the channel. Everybody should know these kind of fundamental concepts.

this channel is a gift for humanity :)

Thanks for tutoring Computer Zen Mastery skills, Scott.

WoW clear the basic concept with live example, loved it. Thank you

A really nice explanation! I enjoyed listening to it.

Thankyou Mr Hanselman, as you might have guessed, I was getting a little antsy trying to understand character encoding before finding this video.

I agree, this is a useful series. There's just a bunch of extra information that's so helpful that you which you'd picked up earlier.

Thanks for making these, this is a great series.

One correction, on Notepad2, when you change Enconding, you are actually getting the displayed characters to a new Code Page. So an "á" or a "ç" would still be an "á" or a "ç" (if supported) but in a different Code Page and with a different byte value (probably). What you meant to use was Enconding -> Recode. That rereads the file from disk interpreting it in the selected Code Page. Also, codepage 437 was used I think mostly in US, elsewhere codepage 850 was more common since it had more accented characters.

Dug this. I encountered the critical importance of character encoding in the wild and it made my life hell lol. So important to get this in ones head from the jump

This video was simply awesome! I love it! Thanks Scott. Can you make one for Networking IP address and VNET from Azure Networking perspective

Awesome.. enlightening. Best explanation ever.

I love this series. Thank you so much.

thanks for the videos, as someone who works with many languages this series is very helpful very informative also, whats the path highlight extension, inside the console

Back in the 1990s, Microsoft tried to get ahead of the curve by making strings made up of 16 bit characters as the preferred format for its APIs. The hope apparently was that every character in every language in the world would be assigned a unique 16 bit number. This was basically UCS2. It soon became apparent that there were too many characters in the world to accomplish this. So, people came up with UTF-16, which has a single 16 bit representation for most common characters, but other characters are represented by two 16-bit words. UTF-16 is what Windows uses internally. Although, typically strings in this format are 16 bits per character, UTF-16 is technically a variable character length encoding, so you don't truly have the advantage of one character per word, so you can't reliably index directly into strings, and string length doesn't necessarily correspond to array length. The UTF-32 encoding does have one character per word, but it wastes quite a bit of space, and isn't very popular. For both UTF-16 and UTF-32 you also have to be concerned with endianness Meanwhile, the internet moved in the opposite direction, with around 95% of all web pages now using UTF-8, in which every character is encoded in either 1, 2, 3, or 4 bytes. As of late 2019, Microsoft seems to have reversed course and is now recommending UTF-8 encoding be used internally for new apps. docs.microsoft.com/en-us/windows/uwp/design/globalizing/use-utf8-code-page * All the discussion above ignores the fact that you can combine two or more Unicode characters to form characters. For example, you might combine a base Unicode character with an accent Unicode character to form the accented form of that base character. So, you can't really always treat a character in a language as a single UTF-32, UTF-16, or UTF-8 character.

Great idea! Looking forward to more

Thanks Scott! Can you make one explaining the Http protocol? And explaining from the bottom top What happens when the user types a URL in the browser? Also another suggestion: how TCP/IP works on the bit level?

Idea for a future video Scott... “Little / Big Endian, What is that all about?!” This video took me back to my school days where we wrote code with a pencil on paper. My school of about 1,500 pupils had 2 computers. I’m still not a coder, but this was fascinating. Thank you

Great Stuff -- as someone dealing with enterprise legacy systems -- aka IBM Cobol reports ... I often deal with reports formatted for "green bar" paper. So this mean reports of a given width and given page length -- typically 132 by 60 .... or is it 133 by 66? The reasons for this and the history may prove interesting and help those that see these legacy report and wonder why? Also -- while you passed over it briefly -- big-enden and little-enden encode of data may also prove interesting -- and for those of us who remember -- don't forget to mention octal notation ... Thanks again for passing along "lore" of computing.

That was da BOM! Seriously very cool even for seasoned people.

Thanks for your lessons. Can have a video about what makes the difference b/w each objective oriented program?

Thanks Scott! Great series

Scott, First, Great videos! In your first two videos you've used hex. A lot of non-CS people that I've worked with don't understand hex, octal, or binary. Actually, even some CS people don't get it. If you're going with basics, maybe a short video on how all of those number systems work would be good.

Thanks Scott! Great stuff.

I love this video! I found it searching for howtos on dealing with file encoding in golang programming. I have written a program that parses a text file with a very old internal syntax (character 127 is the token delimiter, character 212 is the line delimiter etc.) I found that my program runs great if and only if the file input is a utf-8 file. When I fed it a file encoded ISO-8859-1, it broke. Now, I'm trying to figure out what I should do to fix the problem. I think the first step is to "detect the encoding of the input file" and second might be to "convert from the given encoding to utf-8" before processing can begin. I know that I can encode the file on my mac in the shell using iconv, and it works file (the program processes the file with no issues after I do this manual conversion). I'm wondering if you have any comments?

It was so helpful, thank you could you explain the difference between encoding and encrypting

Very informative, even for advanced developers. I couldn't get over the 3 spaces though :P

Awesome Stuff, Is that Hex dump view an Extension? Thanks Scott

I love me a good scott hanselman vid, thank you for putting these together!!!

Finally a simple explanation of code page.

Scott... you're doing good work here. Question... how do you get your VS Code terminal setup fonts/etc to match you Windows Terminal and PowerShell terminals all to match? It seems when I set one up, it doesn't apply to the others.

Great stuff once more!

Thank you sir .... No they never thaught me this, even though we were using it again and again and I asked them again and again. Again ... thanks a lot sir.

Loved the video, and the knowledge you shared. I know you're probably keeping to a specific time frame, but I have a suggestion. Can you make a video on the advanced character encodings and conversions. It's important to know that ISO-8859-* itself has issues and it has various versions that either made improvements and made breaking changes. It'd be important to know that going from ISO-8859-2 (pulling the -2 from memory) and going to UTF-8 has a block of bytes that doesn't convert over so it's lost during conversion.

Ran into this gem a while back when doing some data science stuff. Opened a text file and was seeing Chinese characters. Open notepad, even in Win 10. Type the following without the quotes "colourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolour". Save it, then open it again with notepad.

Great stuff, thank you!

Good job. Thanks for this.

In UTF-8, bytes above 0x7F require special encoding to be valid. As such, those lone 0x80, 0x81, 0x82, etc. are not a valid UTF-8 values, and a proper UTF-8 parser will give you an error there.

sometimes we need basic to reach master.. thank for creating this video.

I was unaware of code pages! thaks.

This was great, thanks!

This is Gold!

I'm 41 years old, and have been wondering what happens to cause my .Txt file to turn to gibberish since the 8th grade. Thank You Scott!

How do applications (notepad/notepad2 whatever) understand the first 3 bytes of a file are BOM or file data?

Great video Scott! You da BOM!

Great video. Thanks!

I went through 4 years of my Computer Science degree not entirely getting to grips with all of this, and it still gives me a headache to this day.

Code pages used to be part of every day computer use, and didn't need to be taught. There were a few Windows code pages for European languages, Wester, Central, Baltic, and way more DOS pages because of fewer open spots after DOS box drawing symbols. Why does the hex dump ignore the top bit and output ABCD again? I've never seen a hex dump behave like that. It should display the complete set of DOS symbols, or replace some with dots, to aid visual extraction of text. I recall a case where there was an incompatibility even within Unicode, where one non-Windows program did a simplistic conversion of Latin-1 into Unicode by putting all characters into the range 0 to 255, intead of their correct spots for things like double quotation marks down into general punctuation, and the copyright sign where that goes. This resulted in files being unopenable if those characters were in the names; and they put all crap into filenames today.

man i really enjoyed this video. taught me a lot

Great video.

16:50 Great crash course introducing the issue! 'Course, could be a post-grad, possibly even undergrad, course devoted to this subject, I would imagine.

I was just wondering whether it is possible to type using a keyboard alt-number combination or other combination any characters beyond the 256 set?

So how can I get my .net console app to display emojs in the new Windows Terminal? Change the code page?

Suggestion: explaining the different bases, hex, octal, binary, bit shifting, xoring... could be an interesting one

7:31 **Sees lots of Asian symbols** "What's that? It looks like crap!"

Approach taken was very practical. I have a question, shoudnt the range of the loop be 0 - 127 instead of 0 - 128??? Why for the second loop (0 - 256) was converted to be integer ??? Byte is 7 bits right ?? So its range is from 0 - 255 ??

Really cool title, rich content, thanks 👍

Some things, like this topic, are just so much better explained in a video. It's very difficult to understand this by reading stackoverflow posts.

awesome channel!

Thanks a lot for this video

Brilliant!

The two things in programming that are causing the most trouble, and will so in decades to come, are character endodings and time formats. Especially in the EDI domain, where different systems and/or standards all use their own take on these, will keep you busy as a developer for sure. It's also hilarious to see that many programming language used symbols not available in certain charsets as these where maintained by competitiors, i.e. the curly brace was not available in IBM driven EBCDIC code pages and as such languages like C couldn't be used initially on those mainframes.

Thank you so much.

question, before fonts existed, how was the ascii letter drawn on screen?

thanks scott, very cool

Why am I finding encoding for blank page"s of a book that is online? If the pages are blank why would you encode it?

Scott- Thank You! One request mate, Can you please tell us a short cut while coding in Vscode.

🤔 I think it is impossible to create a file (name) in/with characters >128 via DOS batch commands. I’ll explain my task. I create a new RTF File, write a Unicode or ANSI term, copy this term and use it as its file name - this works. - BUT if I try to submit a Unicode or ANSI character term as a file name it displays on DIR as silly ASCII characters despite any given code page. - That is somehow code pages do not work together with batch programming file names. Any idea?