Introduction to Tokenization | Writing a Custom Language Parser in Golang

Рет қаралды 3,742

2 ай бұрын

This is the first video in my series covering modern parsing techniques for language & compiler development. In this video we begin writing the lexer which we use for the remainder of the series.
Lexing is the process of transforming the raw source code into meaningful tokens which can later be used by the parser. This process will involve using regular expressions for pattern matching and helper handling functions to simplify the process compared to simpler forms of tokenization.
⭐⭐ GitHub Repo ⭐⭐
- github.com/tlaceby/parser-series
🔔 JOIN THE COMMUNITY 🔔
/ @tylerlaceby
----------------------------------------
-- Social Links --
Discord - / discord
GitHub - github.com/tlaceby/

Пікірлер: 18

@modley_the_m_guy 2 ай бұрын

Really hyped! I first watched your "How to make an interpreted language" about 3 weeks ago, and after rewriting my codebase in C#, I published it on GitHub! It was originally supposed to be a functional language, so I didn't add variables. I'm working on that as well as static typing! Now you can type anything, the only thing I have left to do before I do this GIGANTIC new commit is to add static typing to function return types! ...What am I yapping about? Great video, man! Can't wait for episode 3!

@rakinar2 2 ай бұрын

Loved the video! And yes, I'd love to see new videos about interpreting and compilation!

@thepisewigeon 2 ай бұрын

sameeeee

@martinvacheron3839 2 ай бұрын

Same here! But mostly on an interpreter

@MultiMarcsOfficialChannel Ай бұрын

Super excited to continue learning with this series! I just got finished with exams and wanted to learn more about how LSPs work. Thank you for this!

@DoubleDotStudio 2 ай бұрын

Your videos are always great. They are super easy to follow and learn from. 😃

@Raaampage 2 ай бұрын

Nice ! I'm looking forward the parser videos 😊

@topikyllonenn Ай бұрын

I'm so excited for this project! It's exactly what I've been looking for and you explained everything so well that even I (with basically no experience with tokenization) was able to understand. 😅 I followed along making my own version in TypeScript and even though I don't know anything about Golang, it was surprisingly easy to follow along and create the same functionalities in TypeScript! And yes I'd love to see videos about interpreting too! :)

@Pi7on 2 ай бұрын

Yooo, I literally discovered you a few days ago because I was researching around for my toy language project! Loved your ts series, I'll definitely be sticking around!

@tylerlaceby 2 ай бұрын

Happy to have been of help. New videos coming soon for this series.

@tmanley1985 2 ай бұрын

I'm learning to write dsls right now and this is immensely helpful. I started building an interpreter for a yaml based dsl so that I could learn that portion. But now I'm focusing on the lexing and parsing portion which scared the bejesus out of me. Unemployed at the moment so I'm gonna take as much time as I need to learn to do this. Thanks for the videos!

@tylerlaceby 2 ай бұрын

Hopefully this series will treat you well. Thanks for the kind words. Yea luckily parsing a yaml lioe syntax is much simpler than the one we cover in the series. Best of luck and if you need any assistance feel free to reach out on my discord.

@tmanley1985 2 ай бұрын

@@tylerlaceby I appreciate it! I'm gonna suffer for a while with it but I'm sure this will be a great help.

@NathanWienand 2 ай бұрын

You could make: type TokenKind string ... then define their values in the const declaration block directly, then you dont need a function to determine the value :D Good luck. very cool video and well explained.

@uynilo9 2 ай бұрын

why bro read my mind i was just looking for making language in golang the other day

@kr_24 29 күн бұрын

compile to to machine code would be awesome

@Dviih 2 ай бұрын

Hi, do you have plans to integrate it also with some compiler like LLVM? your language can really take advantage of what LLVM already have for it not only performance but also a bunch of cross language libraries.