Coding A Lexer Part 2

  Рет қаралды 361

Logan

Logan

3 ай бұрын

In this video we are going to continue work on coding our lexer. We will expand our lexical analysis to multiple character tokens. Including variable names and functions and our language keywords.
Please like, subscribe, and share the video if you enjoy!
We are going to dive into one of the toughest subjects to crack in Computer Science, Compilers and Interpreters. Don't dig through exhausting documentation or dense textbooks. Lets build one together in bite-size chunks!
Link to code:
github.com/LoganTylerPatterso...

Пікірлер: 9
@James-zn8nz
@James-zn8nz 2 ай бұрын
BTW, I decided to go with a lightweight custom Trie datastructure implementation for my tokenizer, and its working great. I am currently matching for operators/keywords, identifiers and number literals simultaneously, and picking the longest with a preference for operators/keywords. Is this a good approach?
@Logan-mj3wx
@Logan-mj3wx 2 ай бұрын
Absolutely it is a good approach, and I imagine it is quite a bit cleaner than the good ol fashioned switch block. Personally I havent approached it that way yet :)
@James-zn8nz
@James-zn8nz 2 ай бұрын
@@Logan-mj3wxIt certainly looks nice, and I'm using a macro to instantiate my token trie and enum. Although the latter process of selecting between operators/keywords, number literals and identifiers has a bit to go before i could call it a pretty sight.
@James-zn8nz
@James-zn8nz 2 ай бұрын
3:14 what about a layout-insensitive language, without braces, brackets or dedentation to define scope...
@James-zn8nz
@James-zn8nz 2 ай бұрын
I'm actually working on a grammar for this, but chances are I will use dedentation for nested constructs :( The important part is leading every statement with a keyword/operator
@Logan-mj3wx
@Logan-mj3wx 2 ай бұрын
If you want to have scope you would have to define it somehow, be it by keyword, file, spacing, etc. Otherwise the program would have to be in a global scope.
@James-zn8nz
@James-zn8nz 2 ай бұрын
@@Logan-mj3wxIt's a simple ML/OCaml inpired language, and every statement is prefixed by either a keyword or an operator, which means the compiler could understand scope based on expansion rules. Still there are issues with nested structures as if you were saying that every time a `let` keyword is encountered it is a new statement, then a nested function would not be possible. This particular example can be overcome with lambdas, but other constructs like cases may have to have dedentation.
@Logan-mj3wx
@Logan-mj3wx 2 ай бұрын
@@James-zn8nzAwesome stuff, you are braver than I to create something that is closer to lambda calculus. I love OCaml though, every couple months I apply to Jane Street I want to work with it 😂
@James-zn8nz
@James-zn8nz 2 ай бұрын
@@Logan-mj3wxFunctional languages actually seem quite easy to build. No mutability means simple functions and composition means simple production rules. I'm sure it'll be more difficult than that though. Initially there will be no complex typing system, just the built-ins, however I do plan on building some dependant type magic at some point. Also, I have written a lot more Haskell than OCaml, but I prefer some of the OCaml syntax to Haskell's. In particular, it's the indentation-sensitivity that I don't like. Also, not having to repeat the name of my function tens of times is quite nice.
The Absolute Best Intro to Monads For Software Engineers
15:12
Studying With Alex
Рет қаралды 592 М.
When Steve And His Dog Don'T Give Away To Each Other 😂️
00:21
BigSchool
Рет қаралды 17 МЛН
Smart Sigma Kid #funny #sigma #comedy
00:19
CRAZY GREAPA
Рет қаралды 8 МЛН
ПООСТЕРЕГИСЬ🙊🙊🙊
00:39
Chapitosiki
Рет қаралды 68 МЛН
Ray Marching in Three.js Tutorial
8:00
Nabil Mansour
Рет қаралды 516
Use Arc Instead of Vec
15:21
Logan Smith
Рет қаралды 135 М.
The Master Theorem Explained
6:07
Logan
Рет қаралды 160
All Rust string types explained
22:13
Let's Get Rusty
Рет қаралды 148 М.
What Makes Rust Different?
12:38
No Boilerplate
Рет қаралды 197 М.
Writing A Lexer: Final
2:42
Logan
Рет қаралды 403
Coding a Parser Part 2: Recursive Descent Parsing
5:08
ТОП-5 культовых телефонов‼️
1:00
Pedant.ru
Рет қаралды 11 М.
ПОКУПКА ТЕЛЕФОНА С АВИТО?🤭
1:00
Корнеич
Рет қаралды 1,7 МЛН
Iphone or nokia
0:15
rishton vines😇
Рет қаралды 1,7 МЛН
Bardak ile Projektör Nasıl Yapılır?
0:19
Safak Novruz
Рет қаралды 6 МЛН
Непробиваемый телевизор 🤯
0:23
FATA MORGANA
Рет қаралды 479 М.