"Cursorless: A spoken language for editing code" by Pokey Rule (Strange Loop 2023)

No video

"Cursorless: A spoken language for editing code" by Pokey Rule (Strange Loop 2023)

Рет қаралды 54,222

Күн бұрын

If you could design a spoken language from scratch for editing code, how would it look? What would be your nouns? Would they be tokens? Functions? Lines? What would be your verbs, your adjectives, and your adverbs?
Cursorless is one answer to these questions: a spoken language designed for maximally efficient code editing by voice. Cursorless leverages the tree-sitter real-time parser to enable high-level, "smart" code manipulations while retaining the flexibility to use "dumb" primitives like tokens, lines, delimiter pairs, and regexes when necessary.
Learn how a handful of simple abstractions - actions, modifiers, marks, and scopes - empower Cursorless users to create powerful and concise command chains that would leave even the most seasoned vim user drooling on their keyboard.
Pokey Rule
Creator of Cursorless
@PokeyRule
Pokey Rule is the creator and lead maintainer of Cursorless. He releases all of his code under the MIT license and relies on donations from users to sustain the project. Prior to working on Cursorless, Pokey managed a machine learning team at Globality. He studied programming languages and human-computer interaction at Stanford University.
----
Recorded Sept 22, 2023 at Strange Loop 2023 in St. Louis, MO.
thestrangeloop...

Пікірлер: 84

@samwight 10 ай бұрын

Cursorless literally saved my job. When I was hit by a car and broke my wrist, I couldn't use one of my hands for months for typing. I was able to buy a cheap mic, install and learn Talon, and then be just about as productive as I was before. The Talon community, and cursorless especially, has such a special place in my heart because of that. They made it possible for me to turn a very tumultuous time into one that I was easily able to adapt to.

@NithinJune 9 ай бұрын

wow !

@chromosundrift 8 ай бұрын

Fantastic! So happy for you that such a scary and horrifying experience had a silver lining! Quick note, while cheap mics are pretty good these days, it's worth noting that a _good mic_ (hopefully a mid price) can make voice control much more reliable, especially if it's positioned near and pointed at your mouth. Mic position can be an art but it can make all the difference and many people don't seem to realise. Note too that multiple mics can be better still. Finally, if you change your mic or have multiple setups, you may need to tune things for each. Doing this can be worth the effort.

@redbrick808 9 ай бұрын

The visual aids on this presentation are so well done and helped me grasp the concepts much quicker. Fantastic presentation, this really held my attention!

@brollin_ 9 ай бұрын

Cursorless was a life changing find for me as well. After I lost the use of my hands, I found Talon which made it possible for me to use a computer again, which was big. Cursorless was a HUGE benefit on top of that, that actually made it fun to program again. 1+ years later of using it and it still feels like magic when you string a novel command together on the fly. :D

@JasonStillwell 10 ай бұрын

This is vi for your throat.

@TiredOcean 9 ай бұрын

The part about point-free/tacit programming paradigms being a great fit for interactive use, and drawing the comparison between bash and jq pipelines to Cursorless was a great "aha" moment. It made me realise that maintainability of a programming language is not an unalloyed good, since it very often puts pressure on other aspects of the language that impacts its interactive use - which is a separate, but just as valid use case! (One of the things I like most about doing computer work in general is when you've got a problem you're trying to figure out, and are in the fun phase of interactive experimentation. However, when you have found a potential solution/process, you've got to codify it for future use/automation, and there's almost a feeling of starting from scratch as you have to consider maintainability. I wish that transition was simpler but I have no idea what that would look like.)

@danser_theplayer01 9 ай бұрын

Bro presented literal computer incantations. 10/10

@mattkriese7170 8 ай бұрын

This is incredible. I have been using talon on and off, and though I was amazed with the speed at which some users could utilize the software... I found that it was hard on my voice and in turn I would get voice strain during long sessions. this really does seem to be a more efficient way, and it excites me that there are so many languages supported. Thanks for your work, and for sharing your talent and knowledge. Very cool.

@Waitwhat469 9 ай бұрын

100% agree, accessibility work helps more people than just those that absolutely need it. Another great modern example, look at subtitles! Tons of non-hearing impaired people have them on by default, because there are moments in which that little bit of extra information helps!

@kenneth_romero 5 ай бұрын

one of the coolest talks and tech out there.

@jimhrelb2135 10 ай бұрын

One step for a more coordinated pair programming session. Call it a tool issue if you will, but as a vimmer, I like this a lot :)

@marksmod 9 ай бұрын

This is crazy interesting. Most interesting language I've learnt about in years, its like completely different

@robertsmme 10 ай бұрын

Really interesting. I liked the sign off comment about using restrictions others experience to be an lever to innovation for them and the great good.

@garthgoldwater5256 10 ай бұрын

this is incredible. super inspiring! great take on pointfree and interactive programming

@falkland_pinguin 9 ай бұрын

Unnecessary remark: At 12:58, Pokey was probably referring to the NATO "phonetic" alphabet, also known as International Radiotelephony Spelling Alphabet according to Wikipedia. The International Phonetic Alphabet is /ðɪs/.

@murtaza6464 10 ай бұрын

This is super awesome, and happened to show up in my feed the same week I've been working on a tree sitter parser of my own. Love seeing innovation in HCI especially in niche areas. Excellent presentation style as well

@isaactfa 10 ай бұрын

Great presentation! Apart from the specific design of, say, the vocabulary, is Cursorless inherently voice-oriented? I don't really have the desire to talk while coding, but this rigorous point-free mark->target->modifier(scope)->action pipeline approach is really compelling as a vim user.

@emma70707 10 ай бұрын

I've heard Pokey mention in the Slack a keyboard-based version they're testing out. I'm not sure what the priority level on that is though.

@SynchronizedRandomness 9 ай бұрын

The Kakaoune editing model is also very similar to this, if you’re interested. It’s also been ported via packages to some other editors (eg Emacs).

@hysenndregjoni853 9 ай бұрын

Spoken VIM, nice

@philltopia 10 ай бұрын

Definitely not a point free talk! excellent stuff

@capability-snob 10 ай бұрын

Wow, I need to add this to my system shell, and would be nice to have it in emacs too. Lovely.

@PokeyRuleJams 10 ай бұрын

💯 stay tuned on both of those

@danielrhouck 9 ай бұрын

The thing thatʼs weird about this talk is… he doesnʼt mention a place to *start.* How do I enter a *new* line of code, instead of modifying an existing one? Iʼm sure the resources he mentions at the end cover that, itʼs just weird

@melodyogonna 6 ай бұрын

I thought it was going to be about Vim, instead it's something completely different that addresses a fear I've silently had for a long time - that I might lose my hands someday, and with it the possibility to program. My face split into a wide smile immediately I heard what this does.

@jacobzimmerman3492 10 ай бұрын

Hey Pokey, not sure if you’re reading these but there was a recent short presentation/paper called “A Caret for your Thoughts” that I think you’d find interesting. Cool stuff here, I love that you could teach someone kind of naturally by conversation or pair programming. I bet an interactive tutorial in the form of a podcast/recording could be fun

@MrRonah 10 ай бұрын

Link please? I searched for it and failed to find it :(

@cursorless 10 ай бұрын

Thanks for the pointer! Are you referring to kzfaq.info/get/bejne/qJNdl5iJ0bvdhGg.html > I bet an interactive tutorial in the form of a podcast/recording could be fun Ooh interesting idea. Could you elaborate?

@garthgoldwater5256 10 ай бұрын

is that talk the “A Caret for Your Thoughts?” video on the veztron youtube channel?

@theNoriLi 9 ай бұрын

What a fantastic tool for accessibly programming =] disability and work-related injury make keyboard usage challenging for so many.

@jaysilence3314 9 ай бұрын

Cursorless should be excellent for live coding on Twitch,TV. Yet there seems to be no one doing this.

@josevargas686 9 ай бұрын

it is too difficult, no one is stepping up to the challenge, a gold mine unmined!

@ancbi 10 ай бұрын

with how much this works like a spoken vim, wouldn't it be funny if you have to speak something unintuitive to exit cursorless?

@PokeyRuleJams 10 ай бұрын

Haha. Reminds me of “whale quench”. In Emily’s strange loop talk on voice coding, she said “whale quench” to exit vim, as that’s the talon for pressing wq, and then “whale quench” went viral 😄

@jishcatg 9 ай бұрын

@@PokeyRuleJams Now I'm going to hear that every time I do it.

@alurma 10 ай бұрын

Amazing

@schubertludwig 10 ай бұрын

Super nice! But isn’t the backwards word order an actual usability issue? If you spoke them beginning with a mark, then modifiers, then actions then you could have on screen feedback for your selection as you’re speaking. Same problem that vim had, no?

@PokeyRuleJams 10 ай бұрын

Great question! Flipping the order is something I’m quite curious about, and would actually be trivial to do; ~3 lines of code. I’ve got an open offer to help anyone set that up who’s interested That being said, due to the way the Talon engine parses phrases today, you wouldn’t get live feedback. We don’t know what you’ve said until you’ve completed an entire phrase, so we couldn’t highlight the intermediate targets If you paused after the mark, then in theory we could show you that target, but it’s actually ambiguous, because if you just say a letter in talon, it will type the letter But I really don’t want to discourage anybody from trying the reverse word order, because I’m quite curious how it would work Fwiw the experimental Cursorless keyboard interface does actually go in reverse order of the spoken grammar, and we do highlight the intermediate targets as you type

@julianferrone9620 10 ай бұрын

@@PokeyRuleJams the whole word order vs. reverse order thing reminds me of Haskell's $ vs & operators-i.e. for functions f, g, h you can compose a pipeline as either "f $ g $ h" (apply f to the result of applying g to the result of applying h to the argument) or you can run it as "h & g & f" (apply h then apply g then apply f) Live view would be super cool, I'll have to check out that experimental interface

@schubertludwig 10 ай бұрын

@PokeyRuleJams thanks for the in-depth answer! Once I setup Cursorless and saw it involved Talon I realized the speech recognition wasn’t real time, so probably not immediately a big priority. Also, watching you use the system as an expert in your live coding videos made me think that most targeting doesn’t seem as complicated as I had imagined. What a fun and productive lens through which to look at editing! I didn’t realize how much of my coding tasks were “bring” actions. ;) I will try the keyboard interface soon… also so many ideas involving LLMs for natural language editing commands… what a rabbit hole of ideas you’ve open ended up for me! \o/

@AnthonyBullard 10 ай бұрын

You could take the approach Helix did to vim commands and invert the order and highlight the target range as you speak it, and then when the action is spoken you’ll know what you acted on.

@PokeyRuleJams 10 ай бұрын

@@AnthonyBullard yeah that would be cool; see my answer at the top of the thread for some of the practical difficulties there, though

@andreas_arvidsson 10 ай бұрын

Truly an inspiration for the (coding) children! :D

@randywest984 Ай бұрын

You have created an incredible product. My question is can you think of a way a blind user could interact with your product without the ID marks and therefore without access to the vocabulary to distinguish what tokens are the target

@RockieYang 10 ай бұрын

Thanks for great talk, super inspiring

@yash1152 9 ай бұрын

0:06 "spoken language for editing code" i always wanted helix/kakoune/vim modal navigation & editing keybinds to be considered an actual formal language, rather than just "commands" - just like this person considers his creation to be a programming language.

@yash1152 9 ай бұрын

3:33 no matter how small the delay is, we wanna amortize that woww. amortized analysis of complexity. awesome.

@aredrih6723 10 ай бұрын

Not sure how much it contrast with vim motion (with proper tokenization of the AST) outside of the pointfree aspect. Arbitrary word selection could be one but I'm more expecting of eye tracking tracking for that one (though the tech is a bit lacking there). Unfortunately, the current impl receiving sentence by sentence make cursorless incompatible with time sensitive eye gazing. On the point free exemple, I'm curious how it would behave in a 1k line document with instances on top and on the bottom. Would it replaces everything matching ? Even outside of your view ? What if you want to only replace half ? Also, I guess the usability depends on the work environment (open space might be a bad mix) and how much feedback speed matters to the user (keypress vs spoken word). For the world order, I'm curious what language where the verb is at the end would prefer. Japanese comes to mind (at least to my limited understanding) with flexibility in the sentence structure (lots of context in no particular order (distinguished by following particle) followed by a verb (optional of course 🤡)), and they implicit subject repeat from sentence to sentence (if 2 successive sentence talk about the same thing, only the first has the topic); so `this` as the default mark might less obvious. Either way, good job on making to address your specific condition and building a community around it. I can only wish it goods things. (and (obvious but always true) don't worry about random internet user, you know you're delivering value to your user ~) Good luck on the project.

@4xelchess905 10 ай бұрын

I think one key difference with Vim is that default commands are optimized by syllables, as opposed to key strokes. Syllables are slightly longer to utter, but there are a lot more of them, and we learn them right from birth. My guess is that you can restrict edit to scopes and selections, and you have a lot of ways to finely build selections in one or several sentences. With the help of the LSP, you might even be able to select a variable instead of a mere token and have every only select that variable rather than all the instances of the name in unrelated scopes. In an other comments, pokey talks about an "experimental Cursorless keyboard interface [that] does actually go in reverse order of the spoken grammar, and we do highlight the intermediate targets as you type", ala kakoune/helix. Also, since it is integrated inside VS code (and more generally the goal seems to be a plug-in rather than a standalone editor), I surmise you can actually seamlessly switch between cursorless and regular commands depending on which is the simplest/fastest for your needs.

@yash1152 9 ай бұрын

26:45 tree sitter localizes those errors (i.e. rest doc is still valid) 26:59 treesitter's parse tree output: dialect of scheme (lisp)

@fburton8 10 ай бұрын

Glassk! Shrock!!

@RyanLynch1 9 ай бұрын

wow this is actually so cool! i really want to try this

@hikesandbiking1181 9 ай бұрын

pokey you rule!

@strangeWaters 9 ай бұрын

I use it on and off. It's difficult but workable.

@yash1152 9 ай бұрын

18:37 _"what paradigm cursorless is as a prog lang"_ lemme guess, this aspect of it is "array programming" !?

@yash1152 9 ай бұрын

31:45 yep, uiua tacit array programming language

@ed_halley 9 ай бұрын

Just ran into this, and looked at a few jams. I don't see an example of just typing: how do you create content, spell identifiers you can't yank from elsewhere, etc.? There's gotta be a literal mode somewhere.

@SSJ3Tim 10 ай бұрын

@13:07 I really want to hear how you say "jury" with one syllable!

@PokeyRuleJams 10 ай бұрын

😂 yeah I’ve wondered the same thing. The creator of Talon is apparently able to do it. I personally just have it remapped to “Jane” 😊

@the-pink-hacker 9 ай бұрын

So it's vim for the voice?

@DebanjanBasu 4 ай бұрын

I see some vim like structure in the grammar

@yash1152 9 ай бұрын

12:38 _"little hats on a symbol of token, special word for that symbol"_ ahhw, i'd like to use the IPA flight names then: India Golf Niner Nine Charlie Tango

@yash1152 9 ай бұрын

12:57 on yeah, ofcourse IPA got a mention here (:

@Ben_EH-Heyeh 7 ай бұрын

Looking at Cursorless Git... All of the language support files are written in Scheme, a dialect of Lisp, why is Scheme not a supported language?

@chaquator 8 ай бұрын

ive been interested in programming by dictation since seeing the apple vision pro with eye tracking. i would gladly throw away my mouse and maybe even keyboard and just program with my eyes voice and sometimes hands

@yash1152 9 ай бұрын

16:09 16:16 english: VAO order 16:24 different word order, _probably_ wouldnt be hard thanks a lot. yeah, i would definitely not use this order.

@sathirasilva4958 8 ай бұрын

Does anyone know what presentation tool is he using?

@hytmal 10 ай бұрын

Mandarin is not a written language. Chinese can be written in Traditional or Simplified. Cantonese and Mandarin are spoken dialects.

@PokeyRuleJams 10 ай бұрын

Ha that’s fair. I should probably have thought of that, as someone who studied mandarin in college for 2 years 😅

@ShadowDrakken 8 ай бұрын

probably better to say "simple scopes and smart scopes" rather than "dumb scopes" :P

@ehsanu1 10 ай бұрын

This seems like a great fit for allowing LLMs to edit code efficiently, with some training.

@ueaj4576 9 ай бұрын

was thinking the same thing

@thenwhoami 9 ай бұрын

This is an extra step an LLM would have to take vs. just outputting code directly.

@clray123 8 ай бұрын

@@thenwhoami I think what was meant was editing existing code, not outputting new code. But I suppose for that we already have diff/patch, so there is indeed little use for some relative positioning commands if you can just train the LLM to generate well-formed patches.

@wege8409 9 ай бұрын

I tell you what, coding in the age of AI is really a pleasure

@thenwhoami 9 ай бұрын

This is really cool, but I can't help but wonder how LLMs will upheave this. Sure, this is much more precise, you can edit something in exactly the way you intend on the first shot, every time. An LLM on the other hand will spew out code that may not have been what you wanted initially, but then you can tell the LLM to make specific edits to your code in (using your natural language), and through that process you can still arrive at what you want. Thanks for the talk!

@fennecbesixdouze1794 9 ай бұрын

I think that generative AI could be really powerful for doing day-to-day code editing by voice command. Soon we will be able to give generative AI instructions to refactor sections or blocks of code, out loud in natural language, and let the AI assistant do the typing for us, within the full context of the file and project. Editing as well as navigating project directories, finding files, definitions, functions, etc all by voice in natural language.

@superscatboy 10 ай бұрын

Is "whale" one syllable? I would've said it's two. Way-ul.

@PokeyRuleJams 10 ай бұрын

Ha never thought about that. It’s “jury” people usually go after 😄

@teromc 10 ай бұрын

Idk, but pronunciation in the dictionary is /weɪl/ (phonetic symbols) and Google says "sounds like wayl".

@redpepper74 10 ай бұрын

Ls are kinda weird, sometimes they add like half a syllable if you put them after a vowel. (And yes it is strange to think about fractional syllables but there’s no real reason we can’t)