HomeЛюди и блогиRelated VideosMore From: PyCon 2018

David Beazley - Reinventing the Parser Generator - PyCon 2018

183 ratings | 9484 views
Speaker: David Beazley Writing lexers and parsers is a complex problem that often involves the use of special tools and domain specific languages (e.g., the lex/yacc tools on Unix). In 2001, I wrote Python versions of these tools which can be found in the PLY project. PLY predates a huge number of modern Python features including the iteration protocol, generators, decorators, metaclasses, and more. As such, it relied on a variety of clever hacks to layer a domain specific parser specification language on top of Python itself. In this talk, I discuss a modernization of the PLY project that abandons its past and freely abuses modern Python features including advanced metaclasses, guaranteed dictionary ordering, class decorators, type hints, and more. The result of this work can be found in the SLY project. However, this talk isn't so much about SLY as it is focused on how far you can push Python metaprogramming features to create domain-specific languages. Prepare to be horrified--and to write code that will break your IDE. Slides can be found at: https://speakerdeck.com/pycon2018 and https://github.com/PyCon/2018-slides
Html code for embedding videos on your blog
Text Comments (15)
Jason Doege (26 days ago)
Good talk. The question I have is why do we still separate tokenization from parsing? So many languages have context sensitivity and this practice makes that very difficult to handle. The second question I have is, does SLY support stream-based parsing or does it only work on complete documents?
Yeshwanth Reddy (1 month ago)
What editor is he using???
Aaron Christianson (1 month ago)
He's an emacs user, generally speaking.
Anonymouspock (2 months ago)
This is more hilariously turning the code into Lisp. Quite informational on why that language is how it is: it barely requires a parser!
C-SPAM (5 months ago)
I'm a simple man. If I see David Beazley, I click.
Marcio Machado Pereira (5 months ago)
What tool is being used to execute python code merged into source code?
Daniel Duong (5 months ago)
At 32:49, why doesn't the code raise a NameError?
Aaron Christianson (1 month ago)
If you're using a metaclass, you get the class dictionary before the class body is executed in it. He's probably using a custom mapping that defines a `__missing__` method, which will be triggered whenever a name is looked up that isn't defined in the scope. You can use "magic" mappings in metaclasses to basically arbitrarily redefine the semantics of the language inside of a class body, so long as it's valid Python syntax (NameErrors happen at runtime, not during parsing)
Josef Kvita (5 months ago)
It does raise the error...
Daniel Duong (5 months ago)
Actually a guy asked the same question at the end of the talk, but I didn't really understand the answer.
Stefan Zweig (5 months ago)
Daniel Duong whitespace and tab should be ignored.
Fredrik Mellström (5 months ago)
I’m just going to stand here for 45 minutes and live hand-code a proof-of-concept parser generator… Twice. That’s got to be one of the most ridiculously cool live demos I’ve ever seen. Plus, the guy’s genuinely funny too. 😁
zdenda (4 months ago)
Sometimes he does that, but not this time. This was simple syntax definition for the parsers he coded before. Get well soon.
pamdemonia (6 months ago)
First? Really? Wow!
goku14139268520 (6 months ago)
pamdemonia Nooooooooo! I missed my chance!

Would you like to comment?

Join YouTube for a free account, or sign in if you are already a member.