HomeЛюди и блогиRelated VideosMore From: PyCon 2018

David Beazley - Reinventing the Parser Generator - PyCon 2018

249 ratings | 13358 views
Speaker: David Beazley Writing lexers and parsers is a complex problem that often involves the use of special tools and domain specific languages (e.g., the lex/yacc tools on Unix). In 2001, I wrote Python versions of these tools which can be found in the PLY project. PLY predates a huge number of modern Python features including the iteration protocol, generators, decorators, metaclasses, and more. As such, it relied on a variety of clever hacks to layer a domain specific parser specification language on top of Python itself. In this talk, I discuss a modernization of the PLY project that abandons its past and freely abuses modern Python features including advanced metaclasses, guaranteed dictionary ordering, class decorators, type hints, and more. The result of this work can be found in the SLY project. However, this talk isn't so much about SLY as it is focused on how far you can push Python metaprogramming features to create domain-specific languages. Prepare to be horrified--and to write code that will break your IDE. Slides can be found at: https://speakerdeck.com/pycon2018 and https://github.com/PyCon/2018-slides
Html code for embedding videos on your blog
Text Comments (25)
Et Tu (1 day ago)
Great introduction.
efraimdeluxe (7 days ago)
Dabeaz - a steady stream of excellent talks!
Clint Olsen (1 month ago)
Good talk. I'm not sure if you redeemed yourself from the abhorrent docstring with SLY. This could be more of a lateral move. :D
Domingo Gómez (2 months ago)
The compilers course is a must for all the people interesting in computer science
Modus Ponens (3 months ago)
What's this f'blah blah {template_thing}' trick? That's bad ass. Where do I get it?
Clint Olsen (1 month ago)
+Modus Ponens Yeah, Python appears to be taking lessons from Perl. There are now 3 ways to interpret a string.
Modus Ponens (3 months ago)
Oh, it's Python 3.6. Cool. I must have missed that memo.
Jason Doege (5 months ago)
Good talk. The question I have is why do we still separate tokenization from parsing? So many languages have context sensitivity and this practice makes that very difficult to handle. The second question I have is, does SLY support stream-based parsing or does it only work on complete documents?
Yeshwanth Reddy (6 months ago)
What editor is he using???
Aaron Christianson (6 months ago)
He's an emacs user, generally speaking.
Anonymouspock (7 months ago)
This is more hilariously turning the code into Lisp. Quite informational on why that language is how it is: it barely requires a parser!
C-SPAM (10 months ago)
I'm a simple man. If I see David Beazley, I click.
Kevin Dong (1 month ago)
me too. I search around for Beazley's all talk videos.
john Sabini (2 months ago)
+Edwin He Chicago is not too far#Sarcasm :) He has a compiler course coming up. It's cold there but there are a lot of pubs near his studio.
Edwin He (4 months ago)
me too! love his talks - wished he held courses down under
Marcio Machado Pereira (10 months ago)
What tool is being used to execute python code merged into source code?
Daniel Duong (10 months ago)
At 32:49, why doesn't the code raise a NameError?
Aaron Christianson (6 months ago)
If you're using a metaclass, you get the class dictionary before the class body is executed in it. He's probably using a custom mapping that defines a `__missing__` method, which will be triggered whenever a name is looked up that isn't defined in the scope. You can use "magic" mappings in metaclasses to basically arbitrarily redefine the semantics of the language inside of a class body, so long as it's valid Python syntax (NameErrors happen at runtime, not during parsing)
Josef Kvita (10 months ago)
It does raise the error...
Daniel Duong (10 months ago)
Actually a guy asked the same question at the end of the talk, but I didn't really understand the answer.
Stefan Zweig (10 months ago)
Daniel Duong whitespace and tab should be ignored.
Fredrik Mellström (10 months ago)
I’m just going to stand here for 45 minutes and live hand-code a proof-of-concept parser generator… Twice. That’s got to be one of the most ridiculously cool live demos I’ve ever seen. Plus, the guy’s genuinely funny too. 😁
zdenda (9 months ago)
Sometimes he does that, but not this time. This was simple syntax definition for the parsers he coded before. Get well soon.
pamdemonia (11 months ago)
First? Really? Wow!
goku14139268520 (11 months ago)
pamdemonia Nooooooooo! I missed my chance!

Would you like to comment?

Join YouTube for a free account, or sign in if you are already a member.