Lexical Scanning in Go - Rob Pike

Lexical Scanning in Go, a talk by Rob Pike at Google Technology User Group given on Tuesday, 30 August 2011.

The slides: http://rspace.googlecode.com/hg/slide/lex.html

Category:

Science & Technology

Tags:

• open source
• google
• sydney
• technology
• programming
• golang
• go
• rob pike
• lexer
• lexing
• lexical
• scanning

[DavidPFarrell]

I wrote a lexer and parser API in Go, inspired by this presentation:

github.com/iNamik/lexer.go

github.com/iNamik/parser.go

[argman13]

where's Andrew's talk on using the new template system?

[egcrosser]

@jimmyrcom I see. Honestly, in this particular case it's just a matter of personal preference, the for-loop is "elegant enough" too.

[jimmyrcom]

@egcrosser They didn't want it in the language spec as far as I can remember. It's kinda funny too because you could see people pleading for it trying to describe exactly this type of situation. They were saying something like the stack trace is too important, use loops/gotos instead.

[henriquedante]

@vonschlesien You're right I didn't think the test functions would be so efficient as they are

[vonschlesien]

@henriquedante Unicode is actually not a problem - Go as a language natively supports utf8 as its character encoding for the string type. For example, around 23:00-24:00, where you see "switch r = l.next() {...", r is the next rune in the output - i.e. the next unicode code point assembled through utf8 decoding. If you want to accept only upper case, just call the relevant test function from package unicode on the value of r :-)

[egcrosser]

I wonder if Go has tail call optimization. You might implement the construct described in minutes 14-15 as a tail call instead of a for-loop.

[henriquedante]

Main objection point is that you can't write this way if you're parsing unicode patterns. For example, what would be the accept() string for accepting upper case, lower case, etc. letters ? Nevertheless, the final scanner was extremelly clean and it must have been really fun to write.