2.5 Using PyLly with PLY

Lexers generated by PyLly can be used by PyGgy parsers or by other parser packages. The lexer interface was designed to be similar enough to PLY's lexer interface that PyLly lexers can be used as-is. For other parsers, a small wrapper may class may be needed around the lexer to provide the proper interface.

The example/ply_calc.py example illustrates how to use a PLY parser with a PyLly lexer. This is a small calculator test case based on the example from the PLY web site (http://systems.cs.uchicago.edu/ply/example.html). The example/ply_calc.pyl file specifies the lexer:

code :
    tokens = (
        'NAME','NUMBER',
        'PLUS','MINUS','TIMES','DIVIDE','EQUALS',
        'LPAREN','RPAREN',
    )

    lineno = 1

    class Tok :
        def __init__(self, l, type) :
            self.type = type
            self.lineno = lineno
            self.value = l.value
        def __str__(self) :
            return "Tok(%s,%r,%d)" % (self.type, self.value, self.lineno)

definitions :
    NAME    "[[:alpha:]_][[:alnum:]_]*"
    NUMBER  "[[:digit:]]+"

INITIAL :
    "\+" :  return Tok(self, "PLUS")
    "-" :   return Tok(self, "MINUS")
    "\*" :  return Tok(self, "TIMES")
    "/" :   return Tok(self, "DIVIDE")
    "=" :   return Tok(self, "EQUALS")
    "\(" :  return Tok(self, "LPAREN")
    "\)" :  return Tok(self, "RPAREN")

    "{NAME}":   return Tok(self, "NAME")
    "{NUMBER}":
        try :
            self.value = int(self.value)
        except ValueError :
            print "Integer value too large", self.value
            self.value = 0
        return Tok(self, "NUMBER")

    "[[:blank:]]" : return
    "\n+" :
        global lineno
        lineno += len(self.value)
        return

    "." :
        print "Illegal character '%s'" % self.value
        return

Each token's value is represented by a Tok instance which adheres to PLY's token interface. The specification also defines a list of tokens in the tokens variable as needed by PLY. The examples/ply_calc.py script builds the lexer and uses it to feed a PLY parser:

# build the lexer
import pyggy
l,lexer = pyggy.getlexer("ply_calc.pyl")
tokens = lexer.tokens

[...]

import yacc
yacc.yacc()

while 1:
    sys.stdout.write("calc > ")
    line = sys.stdin.readline()
    if line == "" :
        break

    l.setinputstr(line)
    yacc.parse(lexer=l)

See the PyGgy Home Page.