Lexer

Turns a stream of characters into a stream of tokens.

To create a lexer, use the companion object’s forFile or forString methods.

The lexer is the first main componenent in the language implementation. It reads a stream of characters from a file, or for testing, from a string. It produces a stream of tokens which is further processed by the parser.

The lexer classifies tokens according to their token type. The token type of the next token in the token stream is available from nextTokenType.

The lexer keeps track of line and column positions inside the source code. The first character of the source code is line 1 and column 1. The location of the beginning of the next token is available from startLine and startColumn. The location of the first character after the next token is available from currentLine and currentColumn.

Most token types classify a fixed string such as an operator or keyword. The following token types are special:

EndOfFile ─ Used for every position after the last token in the stream
Identifier ─ The name of the identifier is available from nextTokenText
IntegerLiteral ─ The value of the integer literal is available from nextTokenIntegerValue

Linear Supertypes

AnyRef, Any

Instance Constructors

new Lexer(input: Reader)

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def at(tokenType: TokenType): Boolean

Determine whether the next token has the specified type, but don't process that token.
def atDigit: Boolean

Determine whether the next code point in the input is a digit according to the Scala Language Specification, Chapter 1.
def atEndOfFile(): Boolean
def atEndOfLine(): Boolean
def atLetter: Boolean

Determine whether a given code point can be an operator character according to the Scala Language Specification, Chapter 1
def atOperatorCharacter: Boolean

Determine whether the next code point in the input is an operator character according to the Scala Language Specification, Chapter 1.
def check(tokenType: TokenType): Boolean

Determine whether the next token has the specified type, and process it if it does.
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
var currentColumn: Int
var currentLine: Int
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def expect(tokenType: TokenType): Unit

Process the next token if it has the specified type, and throw an error if it doesn't.
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
def isDigit(codepoint: Int): Boolean

Determine whether the given code point is a digit according to the Scala Language Specification, Chapter 1.
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def isLetter(codepoint: Int): Boolean

Determine whether the given code point is a letter according to the Scala Language Specification, Chapter 1.
def isOperatorCharacter(codepoint: Int): Boolean

Determine whether the given code point is an operator character according to the Scala Language Specification, Chapter 1.
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
var nextCodepoint: Int

next code point in input string, after current position
var nextNextCodepoint: Int

next code point after the next code point after the current position
var nextTokenIntegerValue: Int
val nextTokenText: StringBuilder
var nextTokenType: TokenType
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def readNextCodepoint(): Unit
def readNextToken(): TokenType
def readNotFirstCodepoint(): Int
def registerCodepoint(codepoint: Int): Unit
def skipNextCodepoint(): Unit
def skipWhitespaceAndComments(): Unit
var startColumn: Int
var startLine: Int
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Docs: object Lexer | package pli

class Lexer extends AnyRef

Instance Constructors

new Lexer(input: Reader)

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def at(tokenType: TokenType): Boolean

def atDigit: Boolean

def atEndOfFile(): Boolean

def atEndOfLine(): Boolean

def atLetter: Boolean

def atOperatorCharacter: Boolean

def check(tokenType: TokenType): Boolean

def clone(): AnyRef

var currentColumn: Int

var currentLine: Int

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def expect(tokenType: TokenType): Unit

def finalize(): Unit

final def getClass(): Class[_]

def hashCode(): Int

def isDigit(codepoint: Int): Boolean

final def isInstanceOf[T0]: Boolean

def isLetter(codepoint: Int): Boolean

def isOperatorCharacter(codepoint: Int): Boolean

final def ne(arg0: AnyRef): Boolean

var nextCodepoint: Int

var nextNextCodepoint: Int

var nextTokenIntegerValue: Int

val nextTokenText: StringBuilder

var nextTokenType: TokenType

final def notify(): Unit

final def notifyAll(): Unit

def readNextCodepoint(): Unit

def readNextToken(): TokenType

def readNotFirstCodepoint(): Int

def registerCodepoint(codepoint: Int): Unit

def skipNextCodepoint(): Unit

def skipWhitespaceAndComments(): Unit

var startColumn: Int

var startLine: Int

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from AnyRef

Inherited from Any

Ungrouped