Lexer Struct Reference

#include <lexer.h>

Public Types
enum	Token {   ERROR , BUILD , COLON , DEFAULT ,   EQUALS , IDENT , INCLUDE , INDENT ,   NEWLINE , PIPE , PIPE2 , PIPEAT ,   POOL , RULE , SUBNINJA , TEOF }

Public Member Functions
std::string	DescribeLastError ()
	If the last token read was an ERROR token, provide more info or the empty string.
bool	Error (const std::string &message, std::string *err)
	Construct an error message with context.
	Lexer ()
	Lexer (const char *input)
	Helper ctor useful for tests.
bool	PeekToken (Token token)
	If the next token is token, read it and return true.
bool	ReadIdent (std::string *out)
	Read a simple identifier (a rule or variable name).
bool	ReadPath (EvalString *path, std::string *err)
	Read a path (complete with $escapes).
Token	ReadToken ()
	Read a Token from the Token enum.
bool	ReadVarValue (EvalString *value, std::string *err)
	Read the value side of a var = value line (complete with $escapes).
void	Start (StringPiece filename, StringPiece input)
	Start parsing some input.
void	UnreadToken ()
	Rewind to the last read Token.

Static Public Member Functions
static const char *	TokenErrorHint (Token expected)
	Return a human-readable token hint, used in error messages.
static const char *	TokenName (Token t)
	Return a human-readable form of a token, used in error messages.

Private Member Functions
void	EatWhitespace ()
	Skip past whitespace (called after each read token/ident/etc.).
bool	ReadEvalString (EvalString *eval, bool path, std::string *err)
	Read a $-escaped string.

Private Attributes
StringPiece	filename_
StringPiece	input_
const char *	last_token_
const char *	ofs_

Detailed Description

Definition at line 27 of file lexer.h.

Member Enumeration Documentation

Token

enum Lexer::Token

Enumerator
ERROR
BUILD
COLON
DEFAULT
EQUALS
IDENT
INCLUDE
INDENT
NEWLINE
PIPE
PIPE2
PIPEAT
POOL
RULE
SUBNINJA
TEOF

Definition at line 32 of file lexer.h.

Constructor & Destructor Documentation

Lexer() [1/2]

Lexer::Lexer ( )

inline

Definition at line 28 of file lexer.h.

Lexer() [2/2]

Lexer::Lexer ( const char * input )

explicit

Helper ctor useful for tests.

Definition at line 64 of file lexer.cc.

References Start().

Member Function Documentation

DescribeLastError()

string Lexer::DescribeLastError

(

)

If the last token read was an ERROR token, provide more info or the empty string.

Definition at line 106 of file lexer.cc.

References last_token_.

Referenced by ReadEvalString(), and TEST().

EatWhitespace()

void Lexer::EatWhitespace ( )

private

Skip past whitespace (called after each read token/ident/etc.).

re2c [ ]+ { continue; } "$\r\n" { continue; } "$\n" { continue; } nul { break; } [^] { break; }

Definition at line 471 of file lexer.cc.

References ofs_.

Referenced by ReadEvalString(), ReadIdent(), and ReadToken().

Error()

bool Lexer::Error	(	const std::string &	message,
		std::string *	err )

Construct an error message with context.

Definition at line 25 of file lexer.cc.

References filename_, input_, and last_token_.

Referenced by Parser::Load(), and ReadEvalString().

PeekToken()

bool Lexer::PeekToken

(

Token

token

)

If the next token is token, read it and return true.

Definition at line 463 of file lexer.cc.

References ReadToken(), and UnreadToken().

ReadEvalString()

bool Lexer::ReadEvalString ( EvalString * eval, bool path, std::string * err )

private

Read a $-escaped string.

re2c [^$ :\r
|\000]+ { eval->AddText(StringPiece(start, p - start)); continue; } "\r\n" { if (path) p = start; break; } [ :|
] { if (path) { p = start; break; } else { if (*start == '
') break; eval->AddText(StringPiece(start, 1)); continue; } } "$$" { eval->AddText(StringPiece("$", 1)); continue; } "$ " { eval->AddText(StringPiece(" ", 1)); continue; } "$\r\n"[ ]* { continue; } "$\n"[ ]* { continue; } "${"varname"}" { eval->AddSpecial(StringPiece(start + 2, p - start - 3)); continue; } "$"simple_varname { eval->AddSpecial(StringPiece(start + 1, p - start - 1)); continue; } "$:" { eval->AddText(StringPiece(":", 1)); continue; } "$". { last_token_ = start; return Error("bad $-escape (literal $ must be written as $$)", err); } nul { last_token_ = start; return Error("unexpected EOF", err); } [^] { last_token_ = start; return Error(DescribeLastError(), err); }

Definition at line 623 of file lexer.cc.

References EvalString::AddSpecial(), EvalString::AddText(), DescribeLastError(), EatWhitespace(), Error(), last_token_, and ofs_.

Referenced by ReadPath(), and ReadVarValue().

ReadIdent()

bool Lexer::ReadIdent

(

std::string *

out

)

Read a simple identifier (a rule or variable name).

Returns false if a name can't be read.

re2c varname { out->assign(start, p - start); break; } [^] { last_token_ = start; return false; }

Definition at line 554 of file lexer.cc.

References EatWhitespace(), last_token_, and ofs_.

Referenced by TEST(), and TEST().

ReadPath()

bool Lexer::ReadPath ( EvalString * path, std::string * err )

inline

Read a path (complete with $escapes).

Returns false only on error, returned path may be empty if a delimiter (space, newline) is hit.

Definition at line 80 of file lexer.h.

References ReadEvalString().

ReadToken()

Lexer::Token Lexer::ReadToken

(

)

Read a Token from the Token enum.

re2c re2c:define:YYCTYPE = "unsigned char"; re2c:define:YYCURSOR = p; re2c:define:YYMARKER = q; re2c:yyfill:enable = 0;

nul = "\000"; simple_varname = [a-zA-Z0-9_-]+; varname = [a-zA-Z0-9_.-]+;

[ ]*"#"[^\000
]*"\n" { continue; } [ ]*"\r\n" { token = NEWLINE; break; } [ ]*"\n" { token = NEWLINE; break; } [ ]+ { token = INDENT; break; } "build" { token = BUILD; break; } "pool" { token = POOL; break; } "rule" { token = RULE; break; } "default" { token = DEFAULT; break; } "=" { token = EQUALS; break; } ":" { token = COLON; break; } "|@" { token = PIPEAT; break; } "||" { token = PIPE2; break; } "|" { token = PIPE; break; } "include" { token = INCLUDE; break; } "subninja" { token = SUBNINJA; break; } varname { token = IDENT; break; } nul { token = TEOF; break; } [^] { token = ERROR; break; }

Definition at line 120 of file lexer.cc.

References BUILD, COLON, DEFAULT, EatWhitespace(), EQUALS, ERROR, IDENT, INCLUDE, INDENT, last_token_, NEWLINE, ofs_, PIPE, PIPE2, PIPEAT, POOL, RULE, SUBNINJA, and TEOF.

Referenced by PeekToken(), TEST(), and TEST().

ReadVarValue()

bool Lexer::ReadVarValue ( EvalString * value, std::string * err )

inline

Read the value side of a var = value line (complete with $escapes).

Returns false only on error.

Definition at line 86 of file lexer.h.

References ReadEvalString().

Referenced by TEST(), TEST(), TEST(), and TEST().

Start()

void Lexer::Start	(	StringPiece	filename,
		StringPiece	input )

Start parsing some input.

Definition at line 68 of file lexer.cc.

References filename_, input_, last_token_, and ofs_.

Referenced by Lexer().

TokenErrorHint()

const char * Lexer::TokenErrorHint ( Token expected )

static

Return a human-readable token hint, used in error messages.

Definition at line 97 of file lexer.cc.

References COLON.

Referenced by Parser::ExpectToken().

TokenName()

const char * Lexer::TokenName ( Token t )

static

Return a human-readable form of a token, used in error messages.

Definition at line 75 of file lexer.cc.

References BUILD, COLON, DEFAULT, EQUALS, ERROR, IDENT, INCLUDE, INDENT, NEWLINE, PIPE, PIPE2, PIPEAT, POOL, RULE, SUBNINJA, and TEOF.

Referenced by Parser::ExpectToken(), DyndepParser::Parse(), and ManifestParser::Parse().

UnreadToken()

void Lexer::UnreadToken

(

)

Rewind to the last read Token.

Definition at line 116 of file lexer.cc.

References last_token_, and ofs_.

Referenced by PeekToken().

Member Data Documentation

filename_

StringPiece Lexer::filename_

private

Definition at line 100 of file lexer.h.

Referenced by Error(), and Start().

input_

StringPiece Lexer::input_

private

Definition at line 101 of file lexer.h.

Referenced by Error(), and Start().

last_token_

const char* Lexer::last_token_

private

Definition at line 103 of file lexer.h.

Referenced by DescribeLastError(), Error(), ReadEvalString(), ReadIdent(), ReadToken(), Start(), and UnreadToken().

ofs_

const char* Lexer::ofs_

private

Definition at line 102 of file lexer.h.

Referenced by EatWhitespace(), ReadEvalString(), ReadIdent(), ReadToken(), Start(), and UnreadToken().

---

The documentation for this struct was generated from the following files:

• lexer.h
• lexer.cc
• lexer.in.cc