#include <google/protobuf/io/tokenizer.h>
namespace google::protobuf::io
Class for parsing tokenized text from a ZeroCopyInputStream.
Classes in this file | |
|---|---|
Abstract interface for an object which collects the errors that occur during parsing. | |
This class converts a stream of raw text into a stream of tokens for the protocol definition parser to parse. | |
Structure representing a token read from the token stream. | |
#include <google/protobuf/io/tokenizer.h>
namespace google::protobuf::io
Abstract interface for an object which collects the errors that occur during parsing.
A typical implementation might simply print the errors to stdout.
Members | |
|---|---|
| ErrorCollector() |
virtual | ~ErrorCollector() |
virtual void | AddError(int line, int column, const string & message) = 0Indicates that there was an error in the input at the given line and column numbers. more... |
|
virtual void ErrorCollector::AddError(
int line,
int column,
const string & message) = 0Indicates that there was an error in the input at the given line and column numbers.
The numbers are zero-based, so you may want to add 1 to each before printing them.
#include <google/protobuf/io/tokenizer.h>
namespace google::protobuf::io
This class converts a stream of raw text into a stream of tokens for the protocol definition parser to parse.
The tokens recognized are similar to those that make up the C language; see the TokenType enum for precise descriptions. Whitespace and comments are skipped. By default, C- and C++-style comments are recognized, but other styles can be used by calling set_comment_style().
Members | |
|---|---|
enum | TokenType |
| Tokenizer(ZeroCopyInputStream * input, ErrorCollector * error_collector) |
| ~Tokenizer() |
const Token & | current()Get the current token. more... |
bool | Next()Advance to the next token. more... |
Options | |
enum | CommentStyleValid values for set_comment_style(). more... |
void | set_allow_f_after_float(bool value)Set true to allow floats to be suffixed with the letter 'f'. more... |
void | set_comment_style(CommentStyle style)Sets the comment style. |
Parse helpers | |
static double | ParseFloat(const string & text)Parses a TYPE_FLOAT token. more... |
static void | ParseString(const string & text, string * output)Parses a TYPE_STRING token. more... |
static void | ParseStringAppend(const string & text, string * output)Identical to ParseString, but appends to output. |
static bool | ParseInteger(const string & text, uint64 max_value, uint64 * output)Parses a TYPE_INTEGER token. more... |
enum Tokenizer::TokenType {
TYPE_START,
TYPE_END,
TYPE_IDENTIFIER,
TYPE_INTEGER,
TYPE_FLOAT,
TYPE_STRING,
TYPE_SYMBOL
}| TYPE_START | Next() has not yet been called. |
| TYPE_END | End of input reached. "text" is empty. |
| TYPE_IDENTIFIER | A sequence of letters, digits, and underscores, not starting with a digit. It is an error for a number to be followed by an identifier with no space in between. |
| TYPE_INTEGER | A sequence of digits representing an integer. Normally the digits are decimal, but a prefix of "0x" indicates a hex number and a leading zero indicates octal, just like with C numeric literals. A leading negative sign is NOT included in the token; it's up to the parser to interpret the unary minus operator on its own. |
| TYPE_FLOAT | A floating point literal, with a fractional part and/or an exponent. Always in decimal. Again, never negative. |
| TYPE_STRING | A quoted sequence of escaped characters. Either single or double quotes can be used, but they must match. A string literal cannot cross a line break. |
| TYPE_SYMBOL | Any other printable character, like '!' or '+'. Symbols are always a single character, so "!+$%" is four tokens. |
Tokenizer::Tokenizer(
ZeroCopyInputStream * input,
ErrorCollector * error_collector)Construct a Tokenizer that reads and tokenizes text from the given input stream and writes errors to the given error_collector.
The caller keeps ownership of input and error_collector.
const Token & Tokenizer::current()Get the current token.
This is updated when Next() is called. Before the first call to Next(), current() has type TYPE_START and no contents.
bool Tokenizer::Next()Advance to the next token.
Returns false if the end of the input is reached.
enum Tokenizer::CommentStyle {
CPP_COMMENT_STYLE,
SH_COMMENT_STYLE
}Valid values for set_comment_style().
| CPP_COMMENT_STYLE | Line comments begin with "//", block comments are delimited by "/*" and "* /". |
| SH_COMMENT_STYLE | Line comments begin with "#". No way to write block comments. |
void Tokenizer::set_allow_f_after_float(
bool value)Set true to allow floats to be suffixed with the letter 'f'.
Tokens which would otherwise be integers but which have the 'f' suffix will be forced to be interpreted as floats. For all other purposes, the 'f' is ignored.
static double Tokenizer::ParseFloat(
const string & text)Parses a TYPE_FLOAT token.
This never fails, so long as the text actually comes from a TYPE_FLOAT token parsed by Tokenizer. If it doesn't, the result is undefined (possibly an assert failure).
static void Tokenizer::ParseString(
const string & text,
string * output)Parses a TYPE_STRING token.
This never fails, so long as the text actually comes from a TYPE_STRING token parsed by Tokenizer. If it doesn't, the result is undefined (possibly an assert failure).
static bool Tokenizer::ParseInteger(
const string & text,
uint64 max_value,
uint64 * output)Parses a TYPE_INTEGER token.
Returns false if the result would be greater than max_value. Otherwise, returns true and sets *output to the result. If the text is not from a Token of type TYPE_INTEGER originally parsed by a Tokenizer, the result is undefined (possibly an assert failure).
#include <google/protobuf/io/tokenizer.h>
namespace google::protobuf::io
Structure representing a token read from the token stream.
Members | |
|---|---|
TokenType | type |
string | textThe exact text of the token as it appeared in the input. more... |
int | line"line" and "column" specify the position of the first character of the token within the input stream. more... |
int | column |
stringToken::textThe exact text of the token as it appeared in the input.
e.g. tokens of TYPE_STRING will still be escaped and in quotes.
intToken::line"line" and "column" specify the position of the first character of the token within the input stream.
They are zero-based.