#include <google/protobuf/io/coded_stream.h>
namespace google::protobuf::io
This file contains the CodedInputStream and CodedOutputStream classes, which wrap a ZeroCopyInputStream or ZeroCopyOutputStream, respectively, and allow you to read or write individual pieces of data in various formats.
In particular, these implement the varint encoding for integers, a simple variable-length encoding in which smaller numbers take fewer bytes.
Typically these classes will only be used internally by the protocol buffer library in order to encode and decode protocol buffers. Clients of the library only need to know about this class if they wish to write custom message parsing or serialization procedures.
CodedOutputStream example:
Write some data to "myfile". First we write a 4-byte "magic number"
to identify the file type, then write a length-delimited string. The
string is composed of a varint giving the length followed by the raw
bytes.
int fd = open("myfile", O_WRONLY);
ZeroCopyOutputStream* raw_output = new FileOutputStream(fd);
CodedOutputStream* coded_output = new CodedOutputStream(raw_output);
int magic_number = 1234;
char text[] = "Hello world!";
coded_output->WriteLittleEndian32(magic_number);
coded_output->WriteVarint32(strlen(text));
coded_output->WriteRaw(text, strlen(text));
delete coded_output;
delete raw_output;
close(fd);
CodedInputStream example:
Read a file created by the above code.
int fd = open("myfile", O_RDONLY);
ZeroCopyInputStream* raw_input = new FileInputStream(fd);
CodedInputStream coded_input = new CodedInputStream(raw_input);
coded_input->ReadLittleEndian32(&magic_number);
if (magic_number != 1234) {
cerr << "File not in expected format." << endl;
return;
}
uint32 size;
coded_input->ReadVarint32(&size);
char* text = new char[size + 1];
coded_input->ReadRaw(buffer, size);
text[size] = '\0';
delete coded_input;
delete raw_input;
close(fd);
cout << "Text is: " << text << endl;
delete [] text;
For those who are interested, varint encoding is defined as follows:
The encoding operates on unsigned integers of up to 64 bits in length. Each byte of the encoded value has the format:
In theory, varint could be used to encode integers of any length. However, for practicality we set a limit at 64 bits. The maximum encoded length of a number is thus 10 bytes.
Classes in this file | |
|---|---|
Class which reads and decodes binary data which is composed of varint- encoded integers and fixed-width pieces. | |
Class which encodes and writes binary data which is composed of varint- encoded integers and fixed-width pieces. | |
#include <google/protobuf/io/coded_stream.h>
namespace google::protobuf::io
Class which reads and decodes binary data which is composed of varint- encoded integers and fixed-width pieces.
Wraps a ZeroCopyInputStream. Most users will not need to deal with CodedInputStream.
Most methods of CodedInputStream that return a bool return false if an underlying I/O error occurs or if the data is malformed. Once such a failure occurs, the CodedInputStream is broken and is no longer useful.
Members | |
|---|---|
explicit | CodedInputStream(ZeroCopyInputStream * input)Create a CodedInputStream that reads from the given ZeroCopyInputStream. |
explicit | CodedInputStream(const uint8 * buffer, int size)Create a CodedInputStream that reads from the given flat array. more... |
| ~CodedInputStream()Destroy the CodedInputStream and position the underlying ZeroCopyInputStream at the first unread byte. more... |
bool | Skip(int count)Skips a number of bytes. more... |
bool | GetDirectBufferPointer(const void ** data, int * size)Sets *data to point directly at the unread part of the CodedInputStream's underlying buffer, and *size to the size of that buffer, but does not advance the stream's current position. more... |
bool | ReadRaw(void * buffer, int size)Read raw bytes, copying them into the given buffer. |
bool | ReadString(string * buffer, int size)Like ReadRaw, but reads into a string. more... |
bool | ReadLittleEndian32(uint32 * value)Read a 32-bit little-endian integer. |
bool | ReadLittleEndian64(uint64 * value)Read a 64-bit little-endian integer. |
bool | ReadVarint32(uint32 * value)Read an unsigned integer with Varint encoding, truncating to 32 bits. more... |
bool | ReadVarint64(uint64 * value)Read an unsigned integer with Varint encoding. |
uint32 | ReadTag()Read a tag. more... |
bool | ExpectTag(uint32 expected)Usually returns true if calling ReadVarint32() now would produce the given value. more... |
bool | ExpectAtEnd()Usually returns true if no more bytes can be read. more... |
bool | LastTagWas(uint32 expected) |
bool | ConsumedEntireMessage()When parsing message (but NOT a group), this method must be called immediately after MergeFromCodedStream() returns (if it returns true) to further verify that the message ended in a legitimate way. more... |
LimitsLimits are used when parsing length-delimited embedded messages. After the message's length is read, PushLimit() is used to prevent the CodedInputStream from reading beyond that length. Once the embedded message has been parsed, PopLimit() is called to undo the limit. | |
typedef | int Limit |
Limit | PushLimit(int byte_limit)Places a limit on the number of bytes that the stream may read, starting from the current position. more... |
void | PopLimit(Limit limit)Pops the last limit pushed by PushLimit(). more... |
int | BytesUntilLimit()Returns the number of bytes left until the nearest limit on the stack is hit, or -1 if no limits are in place. |
Total Bytes LimitTo prevent malicious users from sending excessively large messages and causing integer overflows or memory exhaustion, CodedInputStream imposes a hard limit on the total number of bytes it will read. | |
void | SetTotalBytesLimit(int total_bytes_limit, int warning_threshold)Sets the maximum number of bytes that this CodedInputStream will read before refusing to continue. more... |
Recursion LimitTo prevent corrupt or malicious messages from causing stack overflows, we must keep track of the depth of recursion when parsing embedded messages and groups. CodedInputStream keeps track of this because it is the only object that is passed down the stack during parsing. | |
void | SetRecursionLimit(int limit)Sets the maximum recursion depth. The default is 64. |
bool | IncrementRecursionDepth()Increments the current recursion depth. more... |
void | DecrementRecursionDepth()Decrements the recursion depth. |
explicit CodedInputStream::CodedInputStream(
const uint8 * buffer,
int size)Create a CodedInputStream that reads from the given flat array.
This is faster than using an ArrayInputStream. PushLimit(size) is implied by this constructor.
CodedInputStream::~CodedInputStream()Destroy the CodedInputStream and position the underlying ZeroCopyInputStream at the first unread byte.
If an error occurred while reading (causing a method to return false), then the exact position of the input stream may be anywhere between the last value that was read successfully and the stream's byte limit.
bool CodedInputStream::Skip(
int count)Skips a number of bytes.
Returns false if an underlying read error occurs.
bool CodedInputStream::GetDirectBufferPointer(
const void ** data,
int * size)Sets *data to point directly at the unread part of the CodedInputStream's underlying buffer, and *size to the size of that buffer, but does not advance the stream's current position.
This will always either produce a non-empty buffer or return false. If the caller consumes any of this data, it should then call Skip() to skip over the consumed bytes. This may be useful for implementing external fast parsing routines for types of data not covered by the CodedInputStream interface.
bool CodedInputStream::ReadString(
string * buffer,
int size)Like ReadRaw, but reads into a string.
Implementation Note: ReadString() grows the string gradually as it reads in the data, rather than allocating the entire requested size upfront. This prevents denial-of-service attacks in which a client could claim that a string is going to be MAX_INT bytes long in order to crash the server because it can't allocate this much space at once.
bool CodedInputStream::ReadVarint32(
uint32 * value)Read an unsigned integer with Varint encoding, truncating to 32 bits.
Reading a 32-bit value is equivalent to reading a 64-bit one and casting it to uint32, but may be more efficient.
uint32 CodedInputStream::ReadTag()Read a tag.
This calls ReadVarint32() and returns the result, or returns zero (which is not a valid tag) if ReadVarint32() fails. Also, it updates the last tag value, which can be checked with LastTagWas(). Always inline because this is only called in once place per parse loop but it is called for every iteration of said loop, so it should be fast. GCC doesn't want to inline this by default.
bool CodedInputStream::ExpectTag(
uint32 expected)Usually returns true if calling ReadVarint32() now would produce the given value.
Will always return false if ReadVarint32() would not return the given value. If ExpectTag() returns true, it also advances past the varint. For best performance, use a compile-time constant as the parameter. Always inline because this collapses to a small number of instructions when given a constant parameter, but GCC doesn't want to inline by default.
bool CodedInputStream::ExpectAtEnd()Usually returns true if no more bytes can be read.
Always returns false if more bytes can be read. If ExpectAtEnd() returns true, a subsequent call to LastTagWas() will act as if ReadTag() had been called and returned zero, and ConsumedEntireMessage() will return true.
bool CodedInputStream::LastTagWas(
uint32 expected)If the last call to ReadTag() returned the given value, returns true.
Otherwise, returns false;
This is needed because parsers for some types of embedded messages (with field type TYPE_GROUP) don't actually know that they've reached the end of a message until they see an ENDGROUP tag, which was actually part of the enclosing message. The enclosing message would like to check that tag to make sure it had the right number, so it calls LastTagWas() on return from the embedded parser to check.
bool CodedInputStream::ConsumedEntireMessage()When parsing message (but NOT a group), this method must be called immediately after MergeFromCodedStream() returns (if it returns true) to further verify that the message ended in a legitimate way.
For example, this verifies that parsing did not end on an end-group tag. It also checks for some cases where, due to optimizations, MergeFromCodedStream() can incorrectly return true.
typedef CodedInputStream::LimitOpaque type used with PushLimit() and PopLimit().
Do not modify values of this type yourself. The only reason that this isn't a struct with private internals is for efficiency.
Limit CodedInputStream::PushLimit(
int byte_limit)Places a limit on the number of bytes that the stream may read, starting from the current position.
Once the stream hits this limit, it will act like the end of the input has been reached until PopLimit() is called.
As the names imply, the stream conceptually has a stack of limits. The shortest limit on the stack is always enforced, even if it is not the top limit.
The value returned by PushLimit() is opaque to the caller, and must be passed unchanged to the corresponding call to PopLimit().
void CodedInputStream::PopLimit(
Limit limit)Pops the last limit pushed by PushLimit().
The input must be the value returned by that call to PushLimit().
void CodedInputStream::SetTotalBytesLimit(
int total_bytes_limit,
int warning_threshold)Sets the maximum number of bytes that this CodedInputStream will read before refusing to continue.
To prevent integer overflows in the protocol buffers implementation, as well as to prevent servers from allocating enormous amounts of memory to hold parsed messages, the maximum message length should be limited to the shortest length that will not harm usability. The theoretical shortest message that could cause integer overflows is 512MB. The default limit is 64MB. Apps should set shorter limits if possible. If warning_threshold is not -1, a warning will be printed to stderr after warning_threshold bytes are read. An error will always be printed to stderr if the limit is reached.
This is unrelated to PushLimit()/PopLimit().
Hint: If you are reading this because your program is printing a warning about dangerously large protocol messages, you may be confused about what to do next. The best option is to change your design such that excessively large messages are not necessary. For example, try to design file formats to consist of many small messages rather than a single large one. If this is infeasible, you will need to increase the limit. Chances are, though, that your code never constructs a CodedInputStream on which the limit can be set. You probably parse messages by calling things like Message::ParseFromString(). In this case, you will need to change your code to instead construct some sort of ZeroCopyInputStream (e.g. an ArrayInputStream), construct a CodedInputStream around that, then call Message::ParseFromCodedStream() instead. Then you can adjust the limit. Yes, it's more work, but you're doing something unusual.
bool CodedInputStream::IncrementRecursionDepth()Increments the current recursion depth.
Returns true if the depth is under the limit, false if it has gone over.
#include <google/protobuf/io/coded_stream.h>
namespace google::protobuf::io
Class which encodes and writes binary data which is composed of varint- encoded integers and fixed-width pieces.
Wraps a ZeroCopyOutputStream. Most users will not need to deal with CodedOutputStream.
Most methods of CodedOutputStream which return a bool return false if an underlying I/O error occurs. Once such a failure occurs, the CodedOutputStream is broken and is no longer useful. The Write* methods do not return the stream status, but will invalidate the stream if an error occurs. The client can probe HadError() to determine the status.
Note that every method of CodedOutputStream which writes some data has a corresponding static "ToArray" version. These versions write directly to the provided buffer, returning a pointer past the last written byte. They require that the buffer has sufficient capacity for the encoded data. This allows an optimization where we check if an output stream has enough space for an entire message before we start writing and, if there is, we call only the ToArray methods to avoid doing bound checks for each individual value. i.e., in the example above:
CodedOutputStream coded_output = new CodedOutputStream(raw_output); int magic_number = 1234; char text[] = "Hello world!"; int coded_size = sizeof(magic_number) + CodedOutputStream::Varint32Size(strlen(text)) + strlen(text); uint8* buffer = coded_output->GetDirectBufferForNBytesAndAdvance(coded_size); if (buffer != NULL) { The output stream has enough space in the buffer: write directly to the array. buffer = CodedOutputStream::WriteLittleEndian32ToArray(magic_number, buffer); buffer = CodedOutputStream::WriteVarint32ToArray(strlen(text), buffer); buffer = CodedOutputStream::WriteRawToArray(text, strlen(text), buffer); } else { Make bound-checked writes, which will ask the underlying stream for more space as needed. coded_output->WriteLittleEndian32(magic_number); coded_output->WriteVarint32(strlen(text)); coded_output->WriteRaw(text, strlen(text)); } delete coded_output;
Members | |
|---|---|
explicit | CodedOutputStream(ZeroCopyOutputStream * output)Create an CodedOutputStream that writes to the given ZeroCopyOutputStream. |
| ~CodedOutputStream()Destroy the CodedOutputStream and position the underlying ZeroCopyOutputStream immediately after the last byte written. |
bool | Skip(int count)Skips a number of bytes, leaving the bytes unmodified in the underlying buffer. more... |
bool | GetDirectBufferPointer(void ** data, int * size)Sets *data to point directly at the unwritten part of the CodedOutputStream's underlying buffer, and *size to the size of that buffer, but does not advance the stream's current position. more... |
uint8 * | GetDirectBufferForNBytesAndAdvance(int size)If there are at least "size" bytes available in the current buffer, returns a pointer directly into the buffer and advances over these bytes. more... |
void | WriteRaw(const void * buffer, int size)Write raw bytes, copying them from the given buffer. |
void | WriteString(const string & str)Equivalent to WriteRaw(str.data(), str.size()). |
void | WriteLittleEndian32(uint32 value)Write a 32-bit little-endian integer. |
void | WriteLittleEndian64(uint64 value)Write a 64-bit little-endian integer. |
void | WriteVarint32(uint32 value)Write an unsigned integer with Varint encoding. more... |
void | WriteVarint64(uint64 value)Write an unsigned integer with Varint encoding. |
void | WriteVarint32SignExtended(int32 value)Equivalent to WriteVarint32() except when the value is negative, in which case it must be sign-extended to a full 10 bytes. |
void | WriteTag(uint32 value)This is identical to WriteVarint32(), but optimized for writing tags. more... |
int | ByteCount() constReturns the total number of bytes written since this object was created. |
bool | HadError() constReturns true if there was an underlying I/O error since this object was created. |
static uint8 * | WriteRawToArray(const void * buffer, int size, uint8 * target)Like WriteRaw() but writing directly to the target array. more... |
static uint8 * | WriteStringToArray(const string & str, uint8 * target)Like WriteString() but writing directly to the target array. |
static uint8 * | WriteLittleEndian32ToArray(uint32 value, uint8 * target)Like WriteLittleEndian32() but writing directly to the target array. |
static uint8 * | WriteLittleEndian64ToArray(uint64 value, uint8 * target)Like WriteLittleEndian64() but writing directly to the target array. |
static uint8 * | WriteVarint32ToArray(uint32 value, uint8 * target)Like WriteVarint32() but writing directly to the target array. |
static uint8 * | WriteVarint64ToArray(uint64 value, uint8 * target)Like WriteVarint64() but writing directly to the target array. |
static uint8 * | WriteVarint32SignExtendedToArray(int32 value, uint8 * target)Like WriteVarint32SignExtended() but writing directly to the target array. |
static uint8 * | WriteTagToArray(uint32 value, uint8 * target)Like WriteTag() but writing directly to the target array. |
static int | VarintSize32(uint32 value)Returns the number of bytes needed to encode the given value as a varint. |
static int | VarintSize64(uint64 value)Returns the number of bytes needed to encode the given value as a varint. |
static int | VarintSize32SignExtended(int32 value)If negative, 10 bytes. Otheriwse, same as VarintSize32(). |
bool CodedOutputStream::Skip(
int count)Skips a number of bytes, leaving the bytes unmodified in the underlying buffer.
Returns false if an underlying write error occurs. This is mainly useful with GetDirectBufferPointer().
bool CodedOutputStream::GetDirectBufferPointer(
void ** data,
int * size)Sets *data to point directly at the unwritten part of the CodedOutputStream's underlying buffer, and *size to the size of that buffer, but does not advance the stream's current position.
This will always either produce a non-empty buffer or return false. If the caller writes any data to this buffer, it should then call Skip() to skip over the consumed bytes. This may be useful for implementing external fast serialization routines for types of data not covered by the CodedOutputStream interface.
uint8 * CodedOutputStream::GetDirectBufferForNBytesAndAdvance(
int size)If there are at least "size" bytes available in the current buffer, returns a pointer directly into the buffer and advances over these bytes.
The caller may then write directly into this buffer (e.g. using the ToArray static methods) rather than go through CodedOutputStream. If there are not enough bytes available, returns NULL. The return pointer is invalidated as soon as any other non-const method of CodedOutputStream is called.
void CodedOutputStream::WriteVarint32(
uint32 value)Write an unsigned integer with Varint encoding.
Writing a 32-bit value is equivalent to casting it to uint64 and writing it as a 64-bit value, but may be more efficient.
void CodedOutputStream::WriteTag(
uint32 value)This is identical to WriteVarint32(), but optimized for writing tags.
In particular, if the input is a compile-time constant, this method compiles down to a couple instructions. Always inline because otherwise the aformentioned optimization can't work, but GCC by default doesn't want to inline this.
static uint8 * CodedOutputStream::WriteRawToArray(
const void * buffer,
int size,
uint8 * target)Like WriteRaw() but writing directly to the target array.
This is _not_ inlined, as the compiler often optimizes memcpy into inline copy loops. Since this gets called by every field with string or bytes type, inlining may lead to a significant amount of code bloat, with only a minor performance gain.