Merge pull request #39 from huangminghuang/update-protobuf

Update protobuf
huangminghuang · Oct 20, 2024 · 50496de · 50496de
2 parents 0c275f5 + a857deb
commit 50496de
Show file tree

Hide file tree

Showing 6 changed files with 382 additions and 103 deletions.
diff --git a/README.md b/README.md
@@ -4,6 +4,7 @@
 
 Hpp-proto is a lightweight, high-performance Protocol Buffers implementation for C++20. It maps Protocol Buffers messages directly to simple C++ aggregates, using only C++ built-in or standard library types. Apart from UTF-8 validation, the serialization code for these mapped aggregates is entirely header-only, ensuring minimal dependencies and efficient performance.
 
+Compared to Google’s implementation, hpp-proto features a minimalistic design that significantly reduces code size while delivering superior performance in our benchmarks when runtime reflection is not required. This makes hpp-proto an ideal choice for performance-critical, resource-constrained environments where minimizing binary size is a priority.
 # Features
 
 * Significantly smaller code size compared to Google's implementation.
@@ -12,8 +13,9 @@ Hpp-proto is a lightweight, high-performance Protocol Buffers implementation for
 * Aside from [UTF-8 validation](https://github.com/simdutf/is_utf8), all generated code and the core library are header-only.
 * Each generated C++ aggregate is associated with static C++ reflection data for efficient Protocol Buffers encoding and decoding.
 * Includes metadata for JSON serialization in each generated C++ aggregate, utilizing a slightly modified version of the [glaze](https://github.com/stephenberry/glaze) library.
+* All generated message types are equality-comparable, making them useful in unit testing.
 * Completely exception-free.
-* Supports non-owning mode code generation, mapping string and repeated fields to `std::string_view` and `std::span`.
+* Supports non-owning mode code generation, mapping string and repeated fields to `std::string_view` and `hpp::proto::equality_comparable_span` which derives from `std::span` and adds the equality comparator.
 * Enables compile-time serialization.
 
 ## Limitations
@@ -177,28 +179,29 @@ We compared the code sizes of three equivalent programs: [hpp_proto_decode_encod
 
 The comparison highlights a significant reduction in code size when using hpp-proto compared to Google’s Protocol Buffers implementations. On macOS, hpp-proto offers a 22.99x reduction in size compared to google_decode_encode and a 9.68x reduction compared to google_decode_encode_lite. The reduction is even more pronounced on Linux, where hpp-proto reduces the code size by 39.13x compared to google_decode_encode and by 16.91x compared to google_decode_encode_lite.
 
-This drastic reduction is a result of hpp-proto’s minimalistic design, which avoids the overhead associated with Google’s full libprotobuf and libprotobuf-lite libraries. The smaller code size makes hpp-proto an attractive option for performance-critical and resource-constrained environments where minimizing binary size is essential.
 
 ## Getting Started
-This section provides a quick introduction to the basic usage of hpp-proto to help you get started with minimal setup. It covers the essential steps required to integrate hpp-proto into your project and begin working with Protocol Buffers. For more advanced usage scenarios, optimizations, and additional features, please refer to the detailed examples and guides in the tutorial directory of the repository.
+This section provides a quick introduction to the basic usage of hpp-proto to help you get started with minimal setup. It covers the essential steps required to integrate hpp-proto into your project and begin working with Protocol Buffers. For more advanced usage scenarios, optimizations, and additional features, please refer to the detailed examples in the [tutorial](tutorial) directory and [code generation guide](docs/Code_Generation_Guide.md) of the repository.
 
 ### Install google protoc 
 If you haven’t installed the `protoc` compiler, [download the package](https://protobuf.dev/downloads) and follow the instructions in the README.
 
 ### [optional] Install hpp-proto
 
-Hpp-proto can be directly installed locally then use cmake `find_package` to solve the dependency,  or it can be used via cmake `FetchContent` mechanism.
+The hpp-proto library can be directly installed locally then use cmake `find_package` to solve the dependency,  or it can be used via cmake `FetchContent` mechanism.
 
 ```bash
 git clone https://github.com/huangminghuang/hpp-proto.git 
 cd hpp-proto
-# use installed protoc by default or specify '-DHPP_PROTO_PROTOC=compile' to compile protoc 
+# use installed protoc by default or specify '-DHPP_PROTO_PROTOC=compile' to download google protobuf and compile protoc from source
 cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$HOME/local -Bbuild -S .
 cmake --build build --target install
 ```
 
 ### Defining Your Protocol Format 
+
 ```protobuf
+// addressbook.proto
 syntax = "proto3";
 
 package tutorial;
@@ -244,7 +247,9 @@ This generates the following files in your specified destination directory:
 
 
 #### Code generation with CMake
-##### find_package 
+<details open><summary> find_package  </summary>
+<p>
+
 ```cmake
 cmake_minimum_required(VERSION 3.24)
 
@@ -257,13 +262,20 @@ find_package(hpp_proto CONFIG REQUIRED)
 add_library(addressbook_lib INTERFACE addressbook.proto)
 target_include_directories(addressbook_lib INTERFACE ${CMAKE_CURRENT_BINARY_DIR})
 protobuf_generate(TARGET addressbook_lib
-                  LANGUAGE hpp)
+                  LANGUAGE hpp
+# uncomment the next line for non-owning mode                 
+#                 PLUGIN_OPTIONS non_owning 
+)
 
 add_executable(tutorial_proto addressbook.cpp)
 target_link_libraries(tutorial_proto PRIVATE addressbook_lib)
 ```
+</p>
+</details>
+
+<details open><summary> FetchContent  </summary>
+<p>
 
-#### FetchContent
 ```cmake
 cmake_minimum_required(VERSION 3.24)
 
@@ -284,17 +296,27 @@ FetchContent_MakeAvailable(hpp_proto)
 add_library(addressbook_lib INTERFACE addressbook.proto)
 target_include_directories(addressbook_lib INTERFACE ${CMAKE_CURRENT_BINARY_DIR})
 protobuf_generate(TARGET addressbook_lib
-                  LANGUAGE hpp)
+                  LANGUAGE hpp
+# uncomment the next line for non-owning mode  
+#                 PLUGIN_OPTIONS non_owning 
+)
 
 add_executable(tutorial_proto addressbook.cpp)
 target_link_libraries(tutorial_proto PRIVATE addressbook_lib)
 ```
+</p>
+</details>
 
 ## The hpp-proto API
 
 The mapping from proto messages to their C++ counterparts is straight forward, as shown in the following generated code. 
 Notice that the `*.msg.hpp` only contains the minimum message definitions and avoid the inclusion of headers related to the protobuf/JSON 
 encoding/decoding facilities. This makes those generated structures easier to be used as basic vocabulary types among modules without incurring unnecessary dependencies. 
+
+Below are the examples of the generated code for the `addressbook.proto` file in regular and non-owning modes.
+<details><summary> Regular Mode </summary>
+<p>
+
 ```cpp
 // addressbook.msg.hpp
 namespace tutorial {
@@ -327,31 +349,95 @@ struct AddressBook {
 
   bool operator == (const AddressBook&) const = default;
 };
+}
+
+// addressbook.pb.hpp
+#include "addressbook.msg.hpp"
+namespace tutorial {
+  auto pb_meta(const Person &) -> std::tuple<...> ;
+  auto pb_meta(const Person::PhoneNumber &) -> std::tuple<...> ;
+  auto pb_meta(const AddressBook &) -> std::tuple<...>;
+}
+```
+</p>
+</details>
+<details><summary> Non-owning Mode </summary>
+<p>
+
+```cpp
+// addressbook.msg.hpp
+namespace tutorial {
+
+using namespace hpp::proto::literals;
+struct Person {
+  enum class PhoneType {
+    MOBILE = 0,
+    HOME = 1,
+    WORK = 2 
+  };
+
+  struct PhoneNumber {
+    std::string_view number = {};
+    PhoneType type = PhoneType::MOBILE;
+
+    bool operator == (const PhoneNumber&) const = default;
+  };
+
+  std::string_view name = {};
+  int32_t id = {};
+  std::string_view email = {};
+  hpp::proto::equality_comparable_span<const PhoneNumber> phones;
+
+  bool operator == (const Person&) const = default;
+};
+
+struct AddressBook {
+  hpp::proto::equality_comparable_span<const Person> people;
+
+  bool operator == (const AddressBook&) const = default;
+};
+}
 
 // addressbook.pb.hpp
 #include "addressbook.msg.hpp"
 namespace tutorial {
-    auto pb_meta(const Person &) -> std::tuple<...> ;
-    auto pb_meta(const Person::PhoneNumber &) -> std::tuple<...> ;
-    auto pb_meta(const AddressBook &) -> std::tuple<...>;
+  auto pb_meta(const Person &) -> std::tuple<...> ;
+  auto pb_meta(const Person::PhoneNumber &) -> std::tuple<...> ;
+  auto pb_meta(const AddressBook &) -> std::tuple<...>;
 }
 ```
+</p>
+</details>
 
 ### Protobuf encoding/decoding APIs
 
+The hpp-proto library provides convenient functions for encoding and decoding Protobuf messages in C++. The core functions are:
+
+-	write_proto: Used to serialize a C++ structure into a Protobuf message format, typically stored in a binary buffer (e.g., std::vector<std::byte>).
+-	read_proto: Used to deserialize a Protobuf-encoded binary buffer back into the corresponding C++ structure.
+
+These APIs allow you to serialize and deserialize data efficiently, with overloads that return either a success/error status or an expected object (containing the result or an error). Below is the demonstration of how to use these functions in regular and non-owning modes.
+
+</details>
+<details><summary> Regular Mode </summary>
+<p>
+
 ```cpp
 #include "addressbook.pb.hpp"
 
 int main() {
+  using enum tutorial::Person::PhoneType;
   tutorial::AddressBook address_book{
       .people = {{.name = "Alex",
                  .id = 1,
                  .email = "[email protected]",
-                 .phones = {{.number = "1111111", .type = tutorial::Person::PhoneType::MOBILE}}},
+                 .phones = {{.number = "1111111", 
+                             .type = MOBILE}}},
                  {.name = "Bob",
                   .id = 2,
                   .email = "[email protected]",
-                  .phones = {{.number = "22222222", .type = tutorial::Person::PhoneType::HOME}}} }};
+                  .phones = {{.number = "22222222", 
+                              .type = HOME}}} }};
 
   std::vector<std::byte> buffer;
 
@@ -360,21 +446,93 @@ int main() {
     return 1;
   }
 
+  // alternatively, use the overload returning an expected object
+  hpp::proto::expected<std::vector<std::byte>, std::errc> write_result 
+    = hpp::proto::write_proto(address_book);
+  assert(write_result.value() == buffer);
+
   tutorial::AddressBook new_address_book;
 
   if (!hpp::proto::read_proto(new_address_book, buffer).ok()) {
     std::cerr << "protobuf deserialization failed\n";
     return 1;
   }
 
+  // alternatively, use the overload returning an expected object
+  hpp::proto::expected<tutorial::AddressBook, std::errc> read_result 
+    = hpp::proto::read_proto<tutorial::AddressBook>(buffer);
+  assert(read_result.value() == new_address_book);
   return 0;
 }
 ```
+</p>
+</details>
+
+<details><summary> Non-owning Mode </summary>
+<p>
+
+```cpp
+#include "addressbook.pb.hpp"
+#include <memory_resource>
+
+int main() {
+  using enum tutorial::Person::PhoneType;
+  using namespace std::string_view_literals;
+  std::pmr::monotonic_buffer_resource pool;
+  std::pmr::vector<tutorial::Person::PhoneNumber> alex_phones{&pool};
+  alex_phones.push_back({.number = "1111111"sv, .type = MOBILE});
+  std::pmr::vector<tutorial::Person> people{&pool};
+  people.reserve(2);
+  people.emplace_back("Alex"sv, 1, "[email protected]"sv, alex_phones);
+  std::pmr::vector<tutorial::Person::PhoneNumber> bob_phones{&pool};
+  bob_phones.push_back({.number = "22222222"sv, .type = HOME});
+  people.emplace_back("Bob"sv, 2, "[email protected]"sv, bob_phones);
+
+  tutorial::AddressBook address_book;
+  address_book.people = people;
+
+  std::pmr::vector<std::byte> buffer{&pool};
+
+  if (!hpp::proto::write_proto(address_book, buffer).ok()) {
+    std::cerr << "protobuf serialization failed\n";
+    return 1;
+  }
+
+  // alternatively, use the overload returning an expected object
+  hpp::proto::expected<std::pmr::vector<std::byte>, std::errc> write_result 
+    = hpp::proto::write_proto<std::pmr::vector<std::byte>>(address_book);
+  assert(write_result.value() == buffer);
+
+  tutorial::AddressBook new_address_book;
+
+  if (!hpp::proto::read_proto(new_address_book, buffer, hpp::proto::pb_context{pool}).ok()) {
+    std::cerr << "protobuf deserialization failed\n";
+    return 1;
+  }
+
+  // alternatively, use the overload returning an expected object
+  hpp::proto::expected<tutorial::AddressBook, std::errc> read_result 
+    = hpp::proto::read_proto<tutorial::AddressBook>(buffer, hpp::proto::pb_context{pool});
+  assert(read_result.value() == new_address_book);
+
+  return 0;
+}
+```
+</p>
+</details>
 
 ### JSON encoding/decoding APIs
 
-hpp-proto utilizes (glaze)[https://github.com/stephenberry/glaze] for JSON encoding/decoding.
-To support the [canonical JSON encoding](https://protobuf.dev/programming-guides/proto3/#json) of protobuf messages; hpp-proto generates `*.glz.hpp` files to contain the template specializations necessary to meet the specification. The APIs for JSON encoding/decoding is similar to those of protobuf encoding/decoding.
+The hpp-proto library also supports encoding and decoding Protobuf messages to and from [canonical JSON encoding](https://protobuf.dev/programming-guides/proto3/#json) using the modified (glaze)[https://github.com/stephenberry/glaze] library. This ensures compatibility with the canonical JSON encoding of Protobuf messages. The key functions are:
+
+-	`write_json`: Used to serialize a C++ structure into a JSON string.
+-	`read_json`: Used to deserialize a JSON string back into the corresponding C++ structure.
+
+Similar to Protobuf, the JSON APIs provide overloads that return either a success/error status or an expected object. Below is a demonstration of how to use these functions for encoding and decoding in regular and non-owning modes.
+
+</details>
+<details><summary> Regular Mode </summary>
+<p>
 
 ```cpp
 
@@ -388,9 +546,50 @@ if (!hpp::proto::write_json(address_book, json).ok()) {
     return 1;
 }
 
+// alternatively, use the overload returning an expected object
+auto write_result = hpp::proto::write_json(address_book);
+assert(write_result.value() == json);
+
 tutorial::AddressBook new_book;
 if (auto e = hpp::proto::read_json(new_book, json); !e.ok()) {
     std::cerr << "read json error: " << e.message(json) << "\n";
     return 1;
 }
-```
+
+// alternatively, use the overload returning an expected object
+auto read_result = hpp::proto::read_json<tutorial::AddressBook>(json);
+assert(read_result.value() == new_address_book);
+```
+</p>
+</details>
+<details><summary> Non-owning Mode </summary>
+<p>
+
+```cpp
+
+#include "addressbook.glz.hpp"
+
+// ....
+std::pmr::string json{&pool};
+
+if (!hpp::proto::write_json(address_book, json).ok()) {
+    std::cerr << "write json error\n";
+    return 1;
+}
+
+// alternatively, use the overload returning an expected object
+auto write_result = hpp::proto::write_json<glz::opts{}, std::pmr::string>(address_book, hpp::proto::json_context{pool});
+assert(write_result.value() == json);
+
+tutorial::AddressBook new_book;
+if (auto e = hpp::proto::read_json(new_book, json, hpp::proto::json_context{pool}); !e.ok()) {
+    std::cerr << "read json error: " << e.message(json) << "\n";
+    return 1;
+}
+
+// alternatively, use the overload returning an expected object
+auto read_result = hpp::proto::read_json<tutorial::AddressBook>(json, hpp::proto::json_context{pool});
+assert(read_result.value() == new_address_book);
+```
+</p>
+</details>