Can I reuse an existing protobuf binary when marshalling the messages containing it? (protobuf3)-Golang-php.cn

Can I reuse an existing protobuf binary when marshalling the messages containing it? (protobuf3)

WBOY

Release： 2024-02-06 10:30:11

forward

486 people have browsed it

在编组包含它的消息时，我可以重用现有的 protobuf 二进制文件吗？(protobuf3)

Question content

protobuf is defined as follows:

syntax = "proto3"

message hugemessage {
    // omitted
}

message request {
    string name = 1;
    hugemessage payload = 2;
}

Copy after login

In one case, I received a hugemessage from someone and I wanted to pack it with additional fields and then transmit that message to someone else. So I have to hugemessage the binary unmarshal into a go structure, package it into a request and then marshal# it again ##. Due to the hgue size of hugemessage, the cost of unmarshal and marshal is prohibitive. So can I reuse the hugemessage binary without changing the protobuf definition?

func main() {
    // receive it from file or network, not important.
    bins, _ := os.ReadFile("hugeMessage.dump")
    var message HugeMessage
    _ = proto.Unmarshal(bins, &message) // slow
    request := Request{
        name: "xxxx",
        payload: message,
    }
    requestBinary, _ := proto.Marshal(&request) // slow
    // send it.
    os.WriteFile("request.dump", requestBinary, 0644)
}

Copy after login

Correct Answer

The short answer is: No, there is no easy or standard way to achieve this.

The most obvious strategy is to do it the way you currently do it - unmarshal

hugemessage, set it to request, and then marshal again. The Golang protobuf API surface doesn't really provide a way to do more - and for good reason.

That said, there are

ways to achieve what you want to do. But these aren't necessarily safe or reliable, so you have to weigh that cost against what you have now.

One way to avoid unmarshalling is to take advantage of the way messages are normally serialized;

message request {
    string name = 1;
    hugemessage payload = 2;
}

Copy after login

..Equivalent to

message request {
    string name = 1;
    bytes payload = 2;
}

Copy after login

.. where

payload contains the results of calling marshal(...) against some hugemessage.

So, if we have the following definition:

syntax = "proto3";

message hugemessage {
  bytes field1 = 1;
  string field2 = 2;
  int64 field3 = 3;
}

message request {
  string name = 1;
  hugemessage payload = 2;
}

message rawrequest {
  string name = 1;
  bytes payload = 2;
}

Copy after login

The following code:

req1, err := proto.Marshal(&pb.Request{
    Name: "name",
    Payload: &pb.HugeMessage{
        Field1: []byte{1, 2, 3},
        Field2: "test",
        Field3: 948414,
    },
})
if err != nil {
    panic(err)
}

huge, err := proto.Marshal(&pb.HugeMessage{
    Field1: []byte{1, 2, 3},
    Field2: "test",
    Field3: 948414,
})
if err != nil {
    panic(err)
}

req2, err := proto.Marshal(&pb.RawRequest{
    Name:    "name",
    Payload: huge,
})
if err != nil {
    panic(err)
}

fmt.Printf("equal? %t\n", bytes.Equal(req1, req2))

Copy after login

Output

equal? true

It's unclear whether this "quirk" is completely reliable, and there's no guarantee it will continue to function indefinitely. Obviously the

rawrequest type must exactly mirror the request type, which is not ideal.

Another option is to build the message in a more manual way, i.e. using the

protowire package - again, feel free, caution is recommended.

The above is the detailed content of Can I reuse an existing protobuf binary when marshalling the messages containing it? (protobuf3). For more information, please follow other related articles on the PHP Chinese website!