Field presence tracking in Protocol Buffer v3

With Protocol Buffer v3, it was not longer possible to known if a field is unset or set to a default value, except to use workaround with oneof fields or wrapper objects defined in google/protobuf/wrappers.proto. The version 3.12 adds a new exprimental support for field presence allowing to solve this problem.

Explicit presence with optional keyword

The new optional keyword providing by the experimental support allows to support explicitly field presence. It means that, when a field is optional it will be serialize if the field value was initialized (even if it is set to a default value). On contrary, when optional is not used, default values are not serialized, by consequence, if field is set to a default value, it will not be serialized and during deserialization there is not way to retrieve the information if the field was initialized or not.

How to use Protocol Buffer with Gradle into your project

Use Protobuf plugin for Gradle

The Gradle plugin compiles Protobuf definition files (*.proto) in your project, it can be enabled by adding the following line to your build.gradle file.

1
2
3

plugins {
     id 'com.google.protobuf' version '0.8.13'
}

Configure path to the protoc executable

By default the Protobuf plugin needs that the protoc tool is available from the system path. If the tool is not installed, gradle will fails with the following error.

1
2

Execution failed for task ':generateProto'.
> java.io.IOException: Cannot run program "protoc": error=2, No such file or directory

Rather than installing it, the easiest way is to automatically download the tool from repositories by adding the following configuration into the build.gradle file.

1
2
3
4
5

protobuf {
     protoc {
        artifact = 'com.google.protobuf:protoc:3.14.0-rc-1'
    }
}

Customize location of generated files

By default, java files are generated into the build folder of your project. If you want to add those generated files into a versioning tool such as git, a better place should be into a source folder.

Using a source folder dedicated to the generated files seems a good practice, rather than to generate them with the other source files. The destination folder can be specified by initializing the property generatedFilesBaseDir into the build.gradle.

1
2
3

protobuf {
    generatedFilesBaseDir = "$projectDir/src/main/generated"
}

Do not forget to also add the destination folder into the source set of you project.

1
2
3
4
5
6

sourceSets {
    main {
        java {
            srcDir 'src/main/generated/main/java'
        }
}

It is also possible to specify the target package of the generate files directly into the Proto file by adding.

1

option java_package="it.is.my.target.package"

Customize location of the Proto files

By default, proto files are expected to be in the default folder src/main/proto. It can be changed, by addind a new srcDir in the proto source set, for instance, add the following to your build.gradle file if your proto files are located into your resources folder.

1
2
3
4
5
6
7

sourceSets {
    main {
        proto {
            srcDir 'src/main/resources/proto'
        }        
    }
}

Add Protocol Buffer dependencies

Java source files generated by the plugin needs Protocol Buffer libraries in order to compile without error, it can be done by adding the needed dependencies with the following lines:

1
2
3
4
5
6

dependencies {
    // Add dependency on protocol buffer
    compile 'com.google.protobuf:protobuf-java:4.0.0-rc-2'
    // Add dependency on utils to convert Protobuf to JSON
    compile 'com.google.protobuf:protobuf-java-util:4.0.0-rc-2'
}

How to use the experimental support with Protobuf gradle plugin

As described into this issue, the plugin Protobuf does not allow to configure the protoc gradle task with the option --experimental_allow_proto3_optional in order to enable the experimental support.

Nevertheless, it is easily to overcome this limitation by naming your Proto file by include the string test_proto3_optional either in the Proto filename or into the folder containing it.

Sample

The sample below uses two integer fields with only one using the optional keyword and it serializes/deserializes a message to show the behavior of the optional keyword.

1
2
3
4
5
6
7

syntax = "proto3";

message Account {
    string name = 1;
    optional int32 amountOptional = 2;
    int32 amount = 3;
}

Case 1: amountOptional and amount fields are not initialized

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

    Account account = Account.newBuilder().setName("AccountWithUnsetAmountOptionalAndAmount").build();
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    account.writeTo(baos);
    byte[] binary = baos.toByteArray();

    Account decodedAccount = Account.parseFrom(binary);
    System.out.print(decodedAccount.toString());    

    // It displays only the name and not fields that are not initialized
    name: "AccountWithUnsetAmountOptionalAndAmount"

Case 1: amountOptional and amount fields are explicitly initialized to 0 (default value of int32)

The sample show that field presence if take into account and that amountOptional

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

    Account account = Account.newBuilder().setName("AccountWithAmountOptionalAndAmountSetToZero")
                .setAmountOptional(0).setAmount(0).build();
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    account.writeTo(baos);
    byte[] binary = baos.toByteArray();

    Account decodedAccount = Account.parseFrom(binary);
    System.out.print(decodedAccount.toString());

    // It displays the name and the optional field that has been initialized
    name: "AccountWithAmountOptionalAndAmountSetToZero"
    amountOptional: 0

Case 3: amountOptional and amount fields are explicitly initialized to non default value

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

    Account account = Account.newBuilder().setName("AccountWithNonDefaultValue")
                .setAmountOptional(100).setAmount(50).build();
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    account.writeTo(baos);
    byte[] binary = baos.toByteArray();

    Account decodedAccount = Account.parseFrom(binary);
    System.out.print(decodedAccount.toString());

    // It displays all fields
    name: "AccountWithNonDefaultValue"
    amountOptional: 100
    amount: 50