Application service

Applications expose an gRPC service for remote configuration and data exchange. gRPC is an RPC framework which is based on HTTP/2 and Protocol Buffers (Protobuf). To use the gRPC API, you should get familiar with gRPC first. So we encourage you to have a look at the documentation.

The application service is defined in a .proto file which can be found here. The repository contains also descriptions for messages that are requested by inputs or provided by outputs. A full reference of the protocols can be found here.

Usage

This section describes the usage of the application service. The examples are based on the C++ API of gRPC but the usage is similar in other programming languages. Please see the gRPC docs for reference.

Generating the service stub and message classes

The Protobuf compiler protoc is used to generate service stubs and message classes for your programming language. Please make sure, protoc and the gRPC plugin (grpc_cpp_plugin for C++) for the Protobuf compiler is installed correctly. If everything is set up correctly, please run the following commands to create the files:

# set path to the root directory of the api directory
api_dir=<path_to_the_api_dir>

# generate message classes
protoc -I $api_dir --cpp_out=. $api_dir/hemistereo/**/*.proto

# generate gRPC service stubs for the application service
protoc -I $api_dir --grpc_out=. --plugin=protoc-gen-grpc=`which grpc_cpp_plugin` $api_dir/hemistereo/network/services/application.proto

The commands will generate a bunch of files in your current working directory:

.
└── hemistereo
    └── network
        ├── messages
        │   ├── basic.pb.cc
        │   ├── basic.pb.h
        │   ├── matrix.pb.cc
        │   ├── matrix.pb.h
        │   ├── message.pb.cc
        │   ├── message.pb.h
        │   ├── point.pb.cc
        │   └── point.pb.h
        └── services
            ├── application.grpc.pb.cc
            ├── application.grpc.pb.h
            ├── application.pb.cc
            └── application.pb.h

Note

For other languages than C++, the arguments given to protoc have to be changed. Please check the gRPC documentation.

Using the service stub

The file hemistereo/network/services/application.grpc.pb.h contains the classes that are necessary for the application service. To use it, first create a new channel by providing the address to the service endpoint on the sensor. It is also necessary to increase the message size limit for receiving the images and point cloud. This can be achieved by changing the default channel arguments:

grpc::ChannelArguments channelArgs;
// -1 set the message size to unlimited
channelArgs.SetMaxReceiveMessageSize(-1);

The channel arguments can now be used to create a new channel and service stub. By default, the API gateway is listening on port 8888:

// please replace <hostname_or_ip> with the hostname or the IP of the device
auto channel = grpc::CreateCustomChannel("<hostname_or_ip>:8888", grpc::InsecureChannelCredentials(), channelArgs);
auto stub = hs::network::proto::ApplicationService::NewStub(channel);

Note

While the following samples use the synchronous API, gRPC offers also an asynchronous API for interacting with the service. Please check the gRPC documentation for reference.

Setup the client context

The stub object provides functions for accessing the service. For each call of a stub function, a client context has to be provided. This context allows us to add some metadata which will be included in the HTTP header of the call. Because HemiStereo devices could run multiple apps at the same time, we have to add a header field app to address a specific application. The app field will be read from the API gateway to route the request to the right application. For example, to address the stereo application, create the client context the following way:

grpc::ClientContext context;
context.AddMetadata("app", "stereo");

Note

The client context should not be reused. Please create a new context object for each call of a service stub method.

Query the application structure

The function GetApplication can be used for querying the application structure. This gives the caller the information which inputs, outputs and properties are provided by the application. In C++, the request can be done the following way:

hs::network::proto::GetApplicationRequest request;
hs::network::proto::Node node;
auto status = stub->GetApplication(&context, request, &node);
if (!status.ok())
    throw std::runtime_error("failed to get application data from remote device: " + status.error_message());

node.PrintDebugString();

In this example, the content of the node object is printed to stdout.

Data exchange

Message format

The hs.network.proto.messages.Message message is used to exchange data. It contains the timestamp and the data itself packed into a google.protobuf.Any message. Use the PackFrom and UnpackTo methods from the Any type to fill or read the payload.

Reading data

Data is provided by the outputs of the application. There are two ways for getting the data: streaming and polling.

Streaming

Streaming can be used by calling the SubscribeTopic method. It returns a grpc::ClientReader object which can be used for reading the messages:

hs::network::proto::SubscribeTopicRequest request;
// set the path of the output
request.set_path(<path_to_the_output>);

auto reader = stub->SubscribeTopic(&context, request);

while (m_run)
{
    hs::network::proto::messages::Message msg;
    if (!reader->Read(&msg))
        break;

    // do something
    ...
}

auto status = reader->Finish();
if (!status.ok())
    std::cerr << "reading topic \"" << m_path << "\" failed:" << status.error_message() << std::endl;

Note

On high images resolutions or slow network connections, streaming can introduce a growing delay over time. In these cases, prefer polling to read data from the device.

Polling

For polling, data has to be cached internally. To save memory, caching is disabled by default. You can enable the output cache using the SetReadTopicEnabled method:

hs::network::proto::SetReadTopicEnabledRequest request;
// set the path of the output
request.set_path(<path_to_the_output>);
// enable ReadTopic
request.set_value(true);
hs::network::proto::SetReadTopicEnabledResponse response;

auto status = stub->SetReadTopicEnabled(&context, request, &response);
if (!status.ok())
    throw std::runtime_error("failed to set read topic enabled status of output \"" + m_path + "\": " +
                             status.error_message());

Then the ReadTopic method can be used to read the message from the application output:

hs::network::proto::ReadTopicRequest request;
// set the path of the output
request.set_path(<path_to_the_output>);
request.set_prev_message_id(0);
hs::network::proto::messages::Message msg;

auto status = stub->ReadTopic(&context, request, &msg);
if (!status.ok())
    throw std::runtime_error("failed to get read topic enabled status of output \"" + <path_to_the_output> + "\": " + status.error_message());

As you can see, a message id can be provided to the request. In the example above, it is set to zero. This instructs the service to return the data that is currently cached. If you poll in a loop, this can lead to receiving the same data multiple times. To avoid this, the message id of the previous message can be set. ReadTopic will then block until a messages with a newer message id is available.

uint32_t prevMsgId = 0;
while(true)
{
    hs::network::proto::ReadTopicRequest request;
    request.set_path(<path_to_the_output>);
    // set the previous message id here
    request.set_prev_message_id(prevMsgId);
    hs::network::proto::ReadTopicResponse response;

    auto status = stub->ReadTopic(&context, request, &response);
    if (!status.ok())
        throw std::runtime_error("failed to get read topic \"" + <path_to_the_output> + "\": " + status.error_message());

    // access message
    auto msg = response.data();

    // do something
    ...

    // keep the current message id
    prevMsgId = response.message_id();
}
Writing data

Apps can also offer inputs for receiving data from clients. To send a message to the app, WriteTopic can be used:

hs::network::proto::messages::Message msg;
// fill message
...

hs::network::proto::WriteTopicRequest request;
// set the path of the input
request.set_path(<path_to_the_output>);
request.set_data(msg);
hs::network::proto::WriteTopicResponse response;

auto status = stub->WriteTopic(&context, request, &response);
if (!status.ok())
    throw std::runtime_error("failed to get write topic \"" + <path_to_the_output> + "\": " + status.error_message());
Properties

Properties are used for the configuration of an application. At the moment, they are restricted to the following types: Int32, UInt32, Int64, UInt64, Float, Double, Bool, String, Enum.

To get the current value of the property you can use the GetProperty method:

hs::network::proto::GetPropertyRequest request;
// set the path of the input
request.set_path(<path_of_the_property>);
hs::network::proto::Property property;

auto status = stub->GetProperty(&context, request, &property);
if (!status.ok())
    throw std::runtime_error("failed to get property \"" + <path_of_the_property> + "\": " + status.error_message());

The property message has a oneof-field which contains a sub-message depending on the property type. For example, if you want to get the value of an Int32-Property, you have to access the prop_int32 field:

if (property.has_prop_int32())
{
    hs::network::proto::PropertyInt32 propInt32 = property.prop_int32();
    std::cout << propInt32.value() << std::endl;
}

Note

Depending on the property type, the messages contain also additional information (e.g. min, max, step).

To change the value of a property, use the SetProperty method:

hs::network::proto::SetPropertyRequest request;
// set the path of the input
request.set_path(<path_of_the_property>);

// set the property value, use set_value_<value_type> depending on the property type
request.set_value_int32(<int32_value>);
hs::network::proto::Property property;

auto status = stub->SetProperty(&context, request, &property);
if (!status.ok())
    throw std::runtime_error("failed to set property \"" + <path_of_the_property> + "\": " + status.error_message());

Here the property value is set directly in the SetPropertyRequest by setting the value_int32-field. If the property is of a different type, set another field accordingly.

Binning Mode

As described earlier, the custom application running on the sensor may expose several properties for tuning its behavior. One property which might be of special interest for many users is the camera binning mode.

Note

Binning is the (mathematical) combination of several camera pixels into one binned pixel, which therefore has a bigger sensor area and is more sensitive esp. in low-light conditions. The read-out time of the imager chip is decreased, which is beneficial for the rolling shutter effect. The IMX477 chip used in the HemiStereo NX sensor, offers 4:1 binning, meaning a decrease in horizontal and vertical camera resolution of 50%.

The binning mode is controlled by the camera mode, of which currently are only two available:

  • mode 0, which is the default 4032x3040 pixel mode, which is the native resolution of the camera chip

  • mode 1, which is the binned 2016x1520 pixel mode, where each 4 pixels are used in an averaged manner

The mode switch can be triggered using the application service interface, by setting the camera_mode property of the camera stereo source. The export name of this property is application specific, e.g. for the stereo app it is source_camera_camera_mode, which is a UInt64 value, representing the camera mode.

Protocol Documentation

hemistereo/network/messages/matrix.proto

A message representing a matrix.

Field

Type

Label

Description

rows

uint32

The number of rows.

cols

uint32

The number of columns.

channels

uint32

The number of channels.

depth

Matrix.Depth

The depth of the element data type.

data

bytes

The matrix data in row-major order.

Name

Number

Description

Unknown

0

Unknown depth

Int8

1

8-bit signed integer

UInt8

2

8-bit unsigned integer

Int16

3

16-bit signed integer

UInt16

4

16-bit unsigned integer

Int32

5

32-bit signed integer

UInt32

6

32-bit unsigned integer

Int64

7

64-bit signed integer

UInt64

8

64-bit unsigned integer

Float

9

32-bit floating point

Double

10

64-bit floating point

Top

hemistereo/network/messages/message.proto

Represents a message.

Field

Type

Label

Description

timestamp

int64

The message timestamp.

payload

google.protobuf.Any

The message payload.

type

string

The data type of the message payload.

Represents a message map.

Field

Type

Label

Description

map

MessageMap.MapEntry

repeated

A map that contains multiple messages accessible by name.

Top

hemistereo/network/services/application.proto

Field

Type

Label

Description

path

string

The path of the method.

data

messages.Message

The input parameter of the method.

Represents an event.

Field

Type

Label

Description

type

Event.Type

The type of the Event.

path

string

The path to the corresponding object.

input

Input

The Input message. Field is only set on an InputAdded event.

output

Output

The Output message. Field is only set on an OutputAdded event.

property

Property

The Property message. Field is only set on PropertyAdded and PropertyChanged events.

method

Method

The Method message. Field is only set on an MethodAdded event.

Represents a request to get the current application.

Represents a request to get a Node.

Field

Type

Label

Description

path

string

Represents the path to a Node to be requested.

Represents a request to get a Property.

Field

Type

Label

Description

path

string

Represents the path to a Property to be requested.

Represents a request to query if topic reads are enabled.

Field

Type

Label

Description

path

string

The path of the topic.

Represents the response to query if topic reads are enabled.

Field

Type

Label

Description

value

bool

Indicates if topic reads are enabled.

Represents the input to a Node.

Field

Type

Label

Description

type

string

Defines the type of input to the Node.

Field

Type

Label

Description

typeIn

string

Defines the type of the method's input parameter.

typeOut

string

Defines the type of the method's output parameter.

Represents a specific application with corresponding properties, inputs and outputs.

Represents the output from a Node.

Field

Type

Label

Description

type

string

Defines the type of output from a Node.

Represents a property containing a single datatype.

Field

Type

Label

Description

prop_int32

PropertyInt32

A 32 bit signed integer property.

prop_int64

PropertyInt64

A 64 bit signed integer property.

prop_uint32

PropertyUInt32

A 32 bit unsigned integer property.

prop_uint64

PropertyUInt64

A 64 bit unsigned integer property.

prop_float

PropertyFloat

A 32 bit floating point property.

prop_double

PropertyDouble

A 64 bit floating point property.

prop_string

PropertyString

A string property.

prop_bool

PropertyBool

A boolean property.

prop_enum

PropertyEnum

An enumeration property.

displayname

string

Name of the property.

description

string

Description of the property.

Represents a Boolean value.

Field

Type

Label

Description

value

bool

Value of the property, can be true or false.

Represents a double-precision floating-point number.

Field

Type

Label

Description

value

double

The property value.

min

double

Lower bound of the property.

max

double

Upper bound of the property.

Represents an enumeration type.

Field

Type

Label

Description

value

int32

The property value.

value_map

PropertyEnum.ValueMapEntry

repeated

The value map containing possible values and their string representation.

Represents a floating-point number.

Field

Type

Label

Description

value

float

The property value.

min

float

Lower bound of the property.

max

float

Upper bound of the property.

Represents a 32-bit signed integer.

Field

Type

Label

Description

value

int32

The property value.

min

int32

Lower bound of the property.

max

int32

Upper bound of the property.

Represents a 64-bit signed integer.

Field

Type

Label

Description

value

int64

The property value.

min

int64

Lower bound of the property.

max

int64

Upper bound of the property.

Represents a string type.

Field

Type

Label

Description

value

string

The property value.

Represents a 32-bit unsigned integer.

Field

Type

Label

Description

value

uint32

The property value.

min

uint32

Lower bound of the property.

max

uint32

Upper bound of the property.

Represents a 64-bit unsigned integer.

Field

Type

Label

Description

value

uint64

The property value.

min

uint64

Lower bound of the property.

max

uint64

Upper bound of the property.

Represents a request to read a topic.

Field

Type

Label

Description

path

string

The path of the topic.

prev_timestamp

int64

The timestamp of the previous message. The call will block until newer message is available.

Represents a request to set a Property.

Field

Type

Label

Description

path

string

Represents the path to the property to be set.

value_int32

int32

The value of a 32 bit signed integer property.

value_int64

int64

The value of a 64 bit signed integer property.

value_uint32

uint32

The value of a 32 bit unsigned integer property.

value_uint64

uint64

The value of a 64 bit unsigned integer property.

value_float

float

The value of a 32 bit floating point property.

value_double

double

The value of a 64 bit floating point property.

value_string

string

The value of a string property.

value_bool

bool

The value of a boolean property.

value_enum

int32

The value of an enumeration property.

Request to enable/disable topic reads.

Field

Type

Label

Description

path

string

The path of the topic.

value

bool

Enables or disables topic reads.

Represents a request to subscribe to events.

Represents a request to subscribe to a topic.

Field

Type

Label

Description

path

string

The path of the topic.

Represents a request to write to a topic.

Field

Type

Label

Description

path

string

The path of the topic.

data

messages.Message

The message that should be written.

Represents the response to write to a topic.

Name

Number

Description

Unknown

0

An unknown Event occured.

InputAdded

1

An Input has been added.

InputRemoved

2

An Input has been removed.

OutputAdded

3

An Output has been added.

OutputRemoved

4

An Output has been removed.

PropertyAdded

5

A Property has been added.

PropertyRemoved

6

A Property has been removed.

PropertyChanged

7

A Property has been changed.

MethodAdded

8

A Method has been added.

MethodRemoved

9

A Method has been removed.

Method Name

Request Type

Response Type

Description

GetApplication

GetApplicationRequest

Node

Used to get the current Application.

GetNode

GetNodeRequest

Node

Used to get the Node at {path}.

GetProperty

GetPropertyRequest

Property

Used to get the property at {path}.

SetProperty

SetPropertyRequest

Property

Used to set a property at {path} to {value}.

SubscribeTopic

SubscribeTopicRequest

messages.Message stream

Used to subscribe to updates to a topic at {path}.

ReadTopic

ReadTopicRequest

messages.Message

Used to read a topic at {path}.

GetReadTopicEnabled

GetReadTopicEnabledRequest

GetReadTopicEnabledResponse

Used to query if topic reads are enabled for topic at {path}.

SetReadTopicEnabled

SetReadTopicEnabledRequest

SetReadTopicEnabledResponse

Used to enable/disable topic reads for topic at {path}.

WriteTopic

WriteTopicRequest

WriteTopicResponse

Used to write data to a topic at {path}.

SubscribeEvents

SubscribeEventsRequest

Event stream

Used to subscribe to events of the current application.

CallMethod

CallRequest

messages.Message

Used to call a method at {path}.

Scalar Value Types

.proto Type

Notes

C++

Java

Python

Go

C#

PHP

Ruby

double

double

double

float

float64

double

float

Float

float

float

float

float

float32

float

float

Float

int32

Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead.

int32

int

int

int32

int

integer

Bignum or Fixnum (as required)

int64

Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead.

int64

long

int/long

int64

long

integer/string

Bignum

uint32

Uses variable-length encoding.

uint32

int

int/long

uint32

uint

integer

Bignum or Fixnum (as required)

uint64

Uses variable-length encoding.

uint64

long

int/long

uint64

ulong

integer/string

Bignum or Fixnum (as required)

sint32

Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s.

int32

int

int

int32

int

integer

Bignum or Fixnum (as required)

sint64

Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s.

int64

long

int/long

int64

long

integer/string

Bignum

fixed32

Always four bytes. More efficient than uint32 if values are often greater than 2^28.

uint32

int

int

uint32

uint

integer

Bignum or Fixnum (as required)

fixed64

Always eight bytes. More efficient than uint64 if values are often greater than 2^56.

uint64

long

int/long

uint64

ulong

integer/string

Bignum

sfixed32

Always four bytes.

int32

int

int

int32

int

integer

Bignum or Fixnum (as required)

sfixed64

Always eight bytes.

int64

long

int/long

int64

long

integer/string

Bignum

bool

bool

boolean

boolean

bool

bool

boolean

TrueClass/FalseClass

string

A string must always contain UTF-8 encoded or 7-bit ASCII text.

string

String

str/unicode

string

string

string

String (UTF-8)

bytes

May contain any arbitrary sequence of bytes.

string

ByteString

str

[]byte

ByteString

string

String (ASCII-8BIT)