Python - Getting Started

This section gives a brief introduction into the Python SDK / wrapper modules for the gRPC network interfaces, described here.

For this section, the app always refers to the running application service providing backend application, e.g. the pre-installed e.g. the stereo app.

Overview

Repository

The python modules are available from public sources and can be downloaded from gitlab. It can be installed locally on the sensor or your host machine and accesses the raw application service data via gRPC network interfaces.

Next to the application service gRPC api files, the git repository contains following modules:

context: This module contains a Context class for accessing the gRPC functionalities provided by an hemistereo application service providing app, e.g. the stereo app. Results, meaning the gRPC response messages are returned as-is from the Context class member methods. For unpacking, the unpack submodule contains several helper functions. For details, see below.
unpack: This submodule consists of several (free) functions for converting proto response messages into more pythonic types, mostly numpy ndarrays or base types. For details, see below.
imgconvert: This submodule provides simple to use (free) helper functions for plotting of raw images and pointcloud data aquired from the app using numpy, matplotlib and pyntcloud. It is mainly made for jupyter-notebook use. For details, see below.

Jupyter Notebook

The easiest way of getting started with python and the HemiStereo NX platform is installing of the 3dvl-jupyter custom application on the HemiStereo NX sensor.

This provides a jupyter-notebook application on the sensor, which is a python powered environment accessible via web-browser from your host PC. The computations are taking place on the sensor and results are presented as web elements, e.g. a rendered image which is shown in your browser session.

The default 3dvl-jupyter application contains the jupyter-notebook itself, as well as several python modules, which can be used for data processing and AI computations. The app contains: numpy, scipy, matplotlib, pandas, opencv, numba, future, torch, pytorch/vision and grpc. Cuda usage is enabled and possible, especially for pytorch.

For the following steps, you should be logged into the system, as described here and should be familar with the apt based package management and update process, described here.

installation: The jupyter notebook application can be installed via command line:

sudo apt update
sudo apt install 3dvl-jupyter
sudo hemistereo_app_run jupyter-notebook

access

The jupyter notebook, per default, is listening on port 8080 of the sensor, and https is mandatory. The notebook app uses a self-signed ssl certificate, which creates a warning for all browsers on first access.

Go to https://SENSOR_IP:8880 to access the running notebook. For access, the password is 3dvisionlabs.

usage

The notebook application presents the folder contents of /var/lib/hemistereo/data/jupyter-notebook as read and writable directory. You may create new notebooks via the browser, but it is also possible to access the data via the sensors file-system.

For getting started, there is a samples directory linked as read-only filesystem. The origin is located at /usr/local/lib/hemistereo/jupyter-notebook/samples. The folder contains a copy samples notebook as well, opening and executing the notebook, will create a copy of all samples in your writable area. The files will be copied only if there is not a notebook with the same name present already.

Note

as mentioned in the technical specs, the binning mode (camera mode with lower resolution but higher sensitivity) can be enabled via software. For demonstration purposes, a sample notebook has been prepared called Binning, which can be found in the samples folder.

Datatypes

When working with the python SDK, the messages sent to and received from the backend application are serialized in gRPC/Protobuf format. For the backend to be able to deserialize the messages, it is necessary to use datatypes, interoperable with C++ (the backend language). For this, numpy provides fixed-size types which are 1:1 transferrable to C++, e.g. np.uint8. The shall be used for all grpc messages.

Note

for getting an overview of all supported topics (output nodes) and respective types available as well as all currently active properties, with applicable value ranges (if any), their current value and their type info, the getApplication() RPC call can be used.

Writing a property of type bool

Here ctx is an pre-initialized Context instance, which is “connected” to a backend application.

The getApplication() RPC method is used for examining the property, then the setProperty() RPC method is used for setting a new value.

ctx.getApplication()
[...]
properties {
  key: "distance_enabled"
  value {
    prop_bool {
      value: true
    }
  }
}
[...]

ctx.setProperty("distance_enabled",np.bool(1))

Writing a property of type int32

Here ctx is an pre-initialized Context instance, which is “connected” to a backend application.

The getApplication() RPC method is used for examining the property, then the setProperty() RPC method is used for setting a new value.

ctx.getApplication()
[...]
properties {
  key: "stereo_sgm_max_disparity"
  value {
    prop_int32 {
      value: 128
      min: 16
      max: 256
      step: 4
    }
    displayname: "Max. disparity"
    description: "The maximum disparity value."
  }
}
[...]

ctx.setProperty("stereo_sgm_max_disparity", np.uint32(64))

Writing a property of type enum

Here ctx is an pre-initialized Context instance, which is “connected” to a backend application.

The getApplication() RPC method is used for examining the property, then the setProperty() RPC method is used for setting a new value.

ctx.getApplication()
[...]
properties {
  key: "stereo_target_camera_model"
  value {
    prop_enum {
      value: 1
      value_map {
        key: 1
        value: "Equiangular"
      }
      value_map {
        key: 4
        value: "Pinhole"
      }
      value_map {
        key: 5
        value: "Equirectangular"
      }
    }
    displayname: "Target projection model"
    description: "The projection model of the target image."
  }
}
[...]

ctx.setProperty("stereo_target_camera_model", 4)

Here, the python integer 4 is used. Internally, python type integers are treated as enum values by the SDK.

Reading the distance information

Here ctx is an pre-initialized Context instance, which is “connected” to a backend application.

The getApplication() RPC method is used for examining the property, then the readTopic() RPC method is used for setting a new value.

As can be seen, the request will return a matrix object, filled with uint16 elements. These represent (in the case of the distance “image”), the actual distance values in mm.

The output message can be easily deserialized into numpy using unpackMessageToNumpy().

ctx.getApplication()
[...]
outputs {
  key: "distance"
  value {
    type: "hs::types::Matrix<uint16, 1>"
  }
}
[...]

message = ctx.readTopic("distance")
dist = unpackMessageToNumpy( message.data )

Now, the dist object is a numpy array in image dimensions, with each pixel representing the distance.

Reading the RGB image

Here ctx is an pre-initialized Context instance, which is “connected” to a backend application.

The getApplication() RPC method is used for examining the property, then the readTopic() RPC method is used for setting a new value.

As can be seen, the request will return a matrix object, filled with 3-tuples of uint8 elements. These represent (in the case of colored images), the RGB values, represented as uint8 each.

The output message can be easily deserialized into numpy using unpackMessageToNumpy().

ctx.getApplication()
[...]
outputs {
  key: "image"
  value {
    type: "hs::types::Matrix<uint8, 3>"
  }
}
[...]

message = ctx.readTopic("image")
rgb = unpackMessageToNumpy( message.data )

Now, the rgb object is a numpy array in image dimensions, representing the color values of each pixel.

context

The context class manages the gRPC connection to the sensor and a specific application. It opens up a communication channel and gives easy to use access to the most important features of the application.

__init__(self, appname, server, port)

The initializer takes three parameters and directly opens a communication channel.

appname: which is a string, naming the data providing app to connect to, e.g. the “stereo” app, which comes pre-installed with the sensor.
server (None): which is a resolvable hostname or IP address of the target sensor, leave blank (None) if you want to let python figure out the local host (from the containers default route) automatically.
port (8888): this is the port of the reverse-proxy (usually 8080) to access the application service (gRPC) interface.

getApplication(self)

Returns a list of all parameters and outputs available for this application as dictionary object.

setReadTopicEnabled(self, topic)

The application can reduce data conversion/serialization overhead by only enabling outputs which are actually requested. The setReadTopicEnabled method is used to enable (or disable) the output. Parameters are:

topic: the output topic (name of an output) to be enabled
enabled (True): whether to enable or disable the topic

readTopic(self, topic)

Read the specified output value. The output gets enabled automatically on first call.

topic: the output topic (name of an output) to be read

Returns a proto message object for further processing.

Note

If the application is restarted without re-initializing the Context class, the readTopic does not get re-enabled automatically. It is recommended to re-initialize the Context class when restarting the sensor / application.

writeTopic(self, topic, value)

Write the specified input value.

topic: the output topic (name of an output) to be enabled
value: a serialized proto-message to be written.

Returns a proto message object, indicating success or failure.

getProperty(self, prop)

Read the specified property value.

prop: the property name to be read

Returns a proto message object for further processing.

setProperty(self, prop, value)

Write the specified property value.

prop: the property name to be read
value: the value (usually as numpy object) to be written.

Returns a proto message object indicating success or failure.

unpack

This submodule contains several unpack helpers for e.g. the getProperty or readTopic methods of the Context class.

unpackMessageToInt64(message)