Python - Getting Started

This section gives a brief introduction into the Python SDK / wrapper modules for the gRPC network interfaces, described here.

For this section, the app always refers to the running application service providing backend application, e.g. the pre-installed e.g. the stereo app.

Overview

Repository

The python modules are available from public sources and can be downloaded from gitlab. It can be installed locally on the sensor or your host machine and accesses the raw application service data via gRPC network interfaces.

Next to the application service gRPC api files, the git repository contains following modules:

context

This module contains a Context class for accessing the gRPC functionalities provided by an hemistereo application service providing app, e.g. the stereo app. Results, meaning the gRPC response messages are returned as-is from the Context class member methods. For unpacking, the unpack submodule contains several helper functions. For details, see below.

unpack

This submodule consists of several (free) functions for converting proto response messages into more pythonic types, mostly numpy ndarrays or base types. For details, see below.

imgconvert

This submodule provides simple to use (free) helper functions for plotting of raw images and pointcloud data aquired from the app using numpy, matplotlib and pyntcloud. It is mainly made for jupyter-notebook use. For details, see below.

Jupyter Notebook

The easiest way of getting started with python and the HemiStereo NX platform is installing of the 3dvl-jupyter custom application on the HemiStereo NX sensor.

This provides a jupyter-notebook application on the sensor, which is a python powered environment accessible via web-browser from your host PC. The computations are taking place on the sensor and results are presented as web elements, e.g. a rendered image which is shown in your browser session.

The default 3dvl-jupyter application contains the jupyter-notebook itself, as well as several python modules, which can be used for data processing and AI computations. The app contains: numpy, scipy, matplotlib, pandas, opencv, numba, future, torch, pytorch/vision and grpc. Cuda usage is enabled and possible, especially for pytorch.

For the following steps, you should be logged into the system, as described here and should be familar with the apt based package management and update process, described here.

installation

The jupyter notebook application can be installed via command line:

sudo apt update
sudo apt install 3dvl-jupyter
sudo hemistereo_app_run jupyter-notebook
access

The jupyter notebook, per default, is listening on port 8080 of the sensor, and https is mandatory. The notebook app uses a self-signed ssl certificate, which creates a warning for all browsers on first access.

Go to https://SENSOR_IP:8880 to access the running notebook. For access, the password is 3dvisionlabs.

usage

The notebook application presents the folder contents of /var/lib/hemistereo/data/jupyter-notebook as read and writable directory. You may create new notebooks via the browser, but it is also possible to access the data via the sensors file-system.

For getting started, there is a samples directory linked as read-only filesystem. The origin is located at /usr/local/lib/hemistereo/jupyter-notebook/samples. The folder contains a copy samples notebook as well, opening and executing the notebook, will create a copy of all samples in your writable area. The files will be copied only if there is not a notebook with the same name present already.

Note

as mentioned in the technical specs, the binning mode (camera mode with lower resolution but higher sensitivity) can be enabled via software. For demonstration purposes, a sample notebook has been prepared called Binning, which can be found in the samples folder.

Datatypes

When working with the python SDK, the messages sent to and received from the backend application are serialized in gRPC/Protobuf format. For the backend to be able to deserialize the messages, it is necessary to use datatypes, interoperable with C++ (the backend language). For this, numpy provides fixed-size types which are 1:1 transferrable to C++, e.g. np.uint8. The shall be used for all grpc messages.

Note

for getting an overview of all supported topics (output nodes) and respective types available as well as all currently active properties, with applicable value ranges (if any), their current value and their type info, the getApplication() RPC call can be used.

Writing a property of type bool

Here ctx is an pre-initialized Context instance, which is “connected” to a backend application.

The getApplication() RPC method is used for examining the property, then the setProperty() RPC method is used for setting a new value.

ctx.getApplication()
[...]
properties {
  key: "distance_enabled"
  value {
    prop_bool {
      value: true
    }
  }
}
[...]

ctx.setProperty("distance_enabled",np.bool(1))

Writing a property of type int32

Here ctx is an pre-initialized Context instance, which is “connected” to a backend application.

The getApplication() RPC method is used for examining the property, then the setProperty() RPC method is used for setting a new value.

ctx.getApplication()
[...]
properties {
  key: "stereo_sgm_max_disparity"
  value {
    prop_int32 {
      value: 128
      min: 16
      max: 256
      step: 4
    }
    displayname: "Max. disparity"
    description: "The maximum disparity value."
  }
}
[...]

ctx.setProperty("stereo_sgm_max_disparity", np.uint32(64))

Writing a property of type enum

Here ctx is an pre-initialized Context instance, which is “connected” to a backend application.

The getApplication() RPC method is used for examining the property, then the setProperty() RPC method is used for setting a new value.

ctx.getApplication()
[...]
properties {
  key: "stereo_target_camera_model"
  value {
    prop_enum {
      value: 1
      value_map {
        key: 1
        value: "Equiangular"
      }
      value_map {
        key: 4
        value: "Pinhole"
      }
      value_map {
        key: 5
        value: "Equirectangular"
      }
    }
    displayname: "Target projection model"
    description: "The projection model of the target image."
  }
}
[...]

ctx.setProperty("stereo_target_camera_model", 4)

Here, the python integer 4 is used. Internally, python type integers are treated as enum values by the SDK.

Reading the distance information

Here ctx is an pre-initialized Context instance, which is “connected” to a backend application.

The getApplication() RPC method is used for examining the property, then the readTopic() RPC method is used for setting a new value.

As can be seen, the request will return a matrix object, filled with uint16 elements. These represent (in the case of the distance “image”), the actual distance values in mm.

The output message can be easily deserialized into numpy using unpackMessageToNumpy().

ctx.getApplication()
[...]
outputs {
  key: "distance"
  value {
    type: "hs::types::Matrix<uint16, 1>"
  }
}
[...]

message = ctx.readTopic("distance")
dist = unpackMessageToNumpy( message.data )

Now, the dist object is a numpy array in image dimensions, with each pixel representing the distance.

Reading the RGB image

Here ctx is an pre-initialized Context instance, which is “connected” to a backend application.

The getApplication() RPC method is used for examining the property, then the readTopic() RPC method is used for setting a new value.

As can be seen, the request will return a matrix object, filled with 3-tuples of uint8 elements. These represent (in the case of colored images), the RGB values, represented as uint8 each.

The output message can be easily deserialized into numpy using unpackMessageToNumpy().

ctx.getApplication()
[...]
outputs {
  key: "image"
  value {
    type: "hs::types::Matrix<uint8, 3>"
  }
}
[...]

message = ctx.readTopic("image")
rgb = unpackMessageToNumpy( message.data )

Now, the rgb object is a numpy array in image dimensions, representing the color values of each pixel.

context

The context class manages the gRPC connection to the sensor and a specific application. It opens up a communication channel and gives easy to use access to the most important features of the application.

__init__(self, appname, server, port)

The initializer takes three parameters and directly opens a communication channel.

  • appname: which is a string, naming the data providing app to connect to, e.g. the “stereo” app, which comes pre-installed with the sensor.

  • server (None): which is a resolvable hostname or IP address of the target sensor, leave blank (None) if you want to let python figure out the local host (from the containers default route) automatically.

  • port (8888): this is the port of the reverse-proxy (usually 8080) to access the application service (gRPC) interface.

getApplication(self)

Returns a list of all parameters and outputs available for this application as dictionary object.

setReadTopicEnabled(self, topic)

The application can reduce data conversion/serialization overhead by only enabling outputs which are actually requested. The setReadTopicEnabled method is used to enable (or disable) the output. Parameters are:

  • topic: the output topic (name of an output) to be enabled

  • enabled (True): whether to enable or disable the topic

readTopic(self, topic)

Read the specified output value. The output gets enabled automatically on first call.

  • topic: the output topic (name of an output) to be read

Returns a proto message object for further processing.

Note

If the application is restarted without re-initializing the Context class, the readTopic does not get re-enabled automatically. It is recommended to re-initialize the Context class when restarting the sensor / application.

writeTopic(self, topic, value)

Write the specified input value.

  • topic: the output topic (name of an output) to be enabled

  • value: a serialized proto-message to be written.

Returns a proto message object, indicating success or failure.

getProperty(self, prop)

Read the specified property value.

  • prop: the property name to be read

Returns a proto message object for further processing.

setProperty(self, prop, value)

Write the specified property value.

  • prop: the property name to be read

  • value: the value (usually as numpy object) to be written.

Returns a proto message object indicating success or failure.

unpack

This submodule contains several unpack helpers for e.g. the getProperty or readTopic methods of the Context class.

unpackMessageToInt64(message)

Unpacks the raw proto message into a typed proto message, which can be handled by python.

  • message: as received from e.g. getProperty

Returns a hemistereo.api.hemistereo.network.messages.basic_pb2.Int64, which can be handled by python.

unpackMessageToInt32(message)

Unpacks the raw proto message into a typed proto message, which can be handled by python.

  • message: as received from e.g. getProperty

Returns a hemistereo.api.hemistereo.network.messages.basic_pb2.Int32, which can be handled by python.

unpackMessageToUInt32(message)

Unpacks the raw proto message into a typed proto message, which can be handled by python.

  • message: as received from e.g. getProperty

Returns a hemistereo.api.hemistereo.network.messages.basic_pb2.UInt32, which can be handled by python.

unpackMessageToInt64(message)

Unpacks the raw proto message into a typed proto message, which can be handled by python.

  • message: as received from e.g. getProperty

Returns a hemistereo.api.hemistereo.network.messages.basic_pb2.Int64, which can be handled by python.

unpackMessageToUInt64(message)

Unpacks the raw proto message into a typed proto message, which can be handled by python.

  • message: as received from e.g. getProperty

Returns a hemistereo.api.hemistereo.network.messages.basic_pb2.UInt64, which can be handled by python.

unpackMessageToFloat(message)

Unpacks the raw proto message into a typed proto message, which can be handled by python.

  • message: as received from e.g. getProperty

Returns a hemistereo.api.hemistereo.network.messages.basic_pb2.Float, which can be handled by python.

unpackMessageToDouble(message)

Unpacks the raw proto message into a typed proto message, which can be handled by python.

  • message: as received from e.g. getProperty

Returns a hemistereo.api.hemistereo.network.messages.basic_pb2.Double, which can be handled by python.

unpackMessageToBool(message)

Unpacks the raw proto message into a typed proto message, which can be handled by python.

  • message: as received from e.g. getProperty

Returns a hemistereo.api.hemistereo.network.messages.basic_pb2.Bool, which can be handled by python.

unpackMessageToString(message)

Unpacks the raw proto message into a typed proto message, which can be handled by python.

  • message: as received from e.g. getProperty

Returns a hemistereo.api.hemistereo.network.messages.basic_pb2.String, which can be handled by python.

unpackMessageToMat(message)

Unpacks the raw proto message into a typed proto message, which can be handled by python.

  • message: as received from e.g. readTopic

Returns a hemistereo.api.hemistereo.network.messages.matrix_pb2.Matrix, which can be handled by python.

unpackMessageToNumpy(message)

Unpacks the raw proto message into a numpy ndarray (makes sense only for Matrix types and might be a better option over unpackMessageToMat as numpy is easier to use)

  • message: as received from e.g. readTopic

Returns a numpy ndarray which represents a matrix, e.g. an image, depth map or point cloud.

imgconvert

This submodule contains several helpers for plotting matrix and point-cloud data. The most common types of stereo result data are: images (a Matrix with three int8 channels, representing a RGB image), depth maps (a Matrix with one float or int16 channel, representing a distance-from-camera value and point-cloud data (a matrix of XYZ coordinates representing vertex positions in a world-coordinate system). The imgconvert module helps to transfer the raw matrix data into colored images.

formatNumpyToRGB(np, equalize_histogram, color_map)

A helper function for converting matrix types into rgb (opencv) images. The method applies for both, image matrices which have rgb information already and depth maps, which are colorized via a color_map. The equalize_histogram and color_map arguments are applicable for depth maps only.

  • np the numpy-converted matrix object to be transformed into an image. Usually this is aquired via context.Context.readTopic and unpack.unpackMessageToNumpy.

  • equalize_histogram (True): whether or not to equalize the resulting color map (making the colors more evenly distributed). Only applicable for depth map conversions.

  • color_map (jet): the color map scheme to be used for depth map colorization. Only applicable for depth map conversion. Possible values are: jet, winter, bone, ocean, cool, hot and rainbow.

Returns a numpy ndarray object with 3 channels, representing the rgb components of the image. It can be used for opencv operations directly.

formatNumpyToPointCloud(pyc, image, filter_nan, max_dist, scale, every)

A helper function for converting a matrix point-cloud as received by context.Context.readTopic into a (optionally colorized) pointcloud (which also is a numpy ndarray but with a different column layout.

  • pyc: a numpy ndarray representation of the point-cloud as aquired via context.Context.readTopic and unpack.unpackMessageToNumpy.

  • image (None): optional, an RGB convert (as aquired via formatNumpyToRGB) used for colorization of the point-cloud data

  • filter_nan (True): recommended, filter-out all invalid points to avoid plotting issues

  • max_dist (None): optional, used to filter out out-of-range points, which might be noise-artifacts

  • scale (0.01): HemiStereo uses mm as measurement unit, pyntcloud works best for data of magnitude 1. As most sensor data represents spacial information of room-size (around several meters), the scale is for plotting purposes.

  • every (10): strip down the original data set by only using every nth data point, helping the processing time and pointcloud viewer.

Returns a numpy ndarray object with 3 or 6 (colorized) channels, which can be used with the pyntcloud plotter.

rotateCounterClockWise(image)

Wrapper for the opencv cv2.rotate(image, cv2.ROTATE_90_COUNTERCLOCKWISE method, usable for RGB converted matrices.

  • image: an RGB convert (as aquired via formatNumpyToRGB)

Returns a numpy ndarray object with 3 channels, representing the rgb components of the image. It can be used for opencv operations directly.

rotateClockWise(image)

Wrapper for the opencv cv2.rotate(image, cv2.ROTATE_90_CLOCKWISE method, usable for RGB converted matrices.

  • image: an RGB convert (as aquired via formatNumpyToRGB)

Returns a numpy ndarray object with 3 channels, representing the rgb components of the image. It can be used for opencv operations directly.

plotRGB(image, figsize)

Use matplotlib for plotting the RGB converted matrix (image). For use inside jupyter-notebooks.

  • image: an RGB convert (as aquired via formatNumpyToRGB)

  • figsize ((45,30): the scale of the plot

plotPCL(pyc)

Use pyntcloud for converting and plotting the pointcloud data as threejs javascript plot. For use inside jupyter-notebooks.

  • pyc: an pointcloud convert (as aquired via formatNumpyToPointCloud)

Returns the pyntcloud plot object.