.. _python_getting_started: Python - Getting Started ######################## This section gives a brief introduction into the Python SDK / wrapper modules for the gRPC network interfaces, described :ref:`here `. For this section, *the app* always refers to the running :ref:`application service ` providing backend application, e.g. the pre-installed e.g. the :ref:`stereo app `. Overview ======== Repository ---------- The python modules are available from public sources and can be downloaded `from gitlab `_. It can be installed locally on the sensor or your host machine and accesses the raw :ref:`application service ` data via gRPC network interfaces. Next to the application service gRPC api files, the git repository contains following modules: **context** This module contains a :code:`Context` class for accessing the gRPC functionalities provided by an hemistereo application service providing app, e.g. the :ref:`stereo app `. Results, meaning the gRPC response messages are returned as-is from the :code:`Context` class member methods. For unpacking, the :code:`unpack` submodule contains several helper functions. For details, see :ref:`below `. **unpack** This submodule consists of several (free) functions for converting proto response messages into more pythonic types, mostly :code:`numpy` ndarrays or base types. For details, see :ref:`below `. **imgconvert** This submodule provides simple to use (free) helper functions for plotting of raw images and pointcloud data aquired from the app using :code:`numpy`, :code:`matplotlib` and :code:`pyntcloud`. It is mainly made for :code:`jupyter-notebook` use. For details, see :ref:`below `. .. _jupyter_notebook: Jupyter Notebook ---------------- The easiest way of getting started with python and the HemiStereo NX platform is installing of the :code:`3dvl-jupyter` custom application on the HemiStereo NX sensor. This provides a :code:`jupyter-notebook` application on the sensor, which is a python powered environment accessible via web-browser from your host PC. The computations are taking place on the sensor and results are presented as web elements, e.g. a rendered image which is shown in your browser session. The default :code:`3dvl-jupyter` application contains the :code:`jupyter-notebook` itself, as well as several python modules, which can be used for data processing and AI computations. The app contains: :code:`numpy, scipy, matplotlib, pandas, opencv, numba, future, torch, pytorch/vision` and :code:`grpc`. Cuda usage is enabled and possible, especially for :code:`pytorch`. For the following steps, you should be logged into the system, as described :ref:`here ` and should be familar with the :code:`apt` based package management and update process, described :ref:`here `. **installation** The jupyter notebook application can be installed via command line: .. code-block:: bash sudo apt update sudo apt install 3dvl-jupyter sudo hemistereo_app_run jupyter-notebook **access** The jupyter notebook, per default, is listening on port 8080 of the sensor, and https is mandatory. The notebook app uses a self-signed ssl certificate, which creates a warning for all browsers on first access. Go to `https://SENSOR_IP:8880 `_ to access the running notebook. For access, the password is :code:`3dvisionlabs`. **usage** The notebook application presents the folder contents of :code:`/var/lib/hemistereo/data/jupyter-notebook` as read and writable directory. You may create new notebooks via the browser, but it is also possible to access the data via the sensors file-system. For getting started, there is a samples directory linked as read-only filesystem. The origin is located at :code:`/usr/local/lib/hemistereo/jupyter-notebook/samples`. The folder contains a :code:`copy samples` notebook as well, opening and executing the notebook, will create a copy of all samples in your writable area. The files will be copied only if there is not a notebook with the same name present already. .. note:: as mentioned in the :ref:`technical specs `, the binning mode (camera mode with lower resolution but higher sensitivity) can be enabled via software. For demonstration purposes, a sample notebook has been prepared called :code:`Binning`, which can be found in the samples folder. Datatypes ========= When working with the python SDK, the messages sent to and received from the backend application are serialized in gRPC/Protobuf format. For the backend to be able to deserialize the messages, it is necessary to use datatypes, interoperable with C++ (the backend language). For this, numpy provides fixed-size types which are 1:1 transferrable to C++, e.g. np.uint8. The shall be used for all grpc messages. .. note:: for getting an overview of all supported topics (output nodes) and respective types available as well as all currently active properties, with applicable value ranges (if any), their current value and their type info, the :code:`getApplication()` RPC call can be used. Writing a property of type bool ------------------------------- Here :code:`ctx` is an pre-initialized :ref:`Context ` instance, which is "connected" to a backend application. The :code:`getApplication()` RPC method is used for examining the property, then the :code:`setProperty()` RPC method is used for setting a new value. .. code-block:: py ctx.getApplication() [...] properties { key: "distance_enabled" value { prop_bool { value: true } } } [...] ctx.setProperty("distance_enabled",np.bool(1)) Writing a property of type int32 -------------------------------- Here :code:`ctx` is an pre-initialized :ref:`Context ` instance, which is "connected" to a backend application. The :code:`getApplication()` RPC method is used for examining the property, then the :code:`setProperty()` RPC method is used for setting a new value. .. code-block:: py ctx.getApplication() [...] properties { key: "stereo_sgm_max_disparity" value { prop_int32 { value: 128 min: 16 max: 256 step: 4 } displayname: "Max. disparity" description: "The maximum disparity value." } } [...] ctx.setProperty("stereo_sgm_max_disparity", np.uint32(64)) Writing a property of type enum ------------------------------- Here :code:`ctx` is an pre-initialized :ref:`Context ` instance, which is "connected" to a backend application. The :code:`getApplication()` RPC method is used for examining the property, then the :code:`setProperty()` RPC method is used for setting a new value. .. code-block:: py ctx.getApplication() [...] properties { key: "stereo_target_camera_model" value { prop_enum { value: 1 value_map { key: 1 value: "Equiangular" } value_map { key: 4 value: "Pinhole" } value_map { key: 5 value: "Equirectangular" } } displayname: "Target projection model" description: "The projection model of the target image." } } [...] ctx.setProperty("stereo_target_camera_model", 4) Here, the python integer 4 is used. Internally, python type integers are treated as enum values by the SDK. Reading the distance information -------------------------------- Here :code:`ctx` is an pre-initialized :ref:`Context ` instance, which is "connected" to a backend application. The :code:`getApplication()` RPC method is used for examining the property, then the :code:`readTopic()` RPC method is used for setting a new value. As can be seen, the request will return a matrix object, filled with uint16 elements. These represent (in the case of the distance "image"), the actual distance values in *mm*. The output message can be easily deserialized into numpy using :code:`unpackMessageToNumpy()`. .. code-block:: py ctx.getApplication() [...] outputs { key: "distance" value { type: "hs::types::Matrix" } } [...] message = ctx.readTopic("distance") dist = unpackMessageToNumpy( message.data ) Now, the :code:`dist` object is a numpy array in image dimensions, with each pixel representing the distance. Reading the RGB image --------------------- Here :code:`ctx` is an pre-initialized :ref:`Context ` instance, which is "connected" to a backend application. The :code:`getApplication()` RPC method is used for examining the property, then the :code:`readTopic()` RPC method is used for setting a new value. As can be seen, the request will return a matrix object, filled with 3-tuples of uint8 elements. These represent (in the case of colored images), the RGB values, represented as uint8 each. The output message can be easily deserialized into numpy using :code:`unpackMessageToNumpy()`. .. code-block:: py ctx.getApplication() [...] outputs { key: "image" value { type: "hs::types::Matrix" } } [...] message = ctx.readTopic("image") rgb = unpackMessageToNumpy( message.data ) Now, the :code:`rgb` object is a numpy array in image dimensions, representing the color values of each pixel. .. _python_context: context ============= The context class manages the gRPC connection to the sensor and a specific application. It opens up a communication channel and gives easy to use access to the most important features of the application. **__init__(self, appname, server, port)** The initializer takes three parameters and directly opens a communication channel. - *appname*: which is a string, naming the data providing app to connect to, e.g. the "stereo" app, which comes pre-installed with the sensor. - *server* (:code:`None`): which is a resolvable hostname or IP address of the target sensor, leave blank (:code:`None`) if you want to let python figure out the local host (from the containers default route) automatically. - *port* (:code:`8888`): this is the port of the reverse-proxy (usually 8080) to access the :ref:`application service ` (gRPC) interface. **getApplication(self)** Returns a list of all parameters and outputs available for this application as dictionary object. **setReadTopicEnabled(self, topic)** The application can reduce data conversion/serialization overhead by only enabling outputs which are actually requested. The setReadTopicEnabled method is used to enable (or disable) the output. Parameters are: - *topic*: the output topic (name of an output) to be enabled - *enabled* (:code:`True`): whether to enable or disable the topic **readTopic(self, topic)** Read the specified output value. The output gets enabled automatically on first call. - *topic*: the output topic (name of an output) to be read Returns a proto message object for further processing. .. note:: If the application is restarted without re-initializing the :code:`Context` class, the readTopic does not get re-enabled automatically. It is recommended to re-initialize the :code:`Context` class when restarting the sensor / application. **writeTopic(self, topic, value)** Write the specified input value. - *topic*: the output topic (name of an output) to be enabled - *value*: a serialized proto-message to be written. Returns a proto message object, indicating success or failure. **getProperty(self, prop)** Read the specified property value. - *prop*: the property name to be read Returns a proto message object for further processing. **setProperty(self, prop, value)** Write the specified property value. - *prop*: the property name to be read - *value*: the value (usually as numpy object) to be written. Returns a proto message object indicating success or failure. .. _python_unpack: unpack ============= This submodule contains several unpack helpers for e.g. the :code:`getProperty` or :code:`readTopic` methods of the :code:`Context` class. **unpackMessageToInt64(message)** Unpacks the raw proto message into a typed proto message, which can be handled by python. - *message*: as received from e.g. :code:`getProperty` Returns a hemistereo.api.hemistereo.network.messages.basic_pb2.Int64, which can be handled by python. **unpackMessageToInt32(message)** Unpacks the raw proto message into a typed proto message, which can be handled by python. - *message*: as received from e.g. :code:`getProperty` Returns a hemistereo.api.hemistereo.network.messages.basic_pb2.Int32, which can be handled by python. **unpackMessageToUInt32(message)** Unpacks the raw proto message into a typed proto message, which can be handled by python. - *message*: as received from e.g. :code:`getProperty` Returns a hemistereo.api.hemistereo.network.messages.basic_pb2.UInt32, which can be handled by python. **unpackMessageToInt64(message)** Unpacks the raw proto message into a typed proto message, which can be handled by python. - *message*: as received from e.g. :code:`getProperty` Returns a hemistereo.api.hemistereo.network.messages.basic_pb2.Int64, which can be handled by python. **unpackMessageToUInt64(message)** Unpacks the raw proto message into a typed proto message, which can be handled by python. - *message*: as received from e.g. :code:`getProperty` Returns a hemistereo.api.hemistereo.network.messages.basic_pb2.UInt64, which can be handled by python. **unpackMessageToFloat(message)** Unpacks the raw proto message into a typed proto message, which can be handled by python. - *message*: as received from e.g. :code:`getProperty` Returns a hemistereo.api.hemistereo.network.messages.basic_pb2.Float, which can be handled by python. **unpackMessageToDouble(message)** Unpacks the raw proto message into a typed proto message, which can be handled by python. - *message*: as received from e.g. :code:`getProperty` Returns a hemistereo.api.hemistereo.network.messages.basic_pb2.Double, which can be handled by python. **unpackMessageToBool(message)** Unpacks the raw proto message into a typed proto message, which can be handled by python. - *message*: as received from e.g. :code:`getProperty` Returns a hemistereo.api.hemistereo.network.messages.basic_pb2.Bool, which can be handled by python. **unpackMessageToString(message)** Unpacks the raw proto message into a typed proto message, which can be handled by python. - *message*: as received from e.g. :code:`getProperty` Returns a hemistereo.api.hemistereo.network.messages.basic_pb2.String, which can be handled by python. **unpackMessageToMat(message)** Unpacks the raw proto message into a typed proto message, which can be handled by python. - *message*: as received from e.g. :code:`readTopic` Returns a hemistereo.api.hemistereo.network.messages.matrix_pb2.Matrix, which can be handled by python. **unpackMessageToNumpy(message)** Unpacks the raw proto message into a numpy ndarray (makes sense only for Matrix types and might be a better option over unpackMessageToMat as numpy is easier to use) - *message*: as received from e.g. :code:`readTopic` Returns a numpy ndarray which represents a matrix, e.g. an image, depth map or point cloud. .. _python_imgconvert: imgconvert ========== This submodule contains several helpers for plotting matrix and point-cloud data. The most common types of stereo result data are: images (a Matrix with three int8 channels, representing a RGB image), depth maps (a Matrix with one float or int16 channel, representing a distance-from-camera value and point-cloud data (a matrix of XYZ coordinates representing vertex positions in a world-coordinate system). The imgconvert module helps to transfer the raw matrix data into colored images. **formatNumpyToRGB(np, equalize_histogram, color_map)** A helper function for converting matrix types into rgb (opencv) images. The method applies for both, image matrices which have rgb information already and depth maps, which are colorized via a *color_map*. The *equalize_histogram* and *color_map* arguments are applicable for depth maps only. - *np* the numpy-converted matrix object to be transformed into an image. Usually this is aquired via :code:`context.Context.readTopic` and :code:`unpack.unpackMessageToNumpy`. - *equalize_histogram* (:code:`True`): whether or not to equalize the resulting color map (making the colors more evenly distributed). Only applicable for depth map conversions. - *color_map* (:code:`jet`): the color map scheme to be used for depth map colorization. Only applicable for depth map conversion. Possible values are: *jet*, *winter*, *bone*, *ocean*, *cool*, *hot* and *rainbow*. Returns a numpy ndarray object with 3 channels, representing the rgb components of the image. It can be used for opencv operations directly. **formatNumpyToPointCloud(pyc, image, filter_nan, max_dist, scale, every)** A helper function for converting a matrix point-cloud as received by :code:`context.Context.readTopic` into a (optionally colorized) pointcloud (which also is a numpy ndarray but with a different column layout. - *pyc*: a numpy ndarray representation of the point-cloud as aquired via :code:`context.Context.readTopic` and :code:`unpack.unpackMessageToNumpy`. - *image* (:code:`None`): optional, an RGB convert (as aquired via :code:`formatNumpyToRGB`) used for colorization of the point-cloud data - *filter_nan* (:code:`True`): recommended, filter-out all invalid points to avoid plotting issues - *max_dist* (:code:`None`): optional, used to filter out out-of-range points, which might be noise-artifacts - *scale* (:code:`0.01`): HemiStereo uses *mm* as measurement unit, pyntcloud works best for data of magnitude 1. As most sensor data represents spacial information of room-size (around several meters), the scale is for plotting purposes. - *every* (:code:`10`): strip down the original data set by only using *every* nth data point, helping the processing time and pointcloud viewer. Returns a numpy ndarray object with 3 or 6 (colorized) channels, which can be used with the pyntcloud plotter. **rotateCounterClockWise(image)** Wrapper for the opencv :code:`cv2.rotate(image, cv2.ROTATE_90_COUNTERCLOCKWISE` method, usable for RGB converted matrices. - *image*: an RGB convert (as aquired via :code:`formatNumpyToRGB`) Returns a numpy ndarray object with 3 channels, representing the rgb components of the image. It can be used for opencv operations directly. **rotateClockWise(image)** Wrapper for the opencv :code:`cv2.rotate(image, cv2.ROTATE_90_CLOCKWISE` method, usable for RGB converted matrices. - *image*: an RGB convert (as aquired via :code:`formatNumpyToRGB`) Returns a numpy ndarray object with 3 channels, representing the rgb components of the image. It can be used for opencv operations directly. **plotRGB(image, figsize)** Use :code:`matplotlib` for plotting the RGB converted matrix (image). For use inside jupyter-notebooks. - *image*: an RGB convert (as aquired via :code:`formatNumpyToRGB`) - *figsize* (:code:`(45,30)`: the scale of the plot **plotPCL(pyc)** Use :code:`pyntcloud` for converting and plotting the pointcloud data as threejs javascript plot. For use inside jupyter-notebooks. - *pyc*: an pointcloud convert (as aquired via :code:`formatNumpyToPointCloud`) Returns the :code:`pyntcloud` plot object.