Developing Apps for HemiStereo Devices

Preparation

Open a shell on the device

To develop an application for the HemiStereo device, you first have to get a shell on the device. There are two ways for accessing a shell on the device:

Via screen and a keyboard

The easiest way is to connect a screen via DisplayPort and a USB Keyboard. After booting the sensor, you will see a login prompt. Just log in using the following credentials:

username: hemistereo
password: hemistereo

Please note: If you want to use an DisplayPort->HDMI adapter cable, you have to use an active one. Passive adapters are not supported!

Via SSH

By default, the sensor tries to get an IP using DHCP. Therefore you have to connect the sensor via ethernet with a network providing a DHCP server (e.g. a network router). Next, you will have to find out the IP the device got from the DHCP server. The easiest way ist to install the HemiStereo Viewer on a PC which is connected to the same network an scan for devices. The IP will be displayed in the device list.

If HemiStereo Viewer is not able to find the sensor, please make sure, that UDP broadcasting is not blocked in the network.

Alternatively, you can also try to find the IP in the web configuration interface of your router.

After getting the IP, you can connect via SSH (username hemistereo, password: hemistereo):

ssh hemistereo@<IP>

Update the device

Before creating a new app, you should update the sensor to get the latest app templates. For updating, run the following command:

sudo apt-get update && sudo apt-get dist-upgrade

Creating a new app

User apps can consist of one or more Docker containers. The containers are defined in a docker-compose.yml file. If you are not familiar with Docker Compose, we suggest you to have a look at the Getting Started Guide.

Multiple apps can be running in parallel on the HemiStereo sensor with the following restrictions:

1. Dependencies between apps should be avoided. If you have two apps App1 and App2, a service of App1 should not depend on a service of App2. If it is necessary, please merge the apps into one by creating a new docker-compose.yml.
2. Some hardware resources (e.g. cameras) can be accessed only by one service at a time. By default, the hemistereo app is installed which includes services for the stereo processing and for the device discovery from the HemiStereo-Viewer. Because of the app restrictions above, the default hemistereo app has to be disabled and its services must be included in your custom docker-compose.yml. To stop and disable the hemistereo app, run the following commands:

   sudo hemistereo_app_stop hemistereo
   sudo hemistereo_app_disable hemistereo

If you want to enable the app again, just run:

sudo hemistereo_app_enable hemistereo

For creating a new app, please run:

hemistereo_app_create <your-app-name>

The docker-compose.yml file

After creating an app, you will find a new folder in the current working directory with a docker-compose.yml inside. The docker-compose.yml has the following content:

version: '3'
services:
  # The hemistereo service captures images from the cameras and
  # does the stereo calculation.
  hemistereo:
    image: 3dvl/jetson-tx2-hemistereo:r32.3.1-v0.12.0
    privileged: true
    restart: on-failure
    ports:
      # Exposing these ports is necessary for accessing HemiStereo
      # from outside the device (e.g. from the HemiStereo Viewer on
      # you PC. If you don't need that access, you can remove these
      # part.
      - 50051:50051 # non-TLS
      - 50551:50551 # TLS
    volumes:
      - /var/lib/hemistereo:/var/lib/hemistereo
      - /var/run/hemistereo:/var/run/hemistereo # expose unix socket

  # The hemidiscovery service is used for finding sensor from another
  # device. If this is not required, you can remove the service.
  hemidiscovery:
    image: 3dvl/hemidiscovery:latest
    restart: on-failure
    ports:
      - 4321:4321/udp
    volumes:
      - /var/lib/hemistereo:/var/lib/hemistereo:ro

  # This is the service running your app. By default, a sample
  # application is used that records images from the sensor in a
  # given interval. You can adapt it to match you requirements.
  app:
    build: ./src
    restart: on-failure
    depends_on:
      - hemistereo
    privileged: true
    environment:
      - SIMPLE_CAPTURE_OUTPUT_DIR=/record
      - SIMPLE_CAPTURE_INTERVAL=10000
    volumes:
      - /var/run/hemistereo:/var/run/hemistereo
      - ./record:/record

As you can see, three services are defined: hemistereo, hemidiscovery and app. Let's have a closer look at these services:

1. hemistereo This is the main service that provides the functions for the HemiStereo depth calculations. Because this service needs access to the cameras, the privileged flag is set to true. The ports exposed are for accessing the service from outside the HemiStereo device, e.g. from the HemiStereo SDK/Viewer on your PC. If you don't need that access, you can remove it. There are also two host volumes mounted in the container. /var/lib/hemistereo contains some sensor specific files (serial number, calibration data, certificates) that are required by the service. /var/run/hemistereo is the directory, in which the service will store its unix domain socket file. The HemiStereo SDK can use this socket to connect to the service. So you should mount this volume also in your service.
2. hemidiscovery The hemidiscovery service is used for discovering the sensor device from the network. It is mainly used by the HemiStereo SDK/Viewer to find the device if the IP is unknown. If you don't need that feature, you can remove the whole service from the docker-compose.yml.
3. app The app is a sample application which connects to the hemistereo service and captures some images. As one can see, the build is set to the ./src directory. This indicates that the sources of the container image are located in the src directory next to the docker-compose.yml. Docker compose automatically grabs the sources and builds the image, if it does not exist already. For making your own app, you can adapt the sources in the src directory to your needs. We will have a look at the sources in the next section. In the docker-compose.yml, we can also see, that the app depends on the hemistereo services. This ensures that the hemistereo service is running before the app service gets started. Currently it is also necessary to run the container in privileged mode because of accessing some hardware functionality. We are planning to remove the need for enabling the privileged mode in future versions, but for now, it has to be activated. There are also some environment variables set. These are used to configure our application running inside the container. We will have a look at them later. It would be possible to communicate with the hemistereo service over TCP. Nevertheless, it is recommended to use the UNIX domain socket provided by the hemistereo service. Therefore it is necessary to mount the /var/run/hemistereo as volume.

Note: The restart policy for all services is set to on-failure. This restarts the service in case of any failure. If the policy is not set, the application will not start again. Please do not use always or unless-stopped as restart policy because the containers are started automatically by the hemistereo-docker-runner service if the app is enabled.

The source directory

Now let's have a look at the source directory. It contains the following files:

• CMakeLists.txt
• simple_capture.cpp
• Dockerfile
• entrypoint.sh

The CMakeLists.txt and simple_capture.cpp are the files for compiling the application. Please see TODO for how to create them. The Dockerfile contains the build instructions for the container:

FROM 3dvl/jetson-tx2-hemistereo:r32.3.1-0.11.0

WORKDIR /app

RUN apt-get update \
    && apt-get install -y --no-install-recommends \
        build-essential \
        cmake \
        libopencv-dev \
        libboost-filesystem-dev \
        libboost-program-options-dev \
    && rm -rf /var/lib/apt/lists/*        

COPY . src

RUN mkdir -p build \
    && cd build \
    && cmake -Dosp_DIR=/opt/HemiStereo/lib/cmake/osp ../src \
    && make \
    && cp simple_capture ../ \
    && cd .. \
    && rm -r build

RUN rm -r src

COPY entrypoint.sh /app

ENTRYPOINT [ "/app/entrypoint.sh" ]

The first line defines the base image that is used for the container image. We provide Ubuntu 18.04 based images that contain a preinstalled version of the HemiStereo SDK. You should choose one of them. They images have the following schema:

3dvl/{MACHINE}-hemistereo:{L4T_VERSION}-{OSP_VERSION}

Because the HemiStereo sensor is based on a Nvidia Jetson TX2, {MACHINE} is set to jetson-tx2. {L4T_VERSION} is the version of the operating system provided by Nvidia. This defines the versions of some Nvidia libraries that are available inside the container (e.g. CUDA). If you do not have any restrictions, keep it as it is. {OSP_VERSION} is the version of the HemiStereo SDK. Alternatively you can also use latest to use the latest SDK. The next steps describe the build process. First, some dependencies are installed using apt-get. Then the sources are copied and the build process is started. Please note that the osp_DIR variable is set in the CMake call. This is necessary because the HemiStereo SDK is installed under /opt/HemiStereo which is not a default CMake search path. After that, the entrypoint script is copied to the container and the ENTRYPOINT is defined to run that script. This executes the script when the container gets started. This is a good place for parsing some environment variables and pass them to the executable. This allows the user to configure the application from outside without re-building the image. For example, our sample app records images in a given interval. Therefore we could pass the interval and the output directory as environment variable and pass it to the executable:

#!/bin/bash

DEVICE=${SIMPLE_CAPTURE_DEVICE:-auto}
OUTPUT_DIR=${SIMPLE_CAPTURE_OUTPUT_DIR:-/record}
INTERVAL=${SIMPLE_CAPTURE_INTERVAL:-2000} 

/app/simple_capture --device $DEVICE --output_dir $OUTPUT_DIR --interval $INTERVAL

Some of these variables were set in the docker-compose.yml. If they were not provided in the docker-compose.yml, they are set to default values. For example, the SIMPLE_CAPTURE_DEVICE was not provided in the docker-compose.yml, therefore it is set to the default value auto.

Building the app

To build the app, go to the app directory (which contains the docker-compose.yml) and run the following command:

docker-compose build

Running the app

If building was successful, you can run the app using the following command:

hemistereo_app_run <path_to_the_app_directory>

To stop the app again, you can execute the following command:

hemistereo_app_stop <app_name>

The <app_name> is set to the name of the directory in which the docker-compose.yml lives.

If you only run the app, the app will not start again after a reboot. To start the app automatically, you have to enable it:

hemistereo_app_enable <path_to_the_app_directory>

To disable the app again, run:

hemistereo_app_disable <app_name>

Cleaning

Rebuilding or downloading new image versions creates new layers on the filesystem but does not remove old ones. Therefore, sometimes it is necessary to remove unused images and containers to prevent running out of space. To remove all unused images and containers, run the following command:

docker system prune -a

More information about this command can be found here.