Tutorial 4: Run SSD Model with Raintime

  • Author: Ruizhe Zhao vincent.zhao@corerain.com/Chao Xiong
    Date: 2018/10/20

    Run SSD model with Raintime


    I. Introduction

    In Tutorial 3 we showed how to use plumber to export TensorFlow model to SG description and weight data.
    This tutorial focuses on how to use raintime to perform execution on board.

    Raintime is the computational graph runtime and Convolutional Neural Network computation library running on the Corerain RainmanV3 board.
    Raintime receives SG description and weight data file from Plumber. It perform execution process according to the SG description, it also initializes the data which the compute nodes depend on by the weight data file.
    Raintime can allocate different functions, data types, and computing devices to different nodes according to the information of each node provided by the SG description.
    Raintime also provides a call interface that developers can integrates the raintime executions into the target application with just a few lines of code.

    The goal of this tutorial is to explain the working principle and usage of raintime by taking SSD model as an exmple which is converted from original TenserFlow model to the application of raintime on board.

    Generally, this process includes following 3 steps:

    1. Convert the original TensorFlow model into SG description and data file.
    2. Build the application framework and add the raintime function.
    3. Connect the pre-processing and post-processing functions to complete the whole application design.

    SSD means Single Shot Multibox Detector, it doesn't need to analyze the images twice, which means, it can realize target classification and target detection in one single execution. It is commonly used in the algorithm for target detection.

    This tutorial is modified from the original SSD algorithm which released here

    For the basic concept of SG please refer to the Tutorial 3.

    This tutorial is based on plumber in version 1.2.1.

    II. Get the SG description and data of the SSD model

    This tutorial provides a package Tutorial4.tar, extract the package, you can get the directory tutorial-4.tar which includes:

    1. plumber-web-5b, this folder contains SG definition files face_5b_model_hdl_opt_dfg.pbtxt of SSD model which is to execute on the hardware, model weight parameter ./data/float_little/, and post-processing parameter ./post_params
    2. SSD application demo example

    Get the SG description

    SG description is compiled from original TensorFlow model by plumber, to get SG description, you need to call the plumber_cli commands in the order of SG -> Optimize SG -> Optimize SG for Hardware. Please refer to Tutorial 3

    Data files

    The data files used in this tutorial is the weight parameter which is adjusted for hardware execution, the files are wrapped in ssd_hdl_data.zip. Upzip this file, you will see a data directory, which includes:

    • float_little:all floating-point weight data, stored in little endian.
    • fixed_little:all fixed-point weight data which is quantized to low precision, also stored in little endian.
    • README.txt:configuration information for data generation.

    Raintime usually only load the data files in float_little, and automatically quantized these data, converts them into low-precision data format.

    The data in fixed_little is generally used as a test benchmark.

    The data provided in this tutorial is quantized by converting 32-bit floating-point format data into a 16-bit fixed-point format with 9 decimal places, 6 integer bits, and 1 sign bit.

    Each data directory contains a config.json file that describes the one-to-one correspondence relationship between the data files in the directory and the weights in the model. raintime will use data parameters when loading data.

    III. Integrate Raintime in your application

    In last chapter we obtained the SG files and weight data file that raintime need to use, in this chapter, we will perform how to integrate raintime in an application.

    The execution of raintime is based on graph structure: the graph, which is SG description, is passed to raintime, according to the graph structure like nodes and edges, raintime will perform the execution defined by SG.
    Each node corresponds to a neural network layer, like convolution layer, pooling layer and activation layer, and it specifies its computing device and data type.
    The edge with direction between nodes represents the data flow: if the edge points from node A to node B, the edge indicates that the output of A is the input of B.
    Raintime is a C++ library, so that only applications developed through C++ can integrate raintime.
    The integration is the same as other C++ library: refer to raintime header, add the raintime library function to the code of application, link to raintime library while compile.

    We will explain the steps one by one:

    1. Raintime header files

    Raintime related header files are located /usr/local/include/, including raintimeplumber_irlibssdandEigen.
    Among them, the core header file of raintime is /usr/local/include/raintime/core.h, which includes the key raintime library.

    2. Add the library function of raintime to the application

    The core library functions of raintime are divided into the following categories:

    • Load SG Description: Load the local SG description file into memory and make it become an SG object that can be executed.
    • Load weight parameters: Load local data files into memory to initialize constants in the execution graph during execution.
    • Perform executions: Build an Runner to calculate with graph objects and weight parameters and return the results.

    3. Load SG description

    Loading SG description mainly has the following steps:

    1. Initialize the SGDef object, parse the local pbtxt file and merge it into the initialized SGDef object;
    2. Convert SGDef to SG object by using SGBuilder;

    We can use the following codes to complete the above loading process:

    SGDef *LoadFromFile(const char *file_name) {
      auto sg_def = new SGDef();
      int fd = open(file_name, O_RDONLY);
      if (fd < 0) {
        fprintf(stderr, "Cannot open file %s\n", file_name);
      io::FileInputStream in(fd);
      if (!TextFormat::Parse(&in, sg_def)) {
        fprintf(stderr, "Failed to parse file %s\n", file_name);
      return sg_def;
    // in your main function
    SGDef *sg_def = LoadFromFile("xxx.pbtxt");
    SGBuilder builder(sg_def);
    SG *sg = builder.Build();

    LoadFromFile will be added to the core library in subsequent raintime versions.
    ``plumber_irdefines the format ofSGDef`.

    4. Load weight data

    The raintime Runner uses SGDataMap as the main storage data structure of the weight data.
    Loading the weight parameter requires SGDataMap and then calling its LoadFromDir method to complete the process of loading data from the local data directory.

    The codes are shown as below:

    SGDataMap *data_map = new SGDataMap();

    5. load input images

    In addition to the weights, the SSD model requires input images to complete the execution.
    Since the current version of the raintime interface relies on SGDataMap, the input image also needs to be encapsulated into SGDataMap.

    As shown below, this function implements the function of loading an input image and encapsulating it into SGDataMap:

    SGDataMap* LoadFromMem(vector<float>& input_tensor) 
      int length = input_tensor.size();
      // create the shared pointer to the image data to be stored in SGDataMap
      auto data = std::make_shared<std::vector<float>>(length);
      *data = input_tensor;
      // shape is not important in this case
      std::vector<int> shape;
      // create the SGDataMap
      auto data_map = new SGDataMap();
      SGDataMap::ValT val(shape, data);
      data_map->set("img_input", val);
      return data_map;

    field is the name of input node in the SG corresponding to the input picture. The input node name of the SSD model provided in this tutorial is "img_input", so the field here should be "img_input".
    We can find the input node in SG description file through searching for a node with op value of Input.

    6. Build an Runner and perform executions

    At this stage, we have integrated SG description and weight parameters into the application. Next step is to execute the application.

    The executions for raintime are encapsulated in the Runner, called SGRunner. The Runner performs the execution with two steps: Build and Run.
    Build traverses all the nodes of SG, schedules the execution of the nodes according to the dependencies between the nodes, prepares the context for each node's execution, and loads the weight parameters as constants in the context.
    Run will pass the input image data to the input node of SG, and complete the execution of each node according to the execution order.

    As shown in the following example:

    SGRunner runner;
    runner.Build(sg, const_data_map);
    SGDataMap * output = runner.Run(sg, const_data_map, input_data_map);

    build and run are separated into 2 steps to minimize the overhead of running outside of core computing.

    When using an Runner, if you want to perform multiple executions, you only need to build once and run multiple times.

    The result of run is an object of SGDataMap, which contains the output data of all output nodes.

    Link raintime

    The library of raintime is pre-installed in the system image we provide, including header files and library files, as well as other dependent libraries.
    Users can refer to the CMakeLists.txt file in the Tutorial-4/example/ and Tutorial-4/example/single_img/ folders.

    // include path
    // library path
    // link raintime related libs to project
    target_link_libraries(ssd_5b_runner plumber_ir raintime ssd gflags glog pthread -lprotobuf ${OpenCV_LIBS})

    IV. Add pre-processing and post-processing functions

    The previous chapter introduces the core components that should be included in an raintime based application. In order to complete an application, we also need to add application processing codes.

    The application processing codes corresponding to the SSD model of this tutorial includes image pre-processing and post-processing of SSD output results.

    The logic of preprocessing is simple. It only needs to be calculated on the basis of the input image, and then loads the processed input image into the input SGDataMap.
    The post-processing function needs to extract the required content from the output SGDataMap and convert it to the appropriate result.

    Please refer to the contents of example/single_img/ssd_5b_runner.cc for further details.

    V. Summary

    This tutorial focuses on how to convert a TensorFlow model into an application that can be executed on a board.
    This conversion process includes: using plumber to obtain SG description and weight parameters, building a raintime based on application, and completing the application processing.

    A raintime application template is shown below:

    // include raintime/core.h
    // ...
    // namespaces
    using namespace raintime;
    using namespace raintime::sg;
    using namespace plumber_ir;
    // some helper functions
    // ...
    // the main function
    int main(int argc, char *argv[]) {
      // load sg
      SGDef *sg_def = LoadFromFile("xxx.pbtxt");
      SGBuilder builder(sg_def);
      SG *sg = builder.Build();
      // load const data
      SGDataMap *const_data_map = new SGDataMap;
      // pre-processing
      auto img = PreProcess(/* ... */);
      auto shape = GetImageShape();
      auto input_data_map = LoadFromImage("img_input", img, shape);
      // build and run
      SGRunner runner;
      runner.Build(sg, const_data_map);
      SGDataMap *output = runner.Run(sg, const_data_map, input_data_map);
      // post-processing
      auto result = PostProcess(output);
      return 0;

    In summary, applications based on raintime, especially those based on convolutional neural networks, can generally be designed on above template.