Tutorial 3: User Guide of Plumber



  • User Guide of Plumber

    Content

    Plumber is a tool chain for generating high performance FPGA-based CNN inference system. Plumber takes CNN high-level description in TensorFlow as input, and outputs an optimized runnable software program together with a hardware design.

    Glossary

    • SGNode: Plumber represents a high-level description of a set of computation inside the model as SGNode. For example, if there's offset plus and activation function computation in the convolution calculation, SGNode will express them together as one node.

    • SG: Streaming Graph: data construction inside the Plumber, which consists of a series of SGNodes.

    • SG IR: to serialize the SG, and generate the SG IR file(which is in .pb, .pbtxt format), using a cross-platform and high-performance file format called protobuf.

    • Freeze the model: Models trained by TensorFlow are generally saved as checkpoint files, which the parameters and graph structures of the model are scattered across different files. Plumber expects to have a model that merge the model and parameters into one file, so we need a command to freeze the model.

    Get started

    Compile a model takes four steps, generate SG -> optimize SG -> optimize SG for hardware -> export data, these four steps are interdependent, please follow this order to execute the commands.

    1. Freeze the model

    Please refer to instance 1_plumber_freeze.sh.

    plumber_cli freeze ./inference_model/ -d tmp
    

    The first parameter ./inference_model is a checkpoint file saved during the TensorFlow training process, this parameter(file path) must be provided. The folder contains these following files:

    1. *.meta: meta information about the model
    2. *data*: training data
    3. *.index: index file

    If you already have checkpoint file in your folder and the path of model_checkpoint_path is still the absolute path of your training process, it will cause a failure to freeze your model.

    The second parameter -d ./tmp is to specify the path to save the model after freezing, it should be specified as a folder, and can be a folder that doesn't exist, Plumber will automatically create it. model.pb file will be generated inside the folder, this file is based on protobuf protocol.

    Other command option

    --output-file-name

    Generate a file path for the frozen model. You can use this option if you just want to generate a frozen model file. If you are compiling a model, please use -d to specify a file path.

    --output-node-names
    This option specifies the name of output node, multiple output nodes are separated by commas as optional parameters. If it's not specified, Plumber will automatically analyze the model to find possible output nodes and choose for you. For example:

    # plumber_cli freeze ~/tmp/ssd_ckpt -d ~/tml/ssd
    You haven't specified any output node, please select one or many from the following list:
    [  0] ssd_KYnet_v2/softmax_4/Reshape_1
    [  1] ssd_KYnet_v2/block8_box/Reshape
    [  2] ssd_KYnet_v2/softmax_2/Reshape_1
    [  3] ssd_KYnet_v2/softmax/Reshape_1
    [  4] ssd_KYnet_v2/block7_box/Reshape
    [  5] ssd_KYnet_v2/softmax_1/Reshape_1
    [  6] ssd_KYnet_v2/block4_box/Reshape
    [  7] ssd_KYnet_v2/block9_box/Reshape
    [  8] ssd_KYnet_v2/block10_box/Reshape
    [  9] ssd_KYnet_v2/softmax_3/Reshape_1
    Please enter the indices of output nodes, separated by ','. If you want to select all, please enter 'all': all
    Output node names: ['ssd_KYnet_v2/softmax_3/Reshape_1', 'ssd_KYnet_v2/block7_box/Reshape', 'ssd_KYnet_v2/softmax_4/Reshape_1', 'ssd_KYnet_v2/block4_box/Reshape', 'ssd_KYnet_v2/softmax_2/Reshape_1', 'ssd_KYnet_v2/softmax/Reshape_1', 'ssd_KYnet_v2/block8_box/Reshape', 'ssd_KYnet_v2/block9_box/Reshape', 'ssd_KYnet_v2/block10_box/Reshape', 'ssd_KYnet_v2/softmax_1/Reshape_1']
    2018-09-30 15:44:28 INFO Initialising with model directory ...
    Converted 48 variables to const ops.
    Successfully writen the frozen graph to ~/tmp/ssd/model.pb
    

    --use-scope-match

    Boolean type, Plumber finds possible output node patterns automatically, which by default are based on the node's Scope. By this way, Plumber can filter unnecessary training diagrams, so that size of data nodes can be shrunk. If Scope is not used in your model, setting it to False will be better. You can use -m True or --use-scope-match=True

    Notes: If you forget its parameter options, you can use plumber_cli freeze --help to check command options.

    2. Generate SG

    Please refer to instance 2_plumber_freeze.sh.

    plumber_cli sg -d ./tmp -s 1,256,256,3
    

    The first parameter -d ./tmp is to specify the file path for the frozen model, and the file name of the frozen model follows the format of model.pb.

    The second parameter -s 1,256,256,3 is to specify the input shape of the model corresponding to the data format of TensorFlow. According to the input of the model, it should be either NHWC or NCHW.

    3. Optimize SG

    Please refer to instance 3_plumber_SG_opt.sh.

    plumber_cli sg_opt -d tmp
    

    Optimize SG is mainly to analyzes the data width, accuracy and data sparsity of the model. Only need to specify the folder path which contains all files generated by sg command of Plumber.

    4. SG Optimize SG for hardware

    Please refer to instance 4_plumber_HDL_opt.sh.

    plumber_cli hdl_opt ./rainman_board_v3.pbtxt -d tmp
    

    Now you should be familiar with -d tmp, this parameter is used for specifying the save path for generated files. For different boards, this optimization will take different operations and assign different hardware execution devices. And it will generate model_hdl_sg.pbtxt file during the operation, and this file is the input of the Raintime.

    3 optimization will be executed in this step:

    Step Function
    Merge Merge multiple operators into one
    Device Mark the SG node with the execution device
    Design Design space exploration

    5. Export data

    Please refer to instance 5_plumber_export_data.sh.
    Export parameters in the model and the input and output of each SGNode.

    plumber_cli export -d tmp -b 16 -ib 9 -fb 6 --use-hdl=True
    

    -d tmp specifies the file path used by previous steps. -b 16 specifies the bit width of the exported fixed point data. By default, 8-bits, 16-bits, 32-bits are supported, which depends on your model data. On the board we use fixed point data for calculation, so we need to know the data bit width, and also need to specify the data precision, -ib is to specify the integer bit width as 9 bit, -fb decimal bit width as 6Bit, one bit left for the sign bit. --use-hdl=True is to specify the hardware-optimized SG as input to export the data.

    Exporting data is saved under the path /app/tmp/data_hdl, there are four files, each is listing as follow:

    • Big endian floating data is stored in float_big file.
    • Big endian fixed data is stored in fixed_big file
    • Little endian floating data is stored infloat_little file
    • Little endian fixed data is stored infixed_little file

    If you don't know the data size range in your model, you can unspecify -ib and -fb to let Plumber calculate their optimal ratio based on model data to choose the accuracy of the fixed point data.