AI Inference with LibXCoder
Open an AI session
Like the encoder and decoder described previously, to run an AI inference, a session needs to be opened. The session id is a reference to the firmware session during the whole inference lifetime.
At the start of the session an ni_session_context_t structure shall be defined and initialized to refer to the AI session. We can specify the card id to choose different cards to start the AI session with. Passing “NI_DEVICE_TYPE_AI” to ni_device_session_open() creates an AI session.
ni_session_context_t api_ctx = {0};
ni_retcode_t retval;
retval = ni_device_session_context_init(&api_ctx);
api_ctx.hw_id = devid;
retval = ni_device_session_open(&api_ctx, NI_DEVICE_TYPE_AI);
Once an AI session is created successfully, we should pass the model’s network binary file in bytes to the firmware, for it to configure an AI model by calling the function ni_ai_config_network_binary().
On the firmware side, after receiving the whole network binary, it begins to put the content into memory and initializes it by sending init commands to the AI module. When the initialization completes the network layer parameters can be queried out from the firmware and filled in the
ni_network_data_t structure.ni_network_data_t network = {0};retval = ni_ai_config_network_binary(&api_ctx, &network, nb_file);
ni_network_data_t defines the number and details of one model’s input and output layers. Currently this structure supports models with up to NI_MAX_NETWORK_INPUT_NUM input layers and NI_MAX_NETWORK_OUTPUT_NUM output layers. libxcoder uses the ni_frame_t structure to carry the input data, and the ni_packet_t structure to hold the output data. Since there could be multiple layers for a model, the offset pointers of each layer data in the structure’s buffer, will be calculated for future use.
typedef struct _ni_network_layer_info
{
ni_network_layer_params_t in_param[NI_MAX_NETWORK_INPUT_NUM];
ni_network_layer_params_t out_param[NI_MAX_NETWORK_OUTPUT_NUM];
} ni_network_layer_info_t;
typedef struct _ni_network_data
{
uint32_t input_num;
uint32_t output_num;
ni_network_layer_info_t linfo;
struct
{
int32_t
offset; /* point to each input layer start offset from p_frame */
} inset[NI_MAX_NETWORK_INPUT_NUM];
struct
{
int32_t
offset; /* point to each output layer start offset from p_packet */
} outset[NI_MAX_NETWORK_OUTPUT_NUM];
} ni_network_data_t;
With the ni_network_data_t structure we know the size for the whole input and output data. The helper functions are used to allocate page aligned memory for ni_frame_t and ni_packet_t structures, with sizes based on the structure.
ni_session_data_io_t api_src_frame;ni_session_data_io_t api_dst_packet;retval = ni_ai_frame_buffer_alloc(&api_src_frame.data.frame, &network);retval = ni_ai_packet_buffer_alloc(&api_dst_packet.data.packet, &network);
The ni_network_layer_params_t contains the network layer parameters. Its main purpose is to show how to interpret the AI hardware input and output data. Each field is described in the comments.
ni_session_data_io_t api_src_frame;
Typedef struct _ni_network_layer_params_t
{
uint32_t num_of_dims; /* The number of dimensions specified in *sizes */
uint32_t sizes[6]; /* The pointer to an array of dimension */
int32_t
data_format; /* Data format for the tensor, see ni_ai_buffer_format_e */
int32_t
quant_format; /* Quantized format see ni_ai_buffer_quantize_format_e */
union
{
struct
{
int32_t
fixed_point_pos; /* Specifies the fixed point position when the input element type is int16, if 0 calculations are performed in integer math */
} dfp;
struct
{
float scale; /* Scale value for the quantized value */
int32_t zeroPoint; /* A 32 bit integer, in range [0, 255] */
} affine;
} quant_data; /* The union of quantization information */
/* The type of this buffer memory. */
uint32_t memory_type;
} ni_network_layer_params_t;
The models will be kept in the firmware if there are enough resources. This way, if there is another session that would like to do inferences with the same model, it can skip passing the network binary to the firmware and just read back the existing network parameters directly. This helps to speed up the configuration process.
Input data inference
In general, there are two types of input data, the first is picture data in the RBG pixel format, and the other is tensor files, which are in text format.
For pictures, if they are the correct shape the model requires, and if the model format is NHWC, we can directly pass the pixel data “NI_DEVICE_TYPE_AI” to perform inference. We restore the layer data to the ni_frame_t buffer using the pre-calculated offsets of each layer.
int32_t offset = network.inset[input_layer_idx].offset;
uint8_t *p_data = (uint8_t *)api_src_frame.data.frame.p_data[0] + offset;
linesize = pic_width;
for (w = 0; w < pic_height; w++) {
memcpy(p_data, pic_data + w * linesize, pic_width);
p_data += pic_width;
}
For tensor files, since each line in the tensor file is float32 data, and the fact the AI hardware accepts only the vendor specific data format, we first need to convert the float32 data to integer data by calling the helper functions ni_network_layer_convert_tensor() or ni_network_convert_tensor_to_data(). The resulting data is stored in the ni_frame_t buffer pointer p_data.
ni_retcode_t retval;int32_t offset = network.inset[input_layer_idx].offset;uint8_t *p_data = (uint8_t *)api_src_frame.data.frame.p_data[0] + offset;int input_size = ni_ai_network_layer_size(&network.linfo.in_param[input_layer_index]);retval = ni_network_layer_convert_tensor(p_data, input_size, tensor_file, &network.linfo.in_param[input_layer_index]);
Invoke an inference
Once all the layers’ input data is prepared in the ni_frame_t structure, we can pass the input data to the firmware to start an inference, we do this by writing the session. The write may not succeed if there is not enough free buffer available. To retry, the system can poll until there is sufficient buffer available.
do {
ret = ni_device_session_write(&api_ctx, &api_src_frame, NI_DEVICE_TYPE_AI);
if (ret < 0) {
/* error occurs */
break;
} else if (ret >= 0) {
/* write complete */
break;
}
/* can’t write anything. Choose to keep polling the buffer to be written here. */
} while (ret == 0);
After writing the session is complete, we can then poll the output data by reading the session. The read may not succeed immediately if the inference is ongoing. When the inference completes the output data will be stored in the ni_packet_t buffer.
do {
ret = ni_device_session_read(&api_ctx, &api_dst_packet, NI_DEVICE_TYPE_AI);
if (ret < 0) {
/* error occurs */
break;
} else if (ret >= 0) {
/* write complete */
break;
}
/* can’t read anything. Choose to keep polling the buffer to read here. */
} while (ret == 0);
Output data conversion
Any output data from the AI module will be in a hardware vendor specific Integer format. To get the tensor data for postprocessing, we need to convert the output data to the tensor data, based on the output layers network parameters. There are two helper function wrappers to do this conversion, ni_network_layer_convert_output() and ni_network_convert_data_to_tensor(), they accept different input arguments.
Here is an example using ni_network_layer_convert_output(). The first and the second arguments are the output buffer pointer and its size. This buffer is used to store the converted tensor data in the float32 type for the specific layer. The size is equal to the size of float32 multiplied by the ni_ai_network_layer_dims().
retval = ni_network_layer_convert_output(output_buffer,output_buffer_size,
&api_dst_packet.data.packet,
&network, output_layer_index);
Close an AI session
If the inference session is no longer needed, call the ni_device_session_close() to close the session along with which the input and output buffer resources will be reclaimed as well. As mentioned before, releasing the session doesn’t mean releasing the model in the firmware. A reference to the model is kept in case the next session would like to infer the same model.
retval = ni_device_session_close(&api_ctx, 1, NI_DEVICE_TYPE_AI);
AI pre-processing for YUV data
That’s a special case about AI processing, the input and output are both HW frame and the output frame can send to encoder for encoding.
- Open the AI session and set the network nb file to firmware.
ni_session_context_t api_ctx = {0};
ni_retcode_t retval;
retval = ni_device_session_context_init(&api_ctx);
api_ctx.hw_id = devid;
retval = ni_device_session_open(&api_ctx, NI_DEVICE_TYPE_AI);
ni_network_data_t network = {0};
retval = ni_ai_config_network_binary(&api_ctx, &network, nb_file);
- Allocate buffer pool for output buffer. The host call ni_device_alloc_frame() to firmware in order to allocate HW frame, the max buffer number is 8. For host side, use ni_frame_buffer_alloc_hwenc() to allocate ni_session_data_t, the width/height/format should be same as input frame.
// Create frame pool
int format = GC620_I420;
int options = NI_AI_FLAG_IO | NI_AI_FLAG_PC;
/* Allocate a pool of frames by the AI */
retval = ni_device_alloc_frame(&api_ctx,
in_frame->width,
in_frame->height,
format,
options,
0, // rec width
0, // rec height
0, // rec X pos
0, // rec Y pos
8, // frame buffer num
0, // frame index
NI_DEVICE_TYPE_AI);
//Allocate dst_frame on host.
ni_session_data_t dst_frame;
ni_frame_buffer_alloc_hwenc(&dst_frame.data.frame,
in_frame->width, in_frame->height, 0);
Another case is the Model support multi-frame in. The host call ni_device_multi_config_frame() to firmware to set input frames on one api:
ni_frame_config_t frame_in[3];
frame_in[0].picture_width = input_video_width;
frame_in[0].picture_height = input_video_height;
frame_in[0].frame_index = p_frame1->ui16FrameIdx;
frame_in[1].picture_width = input_video_width;
frame_in[1].picture_height = input_video_height;
frame_in[1].frame_index = p_frame2->ui16FrameIdx;
frame_in[2].picture_width = input_video_width;
frame_in[2].picture_height = input_video_height;
frame_in[2].frame_index = p_frame3->ui16FrameIdx;
// Config input frames index on this function.
ret = ni_device_multi_config_frame(&watermark_ctx,frame_in,3,NULL);
3. Prepare output frame buffer and send input HW frame to firmware. Call ni_device_alloc_frame() with NI_AI_FLAG_IO option to prepare the output buffer. Call ni_device_alloc_frame() without option to send input HW frame to firmware.
/* Try to get output buffer */
do{
retval = ni_device_alloc_frame(&api_ctx, in_frame->width,in_frame ->height,
GC620_I420, NI_AI_FLAG_IO, 0, 0,
0, 0, 0, -1, NI_DEVICE_TYPE_AI);
if (retval < NI_RETCODE_SUCCESS) { /* error occurs */
}
}while(retval != NI_RETCODE_SUCCESS);
/* set input buffer */
retval = ni_device_alloc_frame(&api_ctx, 0, 0, 0, 0, 0, 0, 0, 0,
frame_surface->ui32nodeAddress,
frame_surface->ui16FrameIdx,
NI_DEVICE_TYPE_AI);
if (retval != NI_RETCODE_SUCCESS) { /* error occurs */
}
Step4. Read out the HW output frame. Call ni_device_session_read_hwdesc() with NI_DEVICE_TYPE_AI.
do{
retval = ni_device_session_read_hwdesc(
&api_ctx, &dst_frame, NI_DEVICE_TYPE_AI);
if(retval < NI_RETCODE_SUCCESS){ /* error occurs */
}
}while(retval != NI_RETCODE_SUCCESS);
Further Information
For further reading, inside the Quadra Release folder, there is a selection of AI documentation, that covers in more detail Quadra’s AI systems, and also the AI toolkit for model generation.