ni_quadra_roi

The ni_quadra_roi filter analyzes input video to detect faces and generate Region of Interest (ROI) data to improve the encoding quality of the faces. It does this by inferencing the input frames using the AI Engine, identifies the objects’ coordinates and classes in the images and creates ROI side data to be used by the encoder.

The ni_quadra_roi filter loads a yolov4 object detection model down to the AI Engine device specified. It also defines its input and output dimensions. For yolov4 models, the input requires a tensor with shape 416x416 and pixel format BGRPlanar. This means that the filter needs to scale these input images and do the format conversion. The filter supports both hardware frames and software frames. For software frames, only YUV420P is supported for now and the soft FFmpeg scaler is used to do the scaling. Since FFmpeg doesn’t support BGRPlanar, the filter first scales the input image to a 416x416 RGB24 AVFrame, then rearranges the frame to a compact RGRPlanar tensor, and transfers the tensor down to AI module to do inference. For hardware frames, the 2D Engine converts the RGB24 to BGRPlanar and then does the scaling. The scaled hardware frames are then passed to the AI module to do inferences.

Before the inference begins, an output buffer will be preset to the model. when the inference completes, the output tensor will be saved in the output buffer. The filter then fetches the output tensor from firmware. It runs post processing with the output tensor to compute the objects’ boxes and class labels in the images. if there are any objects, their boxes will be converted to the four coordinates relative to the up left zero point of the images. Each objects’ coordinates will be put into the ROI side data including a preset QP offset. All the ROI side data in an image will be appended to and passed down to the encoder along with the images.

Parameters

The following are the ni_quadra_roi parameters:

nb

Specifies the AI module (network_binary_yolov4_head.nb) network binary file path including the file name. This field is required for the filter to work.

Supported Values	string
Default	Null

qpoffset

Specifies the ROI QP offset so that encoder can set the specific QP in these regions based on the QP offset. A negative QP offset will increase the quality in the ROI regions detected by the filter. A positive QP offset will decrease the quality of the ROI regions. A QP offset of 0 means no change in quality. The QP offset range of -1 to +1 translates to an actual QP offset of -25 to +25 in the encoder.

Not Applicable	Not applicable for FFmpeg 4.2.0 or before
Supported Values	-1.0 to 1.0
Default	0.0

devid

Specifies the Device ID of Quadra hardware to use when software frames are used as an input. When a hardware frame is used the filter will be collocated to the same device as the frame.

Supported Values	0 to max device ID
Default	0

obj_thresh

Specifies the yolov4 post processing object threshold. Each region would have its score. When it reaches the obj_thresh, it will be taken as an object. The higher the threshold means the harder to be taken as an object.

Supported Values	0 to 1.0
Default	0.25

nms_thresh

Specifies the yolov4 post processing NMS IOU threshold. The gold is to select the bounding boxes with the highest detection probability and eliminate all the bounding boxes whose intersection over union (IOU) value is higher than given IOU threshold.

Supported Values	floating point value
Default	0.45

keep_alive_timeout

Specifies a session keep alive timeout value. This is a periodic request/response between libxcoder and XCoder firmware that when timed out, terminates the session instance in the XCoder firmware.

Supported Values	Integer in the range 1 to 100
Default	3

buffer_limit

Whether to limit the number of output buffers. If enabled, the buffering count of the filter is its buffer pool size(default 4) plus encoding buffering count if its downstream is encoder. It’s enabled by default in FFmpeg-n6.1, FFmpeg-n7.1, FFmpeg 7.1.3, FFmpeg-n8.0.1 and FFmpeg-n8.1.

Supported Values	Integer 0 and 1
Default	0

The following example shows the ni_quadra_roi filter using hardware frames:

ffmpeg -y -vsync 0 -dec 0 -c:v h264_ni_quadra_dec -xcoder-params 'out=hw' -i cr7_1920x1080.h264  -vf 'ni_quadra_roi=nb=./network_binary_yolov4_head.nb:qpoffset=-0.6' -enc 0 -c:v h264_ni_quadra_enc -xcoder-params 'roiEnable=1:RcEnable=1:bitrate=500000' -an cr7_1080p_roi_b500000.h264

The following example below shows the ni_quadra_roi filter using software frames.

ffmpeg -y -vsync 0 -dec 0 -c:v h264_ni_quadra_dec -i cr7_1920x1080.h264 -vf 'ni_quadra_roi=nb=./network_binary_yolov4_head.nb:qpoffset=-0.6' -enc 0 -c:v h264_ni_quadra_enc -xcoder-params 'roiEnable=1:RcEnable=1:bitrate=500000' -an cr7_1080p_roi_b500000.h264

Please note that this filter is available with FFmpeg-n4.2.1 and higher version since the ROI API was introduced in FFmpeg-n4.2.