ni_quadra_bgr

The ni_quadra_bgr background removal filter analyses input frames, inferences these images using the AI module (segm32), segments the foreground and background of the input images, and then removes the background. The output is kept in RGBA format to allow later overlay to judge fore and background based on value of alpha channel.

The filter takes a segm_32 object segmentation model as input, down to the AI module specified by the user. The imported model will be unfolded and initialized in memory. As for the segm32 model, the dimensions of the input and output are as defined. The segm32 model requires a tensor with shape 256 x 144 (width x height) and a pixel format of BGR Planar. In this case, the filter needs to scale the input frames and do the format translation to adapt to the different resolutions. Currently the filter only supports hardware frames. The 2D Engine is used to do the scaling and format conversion and then the result is passed to the AI module to perform the inferences.

When the inference completes the output tensor is then retrieved by the filter which then performs some post-processing to segment the foreground and background. The result of the post-processing is the alpha data is posted to the alpha channel of the RGBA output. This is then returned back to Quadra HW in the form of a hwupload such that the filter retains HWframe pixel format.

note

The ni_quadra_bgr filter only supports limited color range video (also known as “tv” range). While full range can be passed through the filter, the ni_quadra_bgr filter will clamp to limited color range output before processing. This will cause a slight color shift in the output video.

Parameters

The following are the ni_quadra_bgr parameters:

nb

Specifies the AI module (segm32.nb) network binary file path including the file name. This field is required for the filter to work.

Supported Values	String
Default	Null

skip

Specifies the number of frames to skip between inference. When set to 0, all frames are inferenced. When set to 1, every 2nd frame is inferenced. When set to 2, every 3rd frame is inferenced.

Supported Values	0 or higher
Default	0

The following example shows using the ni_quadra_bgr filter to overlay input stream on top of background image.

ffmpeg –vsync 2 -i theatre.bmp \
-c:v h264_ni_quadra_dec -dec 0 -xcoder-params 'out=hw:scale0=274x154' -i ken0.h264 \
-c:v h264_ni_quadra_dec -dec 0 -xcoder-params 'out=hw:scale0=274x154' -i ken1.h264 \
-c:v h264_ni_quadra_dec -dec 0 -xcoder-params 'out=hw:scale0=274x154' -i ken2.h264 \
-filter_complex "[0:v]format=rgba,ni_quadra_hwupload=0[a0]; \
[1:v]ni_quadra_bgr=nb=segm32_tflite.nb:skip=1[a1]; \
[2:v]ni_quadra_bgr=nb=segm32_tflite.nb:skip=1[a2]; \
[3:v]ni_quadra_bgr=nb=segm32_tflite.nb:skip=1[a3]; \
[a0][a1][a2][a3]ni_quadra_xstack=inputs=4: \
layout=0_0 \
|600_558|810_558|1026_558: \
size=1920_1080 \
|274_154|274_154|274_154, \
ni_quadra_scale=format=yuv420p[out]" \
-map "[out]" \
-c:v h264_ni_quadra_enc -enc 0 xstack_3_ppl_theatre.mkv