ni_quadra_bg

The ni_quadra_bg background removal filter analyses input frames, inferences these images using the AI module (segm32), segments the foreground and background of the input images, and then removes the background.

The filter takes a segm_32 object segmentation model as input, down to the AI module specified by the user. The imported model will be unfolded and initialized in memory. As for the segm32 model, the dimensions of the input and output are as defined. The segm32 model requires a tensor with shape 256 x 144 (width x height) and a pixel format of BGR Planar. In this case, the filter needs to scale the input frames and do the format translation to adapt to the different resolutions. Currently the filter only supports hardware frames. The 2D Engine is used to do the scaling and format conversion and then the result is passed to the AI module to perform the inferences.

When the inference completes the output tensor is then retrieved by the filter which then performs some post-processing to segment the foreground and background. The result of the post-processing is the alpha data needed to mix the new background which is converted to RGBA format to be mixed with the original image using the 2D Engine overlay feature.

The ni_quadra_bg filter only supports limited color range video (also known as “TV” range). While full range (or “PC” range) video is accepted by the filter, the ni_quadra_bg filter will clamp to limited color range video before processing. This clamp will cause a slight color shift in the output video.

Parameters

The following are the ni_quadra_bg parameters:

nb

Specifies the AI module (segm32.nb) network binary file path including the file name. This field is required for the filter to work.

Supported Values	string
Default	Null

bg_img

Specifies the background image full path, including the file name. This field is required for the filter to work, and has no default value.

Supported Values	string
Default	Null

use_default_bg

Specifies whether to use the default background image. When the value is set to 0, the background of input video will be replaced by the customized bg_img. If set to 1, the default background will be used. By default the customized bg_img is used.

Supported Values	0 of 1
Default	0

skip

Specifies the number of frames to skip between inference. When set to 0, all frames are inferenced. When set to 1, every 2nd frame is inferenced. When set to 2, every 3rd frame is inferenced.

Supported Values	0 or higher
Default	0

is_p2p

Specifies if output buffer of the filter is p2p buffer. When is_p2p is set to 1, the output buffer of the filter is set as p2p buffer and can be read out by p2p read. Else the output buffer is normal and cannot perform p2p.

Supported Values	Bool 0 and 1
Default	0

The following example shows using the ni_quadra_bg filter using hardware frames with custom background image:

ffmpeg -y -loglevel debug -vsync 0 -dec 0 -c:v h264_ni_quadra_dec -xcoder-params 'out=hw' -i ./bg_1920x1080.h264  -vf 'ni_quadra_bg=nb=./segm32.nb:bg_img=./bg.png:use_default_bg=0' -enc 0 -c:v h264_ni_quadra_enc ./bg_1920x1080.h264

The following is an example input before and after background replacement:

Before	After