Scaling with LibXCoder

Scaler Demo

A scaler demo is included as part of the ni_xcoder_transcode_filter and ni_xcoder_multithread_transcode programs to demonstrate how to launch a scaler operation with the libxcoder API. It supports two types of operations – scale and drawbox. These are implemented in the same fashion as the video filters in ffmpeg (ni_quadra_scale, ni_quadra_drawbox) in that they are inserted into the transcoding pipeline and transforms the frame between decoder and encoder.

The following are sample commands for running the scale and drawbox filter demo:

ni_xcoder_transcode_filter -i input.264 -o output.265 -m a -n h -d out=hw -f “ni_quadra_scale=width=1280:height=720:format=yuv420p”

This command scales the input video to 1280x720 after decoding and transforms the pixel format to yuv420p if it’s not already in this format. Then the scaled frames are sent encoder for encoding.

ni_xcoder_multithread_transcode -i input.264 -o output.265 -m a -n h -d out=hw -f “ni_quadra_drawbox=x=300:y=150:width=600:height=400”

This command draws a box that’s 600 pixels wide and 400 pixels high, with its top left corner located at 300 pixels from the left border and 150 pixels from the top border. The frames are not scaled so the output video resolution will be the same as the input.

Note that the input frame to the scaler must be a hardware frame, this is because the 2D engine only accepts hardware frames. To make the decoder output a hardware frame, “-d out=hw” must be specified, as shown in the above commands. When a decoded frame is received, the application needs to make sure that the HW frame buffer of the decoded frame is properly recycled once scaler output is read, since the scaler output will use another HW frame buffer. The following code snippet from ni_xcoder_transcode_filter shows this behavior:

if (scale_params.enabled)
{
    p_hwframe = (niFrameSurface1_t *)out_frame.data.frame.p_data[3];
    ni_hw_frame_ref(p_hwframe);
    scale_filter(&sca_ctx, &out_frame.data.frame, &filter_out_frame, 
                    xcoderGUID, scale_params.width, scale_params.height, 
                    ni_to_gc620_pix_fmt(dec_ctx.pixel_format), 
                    scale_params.format);
    ni_hw_frame_unref(p_hwframe->ui16FrameIdx);  //Recycle decoder output
    frame_to_enc = &filter_out_frame;  //Send scaler output to encoder
}
else if (drawbox_params.enabled)
{
    p_hwframe = (niFrameSurface1_t *)out_frame.data.frame.p_data[3];
    ni_hw_frame_ref(p_hwframe);
    drawbox_filter(&crop_ctx, &pad_ctx, &ovly_ctx, &fmt_ctx, 
                      &out_frame.data.frame, &filter_out_frame, 
                    &drawbox_params, xcoderGUID, 
                      ni_to_gc620_pix_fmt(dec_ctx.pixel_format), GC620_I420);
    ni_hw_frame_unref(p_hwframe->ui16FrameIdx);  //Recycle decoder output
    frame_to_enc = &filter_out_frame;  //Send scaler output to encoder
}

Launch a Scaler Operation

Supported scaler operations include SCALE, CROP, PAD, OVERLAY, STACK, and ROTATE. The parameters needed for launching a scaler operation are packed in ni_scaler_input_params_t. Different operations will require different parameters. See init_scaler_params() to learn which parameters are fixed and which parameters can be set by the user for the current operation. After setting the parameters, call launch_scaler_operation():

int launch_scaler_operation(ni_session_context_t *p_ctx, int iXcoderGUID,
                            ni_frame_t *p_frame_in_up,
                            ni_frame_t *p_frame_in_bg,
                            ni_session_data_io_t *p_data_out,
                            ni_scaler_input_params_t scaler_params)
{ 
    // p_frame_in_up and p_frame_in_bg should be the same except it’s an overlay operation
    int ret = 0;
    if (p_ctx->session_id == NI_INVALID_SESSION_ID)
    {
       // If session is not opened yet, open a scaler session
        if (0 != scaler_session_open(p_ctx, iXcoderGUID, scaler_params.op))
      return -1;
  
        // init scaler hwframe pool where output frames of scaler is put
        if (0 != ni_scaler_frame_pool_alloc(p_ctx, scaler_params))
            return -1;
}
 
    // allocate a ni_frame_t structure on the host PC
    if (0 != ni_frame_buffer_alloc_hwenc(&(p_data_out->data.frame),
                                    scaler_params.output_width,
                                    scaler_params.output_height, 0))
    { return -1;}
    niFrameSurface1_t *frame_surface_up;
    // input frame to be scaled (overlay frame in overlay case)
    frame_surface_up = (niFrameSurface1_t *)((p_frame_in_up->p_data[3]));   
    // allocate scaler input frame
    if (0 != ni_scaler_input_frame_alloc(p_ctx, scaler_params, 
         frame_surface_up))
    { return -1;}
    niFrameSurface1_t *frame_surface_bg;
    // input frame to be scaled (main/background frame in overlay case)
    frame_surface_bg = (niFrameSurface1_t *)((p_frame_in_bg->p_data[3])); 
    // Allocate scaler destination frame.
    if (0 != ni_scaler_dest_frame_alloc(p_ctx, scaler_params, 
         frame_surface_bg))
    { return -1;} 
 
    // Retrieve hardware frame info from 2D engine and put it in the ni_frame_t structure.
    ret = ni_device_session_read_hwdesc(p_ctx, p_data_out, 
           NI_DEVICE_TYPE_SCALER);
    if (ret < 0)
    { // free the input and output frame once failed
        ni_frame_buffer_free(p_frame_in_up);
        ni_frame_buffer_free(p_frame_in_bg);
        ni_frame_buffer_free(&(p_data_out->data.frame));
    }
    return ret;
}

Generally, before launching a scaler operation, the hardware frame pool should be initialized. An input hardware frame and destination hardware frame should be allocated. Finally call ni_device_session_read_hwdesc() to get the desired output from the scaler.