Uploading with LibXCoder
Input YUV Frame Preparation / Device Selection
The frame upload preparation for the YUV buffer is the same as what was written in the Encoder section 5.6.2 “Input YUV Frame Preparation”.
Selecting which card to upload to can be done with the “-c" device selection parameter in the ni_xcoder_encode program. Any number > = 0 will directly translate to the device number and a value of -1 will allow the resource allocation to select the least loaded encoder device.
Uploading Loop
The purpose of uploading a YUV frame to the card explicitly is to leverage the advantages of using HW frames. Without an explicit upload, only the decoder would be able to produce HW frames from a source on the host. Data movement on an upload loop is centered around sending a SW frame on the host up to the uploader instance on the device. Outputting a HW frame as a response, this frame is then free to be used and reused among the encoders and 2D engine filters available without extra YUV data movement. For the SW frame upload path, it is very similar to the encoder write path though with some key distinctions.
The uploading loop with output being used to encode is as follows:
while (!end_of_all_streams &&
(send_rc == NI_TEST_RETCODE_SUCCESS ||
receive_rc== NI_TEST_RETCODE_SUCCESS))
{
if (first_frame_uploaded && !eos)
{
p_hwframe = hwupload_frame(&ctx, &upl_ctx, &sca_ctx, &swin_frame,
&in_frame, &scale_frame, pix_fmt,
video_width[0], video_height[0],
input_fp[0], yuv_buf, &eos);
if (upl_ctx.status == NI_RETCODE_NVME_SC_WRITE_BUFFER_FULL)
{
ni_log(NI_LOG_DEBUG, "No space to write to, try to read a packet\n");
p_in_frame = &in_frame;
goto receive_pkt;
} else if (p_hwframe == NULL)
{
ret = NI_TEST_RETCODE_FAILURE;
break;
}
}
// Sending
p_in_frame = is_ni_enc_pix_fmt(pix_fmt) ? &in_frame : &scale_frame;
for (i = 0; i < output_total; i++)
{
ctx.curr_enc_index = i;
send_rc = encoder_send_data3(&ctx, &enc_ctx[i], p_in_frame,
video_width[0], video_height[0], eos);
first_frame_uploaded = 1; //since first frame read before while-loop
if (send_rc < 0) //Error
{
ni_log(NI_LOG_ERROR, "enc %d send error, quit !\n", i);
ni_hw_frame_ref(p_hwframe);
end_of_all_streams = 1;
break;
}
if (!ctx.enc_resend[i])
{
//Increment reference count of HW frame buffer
//after sending to encoder
//It will later be recycled in encoder_receive
ni_hw_frame_ref(p_hwframe);
} else
{
ni_log(NI_LOG_DEBUG, "enc %d need to re-send !\n", i);
ni_usleep(500);
i--;
continue;
}
}
if (end_of_all_streams)
break;
receive_pkt:
receive_rc = encoder_receive(&ctx, enc_ctx, p_in_frame, out_packet,
video_width[0], video_height[0],
output_total, output_fp);
for (i = 0; receive_rc >= 0 && i < output_total; i++)
{
if (!ctx.enc_eos_received[i])
{
ni_log(NI_LOG_DEBUG, "enc %d continues to read!\n", i);
end_of_all_streams = 0;
break;
} else
{
ni_log(NI_LOG_DEBUG, "enc %d eos !\n", i);
end_of_all_streams = 1;
}
}
}
The main libxcoder API functions to complete the YUV conversion from SW frame to HW frame consist of ni_frame_buffer_alloc_pixfmt and ni_frame_buffer_alloc_hwenc to prepare the buffers, and ni_device_session_hwup to make the trade.
It’s possible to have a single HW frame being referenced by multiple encoders, essentially creating a ladder encoding with different encoding configurations. An example ni_xcoder_encode command demonstrating this is shown in section 5.4.2. Reference the ni_xcoder_encode source code for sample implementation details.
Recycling and Implicit Extra Allocation
As always, the HW frame nature of the output requires careful tracking of where the frame goes and when to recycle it with ni_hwframe_buffer_recycle2. Please follow the code example above (from ni_xcoder_encode) for reference.
Usually the framepool used during ni_device_session_init_framepool does not need to be very big. The reason for this is that any downstream entity will automatically expand the allocation count as needed for its purpose.
The clearest example of this would be if a HW upload instance is paired with an encoding instance having the lookahead depth at 40 frames. Naturally, the encoder instance needs at least a buffer count of 40 and if the upload pool was initialized as 3 frames, it would not seem to be enough. However, the encoder will know that it is receiving HW frame inputs and will allocate additional space during encoder open which are compatible with the original upload instance thus expanding the poolsize to be 3 + 40. No code example of this is needed as this is a FW implementation.
Zero Copy
Application should follow these steps to incorporate zero copy while uploading a YUV frame –
- Before uploading each frame, call ni_uploader_frame_zerocopy_check with input frame width, height, linesize (aka stride), and pixel format.
- If zero copy is applicable, call ni_encoder_frame_zerocopy_buffer_alloc with input frame width, height, linesize (aka stride), pointers to each data buffer (p_src in the example), and metadata length (extra_data_len in the example) Example:
Zero Copy while uploading a frame
The following example is based on FFmpeg hwupload filter, which receives an AvFrame (“src” in the following example) with an array of pointers to each of the luma / chroma data buffers, as well as an array of linesizes for each of the buffers. For users who implements applications based on libxcoder APIs (and therefore cannot utilitze FFmpeg or Gstreamer frame structure), please setup linesize array and data buffer pointer array and pass pointers of the arrays to libxcoder APIs
ni_encoder_frame_zerocopy_buffer_alloc extra_len parameter must be NI_APP_ENC_FRAME_META_DATA_SIZE
// check input resolution zero copy compatible or not
if (ni_uploader_frame_zerocopy_check(&ctx->api_ctx, src->width, src->height, (const int *)src->linesize, pixel_format) == NI_RETCODE_SUCCESS)
{
need_to_copy = 0;
p_src_session_data->data.frame.extra_data_ln = NI_APP_ENC_FRAME_META_DATA_SIZE;
// alloc metadata buffer etc. (if needed)
ret = ni_encoder_frame_zerocopy_buffer_alloc( &p_src_session_data->data.frame, src->width, src->height, (const int *)src->linesize, (const uint8_t **)src->data, (int)p_src_session_data->data.frame.extra_data_len);
if (ret != NI_RETCODE_SUCCESS)
return AVERROR(ENOMEM);
}
Please also refer to the following file / function for the complete sample code
FFmpeg libavutil/hwcontext_ni_quad.c ni_hwup_frame ()