Decoding with LibXCoder
Decoding Loop
The decoding process involves sending encoded bitstream packets to the decoder and receiving decoded YUV or HW frames from the decoder. The NETINT decoder engine may need to buffer several bitstream packets before producing decoded frames depending on the stream’s GOP structure (i.e. when frames are encoded out of sequence). The simple and effective way of decoding is to alternate between sending and receiving in a loop that runs until all the bitstream packets have been sent and all the decoded frames have been decoded and returned.
The loop is as follows:
while (send_rc == NI_TEST_RETCODE_SUCCESS || receive_rc == NI_TEST_RETCODE_SUCCESS)
{
// Sending
send_rc = decoder_send_data(&ctx, &dec_ctx, &in_pkt, video_width,
video_height, p_stream_info);
if (send_rc < 0)
{
ni_log(NI_LOG_ERROR,
"Error: decoder_send_data() failed, rc: %d\n",
send_rc);
break;
}
// Receiving
do
{
rx_size = 0;
receive_rc = decoder_receive_data(&ctx, &dec_ctx, &out_frame, video_width, video_height,
output_fp, 1, &rx_size);
if (dec_ctx.hw_action == NI_CODEC_HW_ENABLE)
{
if (receive_rc != NI_TEST_RETCODE_EAGAIN &&
receive_rc != NI_TEST_RETCODE_END_OF_STREAM)
{
p_ni_frame = &out_frame.data.frame;
p_hwframe = (niFrameSurface1_t *)p_ni_frame->p_data[3];
ni_log(NI_LOG_DEBUG, "decoder_receive_data HW decode-only. recycle HW frame idx %u\n", p_hwframe->ui16FrameIdx);
ni_hwframe_buffer_recycle2(p_hwframe);
ni_frame_buffer_free(p_ni_frame);
}
}
else
{
ni_decoder_frame_buffer_free(&(out_frame.data.frame));
}
// Error or eos
if (receive_rc < 0 || out_frame.data.frame.end_of_stream)
{
break;
}
} while (!dec_ctx.decoder_low_delay && rx_size > 0); //drain consecutive outputs
// Error or eos
if (receive_rc < 0 || out_frame.data.frame.end_of_stream)
{
break;
}
}
The bitstream packet sending involves calling libxcoder API function ni_device_session_write, while the YUV frame receiving ni_device_session_read or ni_device_session_read_hwdesc. Check the ni_xcoder_decode program source code for details on preparing the data for sending and receiving, and the ways for the application to notify the decoder of start of stream (sos) and to get notified by the decoder of the end of stream (eos) using attributes in ni_packet_t and ni_frame_t struct.
Libxcoder implements a memory buffer pool for storing YUV raw data retrieved from decoder. This mechanism reduces frequent memory allocation/release and speeds up processing and is the recommended way for decoder YUV retrieval. Libxcoder API functions ni_decoder_frame_buffer_alloc and ni_decoder_frame_buffer_free are provided for this purpose. Check the ni_xcoder_decode program for their example usage.
As previously detailed in Section 5.4.2, if HW frames are used, the memory footprint on the host is drastically reduced and we would use ni_frame_buffer_alloc and ni_frame_buffer_free as provided for a simpler allocation and freeing method. There is also a critical step for HW frames once the application knows a specific HW frame is no longer needed. As described in the bottom of section 5.4.2 “Pathways and Features”, the HW frame needs to be recirculated to the device with ni_hwframe_buffer_recycle2. Check the transcoding, HW upload and encoding parts of the example programs for their example usage.
Multithreading
Parts of the decoding cycle can be augmented with multithreading. For low instance counts or smaller resolutions, this often greatly improves performance. The ni_xcoder_multithread_transcode program demonstrates the multithreaded version of decoding. One thread would handle sending bitstream packets to the decoder, while the other will pull YUV outputs from the decoder.
When taking advantage of multithreading, it is still recommended to keep the decoder open and close in sync, especially when transcoding is involved. This means that for greater reliability, all decoder read/write should be completed before sending a session close.
An example of decoding send and receive thread is shown below:
Decoder Send
void *decoder_send_thread(void *args)
{
dec_send_param_t *p_dec_send_param = args;
ni_demo_context_t *p_ctx = p_dec_send_param->p_ctx;
ni_session_context_t *p_dec_ctx = p_dec_send_param->p_dec_ctx;
ni_session_data_io_t in_pkt = {0};
int retval = 0;
ni_log(NI_LOG_INFO, "decoder_send_thread start: decoder_low_delay %d\n",
p_dec_ctx->decoder_low_delay);
while (!p_ctx->end_all_threads)
{
retval = decoder_send_data(p_ctx, p_dec_ctx, &in_pkt, p_dec_send_param->input_width,
p_dec_send_param->input_height, p_dec_send_param->p_stream_info);
if (retval < 0) // Error
{
ni_log(NI_LOG_ERROR, "Error: decoder send packet failed\n");
break;
} else if (p_dec_send_param->p_ctx->dec_eos_sent) //eos
{
ni_log(NI_LOG_INFO, "decoder_send_thread reached eos\n");
break;
} else if (retval == NI_TEST_RETCODE_EAGAIN) {
ni_usleep(100);
}
}
ni_packet_buffer_free(&in_pkt.data.packet);
// Broadcast all codec threads to quit on exception such as NVMe IO.
if (retval < 0)
{
p_ctx->end_all_threads = 1;
}
ni_log(NI_LOG_TRACE, "decoder_send_thread exit\n");
return (void *)(long)retval;
}
Decoder Receive
void *decoder_receive_thread(void *args)
{
dec_recv_param_t *p_dec_recv_param = args;
ni_demo_context_t *p_ctx = p_dec_recv_param->p_ctx;
ni_session_context_t *p_dec_ctx = p_dec_recv_param->p_dec_ctx;
ni_test_frame_list_t *frame_list = p_dec_recv_param->frame_list;
ni_session_data_io_t *p_out_frame = NULL;
ni_frame_t *p_ni_frame = NULL;
niFrameSurface1_t *p_hwframe;
int retval = 0;
int rx_size = 0;
ni_log(NI_LOG_INFO, "decoder_receive_thread start\n");
for (;;)
{
while (frame_list_is_full(frame_list) && !p_ctx->end_all_threads)
{
ni_usleep(100);
}
if (p_ctx->end_all_threads)
{
break;
}
p_out_frame = &frame_list->frames[frame_list->tail];
p_ni_frame = &p_out_frame->data.frame;
retval = decoder_receive_data(
p_ctx, p_dec_ctx, p_out_frame, p_dec_recv_param->input_width,
p_dec_recv_param->input_height, NULL, 0 /* no save to file */, &rx_size);
if (retval < 0) // Error
{
if (!p_dec_ctx->hw_action)
{
ni_decoder_frame_buffer_free(p_ni_frame);
} else
{
ni_frame_buffer_free(p_ni_frame);
}
ni_log(
NI_LOG_ERROR,
"Error: decoder_receive_thread break in transcode mode!\n");
break;
} else if (p_ni_frame->end_of_stream)
{
frame_list_enqueue(frame_list);
ni_log(NI_LOG_INFO, "decoder_receive_thread reach eos\n");
retval = 0;
break;
} else if (retval == NI_TEST_RETCODE_EAGAIN)
{
if (!p_dec_ctx->hw_action)
{
ni_decoder_frame_buffer_free(p_ni_frame);
} else
{
ni_frame_buffer_free(p_ni_frame);
}
ni_usleep(100);
} else
{
if (p_dec_ctx->hw_action)
{
uint16_t current_hwframe_index = ((niFrameSurface1_t *)p_ni_frame->p_data[3])->ui16FrameIdx;
ni_log(NI_LOG_DEBUG, "decoder recv:%d, tail:%d\n", current_hwframe_index, frame_list->tail);
}
frame_list_enqueue(frame_list);
}
}
ni_frame_buffer_free(&filter_out_frame.data.frame);
// Broadcast all codec threads to quit on exception such as NVMe IO.
if (retval < 0)
{
p_ctx->end_all_threads = 1;
}
ni_log(NI_LOG_TRACE, "decoder_receive_thread exit\n");
return (void *)(long)retval;
}
Decoding Input
The NETINT Quadra decoder engine requires the bitstream to be passed in complete frames so libxcoder expects the same for its input. The example programs provides very simple H.264, H.265 and VP9 parsers to do the frame partitioning.
NETINT Pixel Formats
The NETINT Quadra card supports the following:
- YUV420P(8 and 10-bit) – Decoder output, 2D engine input/output, Upload input, Encoder input
- NV12 ( 8 and 10-bit) – Decoder output, 2D engine input/output, Upload input, Encoder input
- Tiled4x4 (8 and 10-bit) – Decoder HWframe output, Encoder input
- RGBA (2D engine only)
Note that 10 bit video uses 16 bit words in little endian format. For planar format, the ten least significant bits store the component value with the six most significant bits as don’t care. For semi-planar format, the ten most significant bits store the component value and the six least significant bits are not used.
Figure 44 shows the NETINT 8 bit YUV format. The Y components are stored first followed by the U and then the V both subsampled by 2 in each direction. Each component is 8 bits.
Figure 4 NETINT YUV420 8 bit Format
10 bit pixels are stored as 16 bits as shown in Figure 6.
Figure 5 NETINT 10 bit Pixel Format
YUV420 Alignment Requirements
The NETINT Quadra hardware places certain restrictions on the height and width of the YUV data. The picture width (stride) must be 128-byte aligned and height (in pixels) must be even. Picture sizes that do not meet these requirements must be padded before encoding. Minimum width and height requirements of 144 and 128 pixels respectively must also be met and aligned to previous rules. Decoded YUV frames will be output aligned.
2-Pass and AV1 encode input have extra restrictions which are described in the Quadra Integration & Programming Guide[1], section 8.4.5 “Encoder Limitations”. They may require extra input padding.
Figure 6 shows how the padding must be added to meet the alignment requirements. For example, to encode a 1916x1076 stream to HEVC, the width of the YUV picture must be padded to 1920 to be divisible by 128. The height can remain the same. Another example would be a 1200x720 picture would require horizontal padding equal to 80 bytes Luma and 40 bytes chroma for H264/H265 encoding.
Figure 6 NETINT YUV420 Padding
To meet the alignment requirements, encoder input may need to be padded and decoder output may need to be cropped. Libxcoder provides a data structure, attributes and API functions for the frame operations as follows:
typedef struct _ni_frame
{
uint8_t *p_data[NI_MAX_NUM_DATA_POINTERS];
uint32_t data_len[NI_MAX_NUM_DATA_POINTERS];
uint32_t video_width;
uint32_t video_height;
uint32_t crop_top;
uint32_t crop_bottom ;
uint32_t crop_left ;
uint32_t crop_right ;
…
} ni_frame_t ;
In this video frame data structure, p_data has the pointers to the data of picture planes, data_len specifies each plane size in bytes. As illustrated in Figure 7, video_width and video_height is the padded picture size , crop_* specifies a cropping window which is used when decoding to crop the padded picture to original picture size (rectangle area in black in Figure 7):
- crop_left the horizontal pixel offset of top-left corner of rectangle from (0, 0),
- crop_top the vertical pixel offset of top-left corner of rectangle from (0, 0),
- crop_right the horizontal pixel offset of bottom-right corner of rectangle from (0, 0) and
- crop_bottom the vertical pixel offset of bottom-right corner of rectangle from (0, 0).
Figure 7 NETINT YUV420 Decoded Frame and Cropping Window
NETINT Decoder Output Data Layout
In addition to the YUV data (or HW descriptors), the decoder returns other information associated with the frame such as frame type, cropping info, frame offset, and any SEI’s associated with the frame such as closed captions, HDR colour information etc. Figure 8 shows the layout of the decoder output data.
Figure 8 Decoder output layout
YUV/Hwdescriptors and metadata header are mandatory. Metadata header has the information described in the following struct:
typedef struct _ni_metadata_common
{
uint16_t crop_left;
uint16_t crop_right;
uint16_t crop_top;
uint16_t crop_bottom;
union
{
uint64_t frame_offset;
uint64_t frame_tstamp;
} ui64_data;
uint16_t frame_width;
uint16_t frame_height;
uint16_t frame_type;
uint16_t reserved;
} ni_metadata_common_t;
The cropping and resolution information is used to fill in ni_frame_t attributes presented in section “YUV Alignment Requirements”. Other information (shown as auxiliary data) from supported SEIs, if present, is then used to fill in the remaining attributes of ni_frame_t as follows:
typedef struct _ni_frame
{
… …
// SEI info of closed caption: returned by decoder or set by encoder
unsigned int sei_cc_offset;
unsigned int sei_cc_len;
// SEI info of HDR: returned by decoder
unsigned int sei_hdr_mastering_display_color_vol_offset;
unsigned int sei_hdr_mastering_display_color_vol_len;
unsigned int sei_hdr_content_light_level_info_offset;
unsigned int sei_hdr_content_light_level_info_len;
// SEI info of HDR10+: returned by decoder
unsigned int sei_hdr_plus_offset;
unsigned int sei_hdr_plus_len;
// SEI info of User Data Unregistered SEI: returned by decoder
unsigned int sei_user_data_unreg_offset;
unsigned int sei_user_data_unreg_len;
} ni_frame_t;
Each pair of attributes specify the location of the respective SEI raw data: offset from the start, and size of the data. This information will be used to extract and convert the SEI data into a more useable format by libxcoder. The supported SEIs include: T35 closed captions, HDR static metadata (mastering display color volume, content light level), T35 HDR10+ dynamic metadata, and user data unregistered.
Closed caption data is formatted as per the T35 SEI payload data specified by CEA708[7]. The user data unregistered data includes the UUID and is formatted as per the H.264[5] and H.265[6] standards.
The static HDR metadata is defined in the following structures:
ni_dec_mastering_display_colour_volume_tni_content_light_level_info_t
The HDR10+ dynamic metadata is defined by the following structure:
ni_dynamic_hdr_plus_t
These SEIs, together with other information such as ROI (Region of Interest) and reconfiguration data (used for encoding) are called auxiliary data in the libxcoder.
The following sections give more details on the auxiliary data and its usage, and in the encoding section we see that the auxiliary data is passed to the encoder for insertion into the encoded stream.
Auxiliary data
Auxiliary data is information that accompanies a video frame. It is returned by the decoder as a result of the decoding process, e.g. a closed caption that comes with a decoded video frame. It may also be supplied by the user to be included in the encoded bitstream, or for the case of ROI, to permit frame by frame control of encoding quality.
The following auxiliary data types have been defined:
typedef enum _ni_frame_aux_data_type
{
NI_FRAME_AUX_DATA_NONE = 0,
// ATSC A53 Part 4 Closed Captions
NI_FRAME_AUX_DATA_A53_CC,
// HDR10 mastering display metadata associated with a video frame
NI_FRAME_AUX_DATA_MASTERING_DISPLAY_METADATA,
// HDR10 content light level (based on CTA-861.3). This payload contains
// data in the form of ni_content_light_metadata_t struct.
NI_FRAME_AUX_DATA_CONTENT_LIGHT_LEVEL,
// HDR10+ dynamic metadata associated with a video frame. The payload is
// a ni_dynamic_hdr_plus_t struct that contains information for color volume
// transform - application 4 of SMPTE 2094-40:2016 standard.
NI_FRAME_AUX_DATA_HDR_PLUS,
// Regions of Interest, the payload is an array of ni_region_of_interest_t,
// the number of array element is implied by:
// ni_frame_aux_data.size / sizeof(ni_region_of_interest_t)
NI_FRAME_AUX_DATA_REGIONS_OF_INTEREST,
// NETINT: user data unregistered SEI data, which takes SEI payload type
// USER_DATA_UNREGISTERED.
// There will be no byte reordering.a
// Usually this payload would be: 16B UUID + other payload Bytes.
NI_FRAME_AUX_DATA_UDU_SEI,
// NETINT: custom SEI data, which takes SEI payload custom types.
// There will be no byte reordering.
// Usually this payload would be: 1B Custom SEI type + 16B UUID + other
// payload Bytes.
NI_FRAME_AUX_DATA_CUSTOM_SEI,
// NETINT: custom bitrate adjustment, which takes int32_t type data as
// payload that indicates the new target bitrate value.
NI_FRAME_AUX_DATA_BITRATE,
// NETINT: custom VUI adjustment, which is a struct of
// ni_long_term_ref_t that specifies a frame's support of long term
// reference frame.
NI_FRAME_AUX_DATA_VUI,
// NETINT: long term reference frame support, which is a struct of
// ni_long_term_ref_t that specifies a frame's support of long term
// reference frame.
NI_FRAME_AUX_DATA_LONG_TERM_REF,
// NETINT: long term reference interval adjustment, which takes int32_t
// type data as payload that indicates the new long term reference interval
// value.
NI_FRAME_AUX_DATA_LTR_INTERVAL,
// NETINT: frame reference invalidation, which takes int32_t type data
// as payload that indicates the frame number after which all references
// shall be invalidated.
NI_FRAME_AUX_DATA_INVALID_REF_FRAME,
// NETINT: custom framerate adjustment, which takes int32_t type data as
// payload that indicates the new target framerate numerator and denominator values.
NI_FRAME_AUX_DATA_FRAMERATE,
// NETINT: custom maxFrameSize adjustment, which takes int32_t type data as
// payload that indicates the new maxFrameSize value
NI_FRAME_AUX_DATA_MAX_FRAME_SIZE,
} ni_aux_data_type_t;
Note that Quadra currently supports decoding a maximum of 1024 bytes of SEI payload.
Retrieval and Storage
The auxiliary data returned by the decoder is in a low level format ni_query_session_statistic_info and can be converted into a more useable format by calling the ni_dec_retrieve_aux_data API function. The new format can now be used when transcoding to pass on closed captions and HDR SEIs to the encoder.
Libxcoder converts auxiliary data into an easy-to-use format and stores them in the frame, by the following struct:
// struct to hold auxiliary data for ni_frame_t
typedef struct _ni_aux_data
{
ni_aux_data_type_t type;
uint8_t *data;
int size;
} ni_aux_data_t;
typedef struct _ni_frame
{
// frame auxiliary data
ni_aux_data_t *aux_data[NI_MAX_NUM_AUX_DATA_PER_FRAME];
int nb_aux_data;
}
The data attribute in ni_aux_data_t is either a struct defined for that info (e.g. ni_dynamic_hdr_plus_t for HDR10+), or a number of payload bytes (e.g. A53 closed captions or user data unregistered data). These are further detailed in the following sections.
User Data Unregistered SEI
The ni_xcoder_transcode_filter and ni_xcoder_multithread_transcode programs with “-u” parameter enables User Data Unregistered SEI passthrough. If the parameter is enabled and the User Data Unregistered SEI is available, the SEI is retrieved and then stored as a number of payload bytes directly in the ni_aux_data_t. This payload includes 16 bytes of UUID plus other payload bytes as defined in the H.264[5] and H.265[6] standards and shown in Figure 9.
Figure 9 User Data Unregistered SEI data format
Closed caption
The closed caption data, when available, is stored as a number of payload bytes directly in the ni_aux_data_t. The format is the T35 payload as specified by CEA-708[7] as shown in Figure 10 and Figure 11.
| Length (bits) | Name | Type | Value |
|---|---|---|---|
| 8 | itu_t_t35_country_code | uimsbf | 181 |
| 16 | itu_t_t35_provider_code | uimsbf | 49 or 47 |
| 32 | ATSC_user_identifier (only if provider is 49) | ASCII bslbf | GA94 |
| 8 | ATSC1_data_user_data_type_code (only if provider is 47 or 49) | uimsbf | 3 |
| 8 | DIRECTV_user_data_length (only if provider is 47) | uimsbf | variable |
| X*8 | user_data_type_structure | binary | free form |
Figure 10 CEA-708 T35 Payload Format
| Length (bits) | Name | Type | Value |
|---|---|---|---|
| 1 (b7) | process_em_data_flag | flag | 1 |
| 1 (b6) | process_cc_data_flag | flag | 1 |
| 1 (b5) | additional_data_flag | flag | 0 |
| 5 (b0-b4) | cc_count | uimsbf | Variable |
| 8 | em_data (not in CDP data) | uimsbf | 256 |
| cc_count*24 | cc_data_pkt's | bslbf | free form |
| 8 | marker_bits (not in CDP data) | patterned bslbf | 256 |
| 34 | ATSC_reserved_user_data (not in CDP data) | bslbf | free form |
Figure 11 CEA-708 user_data_type_structure
HDR10 Mastering Display Metadata
The HDR10 mastering display metadata, when available, is stored in the following struct and saved in the ni_aux_data_t:
// struct describing HDR10 mastering display metadata
typedef struct _ni_mastering_display_metadata
{
// CIE 1931 xy chromaticity coords of color primaries (r, g, b order).
ni_rational_t display_primaries[3][2];
// CIE 1931 xy chromaticity coords of white point.
ni_rational_t white_point[2];
// Min luminance of mastering display (cd/m^2).
ni_rational_t min_luminance;
// Max luminance of mastering display (cd/m^2).
ni_rational_t max_luminance;
// Flag indicating whether the display primaries (and white point) are set.
int has_primaries;
// Flag indicating whether the luminance (min_ and max_) have been set.
int has_luminance;
} ni_mastering_display_metadata_t;
HDR10 Content Light Level Info
The HDR10 content light level info, when available, is stored in the following struct and saved in the ni_aux_data_t:
// struct describing HDR10 Content light level
typedef struct _ni_content_light_level
{
// Max content light level (cd/m^2).
uint16_t max_cll;
// Max average light level per frame (cd/m^2).
uint16_t max_fall;
} ni_content_light_level_t;
HDR10+ Dynamic Metadata
The HDR10+ Dynamic Metadata, when available, is stored in the following struct and associated sub-struct, and saved in the ni_aux_data_t:
// struct representing dynamic metadata for color volume transform -
// application 4 of SMPTE 2094-40:2016 standard.
typedef struct _ni_dynamic_hdr_plus
{
// Country code by Rec. ITU-T T.35 Annex A. The value shall be 0xB5.
uint8_t itu_t_t35_country_code;
// Application version in the application defining document in ST-2094
// suite. The value shall be set to 0.
uint8_t application_version;
// The number of processing windows. The value shall be in the range
// of 1 to 3, inclusive.
uint8_t num_windows;
// The color transform parameters for every processing window.
ni_hdr_plus_color_transform_params_t params[3];
// The nominal maximum display luminance of the targeted system display,
// in units of 0.0001 candelas per square metre. The value shall be in
// the range of 0 to 10000, inclusive.
ni_rational_t targeted_system_display_maximum_luminance;
// This flag shall be equal to 0 in bit streams conforming to this version
// of this Specification. The value 1 is reserved for future use.
uint8_t targeted_system_display_actual_peak_luminance_flag;
// The number of rows in the targeted system_display_actual_peak_luminance
// array. The value shall be in the range of 2 to 25, inclusive.
uint8_t num_rows_targeted_system_display_actual_peak_luminance;
// The number of columns in the
// targeted_system_display_actual_peak_luminance array. The value shall be
// in the range of 2 to 25, inclusive.
uint8_t num_cols_targeted_system_display_actual_peak_luminance;
// The normalized actual peak luminance of the targeted system display. The
// values should be in the range of 0 to 1, inclusive and in multiples of
// 1/15.
ni_rational_t targeted_system_display_actual_peak_luminance[25][25];
// This flag shall be equal to 0 in bitstreams conforming to this version of
// this Specification. The value 1 is reserved for future use.
uint8_t mastering_display_actual_peak_luminance_flag;
// The number of rows in the mastering_display_actual_peak_luminance array.
// The value shall be in the range of 2 to 25, inclusive.
uint8_t num_rows_mastering_display_actual_peak_luminance;
// The number of columns in the mastering_display_actual_peak_luminance
// array. The value shall be in the range of 2 to 25, inclusive.
uint8_t num_cols_mastering_display_actual_peak_luminance;
// The normalized actual peak luminance of the mastering display used for
// mastering the image essence. The values should be in the range of 0 to 1,
// inclusive and in multiples of 1/15.
ni_rational_t mastering_display_actual_peak_luminance[25][25];
} ni_dynamic_hdr_plus_t;
Decoder Sequence Change
A sequence change is a change in picture size or bit depth of the input bitstream while decoding. To support this with the NETINT decoder, the application must check for differences in the output frame resolution/depth and reallocate frame buffers as needed. The example programs does not currently support this.