Despite the significant coding gains exhibited by the recently released new-generation video coding standards, their serious computational complexity will pose a significant challenge to practical applications.
Therefore, the corresponding low-complexity optimizations assume paramount importance.
In order to address this challenge, the first fast algorithm implementation-oriented open source library, named OpenFastVC, is developed for new coding standards, i,e, the third-generation audio video coding standard (AVS3), versatile video coding (VVC), and video-based point cloud compression (V-PCC).
The main contributions of OpenFastVC are as follows.
- Rapid acquisition of intermediate coding information: The design of fast algorithm generally is based on the statistics and analysis of intermediate coding information generated by the execution of the target module, while these information is not readily available in the reference software. To facilitate the analysis, OpenFastVC implements the macro switch on original coding platform, which furnishes the requisite information directly, thereby economizing researchers' time.
- Efficient integration of optimization schemes: Seamlessly integrating the devised fast algorithm into original encoder requires researchers to have a in-depth comprehension of the logical structure within the code, which is time-consuming. OpenFastVC supplies the application programming interface (API), where low-complexity optimizations, e.g., early termination and mode pruning, can be achieved by inputting the result of proposed algorithm in a preset format.
- Effortless evaluation of coding performance: After completed the integration of the designed algorithm, assessing the coding performance emerges as an indispensable stride. OpenFastVC facilitates this endeavor by proffering the implementation of customary algorithms, thereby affording researchers a convenient and expeditious means to finalize their algorithms.
OpenFastVC for AVS3 is developed based on the reference software of AVS3, named HPM. The deployment of the project follows the README.md of HPM.
In the deployed projected, OpenFastVC provides three macro switches to control the intermediate coding information output, coding decision process, and the reference fast algorithm, respectively. The current version of OpenFastVC primarily offers support for coding unit (CU) partition. The three macro switches are as follows.
#define OFVC_OUTPUT_CU_INFO 0
#define OFVC_CU_DECISION_CTRL 0
#if OFVC_CU_DECISION_CTRL
#define OFVC_CU_DECISION_REF_ALG1 0
#endif
a) Intermediate coding information output
Setting OFVC_OUTPUT_CU_INFO to 1 can enable the output of CU partitioning information,where an example is illustrated below for reference.
x and y are the position of current CU. w and h denote the CU width and height, respectively.
qt_depth reflects how many times the current CU has undergone QT partition thus far.
bet_depth denotes the split times by BT and EQT modes.
split_mode is the optimal split mode of current CU.
x:0-y:0-w:64-h:64-qt_depth:1-bet_depth:0-split_mode:1
x:0-y:0-w:32-h:64-qt_depth:1-bet_depth:1-split_mode:0
b) Designed algorithm integration
Setting OFVC_CU_DECISION_CTRL to 1 can achieve the control over CU partition process. Moreover, the utilization of each partitioning mode is controlled by fast_split_results[6]. The six digit number represent Non-Split, Ver-BT, Hor-BT, Ver-EQT, Hor-EQT, and QT, respectively. The value $0$ represents removing the corresponding partitioning mode from candidate list, and $1$ denotes retaining this mode. The designed fast algorithm can regulate the CU partition by assigning the corresponding value to fast_split_results[6].
int fast_split_results[6];
c) Coding performance evaluation
To facilitate the performance evaluation of the designed algorithm, OpenFastVC also provides reference fast algorithm. In the current version, the fast CU partitioning algorithm proposed in [1] has been implemented, where it can be enabled by setting OFVC_CU_DECISION_REF_ALG1 to 1.
[1] H. Yuan, W. Gao and J. Wang, "Dynamic Computational Resource Allocation for Fast Inter Frame Coding in Video Conferencing Applications," 2021 IEEE International Conference on Multimedia and Expo (ICME), Shenzhen, China, 2021, pp. 1-6, doi: 10.1109/ICME51207.2021.9428275.
OpenFastVC for VVC is developed based on the reference software of VVC, named VTM. The deployment of the project follows the README.md of VTM.
In the deployed projected, OpenFastVC provides six macro switches to control the intermediate coding information output, coding decision process, and the reference fast algorithm, respectively. The current version of OpenFastVC primarily offers support for coding unit (CU) partition and intra prediction. The six macro switches are as follows.
#define OFVC_OUTPUT_CU_INFO 0
#define OFVC_OUTPUT_IP_INFO 0
#define OFVC_CU_DECISION_CTRL 0
#define OFVC_IP_DECISION_CTRL 1
#if OFVC_CU_DECISION_CTRL
#define OFVC_CU_DECISION_REF_ALG1 0
#endif
#if OFVC_CU_DECISION_CTRL
#define OFVC_ISP_DECISION_REF_ALG1 1
#endif
a) Intermediate coding information output
Setting OFVC_OUTPUT_CU_INFO to 1 can enable the output of CU partitioning information,where an example is illustrated below for reference.
LUMA : [x:0-y:0-w:32-h:16]&&[depth:3-qt_depth:2-bt_depth:1-mt_depth:1] => ParentCU[x:0-y:0-w:32-h:32] => ParentCU[x:0-y:0-w:64-h:64] => ParentCU[x:0-y:0-w:128-h:128]
CHROMA : [x:0-y:0-w:16-h:16]&&[depth:2-qt_depth:2-bt_depth:0-mt_depth:0] => ParentCU[x:0-y:0-w:32-h:32] => ParentCU[x:0-y:0-w:64-h:64]
Note that the reference software of VVC only records the final optimal partitioning result. Hence, a Python script (PartitioningInfoExtract.py) is additionally provided in OpenFastVC to obtain intermediate information during CU decision. The processed result is as follows. x and y are the position of current CU. w and h denote the CU width and height, respectively. qt_depth reflects how many times the current CU has undergone QT partition thus far. bt_depth indicates the BT partitioning times of the current CU. mt_depth is the partitioning times by BT and TT. split_mode is the optimal split mode of current CU. As the independent coding of the luma and chroma components in VVC, LUMA and CHROMA indicate whether the current CU type.
LUMA : x:0-y:0-w:16-h:32-qt_depth:2-bt_depth:1-mt_depth:1-split_mode:0
CHROMA : x:0-y:0-w:16-h:16-qt_depth:2-bt_depth:0-mt_depth:0-split_mode:0
Setting OFVC_OUTPUT_IP_INFO to 1 can enable the output of intra prediction information. All the intra prediction modes in the candidate list are output, where MID is the ID of each mode, and ISPF is the ISP flag (-1: disabling ISP; 0: horizontal ISP; 1: vertical ISP). BestMode indicates the optimal intra prediction mode of current CU.
[x:0-y:0-w:8-h:8]<===>[MID:0, ISPF:-1]<===>[MID:1, ISPF:-1]<===>[MID:50, ISPF:-1]<===>[MID:0, ISPF:0]<===>[MID:0, ISPF:1]<===>[MID:50, ISPF:0]<===>[MID:50, ISPF:1]<===>BestMode:[MID:0, ISPF:-1]
b) Designed algorithm integration
Setting OFVC_CU_DECISION_CTRL to 1 can achieve the control over CU partition process. Moreover, the utilization of each partitioning mode is controlled by fast_split_results[6]. The six digits within $fast_split_results[6]$ correspond to Non-Split, Ver-BT, Hor-BT, Ver-TT, Hor-TT, and QT, respectively. The designed fast algorithm can regulate the CU partition by assigning the corresponding value to fast_split_results[6]. The value 0 represents removing the corresponding partitioning mode from candidate list, and 1 denotes retaining this mode.
int fast_split_results[6];
Setting OFVC_IP_DECISION_CTRL to 1 can achieve the control over intra prediction process. The availability of the ISP mode is controlled by the variable fast_ISP_flag (true or false). If the ID of an intra prediction mode is added to the disabling_mode_array, it will be removed from the candidate list. Note that the removing mode count should be assigned to disabling_mode_cnt.
bool fast_ISP_flag;
int* disabling_mode_array;
int disabling_mode_cnt;
c) Coding performance evaluation
To facilitate the performance evaluation of the designed algorithm, OpenFastVC also provides reference fast algorithms. In the current version, the fast CU partitioning algorithm proposed in [2] has been implemented, where it can be enabled by setting OFVC_CU_DECISION_REF_ALG1 to 1. Moreover, a fast ISP decision algorithm, optimized from [3], is also implemented in OpenFastVC. It can be enabled by setting OFVC_ISP_DECISION_REF_ALG1 to 1.
[2] H. Yang, L. Shen, X. Dong, Q. Ding, P. An and G. Jiang, "Low-Complexity CTU Partition Structure Decision and Fast Intra Mode Decision for Versatile Video Coding," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 6, pp. 1668-1682, June 2020, doi: 10.1109/TCSVT.2019.2904198.
[3] X. Dong, L. Shen, M. Yu and H. Yang, "Fast Intra Mode Decision Algorithm for Versatile Video Coding," in IEEE Transactions on Multimedia, vol. 24, pp. 400-414, 2022, doi: 10.1109/TMM.2021.3052348.
OpenFastVC for V-PCC is developed based on the reference software of V-PCC, named TMC2. The deployment of the project follows the README.md of TMC2.
In the deployed projected, OpenFastVC provides three macro switches to control the intermediate coding information output, coding decision process, and the reference fast algorithm, respectively.
The current version of OpenFastVC primarily offers support for coding unit (CU) partition.
The three macro switches are as follows.
#define OFVC_OUTPUT_CU_INFO 0
#define OFVC_CU_DECISION_CTRL 0
#if OFVC_CU_DECISION_CTRL
#define OFVC_CU_DECISION_REF_ALG1 0
#endif
a) Intermediate coding information output
Setting OFVC_OUTPUT_CU_INFO to 1 can enable the output of CU partitioning information,where an example is illustrated below for reference. x and y are the position of current CU. w and h denote the CU width and height, respectively. split_mode:0 or split_mode:1 denotes that the best partitioning mode of current CU is Non_Split or QT, respectively.
Coding CU ==> x:64−y:0−w:64−split_mode :1
Coding CU ==> x:64−y:0−w:32−split_mode :0
b) Designed algorithm integration
Setting OFVC_CU_DECISION_CTRL to 1 can achieve the control over CU partition process. Assigning 1 or 0 to fast_split_flag indicates executing the original CU decision strategy for the current CU of early terminating the current partition.
int fast_split_flag;
c) Coding performance evaluation
To facilitate the performance evaluation of the designed algorithm, OpenFastVC also provides reference fast algorithms. In the current version, the fast CU partitioning algorithm proposed in [4] has been implemented, where it can be enabled by setting OFVC_CU_DECISION_REF_ALG1 to 1.
[4] H. Yuan, W. Gao, G. Li, and Z. Li, "Rate-Distortion-Guided Learning Approach with Cross-Projection Information for V-PCC Fast CU Decision," the 30th ACM International Conference on Multimedia (MM '22), New York, USA,2022, pp. 3085–3093. doi: 10.1145/3503161.3548215.
