Evaluating the Emerging MPEG Video Coding for Machines in Semantic Segmentation

Abstract

Emerging MPEG Video Coding for Machines (MPEG VCM) standardization activities address the growing demand for machine-to-machine visual applications, including video surveillance, autonomous driving, etc. This paper proposes an evaluation methodology tailored to MPEG VCM, with an emphasis on semantic segmentation tasks using the Pandaset dataset. This is a challenging target as standardization works must follow several limitations, such as dataset’s licensing, fixed tools and software, and compliance with existing common test conditions (CTC) for the standard’s development. The proposed evaluation methodology includes a step to align Pandaset and COCO labels. Two task networks, Detectron2 and Mask2Former, are used to evaluate the Rate-Performance behavior for semantic segmentation under various coding configurations. The performance of MPEG VCM is benchmarked against traditional codecs (VVC and HEVC), with a detailed analysis of MPEG VCM’s coding tools. The extensive evaluations reveal interesting observations. (1) Although VCM achieved reasonably good segmentation performance, some of its developed tools, such as temporal resampling and region-of-interest coding, were not well suited for segmentation task. (2) The Hybrid NNVVC Inner Codec outperformed the VVC Inner Codec. (3) VCM’s performance varies significantly for segmented classes. (4) Despite significant differences in human vision performance, VVC and HEVC exhibit relatively similar performance in machine vision. The main contributions of this work are to (1) enable evaluation of MPEG VCM in a real-world semantic segmentation use case, which is one of VCM’s targeted tasks, and (2) to provide a detailed assessment of VCM’s performance in semantic segmentation.

Index terms: video compression, video coding for machines, semantic segmentation, autonomous driving, computer vision

Standardization impact

The evaluation methodology from this work was adopted into the MPEG Common Test Conditions used to develop the Video Coding for Machines standard (MPEG contribution m67677, ISO/IEC JTC 1/SC 29/WG 4, April 2024).

BibTeX

@inproceedings{phamdinh2025vcm,
  author    = {Pham-Dinh, Khoa and Pakdaman, Farhad and Zhang, Honglei and
               Rezazadegan Tavakoli, Hamed and Le, Nam and Ahonen, Jukka and
               Gabbouj, Moncef},
  title     = {Evaluating the Emerging {MPEG} Video Coding for Machines in
               Semantic Segmentation},
  booktitle = {2025 IEEE International Symposium on Multimedia (ISM)},
  address   = {Naples, Italy},
  year      = {2025},
  month     = dec,
  pages     = {141--148},
  publisher = {IEEE},
  doi       = {10.1109/ISM66958.2025.00038}
}