Print
- Advanced Technology Laboratories
Transmission experiment using real-time codec compliant with the latest international standard of point cloud compression
～To easily transmit various and highly expressive 3D contents for practical use in digital twins～
January 24, 2023
KDDI Research, Inc.
KDDI Research, Inc. (head office at Fujimino-shi, Saitama; Hajime Nakamura, President and CEO) has announced today that it has successfully performed the world's first transmission experiment (Note 1) using real-time codecs compliant with point cloud compression (PCC), the latest international standard for 3D point cloud compression. The experiment demonstrated that 3D point clouds, which consist of massive amounts of data, can be substantially compressed without degrading the visual quality, allowing efficient and stable transmission via a mobile network.
The transmission of 3D point clouds corresponding to people and buildings via a mobile network makes it easier to reproduce real-world subjects in the virtual world with high expression and real-time. The transmitted point clouds can be viewed anytime, anywhere (Note 2) through platforms such as a metaverse. The practical use of 3D point clouds will be accelerated in various fields, such as new entertainment appearance in digital twins (Note 3), disaster support, and construction site assistance.
Figure 1: Examples of transmission of 3D point clouds
【Background】
A point cloud is a data format representing a 3D object as a set of XYZ and RGB information. Ultra-realistic contents with 3D point cloud, require compression technology for transmission due to the massive amount of data, such as 1Gbps.
PCC is the latest international standard for point cloud compression. PCC consists of two standards: video-based point cloud compression (V-PCC), has highly efficient compression for 3D point clouds of moving people, and geometry-based point cloud compression (G-PCC), suitable for 3D point clouds of buildings. Accumulating technologies and knowledge for practical use, KDDI Research has been involved in the international standardization of PCC since 2020.
V-PCC and G-PCC significantly reduce the amount of data compared to conventional technologies, enabling transmission of point cloud via a mobile network. On the other hand, the processing load of compression is high, which makes real-time processing difficult. KDDI Research successfully developed a real-time encoder compliant with V-PCC in October 2022 (Note 4) and then performed this transmission experiment.
【Achievements】
KDDI Research has successfully transmitted 3D point clouds via a mobile network using their developed real-time encoders compliant with PCC. These transmission experiments are the world’s first achievements in terms of V-PCC and G-PCC. The details are as follows.
1．Transmission experiment using V-PCC
Date: January 11, 2023
Overview: KDDI Research demonstrated the end-to-end V-PCC real-time transmission. Specifically, a high-density 3D point cloud of a person (approximately 20 million points/second) was generated in advance at a studio. Next, the 3D point cloud was compressed using a real-time V-PCC software encoder at the KDDI Research headquarters (Fujimino-shi, Saitama). Then, the compressed 3D point cloud was stably transmitted via a 5G mobile network to the KDDI research atelier (Minato-ku, Tokyo).
In October 2022, KDDI Research subsequently developed a real-time software encoder compliant with V-PCC on a cluster PC by introducing acceleration technology for V-PCC encoding process, enabling speeds that are approximately 400 times faster.
The transmission of the point cloud using the real-time encoder suggests that a new experience of live music concerts and fashion events on the metaverse will be created where point clouds of a singer and actor are captured in a volumetric studio (Note 5) and transmitted over a mobile network.
For such use cases, KDDI Research also demonstrated the real-time and end-to-end V-PCC real-time transmission: capturing and encoding a point cloud at a volumetric studio and viewing the transmitted point cloud on a holographic display or smartphone (Note 6).
Figure 2: Transmission experiment using V-PCC
2．Transmission experiment using G-PCC
Date: December 27, 2022
Overview: KDDI Research successfully demonstrated the real-time transmission targeting two use cases as follows. The first demonstrated capturing point cloud of the KDDI Research headquarters building (approximately 300,000 points/second) using a 3D scanner and transmitting it in real-time. The other demonstrated only transmitting a broader area point cloud of a dam (data consisting of roughly 20 million points divided into approximately 300,000 points/second for transmission) (Note 7) pre-generated using a drone aerial photography. Both point clouds were compressed using their G-PCC compliant and real-time encoder developed on a laptop PC. The compressed point clouds were stably transmitted from the KDDI Research headquarters (Fujimino-shi, Saitama) to the KDDI research atelier (Minato-ku, Tokyo) via a 5G/LTE mobile network.
KDDI Research had established an acceleration technology for G-PCC encoder process and realized real-time G-PCC encoding for point clouds of about 300,000 points/second by introducing a frame-level parallelism. This encoder can handle real-time processing for point cloud input up to about 2 million points/second.
This transmission experiment raises the expectation for practical use of G-PCC live-streaming. For instance, by utilizing a G-PCC real-time encoder mounted on a drone, digital transformation (DX) will be promoted, such as rapid understanding of disaster conditions and deployment of rescue operations, remote monitoring and technical support at infrastructure construction, etc.
Figure 3: Transmission experiment using G-PCC
<Reference> Details of V-PCC and G-PCC
| V-PCC | G-PCC | ||
|---|---|---|---|
| Features |
A video-based method that compresses 3D point clouds by projecting them onto 2D images. V-PCC is suitable for 3D point clouds where subjective quality is important, such as moving people. V-PCC can be applied to entertainment applications. |
A geometry-based method that compresses 3D point clouds as they are. G-PCC is suitable for 3D point clouds where encoding errors are not allowed, such as buildings that need to be surveyed. G-PCC can be applied to remote monitoring of construction sites. |
|
| Bitrate when uncompressed（in this transmission experiment） | 1.0Gbps（20 million points/second） |
26Mbps（300,000 points/second） |
|
| The amount of data (bitrate) when compressed |
reduced to 1/40 |
reduced to 1/21 |
|
| Compression performance |
Twice compared to conventional point cloud compression technology |
Six times compared to conventional point cloud compression technology |
|
【Future Vision】
KDDI Research will reduce the codec latency of real-time PCC transmission and develop highly interactive smartphone applications. In addition, KDDI Research will also promote activity to popularize 3D point cloud contents, aiming to create new entertainment expression on metaverse platforms and new solutions for disaster support and construction site assistance.
KDDI Research has also developed an interactive synthesis technology of sound fields that can reproduce directions and movements of sound source in the real world to the virtual world. KDDI Research will proceed with verification and evaluation, combining this technology and V-PCC.
This transmission experiment results from research and development supported by “Strategic Information and Communications R&D Promotion Programme (SCOPE)” of Ministry of Internal Affairs and Communications, Grant no. JPJ000595.
【KDDI Research's Initiatives】
KDDI and KDDI Research formulated “KDDI Accelerate 5.0” (Japanese only), based on a concept for the next-generation society envisioned for 2030, and summarized how it can be accomplished in the “Beyond 5G/6G Whitepaper” in terms of the “Future Image” and “Technologies” required to create an ecosystem that spurs innovation.
To develop a new lifestyle, the two companies will continue to invest in R&D on seven different technologies and on an orchestration technology that links them together. The results presented in this study correspond to one of the seven technologies, “XR”.
(Note 1) This is the world's first transmission experiment using real-time encoders compliant with V-PCC and G-PCC. (As of January 24, 2023, KDDI Research survey)
(Note 2) Within the area of mobile networks.
(Note 3) A technology that reproduces real-world subjects in the virtual world by capturing real-world 3D physical information through IoT devices, etc.
(Note 4) World’s first real-time encoder development for point-cloud-compression compliant with the latest international standard. (Announced on October 24, 2022)
(Note 5) A shooting studio that installs many cameras around the subject and creates 3D point cloud content from video data filmed from various angles.
(Note 6) In this demonstration, the 3D point cloud was transmitted through a fixed line and wireless LAN due to the restriction of the network environment at the reception side.
(Note 7) The point cloud obtained as part of a collaboration involving drone utilization by KDDI SmartDrone Inc. and Obayashi Corporation was used.
※The information contained in the articles is current at the time of publication.Products, service fees, service content and specifications, contact information, and other details are subject to change without notice.