DS-NeRV: Implicit Neural Video Representation with Decomposed Static and Dynamic Codes

Video implicit neural representations have recently evolved rapidly to become a new tool for high-quality video representation and compression. However, existing work mainly relies on neural networks to model videos uniformly, neglecting the distinct modeling of static and dynamic components across the entire video. Furthermore, this unified modeling approach underutilizes the temporal correlations present in videos. To exploit the temporal correlations of videos for efficient modeling, we propose DS-NeRV, which decomposes videos into sparse, learnable static and dynamic codes without the need for explicit optical flow or residual supervision. These codes represent the shared static and cross-time dynamic information within the video. Additionally, we design a cross-channel attention-based fusion module to fuse these static and dynamic codes for frame decoding. Our approach achieves a high quality reconstruction of 31.20 PSNR with only 0.35M parameters due to its compact static and dynamic codes design. We outperform existing video implicit neural representations in many downstream tasks and achieve comparable performance to some other conventional video compression methods.

DS-NeRV decomposes the video into learnable static and dynamic codes. Static Codes. The two gold yellow static codes shown above are the two nearest selected, and after being weighted sum, they are passed to the fusion decoder.Dynamic Codes. The original dynamic code distribution is shown in gray, and the interpolated dynamic code is shown in yellow. The selected one is shown in blue which is forwarded to fusion decoder.

(a) The pipeline of Fusion Decoder. Decoder takes as input the static and dynamic codes corresponding to index t and fuses their information to output frame. (b) Architecture of CCA Fusion Module. The CCA module fuses the information of static and dynamic codes by cross-channel attention.

Comparison of video reconstruction results on UVG and DAVIS. (Top) Jockey. (Bottom) Blackswan.