Neural Scene Representation and Neural Rendering

Seminar – Fall Semester 2024

Instructor: Lingjie Liu

Organization | Content | Format | Resources


NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis ECCV, 2020	NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction NeurIPS, 2021

Topics

Paper Presenting Schedule	Presentation
Introduction slides (by Lingjie)	Aug 28
Introduction 2 slides (by Lingjie)	Sep 4
Introduction 2 (Cont.) slides (by Lingjie) MonoNeRF: Learning a Generalizable Dynamic Radiance Field from Monocular Videos Tian et al., ICCV 2023 slides	Sep 9 Presenter 1: Fengrui Tian
3D Gaussian Splatting for Real-Time Radiance Field Rendering Kerbl et al., SIGGRAPH 2023 (Best Paper Award) slides 2D Gaussian Splatting for Geometrically Accurate Radiance Fields Huang et al., SIGGRAPH 2024 slides	Sep 11 Presenter 1: Daniel Alexander Presenter 2: Wentinn Liao
Fast Rendering of Neural Radiance Fields slides (by Lingjie) Instant Neural Graphics Primitives with a Multiresolution Hash Encoding Müller et al., SIGGRAPH 2022 (Best Paper Award) slides TensoRF: Tensorial Radiance Fields Chen and Xu et al., ECCV 2022 + Factor Fields: A Unified Framework for Neural Fields and Beyond Chen et al., SIGGRAPH 2023 + PlenOctrees for Real-time Rendering of Neural Radiance Fields Yu et al., ICCV 2021 (Oral) + Plenoxels: Radiance Fields without Neural Networks Fridovich-Keil et al., CVPR 2022 (Oral) slides	Sep 16 Presenter 1: Hungju Wang Presenter 2: Qiao Feng
Fast Training of Neural Radiance Fields slides (by Lingjie) Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields Barron et al., ICCV 2021 (Oral, Best Paper Honorable Mention) + Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields Barron et al., CVPR 2022 (Oral Presentation) + Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields Barron et al., ICCV 2023 (Oral Presentation, Best Paper Finalist) slides Mip-Splatting: Alias-free 3D Gaussian Splatting Yu et al., CVPR 2024 (Best Student Paper Finalist) slides	Sep 18 Presenter 1: Matthew Leonard Presenter 2: Yunzhou Song
Unbounded Scene Modeling slides (by Lingjie) MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes Reiser et al., SIGGRAPH 2023 + SMERF: Streamable Memory Efficient Radiance Fields for Real-Time Large-Scene Exploration Duckworth and Hedman et al., SIGGRAPH 2024 (Best Paper Honorable Mention) slides Grid-guided Neural Radiance Fields for Large Urban Scenes Xu et al., CVPR 2023 slides	Sep 23 Presenter 1: Chen Wang Presenter 2: Wentinn Liao
Dynamic 3D Gaussian Fields for Urban Areas Fischer et al., arXiv 2024 slides pixelNeRF Neural Radiance Fields from One or Few Images Yu et al., CVPR 2021 slides	Sep 25 Presenter 1: Yiduo Hao Presenter 2: Hungju Wang
Guest Talk: Represent, Reconstruct and Generate the 4D Real World slides	Sep 30 Presenter: Jiahui Lei
Generalization of Neural Fields slides (by Lingjie) PixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction Charatan et al., CVPR 2024 (Oral) slides LRM: Large Reconstruction Model for Single Image to 3D Hong et al., ICLR 2024 (Oral) slides	Oct 2 Presenter 1: Lee Milburn Presenter 2: Fengtui Tian
DreamFusion: Text-to-3d using 2D diffusion Poole et al., ICLR 2023 + ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation Wang et al. NeurIPS 2023 (Spotlight) slides InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models Xu et al. arXiv 2024 + LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation Tang et al. ECCV 2024 (Oral) + One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion Liu et al. CVPR 2024 slides	Oct 7 Presenter 1: Alex Radchenko Presenter 2: Chen Wang
3D GANs slides (by Lingjie) GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting Chen et al., CVPR 2024 slides PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction Wang et al. arXiv 2024 + SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse Views Xu et al. ECCV 2024 slides	Oct 9 Presenter 1: Xiangyu Han Presenter 2: Albert Wang
Guest Talk: Towards Scalable and Knowledgeable Generative Intelligence slides	Oct 14 Presenter: Jiatao Gu
Guest Talk: Generative Embodied AI slides video	Oct 16 Presenter: Ruoshi Liu
3D Gaussian Ray Tracing: Fast Tracing of Particle Scenes Moenne-Loccoz et al., SIGGRAPH Asia 2024 slides NICER-SLAM: Neural Implicit Scene Encoding for RGB SLAM Zhu et al., 3DV 2024 (Oral, Best Paper Honorable Mention) slides	Oct 21 Presenter 1: Lee Milburn Presenter 2: Zainab Afolabi
Guest Talk: Shapes as Fields: Toward Geometry Processing without Discretization slides	Oct 23 Presenter: Guandao Yang
BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis Yariv et al., SIGGRAPH 2023 slides Splatter Image: Ultra-Fast Single-View 3D Reconstruction Szymanowicz et al., CVPR 2024 slides	Oct 28 Presenter 1: Zainab Afolabi Presenter 2: Erik Jagnandan
EG3D: Efficient Geometry-aware 3D Generative Adversarial Networks Chan et al., CVPR 2022 slides 3D generation on ImageNet Skorokhodov et al., ICLR 2023 (Oral)	Oct 30 Presenter 1: Yihang Liu Presenter 2: Matthew Leonard
K-Planes: Explicit Radiance Fields in Space, Time, and Appearance Fridovich-Keil et al., CVPR 2024 slides 4K4D: Real-Time 4D View Synthesis at 4K Resolution Xu et al., CVPR 2024 slides	Nov 4 Presenter 1: Linzhan Mou Presenter 2: Albert Wang
COLMAP-Free 3D Gaussian Splatting Fu et al., CVPR 2024 (Highlight) slides FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flow Smith et al., NeurIPS 2023 slides	Nov 6 Presenter 1: Boshu Lei Presenter 2: Yunzhou Song
TensoIR: Tensorial Inverse Rendering Jin et al., CVPR 2023 slides Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF Decomposition and Ray Tracing Zhang et al., ECCV 2024 slides	Nov 11 Presenter 1: Yihang Liu Presenter 2: Carlos Lopez Garces
	Nov 13
PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics Xie et al., CVPR 2024 slides PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual Observations Zheng et al., ECCV 2024 slides	Nov 18 Presenter 1: Boshu Lei Presenter 2: Xiangyu Han
Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions Haque et al., ICCV 2023 (Oral) slides PlatoNeRF: 3D Reconstruction in Plato's Cave via Single-View Two-Bounce Lidar Klinghoffer et al., CVPR 2024 (Oral) slides	Nov 20 Presenter 1: Erik Jagnandan Presenter 2: Yiduo Hao
LERF: Language Embedded Radiance Fields Kerr et al. ICCV 2023 (Oral) + LERF-TOGO: Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping Rashid et al. CORL 2023 (Best Paper Finalist) slides Learning Visual Parkour from Generated Images Yu et al., CoRL 2024 slides	Nov 25 Presenter 1: William Liang Presenter 2: William Liang
	Nov 27
NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction Wang et al. NeurIPS 2021 (Spotlight) + NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction Wang et al. ICCV 2023 slides Gaussian Opacity Fields: Efficient and Compact Surface Reconstruction in Unbounded Scenes Yu et al., SIGGRAPH ASIA 2024	Dec 2 Presenter 1: Qiao Feng Presenter 2: Linzhan Mou
NeurCross: A Self-Supervised Neural Approach for Representing Cross Fields in Quad Mesh Generation Dong et al., arXiv 2024 slides Flexible Isosurface Extraction for Gradient-Based Mesh Optimization Shen et al., SIGGRAPH 2023 slides	Dec 4 Presenter 1: Carlos Lopez Garces Presenter 2: Alex Radchenko
Tutorial Lecture: NeRF Studio slides (by Chuhao & Xuyi) ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis Yu et al., arXiv 2024	Dec 9 Presenter 1: Xuyi Meng & Chuhao Chen Presenter 2: Daniel Alexander

The following table lists all the topics and papers that will be discussed in the seminar. Once every participant has submitted their choice of topics and papers, the previous list will be updated to show the presenter of each topic. Send us an email if you cannot access a paper for some reason.

Click on each topic to show the papers to be discussed or show all papers hide all papers.

Topic and Papers
Fast Inference BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis Yariv et al. SIGGRAPH 2023 3D Gaussian Splatting for Real-Time Radiance Field Rendering Kerbl et al. SIGGRAPH 2023 (Best Paper Award) 2D Gaussian Splatting for Geometrically Accurate Radiance Fields Huang et al. SIGGRAPH 2024
Fast Training Instant Neural Graphics Primitives with a Multiresolution Hash Encoding Müller et al. SIGGRAPH 2022 (Best Paper Award) TensoRF: Tensorial Radiance Fields Chen and Xu et al. ECCV 2022 + Factor Fields: A Unified Framework for Neural Fields and Beyond Chen et al. SIGGRAPH 2023 + PlenOctrees for Real-time Rendering of Neural Radiance Fields Yu et al. ICCV 2021 (Oral) + Plenoxels: Radiance Fields without Neural Networks Fridovich-Keil et al. CVPR 2022 (Oral)
Antialiasing Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields Barron et al. ICCV 2021 (Oral, Best Paper Honorable Mention) + Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields Barron et al. CVPR 2022 (Oral Presentation) + Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields Barron et al. ICCV 2023 (Oral Presentation, Best Paper Finalist) Mip-NeRF v.s. Mip-NeRF 360 v.s. Zip-NeRF: Common: Address the aliasing artifacts of NeRF. Mip-NeRF: Mitigates aliasing artifacts at different resolutions by replacing point sampling with Gaussian sampling. Mip-NeRF 360: Extends Mip-NeRF to unbounded scenes using a non-linear scene parameterization to allocate appropriate capacity for foreground and background. Zip-NeRF: Addresses z-aliasing artifacts from Mip-NeRF 360's resampling and adapts to an efficient grid representation using multisampling within a conical frustum. Mip-Splatting: Alias-free 3D Gaussian Splatting Yu et al. CVPR 2024 (Best Student Paper Finalist)
Large (Unbounded) Scenes MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes Reiser et al. SIGGRAPH 2023 + SMERF: Streamable Memory Efficient Radiance Fields for Real-Time Large-Scene Exploration Duckworth and Hedman et al. SIGGRAPH 2024 (Best Paper Honorable Mention) MERF v.s. SMERF: Common: Use compact representation to achieve high-quality real-time volumetric rendering. MERF: Proposed a combination of a low-resolution 3D grid and a set of higher-resolution 2D planes. SMERF: Supports real-time rendering on mobile devices; dedicates each viewpoint a MERF for large scenes. Grid-guided Neural Radiance Fields for Large Urban Scenes Xu et al. CVPR 2023 Dynamic 3D Gaussian Fields for Urban Areas Fischer et al. arXiv 2024
Generalization pixelNeRF Neural Radiance Fields from One or Few Images Yu et al. CVPR 2021 PixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction Charatan et al. CVPR 2024 (Oral) (infers a 3D Gaussian scene from two input views in a single forward pass.) LRM: Large Reconstruction Model for Single Image to 3D Hong et al. ICLR 2024 (Oral)
3D Generative Model [Per-scene optimization: diffusion distillation] DreamFusion: Text-to-3d using 2D diffusion Poole et al. ICLR 2023 + ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation Wang et al. NeurIPS 2023 (Spotlight) [Single-view image → Multi-view image → 3D reconstruction] Cat3D: Create Anything in 3D with Multi-View Diffusion Models Gao et al. arXiv 2024 InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models Xu et al. arXiv 2024 + LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation Tang et al. ECCV 2024 (Oral) + One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion Liu et al. CVPR 2024 [Pose-free 3D Generation] PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction Wang et al. arXiv 2024 + SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse Views Xu et al. ECCV 2024 PF-LRM v.s. SpaRP: Common: 3D reconstruction from sparse unknown-posed images. PF-LRM: Explicit matching through pointcloud + differentiable PnP solver. SpaRP: Distill stable diffusion model to predict NOCS images for camera pose estimation. [Native 3D Generation] Splatter Image: Ultra-Fast Single-View 3D Reconstruction Szymanowicz et al. CVPR 2024 [Multi-view ImageNet] EG3D: Efficient Geometry-aware 3D Generative Adversarial Networks Chan et al. CVPR 2022 3D generation on ImageNet Skorokhodov et al. ICLR 2023 (Oral)
Dynamic Scenes & Human Shape of Motion: 4D Reconstruction from a Single Video Wang et al. arXiv 2024 + MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds Li et al. arXiv 2024 K-Planes: Explicit Radiance Fields in Space, Time, and Appearance Fridovich-Keil et al. CVPR 2023 4K4D: Real-Time 4D View Synthesis at 4K Resolution Xu et al. CVPR 2024
Pose Estimation COLMAP-Free 3D Gaussian Splatting Fu et al. CVPR 2024 FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flow Smith et al. NeurIPS 2023
Lighting TensoIR: Tensorial Inverse Rendering Jin et al. CVPR 2023 Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF Decomposition and Ray Tracing Zhang et al. ECCV 2024
Physics Simulation PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics Xie et al. CVPR 2024 (Highlight) PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual Observations Zheng et al. ECCV 2024
Editing & Multi-modality Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions Haque et al. ICCV 2023 (Oral) PlatoNeRF: 3D Reconstruction in Plato's Cave via Single-View Two-Bounce Lidar Klinghoffer et al. CVPR 2024 (Oral, Best Paper Award Finalist)
Robotics LERF: Language Embedded Radiance Fields Kerr et al. ICCV 2023 (Oral) + LERF-TOGO: Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping Rashid et al. CORL 2023 (Best Paper Finalist) LERF v.s. LERF-TOGO: Common: Embed language embeddings into 3D scene representation. LERF: Enables pixel-aligned zero-shot queries on the distilled 3D CLIP embedding. LERF-TOGO: Extends LERF to task-oriented grasping by adding DINO feature grouping. Unifying 3D Representation and Control of Diverse Robots with a Single Camera Li et al. arXiv 2024
Surface Reconstruction NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction Wang et al. NeurIPS 2021 + NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction Wang et al. ICCV 2023 Gaussian Opacity Fields: Efficient and Compact Surface Reconstruction in Unbounded Scenes Yu et al. SIGGRAPH ASIA 2024
Differentiable Mesh Extraction NeurCross: A Self-Supervised Neural Approach for Representing Cross Fields in Quad Mesh Generation Dong et al. arXiv 2024 Flexible Isosurface Extraction for Gradient-Based Mesh Optimization Shen et al. SIGGRAPH 2023

Neural Scene Representation and Neural Rendering

Seminar – Fall Semester 2024

Instructor: Lingjie Liu

TAs: Chuhao Chen, Xuyi Meng

Organization | Content | Format | Resources

Topics