The text provides a comprehensive overview of Gaussian splatting, a new trend in 3D representation. It discusses its representation of 3D scenes using 3D points and Gaussian functions, its image formation model & rendering, optimization, and limitations. The method is compared to NeRF and highlighted for its speed, simplicity, and interpretability.
For more information, please refer to the original article “A Comprehensive Overview of Gaussian Splatting” on the Towards Data Science publication.
“`html
A Comprehensive Overview of Gaussian Splatting
Everything you need to know about the new trend in the field of 3D representations
TL;DR
First and foremost, the main claim to fame of this work was the high rendering speed as can be understood from the title. This is due to the representation itself which will be covered below and thanks to the tailored implementation of a rendering algorithm with custom CUDA kernels.
Representing a 3D world
In Gaussian splatting, a 3D world is represented with a set of 3D points, in fact, millions of them, in a ballpark of 0.5–5 million. Each point is a 3D Gaussian with its own unique parameters that are fitted per scene such that renders of this scene match closely to the known dataset images.
Image formation model & rendering
Given a set of 3D points, the image formation model of Gaussian splatting is almost identical to NeRF, with the only difference being how 𝛼 is computed between the two. However, this small discrepancy results in drastically different rendering speeds.
Optimization
The optimization process involves good initialization, differentiable optimization, and adaptive densification to address under- and over-reconstruction. The scene is optimized for a loss function that is a combination of L1 and D-SSIM between a ground truth view and a current render.
View-dependant colors with SH
Spherical harmonics, SH for short, play a significant role in computer graphics and were first proposed as a way to learn a view-dependant color of discrete 3D voxels in Plenoxels. View dependence is a nice-to-have property that improves the quality of renders since it allows the model to represent non-Lambertian effects, e.g. specularities of metallic surfaces.
Limitations
Despite the overall great results and the impressive rendering speed, the simplicity of the representation comes with a price. The most significant consideration is various regularization heuristics that are introduced during optimization to guard the model against “broken” Gaussians: points that are too big, too long, redundant, etc.
Where to play with it
No blog post can do justice to a method as well as just running it and seeing the results for yourself. Here is where you can play around:
- gaussian-splatting — the official implementation with custom CUDA kernels;
- nerfstudio —yes, Gaussian splatting in nerfstudio. This is a framework originally dedicated to NeRF-like models but since December, ‘23, it also supports GS;
- threestudio-3dgs — an extension for threestudio, another cross-model framework. You should use this one if you are interested in generating 3D models from a prompt rather than learning an existing set of images;
- UnityGaussianSplatting — if Unity is your thing, you can port a trained model into this plugin for visualization;
- gsplat — a library for CUDA-accelerated rasterization of Gaussians that branched out of nerfstudio. It can be used for independent torch-based projects as a differentiatable module for splatting.
If you want to evolve your company with AI, stay competitive, use for your advantage A comprehensive overview of Gaussian Splatting.
Discover how AI can redefine your way of work. Identify Automation Opportunities: Locate key customer interaction points that can benefit from AI. Define KPIs: Ensure your AI endeavors have measurable impacts on business outcomes. Select an AI Solution: Choose tools that align with your needs and provide customization. Implement Gradually: Start with a pilot, gather data, and expand AI usage judiciously.
For AI KPI management advice, connect with us at hello@itinai.com. And for continuous insights into leveraging AI, stay tuned on our Telegram t.me/itinainews or Twitter @itinaicom.
Spotlight on a Practical AI Solution:
Consider the AI Sales Bot from itinai.com/aisalesbot designed to automate customer engagement 24/7 and manage interactions across all customer journey stages.
Discover how AI can redefine your sales processes and customer engagement. Explore solutions at itinai.com.
“`
