The main selling point of Gaussian Splatting over NeRF-based methods is rendering (and training) speed and efficiency.
This does come at the cost of physical correctness as it uses splatting instead of a real radiance field.
This method tries to move back to radiance fields, but with a primitive-based representation instead of neural nets.
Rendering performance seems to be quite poor (30fps on a 4090), and rendering quality improvements seem to be marginal.
I'm not quite sure I understand where this fits in when NeRFs and 3DGS already exist at the opposite ends of the correctness-speed tradeoff spectrum.
Maybe somewhere in the middle?
Primitive-based representations are a lot easier to manipulate (e.g. animate) than NeRFs. Also they can be a lot more efficient/scalable when there's a lot of e.g. empty space.
It’s great how the field of realtime volumetric rendering is alive with so many options and new yet simple approaches are being invented.
Reminds me of the mid-1990s before everyone basically agreed that lots of triangles of any size are the right model representation. There were NURBs and metaballs and reyes micropolygons and who knows what else… Even Nvidia’s first hardware accelerator chip used quads instead of triangles (just before Microsoft made triangles the standard in Direct3D).
Looking forward to seeing where this settles in a couple of years! The intersection of video and 3D is about to get super interesting creatively.
- your dataset is already an analytic representation of stretched spheres. Just assume that they are a hard shape of plasma with uniform density throughout the volume.
- for each pixel perform ray intersection against each spheroid, the thickness that the ray went through is precisely how much thickness the spheroid is contributing light to your pixel (obviously also multiplied by solid angle)
- since it’s light, there is no occlusion, so just stack the contribution from all the ellipsoids together and you’re done.
- since you are rendering a well-defined shape without self occlusion, there is no random popping in and out no matter the angle.
The computation is practically equivalent to rendering CSG shapes, except even easier since you only ever add and not occlude/subtract. It also scales with rat racing hardware directly.
I always assumed this was already how those were rendered, because that's kinda obvious, and raymarching is a standard technique for real-time volumetric rendering.
The main selling point of Gaussian Splatting over NeRF-based methods is rendering (and training) speed and efficiency. This does come at the cost of physical correctness as it uses splatting instead of a real radiance field.
This method tries to move back to radiance fields, but with a primitive-based representation instead of neural nets. Rendering performance seems to be quite poor (30fps on a 4090), and rendering quality improvements seem to be marginal.
I'm not quite sure I understand where this fits in when NeRFs and 3DGS already exist at the opposite ends of the correctness-speed tradeoff spectrum. Maybe somewhere in the middle?
Primitive-based representations are a lot easier to manipulate (e.g. animate) than NeRFs. Also they can be a lot more efficient/scalable when there's a lot of e.g. empty space.
It’s great how the field of realtime volumetric rendering is alive with so many options and new yet simple approaches are being invented.
Reminds me of the mid-1990s before everyone basically agreed that lots of triangles of any size are the right model representation. There were NURBs and metaballs and reyes micropolygons and who knows what else… Even Nvidia’s first hardware accelerator chip used quads instead of triangles (just before Microsoft made triangles the standard in Direct3D).
Looking forward to seeing where this settles in a couple of years! The intersection of video and 3D is about to get super interesting creatively.
We all know where it's going since 1873. What matters is the fun along the way.
maybe something like this:
https://arxiv.org/pdf/2405.16237
Neat, the concept is so elegantly obvious:
- your dataset is already an analytic representation of stretched spheres. Just assume that they are a hard shape of plasma with uniform density throughout the volume.
- for each pixel perform ray intersection against each spheroid, the thickness that the ray went through is precisely how much thickness the spheroid is contributing light to your pixel (obviously also multiplied by solid angle)
- since it’s light, there is no occlusion, so just stack the contribution from all the ellipsoids together and you’re done.
- since you are rendering a well-defined shape without self occlusion, there is no random popping in and out no matter the angle.
The computation is practically equivalent to rendering CSG shapes, except even easier since you only ever add and not occlude/subtract. It also scales with rat racing hardware directly.
I always assumed this was already how those were rendered, because that's kinda obvious, and raymarching is a standard technique for real-time volumetric rendering.
Impressive video: https://twitter.com/alexandertmai/status/1841739387400552826