The era of spatial photography is here, and it Splats
NeRFs, GSplats, and the wonderful world of spatial photography that is about to change how we remember our lives forever.
The way we remember a moment in our minds never seems to be quite the same as it looks in a photo, does it?
Despite collectively taking 1.8 trillion photos worldwide every year (!!!), we all know deep down that there’s still something we aren’t quite capturing. Something is missing when we look back at those images.
2024 will be the year this changes.
It’s already started to change with the emergence of the instant single shot holograms that I wrote about yesterday. But wait, there’s more.
Imagine you could capture a moment from all angles, with the phone or camera you have, and then relive that memory through a spatial special display. Let’s call that display a Looking Glass :)
Even more than relive it — imagine you could go into the memory, to look around, behind objects, through mirrors, like Rick Deckard did in the original Blade Runner with his Esper Machine.
This is not a distant science fiction dream.
You can do this, right now, with your phone and your Looking Glass.
This is thanks to a new technique called 3D Gaussian Splats that exploded onto the computer vision scene a few months ago.
Here’s how you can do this yourself:
Take a video. Just a normal video. I generally shoot these at 60FPS at 4K. I use an iPhone 15 Pro, but you can use whatever you have.
Upload to your favorite splat app. I’ve been using Luma AI a lot lately, but there are other great apps like Polycam and new ones every day it seems.
Then you wait. The GSplats will take around 30 minutes to process. This is slow right now, and at some point it’s going to take just seconds. But the wait in these halcyon days is charming, in a way. Remember the joy of taking photos with film cameras, and the thrill of not quite knowing what you’d get? This is a little like that.
View in spatial 3D in your Looking Glass Go in the real world – no headsets required.
Here’s a dandelion shot by splatter NotANinja 🥷 in my Looking Glass Go. The detail on this one is truly next level. You can even go into the dandelion 🤯.
The dandelion scan in the video above is running in my Looking Glass Go thanks to a prototype splat viewer for the web we made with the folks at Luma AI. For any Looking Glass owners out there that are curious, and not afraid to take a prototype for a spin, head over to our Discord and hop into the #lab-chat channel — we’re sharing the prototype there!
Here are a few of my favorite splats so far, out of hundreds I’ve taken over the past couple months:
→ The first one. Me capturing this fish in the market. This was the very first splat we got to run in a Looking Glass. It’s also the first macro splat scan I did.
→ Scanning with friends. A few weeks ago some friends and I passed around my phone to capture a plate of sushi. The future state of the previous splat.
→ Family hologram. Here are my kids decorating the Christmas tree last night.
→ Holo-Cat. Now famous in Japan.
This is isn’t your normal run-of-the-mill 3D scanning. Trust me, I was the guy ten years ago with a Kinect connected to a laptop, 3D scanning my friends at their wedding parties. Not easy to look cool doing that, and with mediocre results.
Today taking these incredibly high quality spatial scans doesn’t even require a new camera – it works with the phone you have. I know I already said that twice, but it’s so seemingly impossible that I’ve been finding no one gets it until I say it three times.
Sidenote. last week I ran into
, creator of the fantastic Radiance Fields site. He told me he started Radiance Fields to document the rapid rise of spatial photography this year — no one was doing it, and he decided it had to be done for the benefit of future historians. He thinks we’re at the cusp of a shift that will prove to be as significant as the emergence of photography was in the late 1800s / early 1900s. Yes.
Back to the main message. You can do this yourself, right now. The Esper machine is real. And it’s called Gaussian Splats and Looking Glass Go.
So this holiday season, take a few minutes to watch the tutorial above and get GSplats of your friends and family, your dog and your cat, and make sure to preserve that holiday turkey forever in holographic amber before the moment is lost.
Your future self will thank you.
p.s. You might be wondering, how is this all related to what Apple is doing with their new Vision Pro headset? I’m super excited about the Vision Pro. Apple is doing us all a great service in this field, by making spatial 3D more mainstream, and building new support for this sort of thing into their phones. We’ve been waiting for this moment for years.
But, I figure while folks will occasionally want to relive their spatial photos by gearing up in their Vision Pro headset, most of the time they will want to relive those memories in the real world with friends and family, when they take off their headsets. That’s where Looking Glass comes in. Same spatial content, but without the headset.
We can display those spatial photos in 3D in the real world by putting millions of lenses on a display, instead of putting two lenses onto a person’s face. But that’s a post for another time.
hell yeah congrats to you and the team, Shawn! excited to play with the Go!