16 July 2024
Exclusives, VR/AR Trends
By IAN FAILES
By IAN FAILES
In the game Dr. Grordbort’s Invaders, players don a Magic Leap One and use its hand controller to shoot steampunk robots using a steampunk raygun. This happens in augmented reality (AR), meaning the imagery a player sees is computer-generated and is overlaid on the real world around them via the Magic Leap One goggles.
To play Dr. Grordbort’s Invaders, the space the user is in is first scanned with the Magic Leap One. This produces a ‘mesh’ of the area, and allows robots and other elements of the game to appear to be locked in with that space, or occluded by tables or other items in front of the player as he or she moves around.
For several years now, keen observers have been following the progress of Magic Leap as it looked to release a product centered on augmented reality and computer vision. Last year, the company’s first headset, the Magic Leap One, became available to the public, and one of its flagship experiences was Dr. Grordbort’s Invaders.
The game’s developer is Weta Gameshop, which has been in close partnership with Magic Leap for some time, all the way back to some of the first research Magic Leap made into augmented reality, which, among several other areas of computer vision, involves light-fields.
At the center of development of the Dr. Grordbort’s game is Greg Broadmore, studio director at Weta Gameshop. With a history in concept art, creature design and special effects, he invented the world of Dr. Grordbort’s, eventually ushering this world into the AR experience as it now exists for the Magic Leap One. It has been a long journey for Broadmore.
“When I first started, actually, there was no technology,” the Gameshop studio director told VFX Voice at the recent Neuchâtel International Fantastic Film Festival in Switzerland. “I flew to Florida, where Magic Leap is, met the original team, and really all it was then were ideas about a light field thing.”
However, when Broadmore returned to Florida a second time, he was able to see more of Magic Leap in action with what was known as the ‘Benchtop’. “It was also called ‘the Beast,’” reveals Broadmore. “It was basically this thing strapped to a table, a massive aluminum frame with wires poking out of it, just an explosion of science fiction.”
“It was astonishing just to see people smile and run up and interact with the robots.”
—Greg Broadmore, Studio Director, Weta Gameshop
“We’d already had a version of the [Dr. Grordbort’s Invaders] game running on a WD1, but that one couldn’t estimate head pose. The WD2 (Wearable Device Two) was pretty rubbish at estimating head pose, but it could do it, and that was the revelation. We also coupled that with our hand controller, and although it was all connected to this big cable connected to a PC, we could move around within a couple square meters. I could see what the game could be at that point.”
—Greg Broadmore, Studio Director, Weta Gameshop
What Broadmore saw was a light field display in which viewers placed their chin on a cradle and peered through two lenses. “Through that I saw five crosses, very boring crosses,” he recalls. “They weren’t even programmed, they were photos of crosses! Flat black and white images. But the important part was that they were a light field. I looked at the front one and it was in focus. And then I moved up to the back one, and the front one went out of focus. And I was like, ‘Holy shit, that looks real.’”
Broadmore was impressed with the display, but thought those crosses were “truly ugly.” With the world of Dr. Grordbort’s on his mind, he suggested Magic Leap put something more radical in the light field display. “I said, ‘Let’s put ray guns in there!’ So, on that trip I got to see ray guns in there and experiment a little bit with that.”
As research at Magic Leap continued, so too did Weta Gameshop’s development of a Dr. Grordbort’s experience that could be used with augmented reality. While the game works with the Magic Leap One headset now, at one stage it was also displayed on the large Benchtop itself. “It was a very narrow field of view,” notes Broadmore, “but you could see from the top of the robots’ heads to just below their knuckles. Still, we had the game working, and that was a ‘Eureka!’ thing for us. It was very much a hack thing, and we’d just keep on getting better with every update.”
“We are just at the start of this. Every single performance attribute of that device is just going to keep on getting better and better. To me, the field of view is the number one thing. You can design around it, and I think we’ve made an incredible game that deals with it in a really good way, but it’s of course the thing I would like to get better.”
—Greg Broadmore, Studio Director, Weta Gameshop
The goal, of course, was to miniaturize the Benchtop experience to something that could be worn by a user as a headset. The first time Broadmore saw a working prototype of such a device, that also met some of his main imaginations for Dr. Grordbort’s, was with Magic Leap’s WD2 (Wearable Device Two). This featured several sensors that aided in head tracking and other things crucial to lining up CG imagery with the real world for AR.
“We’d already had a version of the game running on a WD1, but that one couldn’t estimate head pose,” says Broadmore. “The WD2 was pretty rubbish at estimating head pose, but it could do it, and that was the revelation. We also coupled that with our hand controller, and although it was all connected to this big cable connected to a PC, we could move around within a couple square meters. I could see what the game could be at that point.”
Finally, with the release of the Magic Leap One, Broadmore could see Dr. Grordbort’s Invaders as intended and have people move freely about to play the game. “It was astonishing,” he says, “just to see people smile and run up and interact with the robots.”
“It’s like we’ve made the first phone camera and it doesn’t do everything we want to do, so now we’ll keep refining it and adding to it until it’s awesome. But even that will never end, there’s always something more. That’s the fun of making technology.”
—Greg Broadmore, Studio Director, Weta Gameshop
Research continues at Magic Leap, and development also goes on at Weta Gameshop. Areas such as field of view, more accurate tracking and occlusion, and further miniaturization of the device, are things being worked on.
“We are just at the start of this,” states Broadmore. “Every single performance attribute of that device is just going to keep on getting better and better. To me, the field of view is the number one thing. You can design around it, and I think we’ve made an incredible game that deals with it in a really good way, but it’s of course the thing I would like to get better.
“And there’s 100 other things that I would like to get better,” adds Broadmore. “But that’s the fun of the job. It’s like we’ve made the first phone camera and it doesn’t do everything we want to do, so now we’ll keep refining it and adding to it until it’s awesome. But even that will never end, there’s always something more. That’s the fun of making technology.”
;
