Inspiration
We work around a lot of machines, and much of that work means assembling parts from a manual that runs a couple hundred pages. You read a step, set the manual down, pick up the part, can't remember which way the bracket was supposed to face, and pick the manual back up. Most of our mistakes come from that back and forth between the page and the parts.
We wanted the instructions on the parts themselves. With AR you can look at the bench and see which piece to grab next and where it goes, instead of matching a drawing to what's in front of you. Forge XR is our first try at that. We built it in 36 hours for the Work track.
What it does
You wear a Meta Quest 3 in passthrough and look at your workstation. Forge draws a box around the next part on the table and shows the instruction above it, for example "1/3: Press the torso onto the legs." When you join the pieces, it detects the change, plays a chime and some confetti, and goes to the next step. You can also pinch a floating Next button to advance yourself.
For a workstation we could set up on a hackathon table, we used a LEGO Land Rover Defender (set 40650). Our first plan was a portable blender, and we switched to LEGO on day one. Forge has two walkthroughs: a driver minifigure in three steps, and mounting a wheel on the chassis in four. We judged success by whether someone who had never seen the set could finish without anyone helping.
How we built it
Phil built the headset side, a Unity 6 app using OpenXR, AR Foundation and the XR Interaction Toolkit. It handles passthrough, table alignment, hand and controller input, and the floating UI. Michael built the walkthroughs. He staged the parts, photographed and cleaned up the reference shots, wrote the instructions, and made desktop versions of both walkthroughs in Unity so we could test the step logic without a headset on.
Before we split up, we wrote down a shared list of part IDs (legs, torso, head, hairpiece, tire, rim, axle-pin) and agreed that each step would name its parts with those IDs. Because of that, we could work separately all morning and connect the walkthroughs to the Quest scene early in the afternoon.
Most of our time went into recognizing the parts. The Quest sends 960×960 camera frames over USB to a Python relay running on a laptop (adb reverse on port 8787). The relay finds the parts, returns bounding boxes, and decides when a step is finished. At first it sent every frame to NVIDIA's Nemotron VL model in the cloud. By the end of the night it was using a YOLO model we fine-tuned ourselves, running on the laptop. We trained it on frames captured from the headset and used Nemotron to draft the labels, so we didn't have to draw every box by hand.
To decide that a step is done, the relay first checks geometry. The step's two parts have to touch (or the rim has to sit inside the tire) in at least 3 of the last 5 detections, for at least a second. Then it asks a vision model a yes/no question about a cropped image of the pair. If no model is available, geometry alone can complete the step after a longer wait. For demos, we can also force a step from the laptop.
We care most about the local model. By default Forge doesn't need internet access or a cloud account, and camera frames from the workbench stay on site. We expect that to be one of the first questions from anyone who would use this on a factory floor.
Challenges we ran into
The cloud model's rate limit took up most of our evening. The free tier allows 40 calls a minute, so
$$\frac{60\ \text{s}}{40\ \text{calls}} = 1.5\ \text{s per call}$$
at best, and each answer took about 2 seconds to arrive. We spaced requests 1.6 s apart and later 2.4 s, added a 20-second pause after any rate-limit error, and sent up to three requests at once. Even so, the boxes lagged behind the pieces as they moved. That's why we trained the local detector, which answers in about 0.1 s, roughly 20 times faster.
Tracking pieces over time and in 3D was harder than we expected. The display runs at 72 Hz, but the detector answers at most five times a second, and in between people move pieces, pick them up and cover them with their hands. We filled the gaps with colour tracking between detections, kept pieces on screen briefly after they left the camera's view, and raised the box for a lifted piece into 3D so it stayed with the hand.
The cloud model also confused similar parts, mixing up the head and the arms or the tire and the rim. It sometimes answered "no" for a finished step when the frame was dark or the pieces were still in someone's hand, and early on that answer could stop the step from completing. We changed the logic so the model can confirm a step but can't block it.
The headset caused problems of its own. The passthrough camera occasionally stalled, so we added a watchdog that restarts it. For a while our completion chime made no sound, until we learned that Horizon mutes apps that are launched while the headset is asleep.
What we learned
We should have gone local sooner. We spent hours working around the cloud rate limit, and the local model removed the problem, ran faster, and kept the data on site.
Agreeing on the part IDs before splitting up saved us more integration time than anything else we did.
Geometry checks with a model as a second opinion, plus a manual override, worked much better than relying on the model's answer alone.
Being able to change settings without rebuilding helped a lot. The headset reloads a JSON file from the laptop in about 3 seconds, and together with the desktop test scenes that kept each round of changes short.
The wheel walkthrough still recognizes only the tire, so its later steps need a Next press. Next we want to add a "What do I do next?" voice prompt that reads the current step aloud and highlights the part again.