Lightform uses a process called visible structured light to create the “smart scan,” an image of the scene as if it were taken with the projector’s point-of-view. Lightform Creator uses this image to help you create content, effects, and masks.
Structured light scanning is the first and most important step of the Lightform workflow. We developed this process of scanning to enable instant content creation that allows for a pixel by pixel alignment of digital content onto real-world objects.
A structured light scan operates by projecting patterns that are captured by a camera (in this case, the Lightform device). This provides a dense pixel correspondence from projector pixels to/from camera pixels. This mapping can be thought of in much the same way as stereo camera calibration and reconstruction, and we apply similar techniques to extract 3D information (disparity and depth) from the scene.
This provides the ability to reconstruct a projector image, in other words, a re-mapping of camera pixels into the projector’s domain to obtain an image of the scene as if taken with the projector’s optical parameters and point-of-view. Usually, a projector only emits light; this process makes it as if the projector can take its own photograph perfectly aligned with the projected image. The projector image is extremely important for creating procedural effects, as it provides an image that can be understood and processed by vision techniques and effect generating algorithms. A user can author content directly on top of this image, and it is automatically aligned with the real world, eliminating the need for traditional mapping workflows.