History of 3D scanning

In his 1895 manual, Canadian surveyor Édouard Deville described a way to locate a landscape feature with black silk thread. Pin a thread at each camera station marked on a plan. Align the threads using corresponding points identified in photographs. Their intersection locates the feature; another view supplies a check. The photographs preserve the scene, but the surveyor still has to recognize the same place in different images and turn those observations into a position. Deville’s photographic surveying manual, sections 129–130.

The thread makes a problem visible that runs through the history of 3D scanning. An instrument records something observable: an image position, the location of a projected stripe, or the delay before light returns. Calibration and geometric models turn observations into range or position estimates. Further work combines them into surfaces. Advances in optics, electronics and computation have shifted this work between people and machines, making new kinds of capture practical. The resulting model still has to answer a separate question: what evidence supports its shape and dimensions?

01Photographs become measurements

A photograph records directions toward the scene. A mountain summit occupies a particular place on the image, but that place alone does not say how far away the summit is. Observe it from another known position and the two viewing directions constrain its location. This is triangulation. In photographic surveying, camera geometry, station positions and the identification of corresponding features make the picture useful for measurement.

Deville separated recovering an object’s proportions from giving it absolute dimensions. A measured length established scale. Survey control located and oriented camera stations; photographs and enlargements needed checks for distortion. His plotting instructions turned perspective into a practical mapping procedure, with much of the interpretation and drawing taking place after the fieldwork. Taking photographs changed the distribution of labor between field and office. Deville, sections 123–130.

In an 1859 abstract for the French Academy of Sciences, Aimé Laussedat outlined how a survey method based on drawings could be extended to photographs. He had considered photography earlier, but reported finding its procedures and handling poorly suited to field conditions. Practical photographic measurement depended on making the process workable as well as understanding perspective. Laussedat’s original abstract, pages 732–734.

The technique developed through work in several countries, and the dates describe different accomplishments. In April 1867, German architect Albrecht Meydenbauer published an account of photographic measurement with wider trials still in preparation. A December report in the same journal described summer experiments at Freyburg an der Unstrut, including photographs of a church and a rod that supplied scale. It separated reconnaissance, photography and office drawing, and cautioned against treating the reported times as representative before a fair comparison with established methods. A publication, a successful trial and routine adoption are different milestones. The original 1867 journal, pages 125 and 471.

These were procedures for producing plans and measurements, carried out long before digital point clouds. They already contained many of the problems that later scanners would inherit: establish scale, control the geometry, choose useful views and check the result. A camera could preserve a great deal of information quickly while leaving substantial work to the person interpreting it.

02Giving a computer something to match

Seeing depth and measuring it are related achievements. Charles Wheatstone’s 1838 experiments showed how different perspective drawings presented to the two eyes could produce a perception of relief. Obtaining coordinates automatically required another step: finding corresponding observations and calculating their geometry. Wheatstone’s original paper, pages 371–374.

In 1958, U. V. Helava described an analytical plotter coupling computer calculations to a viewing and measuring device. Its observer could be a person or an automatic image correlator. Computing projection geometry and finding corresponding image detail were distinct parts of the system. Helava’s original account, page 795.

One route was to put useful markings into the scene. At Japan’s Electrotechnical Laboratory, Yoshiaki Shirai and Motoi Suwa described a system in 1971 that swept a vertical slit of light across an object while a television camera watched from an offset position. The slit’s position in the image, together with the known geometry, supplied range information. Their program detected lines, fitted planes and recognized simple polyhedrons. A computer could work with the observed geometry rather than merely storing a picture. Shirai and Suwa, Recognition of Polyhedrons with a Range Finder.

Geometrically, a projected slit defines a sheet of light. Where a camera sees the illuminated line, its viewing ray intersects that sheet, locating a surface point. A laser stripe can supply the line; depth depends on the baseline and angles. A wider baseline can make depth changes easier to distinguish, with other conditions fixed. It can also leave more surface hidden from one view. A useful point must be both illuminated and visible. Triangulation geometry and occlusion.

The demonstrations also show how much the chosen task matters. The system expected planar faces and exploited relations among them. Its recognition result included assumptions about solids as well as measurements of illuminated lines. Those assumptions made the problem manageable; an arbitrary crumpled or curved surface would pose a different task.

Another route used the scene’s own image detail. A 1996 paper by Takeo Kanade and colleagues described a dedicated multi-camera stereo machine at Carnegie Mellon. It compared small image patches across views, combined the matching evidence from several camera baselines and selected a depth. Calibration and image correction connected those comparisons to the cameras’ geometry. Specialized electronics made dense depth maps available at video rates. The five-author 1996 stereo paper.

Depth also became useful for visual media. The team demonstrated compositing in which a real hand could appear in front of a synthetic object while the person’s body remained behind it. Correct depth ordering could produce a useful effect without establishing the dimensional accuracy needed to inspect a manufactured part. The application determined which properties of the reconstruction mattered.

Correspondence also depends on the path light takes. A feature seen through clear glass may belong to the background, along a ray bent by the glass surfaces. A 2005 study combined known background patterns with extra viewpoints to recover parts of a glass surface, under assumptions about the material and surface crossings. The original light-path triangulation study.

Light can also bounce between surfaces or scatter within a translucent material, changing a projected code. A 2011 study combined patterns suited to different effects and used further captures when they disagreed. Its method still assumed that one kind of indirect light transport dominated at each point. The original pattern-design study.

03Preserving a sculptor’s marks

In 1998–99, the Digital Michelangelo team captured Italian statues with a custom Cyberware laser-stripe triangulation scanner. Capturing chisel marks as geometry would let them cast shadows under new lighting in a rendering. Light could scatter beneath the marble’s surface, complicating the stripe measurements. The paper stated a target of 0.25 mm for range accuracy but reported discrepancies exceeding 1 mm under changed scanning conditions. Small sample spacing did not ensure that overlapping scans agreed. The Digital Michelangelo measurements and calibration.

Point spacing describes the separation between samples. Range resolution concerns a distance change the instrument can distinguish. Precision concerns how tightly repeated readings cluster; repeatability restricts the comparison to a specified, unchanged setup over a short period. Accuracy concerns closeness to the true value. Reported error compares a reading with a stated reference value, and their uncertainties matter when interpreting the difference. Numerical specifications need the tested error and conditions. A system can repeatedly produce the same displaced surface. Adding samples makes that surface denser while leaving the displacement. The metrology vocabulary, sections 2.13–2.26 and 4.14.

Registration brings those partial scans into alignment. Approximate manual placement was refined by matching nearby surface points, then reconciling many overlaps. Several surveyed points on most statues were available; their use as global alignment constraints was future work. The paper supplies no independently established whole-statue error bound. Registration and surveyed points, section 4.1.

The work also exceeded capture time. For David, the team reported 360 hours scanning over 30 days and 1,500 person-hours post-processing so far. That unfinished total shows how much effort can follow acquisition. The project’s Table 1.

04Measuring light’s return

The same campaign used a Cyra time-of-flight scanner for the Accademia gallery, complementing fine statue measurements with an architectural model. The gallery scan, Figure 15.

A pulsed time-of-flight instrument sends light toward a surface and measures its return delay. The light travels out and back; that travel time supplies range. Combining range with the direction in which the instrument points gives a position in space. Repeating this across a scene produces a point cloud: a collection of sampled positions, before any decision about how to join them into surfaces.

One form of phase-based ranging varies the light’s intensity in a repeating cycle and measures the shift of that cycle in the return. At one modulation frequency, the same phase can correspond to several distances. Resolving that ambiguity requires additional information, such as measurements at another frequency. The original range-ambiguity experiments.

Timing changes the measurement problem, but the returning light still has to be interpreted. In an independent 2002 NIST study of one pulsed scanner, black targets became noisier at longer distances, while highly reflective sheeting biased some nearer measurements. The researchers also described points between a box and the floor behind it when the beam covered both surfaces. Those intermediate points did not correspond to a physical surface. These were findings about a particular instrument and experiment, rather than a universal ranking of materials or scanners. NIST’s original calibration experiments, sections 2.5–2.6 and 3.1.3.

05Letting the operator see what is missing

A further change was to shorten the feedback loop. In 2002, Szymon Rusinkiewicz, Olaf Hall-Holt and Marc Levoy described a projected-light system in which the operator could turn an object while watching a model accumulate. Coded stripe boundaries supplied correspondences; fast registration combined partial views. The display revealed missing regions while the object was still available to scan. Real-Time 3D Model Acquisition.

That feedback reduced the delay between making a capture mistake and discovering it. It also depended on operating conditions. Moving too quickly could break pattern tracking; registration could drift, and some shapes did not constrain the alignment well. Final reconstruction still involved offline processing. A live display, a stream of depth samples and a finished model described different stages of the work.

The paper supplies a particularly revealing example of appearance getting ahead of geometry. Noisy observations could accumulate into a thickened surface, while averaging the directions used to shade it made the display look smoother. A pleasant rendering could therefore hide disagreement in the underlying points. The authors explicitly distinguished their quick display from the more careful final reconstruction. The system’s integration, display and reconstruction methods.

A mesh joins samples into faces, and software can bridge gaps that were not observed. A 2014 study at the Technical University of Denmark found that meshing could improve a simple box by filling holes close to the reference shape, while smoothing erased fine details in more complex objects. Completion was useful when its shape assumptions fit the object. Retaining the observations makes it possible to distinguish measured detail from the surface a reconstruction proposes. The DTU study’s point and surface comparisons.

06Depth from a consumer camera

Microsoft’s Kinect reached North American stores in November 2010, putting a depth sensor into a consumer gaming device. Microsoft’s launch announcement. The original model projected infrared speckles and used their displacement to triangulate depth. A separate color camera supplied the visible image. The projected pattern made depth available without relying entirely on the object’s visible texture, while calibration still connected the observations to coordinates. This generation of Kinect used structured light. The 2012 study of Kinect’s measurement principle.

In June 2011, Microsoft released a Windows developer kit for noncommercial applications, with drivers, access to sensor streams, documentation and sample code. The developer-kit release.

Kourosh Khoshelham and Sander Oude Elberink’s independent experiments showed why the device’s output needed careful interpretation. In their door experiment, scatter around a fitted plane grew with distance; the depth values also fell into increasingly separated layers. A separate comparison registered Kinect data to a calibrated laser-scanner point cloud. Plane-fitting scatter, depth quantization and disagreement with a reference were different measurements of quality. None was a test of a completed, fused reconstruction. The experiments and their conditions.

In 2011, Richard Newcombe and colleagues presented KinectFusion, which combined a moving Kinect with GPU computation. It estimated the camera’s pose against the accumulated scene model and fused incoming depth measurements into a grid of values describing the surface’s position. Combining observations could reduce noise and reveal a more coherent surface. Here the volume was a computer representation built from surface observations. KinectFusion’s original method.

Returning to a previously seen place provides a revealing check. If the reconstructed camera path and surfaces disagree at the return, error has accumulated around the loop. KinectFusion’s tabletop experiment showed how tracking against the model could reduce that disagreement compared with tracking consecutive frames. Its authors also identified limits: large flat scenes left some camera motions ambiguous, and extending the method to whole buildings would require addressing memory and drift. The loop experiment and limitations.

Microsoft distributed Kinect Fusion in its Windows developer kit in 2013. The SDK 1.7 release. In 2014, Pagliari and colleagues tested models from those libraries against reference scans. Aligning their smooth statue model to a triangulation reference included fitting its scale. Further experiments improved geometric agreement through camera calibration, depth corrections or constraints based on a turntable’s circular path. The results connected improvements in the finished surface to corrections in the data and estimated camera motion. The original calibration and reconstruction study.

That combination let an operator move one depth sensor and watch a surface accumulate. The working system included the computer, calibration and a person planning the views.

If the surface itself deforms, rigid alignment cannot bring every part into agreement. DynamicFusion addressed this by estimating a deformation field before combining depth observations. Motion in unseen regions still depended on assumptions, and large movements between frames could corrupt the accumulated model. The original dynamic reconstruction study.

07Photographs also locate the cameras

Photographs offer another route to a model, with software estimating where the cameras were as well as the shape they observed. The 2016 paper presenting COLMAP described an open implementation of incremental structure from motion. It found matching image features, checked their geometric consistency and grew a reconstruction by adding cameras and triangulated points. Bundle adjustment refined camera parameters and 3D points together to reduce their disagreement with observed image positions. Schönberger and Frahm, Structure-from-Motion Revisited.

This moved work once done with surveyed camera stations and manually identified landmarks into a computational pipeline. Camera calibration could itself be estimated during reconstruction, subject to a camera model and checks for implausible results. A separate 2016 COLMAP paper described dense stereo: estimating depth and local surface orientation, selecting useful views, and combining image similarity with geometric consistency. It filtered poorly supported estimates and fused consistent ones into a point cloud. Meshing remained an optional next step. The SfM method, the dense stereo method.

Deville’s distinction between proportions and absolute dimensions still applied. Uniformly rescaling a reconstructed scene and the camera separations preserves its ideal perspective projections. A known length or surveyed control can supply the missing scale. Good agreement with image positions therefore answers a different question from agreement with independently measured dimensions.

The 2017 Tanks and Temples benchmark compared image-based reconstructions with laser-scanned reference scenes, including buildings, interiors and sculptures. Its evaluation first aligned and rescaled a reconstruction to the reference. It then measured both how many reconstructed points lay close to the reference and how much of the reference was covered. The alignment made comparison possible; it did not demonstrate that the original image pipeline had recovered absolute scale. The benchmark’s acquisition and evaluation procedures.

These two directions of comparison matter. A small set of well-placed points can have little error while omitting most of a surface. A dense cloud can cover the scene while including many wrong points. The benchmark tested both, and used different distance thresholds for scenes with different scale and sampling. Its reference scans also required registration, cleaning and resampling. A score belonged to that evaluation procedure and those scenes. Tanks and Temples, sections 4 and 6.

08Learning how a scene looks

Texture mapping can project the colors of photographs onto a model’s surfaces. Debevec, Taylor and Malik’s 1996 Berkeley system chose and blended photographs according to the viewer’s position. Windows could look recessed even when the basic model omitted their recesses. A separate stereo stage estimated additional geometry from image correspondences. Photographic detail and geometric detail could therefore enter the reconstruction through different operations. The original modeling and rendering method.

In 2020, Ben Mildenhall and colleagues presented NeRF, a neural radiance field. A neural network represented density controlling opacity at positions in space, together with color that could change with viewing direction. Rendering accumulated those contributions along a virtual camera’s rays. The network was fitted to photographs and their camera geometry, so it could generate views between those supplied during training. View-dependent color helped represent changing highlights. The original NeRF method.

Recovering a surface was the explicit goal of NeuS, presented by Peng Wang and colleagues in 2021. From photographs with known camera poses, it learned a distance field: zero marked the surface, and opposite signs distinguished its inside and outside. The authors reported improved mean geometric results against selected baselines on the DTU dataset, whose reference surfaces were acquired with structured light. Learning could therefore serve surface reconstruction as well as image synthesis, with a geometric test to assess the result. The NeuS method and evaluation, the DTU reference acquisition.

In 2023, Bernhard Kerbl and colleagues presented 3D Gaussian Splatting. Their representation used explicit, overlapping 3D primitives whose positions, shapes, opacity and directional colors were optimized to reproduce photographs. For real scenes, sparse points from structure from motion initialized the process. A renderer projected and blended the primitives efficiently enough for interactive viewing after training. The system used spatial structure without requiring a finished surface mesh as input. The original Gaussian-splatting paper.

NeRF and Gaussian splatting evaluated agreement with held-out images; Gaussian splatting also reported rendering speed, training time and storage. These are useful measures for navigating a captured scene. They answer a different question from a surface comparison against a separately measured surface. Even the same source photographs can support different evaluations: Gaussian splatting used scenes from Tanks and Temples to test rendered images, while the original benchmark compared reconstructed geometry. NeRF’s evaluation, Gaussian splatting’s evaluation, section 7, Tanks and Temples’ geometry metrics.

A convincing view is a valuable result for a virtual visit or visual asset. Establishing dimensions, extracting a usable surface and checking unobserved regions require additional evidence. The evaluation has to follow the intended use of the representation.

09The evidence a task needs

A tactile coordinate measuring machine samples surfaces its stylus can reach. The contact position is inferred from the probe’s location and geometry; a large tip can miss narrow recesses. Contact supplies another kind of observation, with its own calibration and access limits. Contact-probe geometry and surface reconstruction.

X-ray computed tomography records radiation transmitted through an object from many directions and reconstructs how strongly different regions weaken the rays. Locating a material boundary in that volume is another processing step, and instrument alignment affects the resulting dimensions. This supplies interior information that an optical depth map does not acquire merely by being stored in a three-dimensional grid. The industrial CT geometry study.

For a film or game asset, convincing appearance along the intended camera paths may be the central requirement. Cultural documentation may also need the original observations, calibration records and a distinction between observed and completed regions, so future readers can examine what survives in the model.

Making a replacement part adds a different demand: the dimensions that determine its fit must be checked. A tool following a surface needs that surface located relative to the tool, with error small enough for the intended clearance or contact. A reconstructed object’s appearance alone cannot supply those checks. The relevant scale, features, coverage and coordinate relationships have to be established for the task.

The effort also extends across the workflow. Choosing views, preparing an object, calibrating equipment, acquiring data, processing it and checking the result all consume time. Cheaper capture hardware can make an experiment accessible while leaving these other costs substantial. Useful comparisons specify what work their quoted time or price includes.

Deville’s plotting instructions recommended a third intersection to check the result. A modern reconstruction likewise benefits from evidence that can disagree with it: a reference dimension, another observation or an independently measured surface. The instruments and algorithms have changed who performs the work. Connecting the finished representation to observations that can test it remains part of making a scan useful.