How do I get the precise x, y, and z coordinates of the tip of my robot's gripper?

I ask because I use an SO-101, a small robot arm, to build a smooth, precise controller, which is the program that turns a desired path into motor commands. I call it the drawing arm project, but drawing is only one showcase. The real goal is a controller that also works in space, and this post is about generalizing away from the XY plane. Right now the gripper holds a pen, and a Wacom drawing tablet, a flat sensor that reports where a pen touches it, measures the tip of that pen, called the nib. The tablet gives me x and y along its surface. What it can't give me is z, the height above that plane, so that is the question.

The tablet measures the nib only in its own planeSide view of the arm drawing on a tablet. The tablet reports x and y of the nib. A raised nib position above the tablet has an unknown height z.?z = ?height above the plane,not measuredWacom tabletreads the nib x and y in this planenibSide view. x runs along the tablet, z points up, y points into the page.
The tablet reads the nib position while it draws. Lift the pen and the height z has no sensor.

Why measure the tip in space?

The path of the tip shows what a change to the controller does to the motion. On the tablet, that tells me how well the arm draws. But the arm should also carry a tool above the table, and free-space measurements can show whether corrections learned while drawing still help there. They also give me data to build a model of how the arm moves.

The arm already reports something about its own pose, because its encoders, the sensors that read each joint angle, give me those angles. The catch is that the arm can bend between a joint and the tip, and that bending changes the path. So I want an outside reference that measures the result of the whole chain, including the tool holder.

Three ways to measure in space

A friend who works on robots described three routes, and each one extends the measurement beyond the tablet plane in a different way. The first uses a camera that locates a ChArUco board, a checkerboard with coded markers. The second puts markers on the arm and tracks them with sweeping base stations or with infrared motion-capture cameras. The third uses a laser tracker that follows a reflector.

A camera and a ChArUco board

The codes on a ChArUco board tell the camera which part of the pattern it is looking at, and the corners and the known square size give it geometric reference points. Once the camera is calibrated, which means relating image coordinates to rays in space, a known target lets me estimate its position and orientation. OpenCV ChArUco guide

A camera locates a ChArUco target on the toolSide view. A phone camera looks at a reference board fixed beside the tablet and at a ChArUco target on the tool. Rays run from the lens to each pattern. A known offset connects the target to the nib.reference boardtabletphone cameratarget on the toolknownoffsetnibChArUcopatternThe camera sees the pattern. The pattern pose gives the nib.Side view. Fixed: camera, board, tablet. Moves: arm, target, nib.
The camera measures two patterns: one that never moves and one that rides on the tool. Their relative pose, plus the known offset, gives the nib.

The sketch shows the setup. A board beside the tablet sets a reference frame, and a target attached to the tool moves with it. The relation between that target and the nib is fixed, so the target pose turns into a nib position. Using several calibrated views adds a triangulation baseline and helps keep the target visible as the arm moves. Higher image resolution puts more samples across the target, but the geometry matters too: lateral movement shifts the pattern across the image, while depth changes its apparent scale. That makes depth sensitive to target size, viewing angle, and lens calibration.

Depth changes the size of the target in the imageSide view of a pinhole camera. A target of the same size makes a longer mark on the image plane when it is near than when it is far.optical axisnear targetfar targetSimage planelensThe same target makesa longer mark when it is near.s = f · S / Zs: size in the image · S: true size · Z: distance · f: focal length
A target of fixed size S makes a longer mark on the image plane when it is near (thick mark) than when it is far (thin mark). A sideways move shifts the mark along the plane instead.

For a small arm, the tablet and a calibrated camera play complementary roles. The tablet gives a direct reference while drawing, and the camera adds a view above the surface. A phone such as the Pixel makes this route accessible without a dedicated motion-capture installation.

Markers and base stations or motion-capture cameras

The second route follows markers on the arm. Sweeping base stations send infrared laser scans across photodiodes on a tracker, and the system combines these observations with inertial measurements to estimate the tracker's position and orientation. SteamVR study

Sweeping base stations locate a tracker with laser sweepsSide view. Two base stations on the left and right send sweeping laser planes across a tracker with photodiodes on the tool. The time of each hit gives an angle from each station.base station Abase station Bθlaser sweeptracker withphotodiodesA sweep reaches each photodiode at a different time.The delay gives an angle. Two stations give a position.Side view. Fixed: stations, tablet. Moves: arm, tracker.
Each base station flashes, then sweeps a laser plane across the room. The tracker notes when each photodiode is hit, and that delay becomes an angle. Angles from two stations place the tracker in space.

Infrared motion capture works differently: cameras observe reflective markers and triangulate their positions, and a rigid cluster of non-collinear markers gives the orientation of the tool as well. OptiTrack marker tracking

Infrared motion capture triangulates reflective markersSide view. Three cameras look at a rigid cluster of reflective markers on the tool and at single markers on the arm links. Rays from the cameras cross at each marker.cameracameracamerarigid clusterof markersmarkers on the linksshow where it bendsEach camera sees a marker as a point and gives a ray.Rays from several cameras cross at the marker.Side view. Fixed: cameras, tablet. Moves: arm, markers.
Cameras triangulate each reflective marker from several rays. Three or more markers fixed to the tool also give its orientation. Single markers on the links show where the arm bends.

Static and moving accuracy can differ substantially here. Root mean square error (RMSE) gives large misses more weight than small ones, and the SteamVR study's four-station setup reached 0.574 mm aggregate RMSE at 12.5 mm/s, rising to 4.237 mm at 50 mm/s. The dynamic result matters for a moving tool, even when a stationary tracker looks stable.

Placement matters too. Markers near the tool measure its motion, while markers on separate arm links can also reveal where bending occurs. Where I put them determines which part of the mechanical chain the measurement includes.

A laser tracker

A laser tracker combines a distance measurement with angular encoders to locate a reflector, so the result is the reflector center in space. A dedicated pose target adds orientation, which matters when the reflector is offset from a rotating tool tip. Hexagon tracker specifications

A laser tracker measures distance and two angles to a reflectorSide view with a top-view inset. A laser tracker on a stand sends a beam to a reflector on the tool. It measures the beam distance d, the elevation angle beta, and the azimuth angle alpha.laser trackerreflectordistance dβtracker stays fixedtop viewαDistance from the laser, plus two angles from encoders,give one point: the center of the reflector.
The tracker tracks a reflector with its beam. Interferometry or a distance meter gives the distance d. Two angular encoders give the elevation β and the azimuth α. Together they fix the reflector center in space.

This is industrial dimensional measurement equipment. Its appeal is an independent metric reference with tight specified error bounds, and the target and its mounting geometry are part of the measurement system.

Comparing the systems

To compare these systems fairly, a few terms need defining. Rate counts samples per second. Latency is the time from a physical motion to a usable sample in the program. Mean absolute error (MAE) averages the size of the errors, and RMSE, as above, gives large misses more weight. A tolerance or maximum permissible error states a bound under specified conditions. Two charts show the spread, and the table below them keeps every figure with its source link.

Accuracy figures on one log axisA horizontal log-scale chart of the published accuracy figure for each system. Values range from 0.021 millimeters for a laser tracker at 1 meter to 2.164 millimeters for moving SteamVR trackers.0.01 mm0.1 mm1 mm10 mmWacom Intuos Stolerance · Datasheet0.25 mmChArUco cameraMAE · Paper, a calibration rig1.32 mmSteamVR stationsRMSE, static to 25 mm/s · Paper0.1552.164 mmOptiTrack PrimeX 13typical · Datasheet0.3 mmLeica AT960MPE at 1 m to 10 m · Datasheet0.0210.075 mmerror in mm, log scale: smaller is betterDatasheetPaper
The accuracy figures from the table on one axis. Each row uses its own error metric and test setup, so compare the position within a row, not the gap between rows. The Leica range is the datasheet formula evaluated at 1 m and 10 m.
Sample rates on one log axisA horizontal log-scale bar chart of the sample rate of each system, from 60 frames per second for phone video to 1,000 hertz for a laser tracker.10/s100/s1,000/sWacom Intuos S133 reports/s maximum · DatasheetPixel videoup to 60 frames/s · DatasheetSteamVR tracker100 Hz in that study · PaperOptiTrack PrimeX 13240 Hz at full resolution · DatasheetLeica AT9601,000 Hz output · Datasheetsamples per second, log scale
Sample rates from the table. A high rate does not mean a low delay: only OptiTrack lists a camera latency (4.2 ms, Datasheet), and no source here gives an end-to-end figure.
System Accuracy evidence Rate Latency Working area or volume Position and orientation Cost and setup
Wacom Intuos S ±0.25 mm digital accuracy tolerance 133 reports/s maximum No host-chain figure in the cited specifications 152 × 95 mm active plane Pen x and y, pressure, proximity About US$40 (what I paid); already in this setup; fix the tablet pose and map its coordinates
High-resolution camera with ChArUco An omnidirectional calibration rig reported 1.32 mm translation MAE Pixel video supports up to 60 frames/s, depending on model and mode Capture, processing, and transfer all contribute; no end-to-end figure in the cited sources Set by field of view, target size, and visible corners Target position and orientation; a calibrated offset gives the nib position Existing phone plus a printed target or rigid board; requires lens calibration, mounting, and timing alignment
Sweeping base stations: SteamVR Tracking With four stations: 0.155 mm static translation RMSE; 2.164 mm dynamic aggregate RMSE at 25 mm/s 100 Hz tracker stream in that study Tracker-to-program delay not established by that study Reported coverage up to 10 × 10 m with four stations Tracker position and orientation; calibrated offset to the nib US$149 tracker; US$229 per station, plus mounts and a compatible host; requires frame registration
Infrared motion capture: OptiTrack PrimeX PrimeX 13: ±0.3 mm typical position accuracy 240 Hz at full resolution 4.2 ms listed camera latency Vendor accuracy context: 9 × 9 m tracking area; camera overlap defines the usable volume Marker positions; a rigid marker cluster supplies orientation Estimate: about US$12k to 20k for 4 to 6 cameras. PrimeX 13 starts at US$2,499 each; Motive:Tracker is US$999 plus a US$149 key. The rest is my allowance for a switch, calibration tools, markers, and mounts; requires camera calibration
Laser tracker: Leica AT960 ±(0.015 + 0.006L) mm maximum permissible error, with distance L in meters 1,000 Hz output No application-delay figure in the cited specifications AT960-MR: 40 m diameter for reflector position; 20 m with the pose accessory Reflector center position; a dedicated target adds orientation Estimate: about US$75k new or US$17k to 30k used, tracker only. New AT960: US$74,900; used listings: US$29,900 and US$16,900 for an AT960-LR. No target or software included; rental rates are not published; requires frame registration

From the sensor target to the nib

From the sensor target to the nibSide view. The fixed frame origin, the target position t, and the offset r turned by the target rotation R add up to the nib position p. A small rotation error moves the estimated nib.xzorigintarget framettarget positionR rfixed offset,turned by thetarget rotation Rangle error movesthe nib estimatenib position p = t + R rThe sensor sees the target. The offset r carries it to the nib.Side view. Fixed: origin, tablet. Moves: target, nib.
Whatever sensor you pick, it reports the target pose: position t and rotation R. The offset r is fixed in the target frame, so the nib follows from p = t + R r. A small angle error turns the offset and shifts the nib estimate.

None of these systems reports the nib directly. Each one tracks something near it, so I need the geometric relation between what the system sees and the point I care about. If the target position is tt, its rotation is RR, and the nib offset in target coordinates is rr, then

pnib=t+Rr.p_{\text{nib}} = t + Rr.

An orientation error rotates that offset and shifts the estimated nib position, so a target close to the nib reduces this lever arm. Timing has a similar effect: a delayed sample describes where the tool was, so speed turns delay into apparent path error.

Limits

  • Camera depth is the weakest point of the accessible route. Image resolution alone does not establish depth accuracy, and lens calibration, rolling shutter, occlusion, target orientation, and holder flex all affect the result. The ChArUco paper measured a different calibration rig than a Pixel tracking this arm.
  • The reference point and the error metric matter. The tablet has its own tolerance and supplies no calibrated height or tilt. Camera and tracker figures apply to their targets, and a rigid offset cannot account for flex between target and nib. Accuracy, repeatability, and the table's error metrics describe different properties.
  • The quoted conditions differ. SteamVR results include coordinate and sample-rate post-processing. OptiTrack's product page and family sheet, which lists ±0.2 mm, differ. Leica's bound uses specified precision reflectors. None of these figures establishes nib accuracy on this arm.
  • Rate is not latency. OptiTrack's listed camera latency excludes the rest of the application, and the other cited sources do not establish end-to-end delay for the complete recording chains. Tracking volume also depends on visibility, and component prices exclude a complete installation.

Which route fits a small arm?

For a small arm like the SO-101, a calibrated camera is an accessible route from the drawing plane to space. A phone already provides the images, and coded targets supply a geometric reference. The tablet stays useful because it measures the pen directly on the surface.

Sweeping base stations provide wider coverage with compact tracking hardware, but their dynamic error matters for precise paths. Optical motion capture adds synchronized views and rigid-body tracking at the cost of a dedicated installation. A laser tracker fits work that needs an independent dimensional reference. In the end, the choice follows the motion, the point being measured, and the conditions behind each accuracy figure.

← All writing