Key takeaways
- Resolution, 0.01 mm or 0.0005 in, is the size of one step on the display. Accuracy, ±0.02 mm on a good 150 mm caliper, is how far the reading is allowed to be from the true size. A 0.01 mm display is not accurate to 0.01 mm.
- ±0.001 in is 0.0254 mm, not 0.02 mm. On Mitutoyo's 500-196-30 the metric figure is the tighter one. On the Cen-Tech 47257 the inch figure is tighter, because its metric figure is ±0.03 mm. Plan on the looser of a maker's two figures.
- The figure covers outside jaws only. Mitutoyo rates inside, step and depth readings on the 500-196-30 at ±0.04 mm, twice the outside figure. The budget sheets checked here give no inside figure at all.
- A caliper breaks Abbe's principle, because the part sits below the scale. A jaw tilt of 0.01 mm in 50 mm gives 0.008 mm of error at the tip of a 40 mm jaw, 40 percent of the ±0.02 mm allowance. Measure close to the beam when you can.
- If a reading sits closer to your limit than the caliper's accuracy, the caliper cannot settle it. A 0.0945 in reading on a ±0.001 in caliper does not prove a 0.094 in minimum is met.
Two readings of a caliper spec go around. One says a caliper that shows 0.01 mm is good to 0.01 mm, and the other says ±0.02 mm and ±0.001 in are the same limit in two units. Both are wrong. The display step is not the error, and the two accuracy figures differ by about a quarter, in a direction that changes from brand to brand. Below, a Mitutoyo, a Harbor Freight Cen-Tech and an RS PRO go on one scale, and then we look at why the error grows toward the jaw tips.
Resolution and accuracy are two different numbers
A digital caliper is a steel beam with a scale along it, a fixed jaw at one end and a sliding jaw. The display shows how far apart the jaws are, and the spec sheet carries two figures that look alike.
Resolution is the size of one step on the display. Mitutoyo calls it the "digital step". On the 500-196-30 it is 0.01 mm in metric mode and 0.0005 in in inch mode. The display can show 12.70 or 12.71 mm, but nothing in between.
Accuracy is the most the reading may differ from the true size. Mitutoyo prints it as maximum permissible error, or MPE: ±0.02 mm for the 500-196-30. A reading of 12.70 mm means the part is somewhere from 12.68 to 12.72 mm.
Mitutoyo's own measurement guide is blunt about the link: "it is important to realize there is no relationship between resolution and accuracy." It goes on: "Resolution is constant whereas the accuracy obtainable varies dramatically according to how the caliper is used."
So the last digit on the display is a step, not a promise. On a ±0.02 mm caliper, the accuracy band is two display steps either way of the reading.
There is one more term in the small print. Mitutoyo's leaflet for its ABS Digimatic line lists ±0.02 mm with the footnote "quantizing error excluded". Quantizing error is the error that comes from the display working in whole 0.01 mm steps. The leaflet does not give its size, only that the ±0.02 mm does not include it.
±0.001 in is not ±0.02 mm
An inch is exactly 25.4 mm. So:
0.001 in x 25.4 = 0.0254 mm
0.02 mm / 25.4 = 0.00079 in
±0.001 in allows 0.0254 mm where ±0.02 mm allows 0.02 mm, so the inch figure is 0.0254 / 0.02 = 1.27, or 27 percent looser.
The display step has the same gap: 0.0005 in is 0.0127 mm, 27 percent coarser than 0.01 mm. Switching a caliper to inches does not just change the units, it changes the size of every step and every allowance.
Here is the part nobody spells out: makers do not all round the same way.
| Caliper | Metric figure | Inch figure | Inch figure in mm | Metric figure in inches | Looser figure |
|---|---|---|---|---|---|
| Mitutoyo 500-196-30, outside | ±0.02 mm | ±0.001 in | 0.0254 mm | 0.00079 in | Inch, by 27% |
| Mitutoyo 500-196-30, inside, step, depth | ±0.04 mm | ±0.002 in | 0.0508 mm | 0.00157 in | Inch, by 27% |
| Cen-Tech 47257 | ±0.03 mm | ±0.001 in | 0.0254 mm | 0.00118 in | Metric, by 18% |
On the Mitutoyo the metric figure is the tight one, and the inch figure is 0.00079 in rounded up to the next whole thousandth. On the Cen-Tech it runs the other way, since its metric figure is 0.0254 mm rounded up to the next whole hundredth, 0.03 mm. Each sheet gives one tight limit and one rounded copy, and neither says which is which.
The same rule applies as for the two figures on a laser level: where a maker's two numbers disagree, plan on the looser one.
Five caliper specs and the standard on one scale
Mitutoyo's guide prints Table 1 of the Japanese caliper standard, JIS B 7507:2016, and its finest column is for a 0.02 mm graduation. A 0.01 mm digital caliper has no column of its own, so the 0.02 mm column is the nearest yardstick.
| Caliper | Range | Display step | Outside accuracy | Inside, step, depth | What the figure covers |
|---|---|---|---|---|---|
| Mitutoyo 500-196-30 | 0 to 150 mm (6 in) | 0.01 mm / 0.0005 in | ±0.02 mm / ±0.001 in | ±0.04 mm / ±0.002 in | One figure, full range |
| Mitutoyo ABS 0 to 200 mm models (leaflet) | 0 to 200 mm | 0.01 mm | ±0.02 mm | Not in the leaflet | One figure, quantizing error excluded |
| Mitutoyo 500-172-30 (RS 0374109) | 200 mm (8 in) | 0.01 mm | ±0.02 mm | Not listed | One figure, no range note |
| RS PRO 905803 | 150 mm | 0.01 mm / 0.0005 in | ±0.02 mm | Not listed | Up to 100 mm only |
| Cen-Tech 47257 (Harbor Freight) | 0 to 6 in | 0.01 mm / 0.0005 in | ±0.03 mm / ±0.001 in | Not listed | One figure, no range note |
| JIS B 7507, 0.02 mm column | Up to 300 mm | 0.02 mm | ±0.02 mm to 50 mm, ±0.03 mm to 200 mm, ±0.04 mm to 300 mm | Depth: add 0.02 mm | By length |
Mitutoyo holds ±0.02 mm across 200 mm. The standard's 0.02 mm column allows ±0.03 mm past 50 mm, but Mitutoyo's leaflet lists every 0 to 200 mm model at ±0.02 mm, tighter than the column the guide prints. RS's listing for the 200 mm Mitutoyo 500-172-30 gives the same ±0.02 mm.
The RS PRO figure stops at 100 mm. Its data sheet reads "±0.02mm (≤ 100mm)" for a 150 mm caliper and gives no figure for the last 50 mm, so that part of its range has no published accuracy.
The Cen-Tech sits one step wider. Its ±0.03 mm is 50 percent more than Mitutoyo's metric figure, though in inches the two sheets print the same ±0.001 in. The manual also prints its range as "0" - 6", Or 0mm - 200mm". Six inches is 152.4 mm, so the two halves of that line do not match.
Every caliper here shows 0.01 mm. The display step does not separate them; only the accuracy line does.
Worked through: a saw blade plate on two calipers
Take a check from the thin kerf blade and riving knife guide. A DeWalt DWE7491 wants a blade body no thicker than 0.079 in and a kerf of at least 0.094 in. Say you measure a thin kerf blade plate, and the display shows 1.80 mm, or 0.0710 in.
Mitutoyo, metric mode. ±0.02 mm puts the plate between 1.78 and 1.82 mm, which is 0.0701 to 0.0717 in.
Mitutoyo, inch mode. The same plate shows 0.0710 in, and ±0.001 in puts it between 0.0700 and 0.0720 in. It is the same caliper on the same plate, with a band 27 percent wider.
Cen-Tech, metric mode. ±0.03 mm puts it between 1.77 and 1.83 mm, which is 0.0697 to 0.0720 in.
All three bands sit well under 0.079 in, so any of the three settles the plate question.
Now the kerf. Say the cut in a scrap board reads 0.0945 in on a ±0.001 in caliper. The true kerf could be anywhere from 0.0935 to 0.0955 in, and the low end misses 0.094 in, so the reading does not prove the blade passes. On the Mitutoyo in metric mode the same kerf reads 2.40 mm, and ±0.02 mm puts it at 2.38 to 2.42 mm, or 0.0937 to 0.0953 in. That does not settle it either.
The rule that falls out: if the reading is closer to the limit than the caliper's accuracy, the caliper cannot decide. You need a tighter tool, or the knife-through-the-cut test the DeWalt manual describes.
Why error grows toward the jaw tips
Mitutoyo's guide states Abbe's principle this way: "maximum accuracy is obtained when the scale and the measurement axes are common." In plain words, the part should sit on the line the scale measures along.
A caliper cannot, because its scale runs along the beam while the part is held between the jaws, some distance below it. The guide says a caliper "does not conform to Abbe's Principle", and lists it among the major causes of error.
If the sliding jaw leans out of parallel, its face at the part moves, and the scale does not see that move. The guide gives the error as the distance below the scale times the tilt. Its worked example: "Assume that the error slope of the jaws due to tilt of the slider is 0.01 mm in 50 mm and the outside measuring jaws are 40 mm deep, then the error (at the jaw tip) is calculated as (40/50)x0.01 mm = 0.008 mm."
Scale that example to other spots on the same jaw, keeping the tilt at 0.01 mm in 50 mm:
| Where the part sits | Distance below the scale | Error from that tilt | Share of ±0.02 mm |
|---|---|---|---|
| Near the beam | 10 mm | 0.002 mm | 10% |
| Mid jaw | 20 mm | 0.004 mm | 20% |
| Jaw tip | 40 mm | 0.008 mm | 40% |
The guide names two causes of tilt: "excessive force being used on the slider", or a reference edge on the beam that is not straight. It adds that a worn guide face can give an error "even using the correct measuring force". The JIS handling note it quotes says the caliper "does not have a constant-force device", so you have to apply "an appropriate even measuring force", taking "extra care when you measure it with the root or tip of the jaw."
So hold the part as deep in the jaws as it will go, and squeeze the same way each time. The newer ISO 13385-1:2019, as the guide describes it, requires testing at both the root and the tip of the jaw. A figure checked that way covers both ends of the jaw, but it does not cover your thumb pressure.
Inside, step and depth readings get a wider allowance
The headline figure is for the outside jaws only. The guide explains that under ISO 13385-1:2019, all other readings, "inside, depth, step, and I.D. measurement error", count as scale shift error, rated separately.
Mitutoyo rates the 500-196-30's scale shift error at ±0.04 mm, or ±0.002 in, which is twice its outside figure. For depth, the guide says the standard's limit is the Table 1 value plus 0.02 mm.
Small holes add one more error, because the thin knife-edge inside jaws cannot reach the full diameter of a small bore. The guide's table, for a true 5 mm hole:
| Inside jaw thickness plus gap | Reading short by |
|---|---|
| 0.3 mm | 0.009 mm |
| 0.5 mm | 0.026 mm |
| 0.7 mm | 0.047 mm |
At 0.5 mm, the jaw error alone is more than the caliper's ±0.02 mm outside figure. None of the budget sheets above lists any inside figure, and for a small bore the outside accuracy line tells you nothing.
Temperature: the other error that grows with size
The guide says the caliper's scale is stainless steel, with expansion of "(10.2 ±1) ×10-6 / K", the same as ordinary steel. So a steel part and a steel caliper at the same temperature grow together and cancel, and the error comes from a difference between them.
Here is the arithmetic, ours, from that figure. Say a 150 mm steel part is 10 C warmer than the caliper, fresh from a hand or a sunny bench:
150 mm x 10.2 x 10^-6 x 10 = 0.0153 mm
That is about three quarters of the ±0.02 mm allowance. At 25 mm the same 10 C gives 0.0026 mm, so like the jaw tilt it grows with size. The guide adds that other workpiece materials affect accuracy, but it gives no figure for them, so none is worked out here.
Which caliper a job needs
- Tolerances of ±0.1 mm (±0.004 in) or wider. Any caliper above uses a fifth of the band in metric, and the Cen-Tech uses less than a third. The cheap one is enough.
- Tolerances near ±0.05 mm (±0.002 in). A ±0.02 mm caliper uses 40 percent of the band and a ±0.03 mm one uses 60 percent, before jaw tilt or heat. Use the tighter figure and measure deep in the jaws.
- Readings right at a limit. If the reading is within the caliper's accuracy of the limit, it cannot decide, and that holds for every caliper here.
- Inside diameters and depths. Use the scale shift figure, ±0.04 mm on the Mitutoyo. If the sheet gives none, the outside figure does not cover the job.
- Over 100 mm on the RS PRO 905803. There is no published accuracy, so treat readings there as unrated.
For anything else, compare the looser of the two printed figures with the band the job allows.
Where this answer stops applying
- The figures are new-tool ratings. Each is the maker's figure for a caliper in its delivered state, and the guide warns that a worn guide face adds error even with correct force. A dropped caliper may not hold its rating, and nothing on the display says so. Check it on a known gauge block or a reference you trust.
- Quantizing error is outside Mitutoyo's figure. The leaflet says so and gives no size, so this page does not put a number on it.
- Measuring force is yours. The figures assume an even, moderate squeeze. None of the sheets prints a force, so this page cannot say how hard is too hard.
- The standard text was not read. The JIS B 7507 and ISO 13385-1 details come from Mitutoyo's guide, not from the standards.
- No hands-on check. Nobody measured anything for this page: every figure is a published one, and every derived figure has its arithmetic beside it.
Keeping a caliper at its rating
A caliper, like the click wrench in the torque wrench storage guide, holds its number only if it is cared for, and Mitutoyo's guide gives the routine. Wipe the faces before use, close the jaws and check for zero, and reset the origin after a battery change. After use, wipe off water and oil, then apply a light coat of anti-corrosion oil. Remove the battery if it will sit unused for more than three months, and never store it with the jaws fully closed.
Keep it out of "direct sunlight, high temperatures, low temperatures, and high humidity." The humidity side is covered in rust prevention in tool storage. A cut-out in a foam drawer insert, sized as in the Kaizen foam guide, keeps the jaws from taking knocks.
Calipers are the fourth measuring tool decoded here, after the Class II tape measure, the laser level and the 0.5 mm/m spirit level. All four print a number that sounds like a precision, and each one turns out to be a limit with its own unit and its own conditions.