In a vernier caliper, how do the vernier scales
have the ability to measure 0.1mm?

Vernier caliper is an instrument for making very accurate linear measurements introduced in 1631 by Pierre Vernier of France. It utilises two graduated scales: a main scale similar to that on a ruler and a specially graduated auxiliary scale, the vernier, that slides parallel to the main scale and enables readings to be made to a fraction of a division on the main scale.

The most common vernier has a short graduated scale, or "ruler" which slides along a longer scale. The subdivisions on the short rule are nine-tenths as long as the subdivisions on the long scale. Nine small divisions on the large scale are equal to 10 on the small scale. In using the vernier, the large scale is laid along the material to be measured, a small pipe, for example. The small scale is slid until it reaches the end of the pipe. Now we check to see which of its divisions lines up with one of the divisions on the large scale. Suppose the 5, or fifth division from the zero end of the small scale, lines up with 25 on the large scale. Since each division on the small scale is one-tenth smaller than the large divisions, five divisions are equal to only four and one-half on the large scale. Therefore, the end of the small scale rests at 25 - 4.5, or 20.5 on the large scale.
Engineers often use callipers with a vernier attachment. Some read 0.025 millimetre without a magnifier. The setting circles on astronomical telescopes are often equipped with verniers and magnifying devices, allowing a very accurate reading of the position of the celestial object under observation.