How good is a given translation? How should you measure its quality and effectiveness? In the words of some Six Sigma specialists, you cannot manage what you don’t measure. Responding to this need, metrics for the quality of translation seem to be popping up all over the place. Or are they? Mostly they’re not.

The EUATC announced its quality standard, developed under the auspices of CEN, the European standards group. Earlier this month the State Commission for Administration of Standardization announced China’s first set of quality standards for translation. Later this year you can expect to see the ASTM Standard Guide to Quality Assurance in Translation and Localization (ASTM Subcommittee F15.48). These will join existing efforts such as the SAE’s J2450, DIN 2345, the LISA QA model, and numerous internal corporate efforts.

As you look for ways to measure the quality of translation work done by in-house staff or by your language service providers, be sure that you pick the appropriate standard. What the CEN and ASTM standard do is measure process rather than the actual quality of the translation. As such, they are more akin to ISO9000 certifications than they are about ensuring that a given document conveys exactly the right meaning and tone of the original. On this point the ASTM is quite clear, pointing out that it is a general framework for developing translation specifications that take into account audience, purpose, and text type. The authors of the ASTM specification do not intend it to be a way for directly evaluating quality, but instead suggest that those interested in quality look at a translation quality metric such as the SAE J2450. That metric is geared to a specific audience of automotive technicians and is intended to provide translation of sufficient quality to let them perform their service work accurately and efficiently. It is limited to one type of document — automotive maintenance and repair manuals — although we have seen some non-auto companies use it to measure their output.

At the end of the day, this presents a conundrum for many companies. Managing and measuring translation process is easy, but outside of controlled language and some industries that have invested in detailed specifications for evaluating language quality on very narrow criteria such as SAE I2450, assessing the quality of a given translation will remain a subjective exercise.