Machine translation is widely used and plays a crucial role in knowledge sharing and and transferring.
Last year, I had the opportunity to work on a research project concerning language translators. The goal was to compare the most popular online translators and assess the accuracy of their translations from Chinese to English on Social Science articles. The results indicated that Google Translate is the most accurate online translation tool for social science articles. Among the tools evaluated, Baidu, Dictionary, and Bing have similar translation accuracies while Babylon has the lowest accuracy.
I chose five translation sites based on recommendations from published papers and online reviews. I also conducted a quick test where I used each site to translate a few simple-phrased sentences. Google, Bing, Dictionary.com, Babylon, and Baidu were the top sites chosen. From the Journal of Social Science in China, seven article types were selected that included Religion, Politics, Law, Economics, Language and Literature and other Social Sciences.
I randomly sampled 30 from 180 articles abstracts, selecting 3 to 8 phrases from each of the abstracts. A total of 180 phrases were used, and 900 translations (180 phrases from each of the 5 online translators) were tested. I designed a score-rating method based on the amount of errors and the comprehensibility of a translation.
I determined an experimental design to test if there were any significant differences of translation accuracy. Three testers compared each of the results with the original English translation, and scored them based on a rating rubric. In total, 2700 ratings were given. After all of the scores were gathered, I performed two statistical tests, a two-sample T-test and ANOVA, to determine whether there were any statistically significant differences among the translation tools and article types. Contradicting my hypothesis, the analysis result indicated that Google Translate performed the best. It had the highest average score that was significantly better than its competitors.
On the other hand, Baidu, Dictionary.com, and Bing had similar average scores, without any statistically significant differences. Finally, Babylon was significantly worse than other tools, with the lowest accuracy among all sites that were tested. The accuracy of translation also varied slightly with different article types. Google had the highest average score among all types of articles except Language and Literature. Dictionary.com had a slightly higher average score than Google in the Language and Literature category, but not at a statistically significant level. Babylon.
com had the lowest scores across all types of articles. While Google was the best translation tool for social science articles when I conducted this research, its average rating reflected room for improvement. The complex Chinese language make it difficult for Google to use its statistical rule-based algorithm to translate accurately.
I found that some phrases that Google translated contained some errors in grammar, words, and even the entire meaning. I documented these findings and reported them to the Google support team. In November 2016, three months after my project, Google released a new translation system that uses a neural machine translation, and apparently significantly reduced translation errors. Using some of the inaccurately-translated phrases from my research, I tested the new version of Google Translate.
After comparing the new and old translation results, I confirmed that the new version of Google has significantly improved its translation accuracy.
