A lot of people keep telling me that online machine translators “give you an idea anyway”. A while ago, I found a sentence that is quite useful to test these widgets. It is worth noting that it’s a very simple sentence, with a purely communicative function, but it contains four terms – fan, club, match and cup – all of which have two very different meaning and can be quite ambiguous for a machine to translate. Let me clarify why, for the non-bilingual. The Italian word for fan as in the device that produces air current is ventilatore, while a supporter is tifoso or fan; club as in what Fred Flintstone carried around is clava or mazza while a group of people with a common interest is called a club in Italian too. Match as in what you use to light your cigarette is fiammifero or cerino but it’s partita if you mean a game, and cup is translated as tazza if it’s what your drink your tea in, but as coppa if you mean a trophy. So, the sentence is
“the Italian fan club was ready for the last match of the World Cup”
The experiment is a simple one. Translating the sentence from English to Italian, then back to English. I used it to test Babel Fish, WorldLingo, Reverso, Bing Translator, Systran and Google Translate, with surprising outcomes.
Yahoo! Babel Fish has – incomprehensibly – quite a reputation, but that might be due more to its old age than to the results it achieves:
EN > IT
“il randello di ventilatore italiano era pronto per l’ultimo fiammifero della tazza di mondo”
IT > EN
“___ randello of Italian fan the ready era for l’ last match of the world cup”
Right away, translating into Italian, Babel Fish fails on all our ambiguous words, and misses a preposizione articolata (that’s when the prepositions merge with the articles), so that, despite maintaining the sentence’s structure, the result is incomprehensible. Going back to English, it loses two articles, leaves one in Italian, mistakes a verb for a noun and doesn’t translate the noun randello. The resulting English sentence is not just incomprehensible, it’s not English at all. My rating: 1/10
WorldLingo suffers from similar problems:
EN > IT
“il randello di ventilatore italiano era aspetta per l’ultimo fiammifero della tazza del mondo”
IT > EN
“the randello of Italian fan it was waits for for the last match of the cup of the world”
Let’s try Reverso:
EN > IT
“il bastone di ventilatore italiano era pronto per l’ultimo fiammifero della Coppa del Mondo”
IT > EN
“the baton of Italian fan was ready for the last match of the World Cup”
A little step forwards. Finally World Cup is recognised as Coppa del Mondo, but nonetheless we’re still talking about an ultimo fiammifero. Three out of fours ambiguous words are not translated correctly. Club here becomes bastone (stick) instead of randello, but nothing changes. Era pronto is correctly translated into was ready. Nonetheless, the sentence we get is still not understandable. Going back to English, the first half of the sentence is definitely absurd, bastone becomes baton and the word order doesn’t make much sense. My rating: 4.5/10
Note how these first three softwares used randello and bastone to translate club, which actually corresponds in most cases to clava (which is also a cognate), or mazza (e.g. golf club = mazza da golf).
Let’s move to Microsoft‘s Bing Translator :
EN > IT
“il fan club italiano era pronto per l’ultima partita di Coppa del mondo”
IT > EN
“the fan club Italian was ready for the latest batch of ___ World Cup”
Hell! The English to Italian translation is impeccable, except for a capital. Maybe Bill Gates will prove himself once more? Unfortunately, going back to English things unravel. There is a problem with the word order (fan club Italian instead of Italian fan club). Those four ambiguous words go back to what they were, but – unexpectedly – that unfortunate latest batch comes up, an ultima partita in a commercial sense, and that really kills the sentence. Then, to be painfully strict, an article is lost in translation. It would be hard to get the original meaning out of the final sentence, if you weren’t reading this article. To be fair, though, compared to the competitors above, we start seeing some logic in the translation process. My rating: 5/10.
Now, let’s move to the free translator on Systran‘s website, the first translation software ever and the leading supplier of translation software.
EN > IT
“il fan club italiano era pronto per l’ultima partita della coppa del Mondo”
IT > EN
“the fan Italian club ready era for the last game of the World Cup”
Once again, four out of four ambiguous words are translated correctly. Except for a capital, the English to Italian translation is fine. Revert to English, and we grimace in disappointment (or is it relief?) because, inexplicably, a verb is once again mistaken for a noun. A fundamental mistake that affects the comprehension. Then, there is a word order problem (the fan Italian club). Shame about that era translated as a noun, which really kills the final English sentence. My rating: 5/10
Finally, let’s see how Google Translate performs.
EN > IT
“Il fan club italiano era pronto per l’ultima partita della Coppa del Mondo”
IT > EN
“___ Italian fan club was ready for the final match of ___ World Cup”
Bloody hell, doesn’t Google always do things properly? The translation from English to Italian is perfect. Going back to English, we even see a lexical correction, where the machine uses final instead of last – very appropriately, given the context. Strangely, though, Google Translate fails with the article il and the preposizione articolata della, ignoring them completely, and producing a sentence which is perfectly understandable but still sounds funny. In any case, it is pretty clear that this new widget comes off as the prodigy here – but we’re talking about Google, should we really act surprised? My rating: 8.5/10
To sum up, if you need to grasp the general meaning of a sentence, Google Translate (and Google Translate only) seems to guarantee a minimum of reliability. Nevertheless, let’s make clear that as soon as we move from simple informative sentences to more complex ones, even Google Translate is not to be trusted. I will soon analyse a recent case to demonstrate this. In the meantime, if it is a translation that you need, I warmly advise you to contact a flesh-and-blood professional translator.
Smuggled Words 
@Giuseppe Manuel Brescia: The experiment is a simple one. Translating the sentence from English to Italian, then back to English.
Imagine that you translate from English to a language that you do not know. Then, translate back to English. You do not know whether the translation to the unknown language is good or bad. You know only whether the translation back to English is good or bad.
Google translates the English sentence ‘The event came out of the blue’ to the Italian sentence ‘L’evento è venuto fuori dalla blu.’ Although the Italian translation has no meaning, the sentence is translated back to English correctly. (This test was done on 2008-12-12.)
Welcome, Mike.
Sure, that is yet another level of the problem. I did the translation back to English precisely to show how in many cases the sentence is not even translated back to English correctly. In the case of ‘The event came out of the blue’ that worked, but as you can see that’s not always the case, which makes it all so much worse, in my opinion, because there are mistakes at multiple level.
The point is, as you said, that ‘You do not know whether the translation to the unknown language is good or bad’ and the same can be said for translation from an unknown language.
Bottom line: these programs are fascinating and Google certainly shows some potential, but the amount of programming work and resources needed seems not to be worth the result, which could be achieved so much better by the good old human translators.
[…] recognition might be (think of noise, and above all dialects and idiolects). Secondly, judging by the shortcomings of machine translation, I wouldn’t be to sure of that text-to-text part, either. Synthesis is probably the only […]
[…] also shows how machine translation is perfectly useless for literary texts. A few months ago, I compared the most used free automatic translators, in an effort to show their users how easily things can go wrong. I was quite surprised to find […]