imported>Conan: Bot: Removing category :Category:Applications of artificial intelligence because it's already under :Category:Natural language processing

2024-06-23T19:13:00Z

Bot: Removing category Category:Applications of artificial intelligence because it's already under Category:Natural language processing

New page

The sections below give objective criteria for evaluating the usability of [[machine translation]] software output.

== Stationarity or canonical form ==
{{main|Round-trip translation}}
Do repeated translations converge on a single expression in both languages? I.e. does the translation method show [[stationary process|stationarity]] or produce a [[canonical form]]? Does the translation become stationary without losing the original meaning? This metric has been criticized as not being well correlated with [[BLEU]] (BiLingual Evaluation Understudy) scores.<ref name="somers05">{{cite journal|last1=Somers|first1=Harold|title=Round-trip translation: What is it good for?|journal=Proceedings of Australasian Language Technology Workshop ALTW 2005|date=2005|pages=127–133|url=http://personalpages.manchester.ac.uk/staff/harold.somers/RoundTrip.doc|location=Sydney}}</ref>

== Adaptive to colloquialism, argot or slang ==

Is the system adaptive to [[colloquialism]], [[argot]] or [[slang]]? The [[French language]] has many rules for creating words in the speech and writing of [[popular culture]]. Two such rules are: (a) The reverse spelling of words such as ''femme'' to ''meuf''. (This is called [[verlan]].) (b) The attachment of the suffix ''-ard'' to a noun or verb to form a proper noun. For example, the noun ''[[faluche]]'' means "student hat". The word ''faluchard'' formed from ''faluche'' colloquially can mean, depending on context, "a group of students", "a gathering of students" and "behavior typical of a student". The Google translator as of 28 December 2006 doesn't derive the constructed words as for example from rule (b), as shown here:

<blockquote>
Il y a une chorale falucharde mercredi, venez nombreux, les faluchards chantent des paillardes! ==> ''There is a choral society falucharde Wednesday, come many, the faluchards sing loose-living women!''
</blockquote>

French argot has three levels of usage:<ref>[http://chitlinsandcamembert.blogspot.com/2005/10/agony-of-argot.html "The Agony of Argot", Chitlins & Camembert, October 28, 2005]</ref>
#''familier'' or friendly, acceptable among friends, family and peers but not at work
#''grossier'' or swear words, acceptable among friends and peers but not at work or in family
#''verlan'' or ghetto slang, acceptable among lower classes but not among middle or upper classes

The United States [[National Institute of Standards and Technology]] conducts annual evaluations [https://www.nist.gov/speech/tests/mt/] of [[machine translation]] systems based on the [[BLEU]]-4 criterion [https://www.nist.gov/speech/tests/mt/doc/mt06_evalplan.v4.pdf]. A combined method called IQmt which incorporates BLEU and additional metrics NIST, GTM, ROUGE and METEOR has been implemented by Gimenez and Amigo [http://www.lsi.upc.edu/~nlp/IQMT/IQMT.v1.0.pdf].

== Well-formed output ==

Is the output grammatical or [[well-formedness|well-formed]] in the target language? Using an interlingua should be helpful in this regard, because with a fixed interlingua one should be able to write a grammatical mapping to the target language from the interlingua. Consider the following [[Arabic language]] input and [[English language]] translation result from the Google translator as of 27 December 2006 [http://www.google.com/language_tools?hl=en]. This Google translator output doesn't parse using a reasonable [[English grammar]]:

<blockquote>
وعن حوادث التدافع عند شعيرة رمي الجمرات -التي كثيرا ما يسقط فيها العديد من الضحايا- أشار الأمير نايف إلى إدخال "تحسينات كثيرة في جسر الجمرات ستمنع بإذن الله حدوث أي تزاحم".
==>
And incidents at the push Carbuncles-throwing ritual, which often fall where many of the victims - Prince Nayef pointed to the introduction of "many improvements in bridge Carbuncles God would stop the occurrence of any competing."
</blockquote>

== Semantics preservation ==

Do repeated re-translations preserve the [[semantics]] of the original sentence? For example, consider the following English input passed multiple times into and out of French using the Google translator as of 27 December 2006:

<blockquote>
Better a day earlier than a day late. ==>

''Améliorer un jour plus tôt qu'un jour tard.'' ==>

To improve one day earlier than a day late. ==>

''Pour améliorer un jour plus tôt qu'un jour tard.'' ==>

To improve one day earlier than a day late.
</blockquote>

As noted above and in,<ref name="somers05" /> this kind of round-trip translation is a very unreliable method of evaluation.

== Trustworthiness and security ==

An interesting peculiarity of [http://www.google.com/translate_t?langpair=en|es Google Translate] as of 24 January 2008 (corrected as of 25 January 2008) is the following result when translating from English to Spanish, which shows an embedded [[joke]] in the English-Spanish dictionary which has some added poignancy given recent events:

<blockquote>
[[Heath Ledger]] is dead ==>

''[[Tom Cruise]] está muerto''
</blockquote>

This raises the issue of trustworthiness when relying on a machine translation system embedded in a [[Life-critical system]] in which the translation system has input to a [[Safety Critical Decision Making]] process. Conjointly it raises the issue of whether in a given use the software of the machine translation system is [[Security|safe]] from [[Hacker (computer security)|hackers]].

It is not known whether this feature of Google Translate was the result of a joke/hack or perhaps an unintended consequence of the use of a method such as [[statistical machine translation]]. Reporters from [[CNET Networks]] asked Google for an explanation on January 24, 2008; Google said only that it was an "internal issue with Google Translate".<ref>[http://www.news.com/8301-13577_3-9857280-36.html?tag=newsmap "Google Translate bug mixes up Heath Ledger, Tom Cruise", by Caroline McCarthy], [[CNET Networks]], January 24, 2008</ref> The mistranslation was the subject of much hilarity and speculation on the Internet.<ref>[http://gawker.com/5002510/tom-cruise-is-spanish-for-heath-ledger '"Tom Cruise" is Spanish for "Heath Ledger"', gawker.com, January 24, 2008] {{webarchive|url=https://web.archive.org/web/20080128172538/http://gawker.com/5002510/tom-cruise-is-spanish-for-heath-ledger |date=January 28, 2008 }}</ref><ref>[http://rayhey2.blogspot.com/2008/01/tom-cruise-est-muerto.html "Tom Cruise está muerto", Ray Leon Blog Project, January 24, 2008] {{webarchive|url=https://web.archive.org/web/20081029004505/http://rayhey2.blogspot.com/2008/01/tom-cruise-est-muerto.html |date=October 29, 2008 }}</ref>

If it is an unintended consequence of the use of a method such as [[statistical machine translation]], and not a joke/hack, then this event is a demonstration of a potential source of critical unreliability in the statistical machine translation method.

In human translations, in particular on the part of [[Interpreting|interpreters]], selectivity on the part of the translator in performing a translation is often commented on when one of the two parties being served by the interpreter knows both languages.

This leads to the issue of whether a particular translation could be considered ''verifiable''. In this case, a converging round-trip translation would be a kind of verification.

==See also==
*[[Comparison of machine translation applications]]
*[[Evaluation of machine translation]]
*[[Round-trip translation]]
*[[Translation]]

==Notes==
{{reflist}}

==References==

* Gimenez, Jesus and Enrique Amigo. (2005) [http://www.lsi.upc.edu/~nlp/IQMT/IQMT.v1.0.pdf IQmt: A framework for machine translation evaluation].
* NIST. [https://www.nist.gov/speech/tests/mt Annual machine translation system evaluations] and [https://www.nist.gov/speech/tests/mt/doc/mt06_evalplan.v4.pdf evaluation plan].
* Papineni, Kishore, Salim Roukos, Todd Ward and Wei-Jing Zhu. (2002) BLEU: A Method for automatic evaluation of machine translation. Proc. 40th Annual Meeting of the ACL, July, 2002, pp. 311–318.

[[Category:Computational linguistics]]
[[Category:Machine translation software|*]]
[[Category:Natural language processing]]

Machine translation software usability - Revision history

imported>Conan: Bot: Removing category :Category:Applications of artificial intelligence because it's already under :Category:Natural language processing