/
AI Quality Evaluation

AI Quality Evaluation

Guide to Using AI Quality Evaluation for Translation Evaluation

Overview

AI Quality Evaluation is an AI-powered tool designed to help businesses quickly and accurately assess the quality of their translations and other content. By combining AI efficiency with human expertise, we ensure your content consistently meets the highest standards. This tool evaluates uploaded content, speeds up the process, and guarantees accuracy while reducing costs.

Purpose

The primary function of AI Quality Evaluation is to serve as a quality checker for translations within your project workflow. Once a document is uploaded and translated, the tool evaluates the content using AI tokens, which are consumed during the process. Based on the quality score and the types of content evaluated, users can decide whether further human review is needed, helping to reduce manual review efforts.

Features

  • AI Quality Evaluation can be integrated as a step in your project workflow templates or added to existing projects.

  • Users can purchase AI Quality Evaluation jobs based on the token costs required to evaluate the content.

  • Quality evaluation results gives a preview of the segment quality evaluated by AI system.

  • Users can set a quality threshold, and in the Workbench, translations that meet or exceed this threshold are highlighted and can be treated as approved.

  • Users can purchase AI Quality Evaluation for multiple target languages in a single action.

  • AI Quality Evaluation provides support for 156 languages on the Enterprise platform.

  • Supported Languages: https://lingotek.atlassian.net/wiki/x/BgCdwg

  • Supported File Formats: https://lingotek.atlassian.net/wiki/x/BoCgwg

Steps to Use AI Quality Evaluation

1. Creating Project Workflow Templates with AI Quality Evaluation

a) AI Quality Evaluation is a system phase. You need a workflow template with this phase step to use the feature.

b) Go to Templates > Workflow Templates to create a new workflow template by selecting the ‘+’ icon.

c) We recommend placing the AI Quality Evaluation phase after a Leverage TM and/or Machine Translate/Translate phases.

d) Alternatively, you can add the AI Quality Evaluation phase as a custom phase to an existing project workflow.

2. Create and Upload the Document

a) Create a new project using a workflow template that includes AI Quality Evaluation as a system phase step.

b) Configure initial settings for the Leverage TM and Machine Translate phases.

c) In Project view, under Project Workflow > AI Quality Evaluation, adjust the quality threshold if needed. Translations scoring at or above this value are approved. Default is 100%.

d) Upload a document that meets the supported file type requirements, or input the content which is converted and processed text file (.txt).

e) Select the target language(s) supported by AI Quality Evaluation.

3. Quality Evaluation Purchase and Process

a) The uploaded content will undergo the Leverage TM and/or Machine Translate phases according to the project workflow.

b) Once translated, the content is sent to AI Quality Evaluation for an automated review. The system will generate a token cost.

c) The token balance is associated with the Community and can be viewed in the Document Summary section.

d) The token cost is displayed in the Resources column for the AI Quality Evaluation phase, alongside the target language.

e) To proceed with the quality evaluation, click the ellipsis button in the AI Quality Evaluation phase and select Purchase AI Quality Evaluation.

f) When the Confirm AI Quality Evaluation Purchase dialog appears, users can choose to purchase or skip AIQE. Skipping will bypass the evaluation phase; otherwise, the evaluation will begin.

g) After AI Quality Evaluation completes, users must refresh the targets to see the quality score displayed as a percentage in the Resources column.

h) Clicking the score will display a breakdown of segments evaluated by the AI, categorised as:

  • Best

  • Good

  • Acceptable

  • Bad

  • No Score

  • Untranslated

i) Depending on the quality evaluation results, users can configure further human review phases as and when required.

j) If the target language is not supported by AIQE or the content is an exact match, the Resources column will show "Language not supported by AI Quality Evaluation" or "AI Quality Evaluation not needed for exact match," respectively.

4. Review Translations with AI Quality Evaluation results

a) Once the quality score is ready, users can open the AI Quality Evaluation phase, which will launch in the Workbench.

b) Users can view the quality score for each translation as a percentage next to the segment. Segments are locked and cannot be edited in the AI Quality Evaluation phase.

c) To review translations, the user/reviewer can open the Review phase, which will launch in the Workbench.

d) If a translation score meets or exceeds the quality threshold, the segment will be locked in the Workbench. Hovering over the lock icon displays the quality score and an "AIQE Approved" message. The user/reviewer can approve it directly by clicking the lock icon and the Approve button, or edit the translation before approving if needed. This helps reduce the effort required for manual review.

e) If a translation score is below the quality threshold, the segment will be editable by default. The reviewer can update the translation if needed and approve it.

f) Once AIQE is purchased, the AI Quality Evaluation phase will be marked as complete, and the "Purchase AI Quality Evaluation" option will no longer be available from the ellipsis button.

5. Re-evaluate translations or evaluate for multiple target languages in a single action.

a) If the user/reviewer changes translations in subsequent phases after purchasing AIQE, the quality score becomes invalid. To re-evaluate the updated translations, users must repurchase AIQE using the option under Target Actions. This option also allows evaluate AIQE for multiple target languages in a single action.

b) Select the target(s) to re-evaluate or evaluate, then go to Targets > Actions > Purchase AIQE. This will open the Purchase AI Quality Evaluation dialog.

c) The system will then calculate the AI token cost for the selected targets based on their latest translations and display it. A Purchase AIQE button will appear next to each target, click it to proceed with the purchase.

d) Once the evaluation is complete, the button will turn green; if it fails, it will turn red. Users can either wait in the dialog or close it and refresh the targets later to view the scores.

e) If the language is not supported by AI Quality Evaluation or the document content is an exact match, users will see the respective messages for those targets instead of the AI token cost in this dialog.

Here is a complete demo video guiding you through the steps above. Follow it to set up the workflow and project for AI Quality Evaluation, upload documents, and evaluate translations.

AI Quality Evaluation Demo