Introduction

What is a scripted job?

A scripted Job is custom JavaScript code executed as a process step. The script can react to specific events that occur in the project and perform operations on the job or communicate with remote services.

The code logic can be anything JavaScript offers, including file processing (e.g., changing XML file content), remote Web Service execution (e.g., OCR, QA check), or modifying project attributes (via the Home Portal API). The use cases include, for example, integration with LtCloud (language detection service), Systran.io (machine translation and voice-to-text systems), OCRweb (OCR), Slack, and Twitter (for communication and newsfeed), among others.

The code execution is triggered by a process state change (e.g., a new file uploaded in a previous step).

Human-inserted job vs. scripted job

Scripts in XTRF are primarily suited to initiate and process Jobs, as well as automate the distribution of files between XTRF and external services. For that reason, some attributes are not configured by the scripts, like communication channels, finances, the stages of sending a purchase order, and vendor selection. For the same reasons, job statuses like ‘Offers sent’ and ‘Accepted’ are also absent from scripted jobs. The script decides whether the delivered files are ‘Verified’ or ‘Need Correction.' If the script does not specify the conditions for ‘Need Correction,' the files are classified as ‘Verified’ by default.

How does the XTRF script work?

XTRF scripts use a synchronous execution model, in which, after a series of script requests and Web Service (WS) responses, the WS responds with URLs to completed documents. Then, the XTRF script either downloads and adds the documents to the project or job or adds the links to the remote documents.

Synchronous scripts are simple to implement, as the entire logic is included in the script, and the terms of conclusion are easy to determine. No external WS is required either.

On the flip side, synchronous scripts may struggle with large files and longer processing times.

API References

Event handlers (callback functions)

Events are actions or incidents in the system you’re programming. The system informs you about them, so you can code an appropriate response, which is an event handler. In our case, an event handler is a JavaScript function that runs in response to an event. For example, when a user clicks a button on a website, you may display an information popup.

Event handlers (callbacks) are necessary because many JavaScript actions are asynchronous: they don’t stop other functions from running simultaneously. In such cases, with the callback, actions will continue to be executed in the background while the rest of the code runs.

Event object properties

project – a project or a quote where the job belongs.

job – an automatic job that executes the script.

file – a file that has been shared with the job.

languageCombination – a language combination that has been added to the job.

List of event handlers

Callback function	Event description
`onWorkFileSharedWithJob(event)`	A file was shared with the Job as a Work File (or had been shared before the Job was started).

Object Methods

The ‘Object’ class represents one of JavaScript's data types used to store various keyed collections and more complex entities. Objects usually come with a set of properties, and a JavaScript object method is a property that contains a function definition.

Project and quote object methods

Method name	Returns	Description
`id()`	`String`	An internal identifier of the project or quote.

Job object methods

Method name	Returns	Description
`id()`	`String`	An internal identifier of the job.
`sharedWorkFiles()`	`File[]`	A list of files that are shared with the job as work files.
Method name	Parameters	Description
`addFile(file)`	`file` – file to be added to the job.	Adds a file to the project as a file delivered in the job. The following properties have to be specified for the file (an exception is thrown otherwise): name. category.

File object methods

Method name	Details	Description
`File(httpResponse)` (constructor)	Parameters: `httpResponse (Response)` – provides the file name and the content for the new file, e.g., the result of `utils.transport.download(url)`. Returns: `File`	Creates a new File object based on an HTTP Response object, which provides the file name and the content for the new file.
`File(content, name)` (constructor)	Parameters: `name (String)` – provides the file name for the new file. `content (String)` – provides the file content for the new file. Returns: `File`	Creates a new File object based on the provided content and name.
`id()`	Returns: `String`	An internal identifier of the file.
`name()`	Returns: `String`	Returns the file name.
`name(newName)`	Parameters: `newName`	Sets a name for the file.
`category()`	Returns: Category key `(String or null)`. Available values: `BILINGUAL_DOC` `CAT_ANALYSIS` `CAT_PACKAGE` `CAT_PACKAGE_RETURN` `FILTERING_RULES` `FORMATTED_DOCUMENT` `MEMOQ_LIGHT_RESOURCES` `OTHER` `QA_REPORT` `REFERENCE` `SEGMENTATION_RULES` `SOURCE_DOCUMENT` `SOURCE_TO_BE_PREPARED` `TERMINOLOGY` `TM` `TRANSLATED_DOCUMENT`	Returns the file's current category (defined by the category key). It can be `null`, e.g., in `onFileUploaded`. Default: `null`. Use the `analyze()` method to automatically detect the file's category.
`category(newCategory)`	Parameters: `newCategory (String)` – key of the category. Available values: `BILINGUAL_DOC` `CAT_ANALYSIS` `CAT_PACKAGE` `CAT_PACKAGE_RETURN` `FILTERING_RULES` `FORMATTED_DOCUMENT` `MEMOQ_LIGHT_RESOURCES` `OTHER` `QA_REPORT` `REFERENCE` `SEGMENTATION_RULES` `SOURCE_DOCUMENT` `SOURCE_TO_BE_PREPARED` `TERMINOLOGY` `TM` `TRANSLATED_DOCUMENT`	Sets the file's category (defined by the category key). Default: `null` Use the `analyze()` method to automatically detect the file's category.
`languages()`	Returns: an array of objects. Each object in the array represents a language combination. Each language in the language combination is represented by a Language object (see the Language object description) or null (which means "Any"). An empty array means no language combination is specified for the file. Example: `[ { "sourceLanguage": null, "targetLanguage": Language }, ... ]`	Returns a list of language combinations of the file. Can be empty when no language combination is specified for the file. Can be null, e.g., in onFileUploaded. Default: `[]` Use the `analyze()` method to automatically detect the file’s languages.
`languages(newLanguages)`	Parameters: `newLanguages (Array)` – an array of objects. Each object in the array represents a language combination. Each language in the language combination is represented by a Language object (see Language object description) or null (which means "Any"). An empty array means no language combination is specified for the file. Example: `[ { "sourceLanguage": null, "targetLanguage": Language }, ... ]`	Sets a list of language combinations for the file. An empty array means no language combination is specified for the file. Language objects can be either obtained as a result of some other method call or created manually by using a Language object constructor. See the Language object description for details. Use the `analyze()` method to automatically detect the file’s languages.
`verificationStatus()`	Returns: `Boolean`	Returns the current verification status of the file. Default: `true`.
`verificationStatus(newStatus)`	Parameters: `newStatus (Boolean)`	Sets the file's verification status. Default: `true`.
`analyze()`	Returns: `{ detectedCategory, detectedLanguages, }`	Analyzes the file to suggest the best-matching file’s category and languages.

Language object methods

Method name	Details	Description
`Language(symbol)` (constructor)	Parameters: `symbol` `(String)` – symbol of the language the object should represent. Returns: `File`	Creates a new Language object that represents a language defined in System Configuration in XTRF Home Portal. The language is looked up by its symbol. If not found, the constructor throws an exception.
`name()`	Returns: `String`	Returns the language name.
`localizedName(locale)`	Parameters: `locale (String)` – language code of the locale (according to standard Java implementation). Returns: `String`	Returns the language name localized to the specified language if available, or the same as the name() otherwise.
`symbol()`	Returns: `String`	Returns the language symbol.
`isoCode()`	Returns: `String`	Returns the language ISO 639-1 code.
`isoCode3()`	Returns: `String`	Returns the language ISO 639-2 code.
`multiTermAlias()`	Returns: `String`	Returns the language SDL MultiTerm alias.

Utilities Available for a Script

JavaScript utilities are used to add common functionalities to the script.

HTTP Transport

Method name

Details

Description

utils.transport.download(url)

Parameters:

url (String) – file URL address, supported protocols: file, ftp, ftps, sftp, http, https.

Returns: Response

Downloads the file and returns the Response object that can be used to create a new File.

utils.transport.post(File)

Parameters:

httpResponse (Response) – provides the file name and content for the new file, e.g., result of utils.transport.download(url), also takes File as a parameter.

Returns: Response

Attaches the file content and name to the HTTP request and sends the request.

Use Cases

Changing file content in the script

Case description

Before the translation step, each XML source file should be modified so that the phrase “targetLangauage=xx” is replaced with a real target language ISOcode(s). For each input file, there is one result file for each target language in the project.

Solution 1 – one job for each language combination

A job is executed once for each file for all language combinations. This solution requires splitting jobs into individual language combinations, so there is a dedicated job executed in each language combination.

function onWorkFileSharedWithJob(event) {
    var targetLanguage = event.job().languages()[0].targetLanguage()
    changeAndStore(event.file(), targetLanguageName)
}

function changeAndStore(file, targetLanguage) {
    if (file.name().endWith(".xml")) {
        var content = file.asString()
        var convertedContent = content.replace("targetLangauage = xx", targetLanguage.symbol)
        var resultFile = new File(convertedContent, file.name())
        resultFile.category(file.category())
        resultFile.languages([{"sourceLanguage": null, "targetLanguage": targetLanguage.symbol}])
        event.job().addFile(resultFile)
    }
}

Solution 2 – one job for all language combinations

This solution does not require a split into separate language combinations. However, to add a new language combination after the job is executed, an additional action is required. All Work Files that have already been processed need to be converted to the newly added language. It is handled by implementing the onLanguageCombinationAdded() method.

function onWorkFileSharedWithJob(event) {
    for (lc in event.job().languages()) {
        var targetLanguage = lc.targetLanguage()
        changeAndStore(event.file(), targetLanguage) //as defined in a previous example
    }
}

function onLanguageCombinationAdded(event) {
    for (file in event.job().sharedWorkFiles()) {
        var targetLanguage = lc.targetLanguage
        changeAndStore(file, targetLanguage) //as defined in a previous example
    }
}

Converting a file using an external (Web) service

Case description

Before the translation, a file requires Optical Character Recognition (OCR). This action uses an external OCR service to convert the file to the .docx format. Each file may be processed separately. OCR service works synchronously. For this script, we used http://ocrwebservice.com to provide the OCR service; however, you can use most of the OCR exposing REST API. The detailed documentation can be found in http://www.ocrwebservice.com/api/restguide

Solution

function onWorkFileSharedWithJob(event) {
    var userName = "test"
    var licenseCode = "licenseCode"
    var authentication = "Basic " + utils.Base64.encodeToBytes(user_name + ":" + license_code)
    var urlParams = " ? language = " + file.language.name + " & outputformat = docx"
    var url = "https://www.ocrwebservice.com/restservices/processDocument" + urlParams
    var result = utils.transport.http().target(url)
        .header("Authentication", authentication)
        .header("Content-Type", "application/json")
        .request().post(event.file());
    if (result.getResponseCode() == 200) {
        var resultFileURL = result.asJson().outputFileUrl
        var response = utils.transport.download(resultFileURL)
        var ocrFile = new File(response)
        ocrFile.category("SOURCE_DOC")
        ocrFile.languages(event.job().languages())
        ocrFile.name(event.file().name() + ".docx")
        event.job().addFile(ocrFile)
    } else {
        throw 'service OCR exception' + result.getResponseCode() + getResponseDescription();
    }
}

For production use, the above solution requires additional handling of prepayment status. The language names need to be adjusted to the service's specific requirements (e.g., if it does not support the ISO code, use the name instead).

Converting a file – a local drive approach

Case description

An OCR service is installed on a local infrastructure that shares a file system with the XTRF server, so there is no need to push the file via http call. The scripted job moves the file to a dedicated location and triggers OCR execution via a synchronous API call.

Solution

Assuming that XTRF has RW access to /tmp/QAshared folder, which is shared with the remote QA service, the script would look as below.

function onWorkFileSharedWithJob(event) {
    // make a local copy of a file out of internal XTRF repository
    var localFilePath= utils.localTransport().save(file)
    var urlParams = "?language=" + file.language.name +
        "&outputformat=docx" + "&filePath=" + localFilePath
    var url = "http://ocr.localdomain/restservices/processDocument" + urlParams
    var result = utils.transport.http().target(url)
        .header("Content-Type", "application/json")
        .request().post(event.file().content().toString());
    if (result.getResponseCode() == 200) {
        var resultLocalFIlePath = result.asJson().outputFilePath
        var ocrFileProperties = {
            category = "SOURCE_DOC",
            languages = event.job().languages()
        }
        ocrFile = utils.localTransport().open(resultLocalFIlePath)
        ocrFile.properties(ocrFileProperties)
        event.job().addFile(ocrFile)
    } else {
        throw 'service OCR exception' +
            result.getResponseCode() + result.getResponseDescription();
    }
}

The above solution is particularly useful when migrating from Classic Projects’ automatic actions.

Changing project data

Case description

Each project contains a custom field ‘numberOfFilesTranslated’ with a total number of files translated. The scripted job updates this number after a new file is translated.

Solution

The scripted job is located just after the translation step.

function onWorkFileSharedWithJob(event) {
    currentNumber = event.project().customField("numberOfFilesTranslated") + 1
    event.project().customField("numberOfFilesTranslated", currentNumber)
}

The above solution assumes only one scripted job is executed in a given project and process step. It corresponds to an underlying event-queue model; however, this may change in the future.

There are some other project parameters one can change using ScripptedJobs SDK. However, to have more control over other XTRF entities (Vendors, Clients, etc.), consider using the Home Portal API called as any RESTful service from within the script.

Batch file processing using file metadata

Case description

QA KPI is calculated as the average of each translated file’s QA score. It is to be stored in a project custom field – overAll_QA.

Solution

To speed up the process, QA is calculated for each file individually. Once the entire step is complete, a final calculation is performed. We use a feature that enables us to add userMetaData to each file.

function onWorkFileSharedWithJob(event) {
    var fileQAScore = 0
    // a webservice can be called to calculate the QA (see previous examples)
    // or the calculation can be made in the script code
    file().userData("qaScore", fileQAScore)
}

function onAllPreviousJobsFinished(event) {
    var numberOfFiles = 0;
    var totalScore = 0;
    for (file in event.job().deliveredFiles()) {
        totalScore = totalScore + file().userData("qaScore")
        numberOfFiles = numberOfFiles + 1
    }
    result = 0;
    if (numberOfFiles > 0) {
        result = totalScore / numberOfFiles
    }
    event.project().customField("overAll_QA", currentNumber)
    event.job().finish() //needs to be called explicitly when overriding onAllPreviousJobsFinished
}

Jobs’ user context – passing data between method invocations in one job

Case description

To access a remote service and process all the files within the step, an individual access token is required.

Solution

function onJobCreated(event) {
    var token = ""
    // a webservice call to obtain a token (see previous examples)
    event.job().userData("token", token)
}

function onWorkFileSharedWithJob(event) {
    var token = event.job().userData("token")
    // a webservice can be called which is using a token  (see previous examples)
}

There could be another solution where the token is stored in a project custom field. However, having multiple script types in a single process can lead to concurrency and data protection issues.

Handling temporary & permanent remote service errors

Case description

Due to stability requirements, a remote service may sometimes not be accessible. Instead of raising an exception and forcing the PM to manually retry the operation, it should be retried automatically 5 times after a given period.

Solution

function onWorkFileSharedWithJob(event) {
    // some webservice operation – refer to the previous examples
    if (result.getResponseCode() == 404) {
        if (event.retryNumber < 5) {
            event.retry(10000) //in milliseconds
        } else {
            //seems to be a permanent error
            throw 'service OCR exception' + result.getResponseCode() + result.getResponseDescription();
        }
    }
}

Please note that the number of retries could also be limited by the XTRF system configuration.

Asynchronous service call

Case description

The OCR service used in the case ‘Converting a file using external (Web) service’ took a lot of time: over 10 minutes, exceeding the Scripted Job runtime. One needs to switch to a service supporting asynchronous invocation and callbacks.

Solution

function onJobCreated(event) {
    //we need to count the number of callbacks we await from a remote OCR service
    //as far as store information if all the previous jobs are done
    event.job().userData(“numberOfActiveExecutions”, 0)
    event.job().userData(“allPreviousJobFinished”, 0)
}

function onWorkFileSharedWithJob(event) {
    var fileUploadURL = event.job().fileUploadUrl()
    var urlParams = "?language=" + file.language.name +
        "&outputformat=docx" + "&fileUploadURL=" + fileUploadURL
    var url = "https://www.ocrwebservice.com/restservices/processDocumentAsync" + urlParams
    var result = utils.transport.http().target(url).
        request().post(event.file().content().toString());
    //handle http response code etc.
    //we need to count the number of callbacks we await from a remote OCR service
    event.job().userData("numberOfActiveExecutions",
        event.job().userData("numberOfActiveExecutions") + 1)
}

function onFileUploaded(event) {
    event.job().userData("numberOfActiveExecutions",
        event.job().userData("numberOfActiveExecutions") - 1)
    finishIfPossible(event)
}

function onAllPreviousJobsFinished(event) {
    event.job().userData("allPreviousJobFinished", true)
    finishIfPossible(event)
}

function finishIfPossible(event) {
    if (event.job().userData("allPreviousJobFinished") &&
        event.job().userData("numberOfActiveExecutions") <= 0) {
        event.job().finish()
    }
}

The above solution assumes only one scripted job is executed in a given project and process step. It corresponds to an underlying event-queue model; however, this may change in the future.

The OCR web service does not support callbacks, so the above example only illustrates the potential solution for other services that do.

Getting external CAT tool analysis

Case description

A company uses a proprietary CAT tool. For each language combination, source files should be sent to the CAT tool. CAT analysis generated by the CAT tool should be used in Receivables to calculate the price for the client. The value should be updated each time a new file is added to the project. A new receivable should be generated when a new language combination is added to the project.

Solution

This solution assumes a valid configuration of finance automation. The automation should be configured so that each update of the CAT analysis in the project automatically creates or updates the project finances.

function onJobCreated(event) {
    //we create an empty project in a corresponding CAT tool
    //and store catProejctid in a context of a job
    var url = "https://www.mycat.local/restservices/createProject"
    var result = utils.transport.http().target(url).
        .param("name", event.project().name)
        .param("customerName", event.project().client().name)
        .request().put().asJson();
    event.job().userData("catToolProjectId", result.projectId)
}

function onWorkFileSharedWithJob(event) {
    for (lc in event.job().languages()) {
        addNewFileToCATProject(event)
        updateCatAnalysis(event, lc)
    }
}

function onLanguageCombinationAdded(event) {
    var url = "https://www.mycat.local/restservices/addLanguageCombinationToProject"
    var result = utils.transport.http().target(url)
        .param("id", event.job().userData("catToolProjectId"))
        .param("sourceLanguage",
            event.languageCombination().sourceLanguage().name())
        .param("targetLanguage",
            event.languageCombination().targetLanguaget().name())
        .request().post();
    //the CAT tool logic uses all source files in the newly added LC
    updateCatAnalysis(event, lc)
}

function addNewFileToCATProject(event) {
    var catProjectId = event.job().userData("catToolProjectId")
    var url = "https://www.mycat.local/restservices/addFileToProject"
    utils.transport.http().target(url)
        .param("id", catToolProjectId)
        .request().post(event.file().content.toString());
}

function updateCatAnalysis(event, lc) {
    url = "https://www.mycat.local/restservices/getAnalisys?projectId=" +
        event.job().userData("catToolProjectId") + "&sourceLanguage=" +
        lc.sourceLanguage().name() + "&targetLanguage=" + lc.targetLanguage().name()
    result = utils.transport.http().target(url).request().get().asJson()
        //analysis is in a file accessible via a http link
    var analysisFile = utils.transport.download(result.CATAnalysisUrl)
    var analysisFileName = result.fileName
    var analysisFileProperties = {
        category = "CAT_ANALYSIS",
        languages = [lc]
    }
    analysisFile.name(analysisFileName)
    analysisFile.properties(analysisFileProperties)
    event.job().addFile(analysisFile)
    // due to the automatic finance configuration
    // receivables value will be automatically updated
    // basing on a newly added CATanalysis file
}

As CAT analysis may take some time, the above logic could be changed to an asynchronous model (a callback function registered in the CAT tool) – compare with the case ‘Asynchronous service call’. HTTP communication errors are not handled to keep the example short and clear.

Sending files to Google Translate

Case description

Use whenever Google Translate service is required: it supports callbacks.

Solution

function onWorkFileSharedWithJob(event) {
    for (lc in event.job().languages()) {
        var targetLanguage = lc.targetLanguage()
        translate(event.file(), targetLanguage) //as defined in a previous example
    }
}

function onLanguageCombinationAdded(event) {
    for (file in event.job().sharedWorkFiles()) {
        var targetLanguage = lc.targetLanguage()
        translate(file, targetLanguage) //as defined in a previous example
    }
}
function translate(file, targetLanguage) {
    /// TO BE SCRIPTED
}

1 Introduction
2 API References
- 2.1 Event handlers (callback functions)
  - 2.1.1 Event object properties
  - 2.1.2 List of event handlers
- 2.2 Object Methods
  - 2.2.1 Project and quote object methods
  - 2.2.2 Job object methods
  - 2.2.3 File object methods
  - 2.2.4 Language object methods
- 2.3 Utilities Available for a Script
  - 2.3.1 HTTP Transport
3 Use Cases

XTRF guide to scripted jobs

Introduction

What is a scripted job?

Human-inserted job vs. scripted job

How does the XTRF script work?

API References

Event handlers (callback functions)

Event object properties

List of event handlers

Object Methods

Project and quote object methods

Job object methods

File object methods

Language object methods

Utilities Available for a Script

HTTP Transport

Use Cases

Changing file content in the script

Converting a file using an external (Web) service

Converting a file – a local drive approach

Changing project data

Batch file processing using file metadata

Jobs’ user context – passing data between method invocations in one job

Handling temporary & permanent remote service errors

Asynchronous service call

Getting external CAT tool analysis

Sending files to Google Translate