Opus bitext and monolingual data install#

The official repository contains a few example scripts that you can use directly depending on your use cases:

cross-encoder: two sentences are presented simultaneously to the transformer network and a score (0…1) is derived indicating the similarity or a label.

clustering: grouping sentences together based on their similarity.

text summarization: given a long document, find the k sentences that give a good and short summary of the content.

This allows for an image search given a user query.

image search: map images and text to the same vector space.

semantic search: given a sentence, find in a large collection semantically similar sentences.

It can be used for various applications such as: This Python package is not limited to just mining parallel sentences.

Opus bitext and monolingual data install#

Then, run the following command to install sentence-transformer, a BERT model for multilingual sentence, paragraph, and image embeddings. Install it as follows based on your use cases: # CPU pip install faiss-cpu # GPU pip install faiss-gpu There is a repository which provide python wheels for both cpu and gpu installation. It is highly recommended to create a new virtual environment before you continue with the installation.īefore that, you need to have FAISS installed in your system. Hello World!Īnd you got another Italian corpus as follows: Salve, mondo! L'esame è dietro l'angolo Buongiorno, Peter.īy utilizing LaBSE model, you will be able to perform bitext mining in order to obtain the following correct parallel sentences: Hello World! Salve, mondo! Good morning, Peter. Let’s say that an English corpus contains the following sentences This is an example sentences. Hence, you can use this concept to identify the translation pairs that share high similarity value on two different corpora. Shares the most similarity with the original sentences and thus could be its translated text. It indicates that Questa è una persona felice Questo è un cane felice 0.624 Questa è una persona molto felice 0.757 Oggi è una giornata di sole 0.422 Questa è una persona felice 0.830

Let’s reuse the previous sentence That is a happy personĪnd compare it against the following translated text. In fact, you can identify sentence similarity on translated text as well. You will obtain the following sentence similarity when using the embedding from LaBSE model: That is a happy dog 0.741 That is a very happy person 0.897 Today is a sunny day 0.450 So it can be used for mining for translations of a sentence in a larger corpus.įor example, given the following text: That is a happy person … is trained and optimized to produce similar representations exclusively for bilingual sentence pairs that are translations of each other. In this tutorial, you will learn to use a language-agnostic BERT sentence embedding model called LaBSE.įrom the Tensorflow Hub, it states that LaBSE model The topic for today is about bitext mining on two monolingual files (unordered sentences) to find parallel sentences.