1. Pretrained Word2Vec embeddings

So far, we constructed word representations directly from a tiny corpus.

We can also use pretrained embeddings: vectors that have already been learned from a much larger collection of text.

In this section, we will use a pretrained Word2Vec model.

The workflow is:

pretrained Word2Vec model
          ↓
         word
          ↓
high-dimensional embedding vector
          ↓
similarity or visualization

Importing libraries

We will use TensorFlow Hub to load the pretrained model.

import tensorflow_hub as hub

We will also use PCA from scikit-learn later for visualization:

from sklearn.decomposition import PCA

Loading the pretrained model

Load the model:

print("Loading Word2Vec model...")

embed = hub.load(
    "https://tfhub.dev/google/"
    "Wiki-words-250-with-normalization/2"
)

print("Model loaded successfully!")

The loaded model can return a numerical embedding for a word.

Define a helper function:

def get_word_embedding(word):

    return embed([word])[0].numpy()

Retrieving a word vector

Retrieve the vector for king:

king_vector = get_word_embedding("king")

print(king_vector)

Check the number of dimensions:

print(king_vector.shape)

To inspect only the first ten values:

print(king_vector[:10])

The word is therefore represented computationally as a sequence of numerical values:

king
 ↓
[dimension 1, dimension 2, ..., dimension n]

Retrieving several embeddings

Create a small list of words:

words = [
    "king",
    "queen",
    "man",
    "woman",
    "child"
]

Retrieve their embeddings:

embeddings = [
    get_word_embedding(word)
    for word in words
]

Inspect the first ten dimensions:

for word, vector in zip(
    words,
    embeddings
):
    print(
        word,
        vector[:10]
    )

The individual dimensions are usually not interpreted one at a time. Instead, we compare the vectors as complete representations.

2. Comparing pretrained word embeddings

We can reuse cosine similarity to compare the pretrained vectors.

king_vector = get_word_embedding("king")
queen_vector = get_word_embedding("queen")
child_vector = get_word_embedding("child")

Compare king and queen:

similarity = cosine_similarity(
    king_vector,
    queen_vector
)

print(
    "Similarity between "
    "'king' and 'queen':",
    similarity
)

Compare king and child:

similarity = cosine_similarity(
    king_vector,
    child_vector
)

print(
    "Similarity between "
    "'king' and 'child':",
    similarity
)

The numerical values allow us to compare how similar the word representations are in the pretrained embedding space.

Try it yourself

Add at least five words of your choice:

words = [
    "king",
    "queen",
    "man",
    "woman",
    "child",

    # Add your words here
]

Then:

retrieve their embeddings;
print the first ten dimensions of each vector;
select at least two new word pairs;
compare them using cosine similarity.

3. Visualizing pretrained embeddings with PCA

Pretrained word embeddings contain many dimensions.

To display them on a two-dimensional plot, we can use principal component analysis, or PCA, to project the vectors into two dimensions.

def plot_embeddings(words, embeddings):

    pca = PCA(
        n_components=2
    )

    reduced = pca.fit_transform(
        embeddings
    )

    plt.figure(
        figsize=(8, 8)
    )

    for i, word in enumerate(words):

        plt.scatter(
            reduced[i, 0],
            reduced[i, 1]
        )

        plt.annotate(
            word,
            (
                reduced[i, 0],
                reduced[i, 1]
            )
        )

    plt.title(
        "Word embeddings visualization (PCA)"
    )

    plt.show()

Plot the selected words:

plot_embeddings(
    words,
    embeddings
)

PCA is being used here for visualization. The two-dimensional positions should therefore be interpreted as a reduced view of the original high-dimensional embedding space.

Try it yourself

Add your additional words to the plot.

Look for:

words that appear close together;
groups of semantically related words;
unexpected neighbors.

4. Finding similar words

We can also search a vocabulary for the word whose vector is most similar to a query word.

First, define a small vocabulary:

sample_vocab = [
    "king",
    "queen",
    "man",
    "woman",
    "child",
    "prince",
    "princess"
]

Define a function:

def find_closest_word(
    query_word,
    vocab
):

    query_vector = get_word_embedding(
        query_word
    )

    similarities = []

    for word in vocab:

        word_vector = get_word_embedding(
            word
        )

        similarity = cosine_similarity(
            query_vector,
            word_vector
        )

        similarities.append(
            (
                word,
                similarity
            )
        )

    similarities = [
        item
        for item in similarities
        if item[0] != query_word
    ]

    similarities = sorted(
        similarities,
        key=lambda x: x[1],
        reverse=True
    )

    return similarities[0]

Find the closest word to king within this vocabulary:

closest_word, closest_similarity = (
    find_closest_word(
        "king",
        sample_vocab
    )
)

print(
    "Closest word:",
    closest_word
)

print(
    "Cosine similarity:",
    closest_similarity
)

Notice that this function searches only the words included in sample_vocab.

It is therefore finding:

the most similar word
among the candidate words we provided

rather than searching every possible word.

Try it yourself

Extend sample_vocab with at least five additional words.

Then select one of your new words as a query and find its closest word in the candidate vocabulary.

Next: GloVe embeddings