Back: Pretrained Word2Vec embeddings

1. GloVe embeddings

We will now work with another pretrained word-embedding model: GloVe.

In this tutorial, we will load vectors from a GloVe text file and store them in a Python dictionary.

The basic workflow is:

GloVe text file
      ↓
word → vector dictionary
      ↓
similarity, analogy, and visualization

Preparing the GloVe file

Download the GloVe data described in the original lab instructions and make the extracted vector file available in your Colab environment.

The code below uses:

glove.6B.100d.txt

so make sure that this file is available before running the next section.

Loading GloVe vectors

Define a loading function:

def load_glove(
    file_path,
    expected_dim=100
):

    word2vec = {}

    with open(
        file_path,
        "r",
        encoding="utf-8"
    ) as file:

        for line in file:

            parts = (
                line
                .strip()
                .split()
            )

            word = parts[0]
            vector = parts[1:]

            if len(vector) != expected_dim:
                continue

            word2vec[word] = np.array(
                vector,
                dtype=np.float32
            )

    return word2vec

Load the embeddings:

glove = load_glove(
    "glove.6B.100d.txt",
    expected_dim=100
)

print(
    "Loaded words:",
    len(glove)
)

The resulting glove object is a Python dictionary.

A word is used as a key:

print(
    glove["coffee"]
)

To inspect only the first ten dimensions:

print(
    glove["coffee"][:10]
)

Conceptually:

"coffee"
    ↓
dictionary lookup
    ↓
GloVe vector

2. Finding nearest neighbors in GloVe

We can compare a query word with other words in the GloVe vocabulary using cosine similarity.

Define:

def most_similar(
    word,
    glove,
    topn=10
):

    if word not in glove:
        return []

    query_vector = glove[word]

    similarities = []

    for candidate, vector in glove.items():

        if candidate == word:
            continue

        similarity = cosine_similarity(
            query_vector,
            vector
        )

        similarities.append(
            (
                candidate,
                similarity
            )
        )

    similarities = sorted(
        similarities,
        key=lambda x: x[1],
        reverse=True
    )

    return similarities[:topn]

Try a query:

most_similar(
    "coffee",
    glove,
    topn=10
)

Try several words:

queries = [
    "coffee",
    "happy",
    "university"
]

for query in queries:

    print(
        "\nQuery:",
        query
    )

    print(
        most_similar(
            query,
            glove,
            topn=5
        )
    )

The returned words are nearest neighbors according to cosine similarity in the GloVe vector space.

3. Vector arithmetic and analogies

Word vectors can also be combined mathematically.

The original lab uses the following analogy structure:

x1 : x2 :: y1 : ?

A target vector is constructed as:

vector(y1)
+
[
vector(x2)
-
vector(x1)
]

For example:

man : king :: woman : ?

Define the analogy function:

def analogy(
    x1,
    x2,
    y1,
    glove,
    topn=5
):

    if (
        x1 not in glove
        or x2 not in glove
        or y1 not in glove
    ):
        return (
            "One of the input words "
            "is not in the vocabulary."
        )

    target_vector = (
        glove[y1]
        + (
            glove[x2]
            - glove[x1]
        )
    )

    scores = {}

    for word, word_vector in glove.items():

        if word in [
            x1,
            x2,
            y1
        ]:
            continue

        similarity = cosine_similarity(
            target_vector,
            word_vector
        )

        scores[word] = similarity

    return sorted(
        scores.items(),
        key=lambda x: x[1],
        reverse=True
    )[:topn]

Try:

analogy(
    "man",
    "king",
    "woman",
    glove
)

The original lab also experiments with examples such as:

print(
    "australia : beer :: france : ?",
    analogy(
        "australia",
        "beer",
        "france",
        glove
    )
)

print(
    "pencil : sketching :: camera : ?",
    analogy(
        "pencil",
        "sketching",
        "camera",
        glove
    )
)

print(
    "tall : tallest :: long : ?",
    analogy(
        "tall",
        "tallest",
        "long",
        glove
    )
)

The results allow us to examine whether relationships between vectors correspond to recognizable relationships between words.

Try it yourself

Create five analogy questions.

For each question, specify:

a : b :: c : expected answer

For example, store them as tuples:

analogies = [
    # ("a", "b", "c", "expected")
]

Then compare the model’s top prediction with your expected answer.

4. Visualizing a semantic field with GloVe

We can use PCA again to display selected GloVe vectors in two dimensions.

def display_pca_scatterplot(
    glove_dict,
    words
):

    words = [
        word
        for word in words
        if word in glove_dict
    ]

    word_vectors = np.array([
        glove_dict[word]
        for word in words
    ])

    twodim = PCA(
        n_components=2
    ).fit_transform(
        word_vectors
    )

    plt.figure(
        figsize=(10, 10)
    )

    plt.scatter(
        twodim[:, 0],
        twodim[:, 1]
    )

    for word, (x, y) in zip(
        words,
        twodim
    ):

        plt.text(
            x + 0.05,
            y + 0.05,
            word
        )

    plt.title(
        "PCA projection of GloVe word vectors"
    )

    plt.grid()
    plt.show()

Try a small semantic field:

domain_words = [
    "coffee",
    "tea",
    "beer",
    "wine",
    "water",
    "pizza",
    "sushi",
    "hamburger",
    "school",
    "college",
    "university"
]

display_pca_scatterplot(
    glove,
    domain_words
)

Because we select the words before applying PCA, the visualization describes the relationships among the words included in this particular set.

Try it yourself

Choose approximately ten words from a domain that interests you.

For example:

  • food;
  • sports;
  • education;
  • computer science;
  • language;
  • another domain relevant to your research.

Visualize the vectors and consider:

  1. Do any words form visible clusters?
  2. Which words appear unexpectedly close together?
  3. How should the two-dimensional PCA representation be interpreted in relation to the original high-dimensional vectors?

Back to Module 6