Vector semantics 1. Count-based models and SVD
1. Count-based word representations
A simple distributional approach begins by counting how often words occur near other words.
Suppose we have the following small corpus:
sentences = [
"I like apples",
"you like bananas",
"we enjoy fruit",
"they eat fruit"
]
We will use the following fixed vocabulary:
vocab = [
"I", "you", "we", "they",
"like", "eat", "enjoy",
"apples", "bananas", "fruit"
]
The goal is to construct a matrix that records which words occur near one another.
The overall process is:
corpus
↓
tokens
↓
co-occurrence matrix
↓
SVD
↓
lower-dimensional word vectors
Importing libraries
We will use NumPy for numerical operations, pandas for displaying matrices, and Matplotlib for visualization.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
Creating a word-to-index mapping
A matrix uses numerical row and column positions rather than word strings. We therefore create a dictionary that maps each vocabulary item to an index.
word_to_idx = {
word: i
for i, word in enumerate(vocab)
}
print(word_to_idx)
For example, if I has index 0, then row 0 and column 0 of the matrix correspond to I.
Understanding a context window
A context window determines how far we look to the left and right of a target word when counting neighboring words.
Consider:
I like apples
With window_size = 1, only immediately adjacent words are counted as context.
For the target word like, the context words are:
I ← like → apples
For the target word I, only like is within the window.
With a larger context window, more distant words can also be counted as context.
The choice of context window therefore changes the co-occurrence patterns that the model records.
Building a co-occurrence matrix
We can now define a function that counts how often vocabulary items occur within a specified context window.
def build_cooccurrence_matrix(sentences, vocab, window_size=1):
V = len(vocab)
M = np.zeros((V, V), dtype=float)
word_to_idx = {
word: i
for i, word in enumerate(vocab)
}
for sentence in sentences:
tokens = sentence.split()
for target_position, target_word in enumerate(tokens):
if target_word not in word_to_idx:
continue
target_index = word_to_idx[target_word]
start = max(
0,
target_position - window_size
)
end = min(
len(tokens),
target_position + window_size + 1
)
for context_position in range(start, end):
if context_position == target_position:
continue
context_word = tokens[context_position]
if context_word not in word_to_idx:
continue
context_index = word_to_idx[context_word]
M[target_index, context_index] += 1
return M
This function produces a vocabulary-by-vocabulary matrix.
Each row represents a target word, and each column represents a context word.
The value:
M[i, j]
indicates how many times the word represented by column j occurred within the selected context window of the word represented by row i.
Inspecting the co-occurrence matrix
Build a matrix using a context window of 1:
M = build_cooccurrence_matrix(
sentences,
vocab,
window_size=1
)
Convert it to a pandas DataFrame so that we can see the word labels:
df_M = pd.DataFrame(
M,
index=vocab,
columns=vocab
)
display(df_M)
A row can be interpreted as a simple distributional representation of a word.
For example, if two words frequently occur with similar neighboring words, their rows may show similar patterns.
However, this representation has one dimension for every vocabulary item. With a large vocabulary, the matrix can therefore become very high-dimensional.
2. Reducing the matrix with SVD
We can use singular value decomposition, or SVD, to reduce the dimensionality of the co-occurrence matrix.
In this tutorial, the workflow is:
high-dimensional co-occurrence matrix
↓
SVD
↓
lower-dimensional word representations
Apply SVD using NumPy:
U, S, VT = np.linalg.svd(M)
The original matrix is decomposed into three components:
M = U × S × VT
We do not need to retain all dimensions.
For example, we can keep only two dimensions:
k = 2
U_k = U[:, :k] * S[:k]
The resulting rows can be used as two-dimensional word vectors.
df_vectors = pd.DataFrame(
U_k,
index=vocab,
columns=["dim1", "dim2"]
)
display(df_vectors.round(3))
Each word is now represented by two numerical values.
For example:
word → [dim1, dim2]
These dimensions are not linguistic categories that we named in advance. They are dimensions produced by the mathematical decomposition of the co-occurrence patterns.
Creating a reusable SVD function
We can put the same process into a function:
def compute_word_vectors(M, vocab, k=2):
U, S, VT = np.linalg.svd(M)
U_k = U[:, :k] * S[:k]
return pd.DataFrame(
U_k,
index=vocab,
columns=[
f"dim{i + 1}"
for i in range(k)
]
)
Use it with our matrix:
df_vectors = compute_word_vectors(
M,
vocab,
k=2
)
display(df_vectors.round(3))
3. Visualizing word vectors
Because we retained two dimensions, we can display the word vectors in a two-dimensional scatter plot.
def plot_vectors(df_vectors, title):
np.random.seed(42)
jitter_strength = 0.05
coordinates = (
df_vectors.values
+ np.random.normal(
0,
jitter_strength,
df_vectors.shape
)
)
plt.figure(figsize=(7, 7))
plt.axhline(
0,
linestyle="--",
linewidth=0.5
)
plt.axvline(
0,
linestyle="--",
linewidth=0.5
)
for (x, y), word in zip(
coordinates,
df_vectors.index
):
plt.scatter(x, y)
plt.text(
x + 0.05,
y + 0.05,
word,
fontsize=12
)
plt.title(title)
plt.xlabel("dim1")
plt.ylabel("dim2")
plt.grid(
True,
linestyle="--",
alpha=0.5
)
plt.show()
Plot the vectors:
plot_vectors(
df_vectors,
title="Word vectors from SVD"
)
Words that appear near one another in the plot have similar positions in this reduced two-dimensional representation.
However, remember that this visualization retains only two dimensions. A two-dimensional plot is therefore a simplified view of the information contained in the original matrix.
4. Comparing context window sizes
The definition of context affects the resulting word representations.
We can compare a context window of 1 with a context window of 2.
results = {}
for window_size in [1, 2]:
print(
"\n=============================="
)
print(
f"Window size = {window_size}"
)
print(
"=============================="
)
M = build_cooccurrence_matrix(
sentences,
vocab,
window_size=window_size
)
df_M = pd.DataFrame(
M,
index=vocab,
columns=vocab
)
print("\nCo-occurrence matrix:")
display(df_M)
df_vectors = compute_word_vectors(
M,
vocab,
k=2
)
print("\nWord vectors:")
display(df_vectors.round(3))
plot_vectors(
df_vectors,
title=(
"Word vectors "
f"(window size = {window_size})"
)
)
results[window_size] = df_vectors
Compare the matrices and plots.
A larger window allows a target word to co-occur with more distant words. As a result, changing the window size changes the distributional information used to construct the vectors.
5. Measuring similarity with cosine similarity
A common way to compare two vectors is cosine similarity.
Cosine similarity compares the directions of two vectors.
We can define:
def cosine_similarity(vec1, vec2):
return np.dot(
vec1,
vec2
) / (
np.linalg.norm(vec1)
* np.linalg.norm(vec2)
)
Compare selected word pairs:
pairs = [
("apples", "bananas"),
("eat", "enjoy"),
("I", "you")
]
Calculate their similarities under each context-window setting:
for window_size, df_vectors in results.items():
print(
f"\nWindow size = {window_size}"
)
for word1, word2 in pairs:
similarity = cosine_similarity(
df_vectors.loc[word1],
df_vectors.loc[word2]
)
print(
f"cosine({word1}, {word2}) "
f"= {similarity:.3f}"
)
This lets us examine whether the same pair of words becomes more or less similar when we change the definition of context.