Building wordsim

Best viewed in dark mode.

It’s been a week or two since I launched wordsim, and I’ve had a lot of questions from people on how and why it works behind the scenes. In this post, I want to expand on these topics and my own experience in building this simple but surprisingly addictive game.

In the first part, we’ll go over the fundamental concepts of representing words and determining word similarities. I want to go into details of representations with much more detail in a future post, but here I’ll try to keep the discussion here to a minimum.

Then, I’ll give a technical breakdown of how everything is implemented and how the pipeline works. We’ll have a short section on adapting the game to the Turkish language, and about some unexpected challenges in the process.

Let’s start!

About wordsim, and how it works

There comes a time for many people where they make a simple game about guessing about words or numbers. wordsim is such a game focusing on guessing a hidden word using semantic similarities.

It started as a spin-off of my curiosity in exploring how deep language models internally represent various concepts and the relations between these concepts. I had some prior experience in working with embeddings of vision-language models like CLIP (Radford et al., 2021), or leveraging them for slightly more complex tasks such as jazz standard recognition. I then wanted to explore the geometry of word embeddings.

The core mechanism of word similarity games

This particular genre dates itself back to the childhood classic “hot or cold” guessing game, where one player hides an object within a room and guides another player to find that object using clues such as “cold” and “warm”. This idea naturally extends to a word guessing game: one player thinks of a word, and upon receiving a guess, gives clues about the semantic “closeness” of their guess until the word is eventually found. That is precisely how wordsim is played.

Of course, implementing this idea is not nearly close to being a novel thought: looking for such a game leads one to games like Contexto, the daily semantic-guessing game Semantle, and many more implementations. I am not aware of any Turkish-focused word similarity game of this nature other than wordsim at the moment, so perhaps there is a small but unique contribution of the project there.

To successfully implement wordsim, conceptually we need two things:

1. A way to represent words.
2. A way to compute similar words to a target word.

Let’s start with item #2 first.

Thinking about word similarities

How does one determine which words are similar? Let’s start from some simple ideas:

• “forest” reminds us of “tree”, because forests are made of trees. Relations such as “has a”, “is a”, and words with the same or very similar meanings exhibit naturally strong similarity.

• There are multiple angles to approach a word. You can reach “earth” by starting from the solar system and the planets, as well as starting from seeds, farming and soil. Although they are seemingly unrelated paths, these two concepts should both lead to “earth” via increasing similarity.

• Similarity should be somewhat transitive: if we only report high similarity for immediate neighbors of a word, it’s very hard to reach that word from a random starting point. If the word is “egg”, then we expect “chicken” to be a very similar word, and “feather” to be similar to “chicken”. Thus, “egg” and “feather” should be somewhat similar as “chicken” is highly similar to both of them.

…and so on. We can’t establish all of these requirements by hand, as they would be too specific and brittle. We’d like to have a system that implicitly infers these kinds of relations.

Luckily for us, this is the natural area of language modeling in machine learning. With enough data and an appropriate model, we might be able to satisfy all of these requirements directly by learning them from the data.

How do we represent words?

At the most basic level we need to find a way to represent a word so that we can give it to a model. In natural language processing (NLP), the traditional way to do this was to build “one-hot encodings” for each word to give to a model:

1. Collect all words that you know from the text. Let’s say we collected 8 words, this is now our vocabulary.
2. Give them a unique ordering, so that each word has its unique index.
3. Represent a word by a series of 0s and 1s, such that only its unique index is 1 while all other entries are 0. This is called a “one-hot encoding”.

We can afford skipping grammatical details like word normalization, filtering, and out-of-dictionary words to simplify the general idea. Let’s go over a small example:

docs = [
    ['this', 'is', 'a', 'dog'], # an example sentence from the dataset
    ['cat', 'and', 'dog'],
    ['a', 'mouse'],
    ['cat', 'and', 'mouse', 'game']
]
words = set()
for doc in docs:
    for word in doc:
        words.add(word)
vocab = sorted(list(words))
print(vocab) # ['a', 'and', 'cat', 'dog', 'game', 'is', 'mouse', 'this']

The code above builds a vocabulary of known words from all of our text documents. We can assign a unique representations made of 0s and 1s for each word, which would look like:

print(onehot(vocab, 'cat'))   # [0, 0, 1, 0, 0, 0, 0, 0]
print(onehot(vocab, 'mouse')) # [0, 0, 0, 0, 0, 0, 1, 0])

With this sequence-of-numbers representation, we can achieve many nice things. We could represent a sentence by summing these one-hot vectors together (called a bag of words), or feed them one by one as features into a neural network model to be used in tasks like sentiment analysis and next word prediction.

However, there is one problem with this way to representing words. Do you see it?

Why one-hot encoding isn’t ideal

When representing words as one-hot vectors, all words inside a dictionary $V$ of size $\lvert V \rvert$ occupy a discrete $\lvert V \rvert$-dimensional space filled where each dimension is either $0$ or $1$. Each word representation $w$ has the same norm $\lVert w \rVert = 1$ and the same Euclidean pairwise distance of

independent of the specific pair of words selected. Geometrically, they are all equally far apart points on the surface of a high-dimensional sphere. The embedding space of words here is discrete and devoid of any meaningful distance between words.

This isn’t ideal for our task of simple and cheap similarity computation, and with this approach we don’t have a semantically meaningful word embedding space. Can we do better?

Continuous word embeddings and word2vec

Just when we need it the most, “A Neural Probabilistic Model” (Bengio et al., 2003) comes to our rescue. It focuses on the concept of training neural networks that learn distributed word embeddings, where the representations of words are learned from the data as dense, continuous vectors in an embedding space. Although this idea isn’t immediately established as the defacto method of representing words until later, it forms the foundation of modern language modeling.

What advantages does it provide?

With continuous word embeddings, we now have the ability to represent words as unique directional vectors within a smaller but meaningful vector space.

• We don’t need a new dimension to uniquely identify each word, reducing the problem of the curse of dimensionality.
• A model using continous word embeddings is able to learn a representation for a word that puts it semantically closer to similar words.
• Learning embeddings is made possible by backpropagation from a loss function all they way back to the embeddings.
• It lets us explicitly and cheaply compute a notion of similarity between words $w_{1}$ and $w_{2}$ using simple cosine similarity:

While this work is 20+ years old at this point, it wasn’t until the simple and efficient methods introduced by the 2013 word2vec paper, “Efficient Estimation of Word Representations in Vector Space” (Mikolov et al., 2013), that continuous word embeddings really took off. In particular, the geometry of word2vec vectors shows us an extremely interesting property of word embeddings: composition of concepts using word vector addition is possible in representation space. The king - man + woman = queen is a classic example of “word arithmetic”:

This is a very promising development in our search for cheap and efficient semantic similarity estimation.

Modern word embeddings

From 2017, there have been massive changes across NLP.

• The Transformer architecture (Vaswani et al., 2017) was initially about sequence-to-sequence translation, but it has entirely transformed NLP (and vision, and everywhere else in deep learning) due to its efficiency and incredible scalability to enormous amounts of data.
• Massively scaled language models like the GPT family (Radford et al., 2018) of decoder-only Transformer architectures and the encoder-only BERT family of models (Devlin et al., 2019) have been hugely successful in multiple NLP tasks.

Modern large langugage models (LLMs) build contextual representations of words and sentences from token embeddings, and so their word representations are potentially better than the static word2vec mappings. However, LLMs are in general used for sentence-level tasks, while here we are looking at individual words.

For wordsim, I use a transformer-based embedding model particularly trained for producing general-purpose embeddings, as on paper it provides the best of both worlds.

With everything set in place, we are now ready to talk about how everything works under the hood.

How wordsim is implemented

wordsim is composed of an offline generation module in Python in the backend and a responsive frontend that serves the puzzles and the UI with TypeScript. The code for the project is on GitHub.

The generation pipeline

The generator’s task is to prepare an initial list of possible vocabulary words, process and filter them according to the grammar rules, compute representations of words, and precompute similarities for each puzzle to pass to the frontend. The vocabulary-building process is something like this:

graph TB
    A[1. Determine list of words for the game dictionary] --> B[2. Preprocess words into canonical forms] --> C[3. Extract representations of words] --> D[4. Build static word embedding table]

For English, these steps are all fairly straightforward.

• The wordfreq repository is a pretty good resource that we can use to determine a list of frequent words in the English dictionary based on multiple corpora. This ensures a nice and rich dictionary of words. For English, a vocabulary of approx. 30000 most frequent words is satisfactory for an enjoyable game experience.
• We don’t even have to preprocess words unless the candidate word list is contaminated with plural endings or past tenses, which is generally not the case for English puzzles.
• LLMs are very good at learning semantic information about tokens that make up words in the English language. This is due to the abundance of English training data and downstream tasks of LLMs requiring a good knowledge of the language.
• As a consequence of the above factor, it turns out language models that focus on producing embeddings (e.g., Google’s EmbeddingGemma, Qwen’s Qwen3-Embedding series and many others) produce good enough representations of words to encode semantically-related words together. Empirically, we confirm this on based on user feedback on generated English puzzles.

Since a word’s representation doesn’t change, we can think of the first part as building a static word -> embedding mapping.

We can also do a high-level visualization of embeddings of all the words in the vocabulary with a 2D UMAP projection:

UMAP lets us broadly visualize the organization of the embedding space. Without interpreting the distances too literally, we can see same-category puzzles are generally in the vicinity of each other. This is most true for categories such as food and animal, but more abstract categories such as adjective and action seem to be more dispersed rather than clustered together. We can also see that the general object category is perhaps too vague, perhaps we could narrow it down a little in the future!

Puzzle generation

The puzzle generation process is then simply:

graph TB
    A[1. Select a target word from the vocabulary] --> B[2. Retrieve representation of word] --> C[3. Compute similarity against other words in the dictionary] --> D[4. Compute rankings and store them]

where we have already computed and stored each word representation in the dictionary beforehand. The current version uses EmbeddingGemma-300m (Vera et al., 2025) for word representations in English, as it is designed as a multilingual embedding model that is both lightweight and powerful (and also supports Turkish), that has also been trained with semantic similarity as a training task.

Speaking of Turkish support, we’ll come back to this topic in a little bit.

Frontend logic

The frontend is responsible for serving the generated puzzles, as well as general responsiveness of the game through subtasks such as category-based filtering, puzzle state management, and session management between visits to the site.

It has a whole bunch of important features for improving user experience and quality of life including

• Hint support, validation of user word inputs
• Easier navigation through puzzles using the grid system
• Internationalization of the platform (and the puzzles) in the currently languages
• Tutorials and explanations on game logic

…and so on.

The frontend arguably took more time and effort to polish it into its current state compared to the generation pipeline and definitely deserves more attention, but the heart of the game is about word semantic similarity.

What about Turkish, though?

Turkish is very different from English for a multitude of reasons. There are no gendered nouns, subject-object-verb ordering is different, the rules are have a bunch of gotchas about Turkish support for application support for developers, however we’ll focus on one detail here: it is highly agglutinative. What this means is that words are encoded all sorts of meanings primarily through various suffixes that can be chained together to further enrich their meaning. As an example, some words that look relatively normal to my eyes include

farklılaştırmak

fark-lı-laş-tır-mak

meaning “to turn something into something different”

kalabilseydiler

kal-a-bil-se-(y)-di-ler

meaning “if they were able to stay”

…and so on. Of course, one can talk about the limits of suffix chaining with the examples from this page on longest Turkish words such as

Muvaffakiyetsizleştiricileştiriveremeyebileceklerimizdenmişsinizcesine

but that’s just to demonstrate the capabilities of the language.

The point is that words have many forms depending on their suffixes, and these words are very frequent, which messes up our usual flow in a couple of ways.

Just use EmbeddingGemma?

Let’s recall that we specifically chose EmbeddingGemma because it is multilingual (with Turkish support as well), so can’t we use it here? It’s not that easy unfortunately, due to some problems that are progressively harder to fix.

First of all, there’s the issue of overcrowding the vocabulary due to grammatically similar words. For both nouns and verbs, there is a tendency to include words that are a result of simple suffix chaining.

Take two common words such as “ev” (house) and “koşmak” (to run). Here are some common forms:

Although these are all semantically related to the target word, we don’t want to pollute the vocabulary or the rankings with slight modifications of the target word.

Most of these should be trimmed or normalized (simple tenses, some plural forms), but some of these derived words are standalone concepts and are essential to the language. We have to get our hands dirty here.

Preprocessing. We have to implement some form of normalization in the pipeline so that our vocabulary isn’t crowded by common forms. wordsim uses zeyrek here, which is a Python port of the zemberek-nlp Turkish NLP project. At a high level, we process each candidate word in the list of common Turkish words by analyzing each word. We remove those that contain suffixes about plurality/tense/possessiveness, remove proper nouns, convert verbs into infinitives, and some other steps.

Our next problem is quite interesting.

Even without knowing a bit of Turkish, it’s apparent what the system is doing: it’s listing words that look/sound similar to parts of the original word. What gives?

Tokenization. Tokenization is the act of transforming an input word, sentence or document into a series of chunks called “tokens” that the model learns to understand. Tokens come in many shapes and sizes, they can be characters, subwords, entire words (or patches of images — see vision transformers, honestly anything can be a token), and through the process of training the model learns to extract meaning from tokens and subsequent sequences of tokens. See OpenAI’s tokenizer for an interactive example.

One likely guess is that the model does not learn the tokens of sin and cap correctly in Turkish. In particular, EmbeddingGemma’s multilingual support means tokens for sin and cap have multilingual context (they are also English words themselves), composing them together might not suggest to the model that they together form the Turkish word meaning “squirrel”. It could also just be that the model doesn’t have good representation geometry of Turkish words in general, due to its rarity compared to English.

This general issue is apparently something that other researchers have noticed such as in (Bayram et al., 2026). Trying out their Turkish-aware embedding model, EmbeddingMagibu-200m, definitely did seem to improve embeddings in this direction, but ultimately to a limited extent for this project in my experience.

One final problem in the Turkish version that I experienced during development that I’d like to show is the following:

The “-siz/-sız” suffix encodes the meaning of “without” in Turkish; the words listed are not really related except the “without” meaning added to each word.

I’m not particularly confident about this one. It might be the case of tokenization again, but it feels as though this is more about retrieving similar words based on the wrong kind of semantic similarity.

Solution: The simplest solution for Turkish is to go back to our trusted baseline (word2vec embeddings), and search for a better solution in the process. Recently researchers have shown that on the specific task of semantic similarity, a word2vec model with the skip-gram method can perform better than BERT-based models (Sarıtaş et al., 2024) for the Turkish language. Following this result, the Turkish version currently uses word2vec embeddings with skip-gram training.

What this means for Turkish puzzles is that with the current approach we are much better in removing suffix-induced semantic neighbors, improving neighbor diversity at the cost of a slightly suboptimal list of close words in the neighboring word list, which makes the puzzles a little bit harder.

Similar to the English version, we can also look at UMAP projections of Turkish word embeddings as well. You might recall this figure from the teaser:

We see that overall, the category trends from the English version are also present in the Turkish version. It is quite interesting to see that the verbs are so far removed from other categories. This also makes sense as to why the closest words to a target puzzle verb are generally verbs. One more thing is that these are currently from word2vec embeddings, while English is from a transformer-based Embedding model; in fact I expected less similarity between them.

I’m still working towards improving the Turkish model for a more enjoyable experience for all.

What is next for wordsim?

I want to analyze some parameters that control the entire pipeline. For example, we haven’t talked about the embedding dimension, which is the dimensionality of the space in which word embeddings reside in. For English we have $d_{\mathrm{embed}}=768$, meaning 768-dimensional representations of words. However, what is the effect of increasing or decreasing this dimensionality?

• Does decreasing dimension hurt the representational capability of the embedding model for semantic tasks?
• When increasing the dimensionality, how is the system affected by the curious behavior of Euclidean distance in high dimensional space?

There are many more questions about this parameter and other ones in the process (e.g., raw embedding vs prompt-guided embeddings)

There’s also the obvious next task of improving the Turkish version of the game with better semantic understanding. I also want to add new languages so that it is more accessible for more people.

Want to join the effort? The code is available on GitHub. Feel free to contribute, propose ideas, or report bugs and issues that you might find while playing.