AI/ML · Language

Embeddings

Each word becomes a list of numbers: a point in space, where directions carry meaning.

Words as points in space

Similar words sit close together. Drag to turn the space.

Positions are illustrative: real GloVe vectors have 50 to 300 dimensions, not 3.

GloVe embeddings

https://nlp.stanford.edu/pubs/glove.pdf

Reading embeddings


        import numpy as np
        import gensim.downloader as api
        # load pre-trained 'glove-wiki-gigaword-50'
        model = api.load('glove-wiki-gigaword-50')
        # size of the model (no. of words / tokens)
        len(model)
        # ans: 400000
        # the embedding for any word
        model['india']
        # ans: an array of 50 numbers
        model['king']
      

Reading embeddings: arithmetic on meaning


        # meaningful relationships
        vector1 = model['man'] + model['queen'] - model['king']
        model.most_similar(vector1, topn=1)
        # ans: a word close to 'woman'
        vector2 = model['paris'] - model['france'] + model['india']
        model.most_similar(vector2, topn=1)
        # ans: India's capital, by the same logic
      

Passing a raw vector does not exclude the input words, so they can also rank near the top.