M 4.3: Natural Language Processing NLP

NLP consists of a large number of (mainly mathematical) techniques that have recently been extended to include the possibilities of deep learning with huge data sets to recognise text areas (word types, mood, meaning, alternative terms) or to fill in missing information (missing words) based on context.

ChatGPT and others are based on NLP. (Due to time constraints, we will only look at technologies related to chatbots. This means that e.g. ML-generated images are not covered here.)

Simplified Example

The use of artificial neural networks is shown here as a highly simplified example to demonstrate an application of the scientific discipline of Natural Language Processing NLP, namely the prediction of words.

The first step is to convert texts into numbers so that they can be processed by the computer, i.e. the artificial neural network.

One Hot Encoding of texts

In the small example, the hole vocabulary consists of only 5 words:
['the', 'cat', 'dog', 'meows', 'barks']. This is not yet meaningful.

So the AI needs to learn that a cat meows and a dog barks, then it will be able to predict this, or in other words, it will complete both sentences:

the cat [...] => meows
the dog [...] => barks

The so-called One Hot Encoding refers to the vocabulary and is used here to convert from text format to numbers and vice versa:

the cat meows <=> [1,1,0,1,0]
the dog barks <=> [1,0,1,0,1]

A 1 indicates the presence of a word at a given position in the vocabulary.

Training of a small artificial neural network

This data is used to train a small artificial neural network with a hidden layer:

Training of a small artificial neural network

In this case the training is supervised. Predictions are made by inputs and compared with the corresponding output. The weights of the network have been corrected backwards. This has to be done several times until the error is small enough.
Predictions can then be made with small deviations from the real values.

The trained ANN works well, there are only small errors.

Interpretation of results

My example program on github has produced the following output. (Eick, 2024)

Prediction for 'the cat meows:'
[[ 0.9999522  0.9999451 -0.0000115  0.9999725  0.00006464]]

The convergence is quite good. As you can see, the deviation from the real values is very small, as the test shows.

Word Embeddings

Let us look at the trained values of weights_0_1, which connect the input layer to the hidden layer. These weights are more meaningful than the One Hot Encodings. They form the so-called word embeddings, i.e. the context in which the target word is embedded.

weights_0_1 for ['the',      'cat',       'dog'] represent 'word embeddings'
            [[ 0.00582719 -0.50082475  0.51814336]  <=> meows
             [-0.00413838 -0.7001615   0.69884676]] <=> barks

First of all, it is clear that the word 'the' has no influence on the prediction 'meows' or 'barks', because it is present in both sentences. For this reason, the values of these weights are close to zero.

The weights for 'cat' in relation to 'meows' and 'barks' are in the opposite direction. This means that 'cat' is pulled close to 'meows' and pushed far away from 'barks' with the same force. The same applies to the 'dog' weights for 'barks' and 'meows'.
(These outputs of the program vary due to the initialisation of the weights with random values. Therefore, the 'cat' weights are named a and -a and the 'dog' weights are named b and -b in the image above.)

Vector analysis

If we think of the values as vectors, we can plot them as follows:

Cat vector and dog vector

It is easy to see that the 'cat' vector and the 'dog' vector are the same length but have opposite directions. This corresponds to their influence on the targets 'meows' and 'barks'.
(The 'the' vector is too short to be plotted. )

Remember that real NLP applications are trained on millions of words. In much larger neural networks, the word embeddings have many more dimensions. The vectors are therefore in many hundreds of dimensions. But the principle is the same. The possibilities are many. Here are two of them:

Similar words

(Note that these real vectors are reduced to 2 dimensions in the following graphics for better visualisation.)

Two vectors represent two words. If they have similar angles and lengths, they have a very similar meaning:

Similar words as vectors

It is possible to mathematically find the closest matching words in the learned vocabulary.

Some other examples to illustrate the importance of this are:

does, did
jump, bounce
web, internet
...

Word analogies

An alternative word can be found by mathematically comparing the vectors.
You can also use vector addition and subtraction to find analogue words.

A famous example is (Mikolov et al., 2013):

Vector addition and subtraction to find word analogies.

Imagine you are looking for a word that represents a female king. Man, woman and king are known.

The formula is: Queen = King + Woman - Man

\( \vec{ Man} + \vec{ gender} = \vec{ Woman} \\ \Rightarrow \vec{ gender} = \vec{ Woman} - \vec{ Man}\\ \vec{ Queen} = \vec{ King} + \vec{ gender} = \vec{ King} + \vec{ Woman} - \vec{ Man} \)

Some other examples to illustrate the importance of this are:

in, out => up, down
go, green => stop, red
wheel, bike => tire, car
small, petit (french) => big, grand (french)
...

Reference list:

Eick, W. (May 2024). Moodle KI: SimpleWordEmbedding.ipynb [Software]. Github. https://github.com/iamtrask/Grokking-Deep-Learning/blob/master/Chapter3%20-%20%20Forward%20Propagation%20-%20Intro%20to%20Neural%20Prediction.ipynb (Accessed 5 May 2024)

Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. https://www.semanticscholar.org/paper/Linguistic-Regularities-in-Continuous-Space-Word-Mikolov-Yih/c4fd9c86b2b41df51a6fe212406dda81b1997fd4 North American Chapter of the Association for Computational Linguistics. (Accessed 5 May 2024)