Attention
Older networks read text word by word and often “forgot” the beginning of a sentence. A transformer looks at all the words at once and, for each word, works out which other words matter to it. In the sentence “The cat did not jump on the table because it was tired”, it can tell that “it” refers to the cat.
Parallel training
Because a transformer processes text all at once, it can be trained on thousands of graphics cards at the same time. This made it possible to build huge models trained on a large part of the internet.
Not just text
Transformers today also process images, sound, video and proteins. Both chatbots and image generators are built on the same foundation.