How “Attention Is All You Need” Was Born: The Story Behind the Transformer Paper
A great recounting written by Madhumita Murgia on how “All you need is attention” – the Transformers paper – was born out of a “serendipitous assembly of a group of scientists”: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin.
“In early 2017, two Google research scientists, Ashish Vaswani and Jakob Uszkoreit, were in a hallway of the search giant’s Mountain View campus, discussing a new idea for how to improve machine translation, the AI technology behind Google Translate…
During their chat in the hallway, Uszkoreit and Vaswani were overheard by Noam Shazeer… So when he heard his colleagues talking about this idea of “self-attention”, he decided to jump in and help. “I said, I’m with you . . . let’s do it, this is going to make life much, much better for all AI researchers,” Shazeer says.
The chance conversation formalised a months-long collaboration in 2017 that eventually produced an architecture for processing language, known simply as the “transformer”. The eight research scientists who eventually played a part in its creation described it in a short paper with a snappy title: “Attention Is All You Need.” ”
Especially love this:
“Each was drawn towards what was widely seen as an emerging field of AI research: natural language processing. The group’s educational, professional and geographic diversity — coming from backgrounds as varied as Ukraine, India, Germany, Poland, Britain, Canada and the US — made them unique.”
(X-post on Mastodon)
Originally posted on LinkedIn.
References
[1] Madhumita Murgia. “Transformers: the Google scientists who pioneered an AI revolution.” Financial Times. https://www.ft.com/content/37bb01af-ee46-4483-982f-ef3921436a50