Skip to content
Arne Noori
Aboutarne.ai
← All artifacts
Attention Is All You Need

Attention Is All You Need

Image creditdvgodoy, via Wikimedia Commons (a redrawn diagram of the Transformer architecture)
CC BY 4.0 ↗

Attention Is All You Need

Year
2017
Creator
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin
Company
Google Brain and Google Research (with the University of Toronto)
Location
Mountain View, California

Description

This research paper by eight Google researchers describes the Transformer, a neural network that uses a method called attention to process text. The authors posted it to arXiv on June 12, 2017.

Importance

Soon after Google published the paper, OpenAI researchers used the Transformer to make the first GPT models. The Transformer is also the base of ChatGPT, which is in this room. By 2024, all eight authors had left Google, and most of them had started new AI companies.

Interesting fact

A footnote in the paper says that all eight authors contributed equally and that the order of their names is random.

Sources

  1. Attention Is All You Need (Vaswani et al.) ↗arXiv / NeurIPS 2017 · 2017-06-12
  2. 8 Google Employees Invented Modern AI. Here's the Inside Story ↗Wired · 2024-03-20
  3. Introducing ChatGPT ↗OpenAI · 2022-11-30