Sutskever at NeurIPS 2024: Pre-Training Era Is Over

Ilya Sutskever’s Test of Time award talk at NeurIPS 2024 — pre-training is over, next up is agents/synthetic data/inference-time compute, and reasoning is unpredictable.

Ilya Sutskever’s Test of Time talk at NeurIPS 2024. Image: LinkedIn / NeurIPS.

Ilya Sutskever’s Test of Time award talk at NeurIPS 2024. The main points:

• Pre-training era is over because data (internet) is exhausted.

• Next up: agents, synthetic data, inference time compute (among other things).

• Reasoning is unpredictable.

For the last point it’s also the nature of discrete math problems: they scale poorly due to their combinatorial nature. Solving them often requires different strategies.

https://www.youtube.com/watch?v=WQQdd6qGxNs

Originally posted on LinkedIn.