Transformers Learn Higher-Order Optimization Methods for In-Context Learning: A Study with Linear Models

1 code implementation • 26 Oct 2023 • Deqing Fu, Tian-Qi Chen, Robin Jia, Vatsal Sharan

Transformers excel at in-context learning (ICL) -- learning from demonstrations without parameter updates -- but how they do so remains a mystery.

In-Context Learning