From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformers

1 code implementation • 18 Oct 2023 • Shaoxiong Duan, Yining Shi, Wei Xu

We then introduce Attention Bias Calibration (ABC), a calibration stage that enables the model to automatically learn the proper attention biases, which we show to be connected to mechanisms in relative position encoding.

Position