Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models

no code implementations • 6 May 2024 • Emre Onal, Klemens Flöge, Emma Caldwell, Arsen Sheverdin, Vincent Fortuin

Fine-tuned Large Language Models (LLMs) often suffer from overconfidence and poor calibration, particularly when fine-tuned on small datasets.

Bayesian Inference

Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networks

no code implementations • 5 Aug 2023 • Liam Parker, Emre Onal, Anton Stengel, Jake Intrater

We examine a variety of network architectures, activations, and datasets, and demonstrate that some degree of (NC) emerges in most of the intermediate hidden layers of the network, where the degree of collapse in any given layer is typically positively correlated with the depth of that layer in the neural network.