Search Results for author:
Found 4 papers, 1 papers with code
Exploring Context Window of Large Language Models via Decomposed Positional Vectors
Based on our findings, we design two training-free context window extension methods, positional vector replacement and attention window extension.
Base of RoPE Bounds Context Length
We revisit the role of RoPE in LLMs and propose a novel property of long-term decay, we derive that the \textit{base of RoPE bounds context length}: there is an absolute lower bound for the base value to obtain certain context length capability.
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
As Large Language Models (LLMs) continue to advance in performance, their size has escalated significantly, with current LLMs containing billions or even trillions of parameters.
Baichuan 2: Open Large-scale Language Models
Large language models (LLMs) have demonstrated remarkable performance on a variety of natural language tasks based on just a few examples of natural language instructions, reducing the need for extensive feature engineering.
