BLOG POST

Consider a scenario in extending a model’s context window for tasks like analyzing extensive legal transcripts that could reach 100,000 tokens. Various approaches might be tried, such as scaling rotary positional embeddings (RoPE), fine-tuning with methods like LongLoRA, or increasing computational resources. However, when testing on longer sequences not seen