Computational linguistics

A transformer language model can condition its prediction only on the most recent 2,000 input tokens. A relevant sentence appears 2,500 tokens before the prediction point and is not otherwise represented. What follows?