opus: Buffer the samples from the correct offset