Recently, I started working on DNA sequence modeling and decided to explore linear attention, mainly because DNA sequences can easily reach 1M tokens, making standard softmax attention extremely expensive in terms of memory and computation. The model performed reasonably well on several benchmarks,
