Have URMs and UTs been integrated into frontier models? Or did they disappear into the dustbin of forgotten papers? [D]

UT-based small models, despite being trained from scratch on these tasks without internet-scale pre-training, consistently outperform most standard Transformer-based Large Language models (LLMs) by a significant margin. 👈 The Universal Transformer (UT) extends the standard Transformer by introducing