I like tinkering with stuff and I think that now I understand at least on surface-level how a dense model like 27B works and what needs to be finetuned for a specific configuration. Then, like everyone else, I discovered Strata and running a large moe locally became possible. The problem is that I c
