Not all models scale up well with more parameters.
VAEs are not really exclusively a vision thing, they have been used in a variety of settings. Using VAEs for NLP is also nothing new, an early example is Bowman et al, 2015.
What kind of bottleneck are you talking about? VAEs as such are really in a different category from transformers. They are primarily a tool to get to more structured latent spaces, which is not something transformers are good at in the first place.
VAEs are not really exclusively a vision thing, they have been used in a variety of settings. Using VAEs for NLP is also nothing new, an early example is Bowman et al, 2015.
https://arxiv.org/abs/1511.06349