Junyang Lin’s Post

Qwen3-Next, or to say, a preview of our next generation (3.5?) is out! This time we try to be bold, but actually we have been doing experiments on hybrid models and linear attention for about a year. We believe that our solution shoud be at least a stable and solid solution to new model architecture for super long context! GDN plus hybrid is based on a lot of trials and errors, and the implementation of attention gate is something just like a free lunch to get benefits. Moreover, we continue our research on MoE and carefully further increase the sparsity to make it more efficient and effective! What makes us suffer a lot is that you need to run the whole process of training to evaluate new model architecture, which means pre-training + post-training (notably reinforcement learning). We have proven it working and we release the instruct and thinking models both after RL. Nevertheless, as this is for the first time that we release something totally new, we are still unsure about what we have done right or wrong, and we need the support from the community. Specifically, many thanks to Hugging Face, vLLM, and SGLang. They have done quite a lot of efforts helping us deliver this new model to you all! Welcome to try and send us feedback! Hope it is a good start of a new journey 🚗