GLM Architecture & Novel Model Design
The General Language Model (GLM) family is pretrained by autoregressive blank infilling, one objective that covers both understanding and generation. I am studying that design and prototyping modifications to it.
Focus areas
- Studying the General Language Model (GLM) architecture's autoregressive blank-infilling pretraining objective, which unifies language understanding and generation in one framework.
- Exploring hybrid attention and encoder-decoder design variations to improve bidirectional context modeling while keeping autoregressive generation efficient.
- Prototyping architectural modifications such as mixture-of-experts routing and modified positional encodings to improve scalability and long-context reasoning.
Topics
General Language Model (GLM)Architecture InnovationDeep Learning