Words are the surface of language, but structure is its bloodstream. Modern AI language models have become astonishing at predicting the next word, yet they often stumble when grammar and long-range dependencies demand a more deliberate shape of thought. That gap between fluent word output and robust syntactic understanding is what a new study from ShanghaiTech University and Ant Group sets out to explore. The researchers don’t just propose a taller tower of a single model; they sketch a whole framework for compositional syntactic language models (SLMs) that explicitly braid parse trees into the fabric of Transformer-based generation. The aim isn’t merely to add grammar as decoration, but to test how different design choices around syntax change what the model learns, how it generalizes, and how efficiently it runs. The work, led by Yida Zhao, Hao Xve, Xiang Hu, and Kewei Tu, and involving collaborations with Ant Group, also comes with a public code release to spur further exploration.
What follows is a guided tour through their ideas, findings, and what they imply for the near future of AI-assisted writing, summarization, and dialogue. It’s a story about how a careful blend of linguistic structure and neural computation can push models beyond raw memorization toward more human-like language skills. And it’s a reminder that the best AI progress often comes not from bigger models alone, but from smarter, more thoughtful ways to organize knowledge inside those models.
One striking takeaway from the paper is not a single magic trick, but a mapping of design space. The researchers show that sixteen variants—16 different ways to pair parse trees with generation—live inside a single, unified framework. Across tasks that matter for real-world use, some configurations chase raw perplexity, while others chase syntactic talent or the ability to summarize and converse with a sense of structure. In other words: syntax can be a compass, not a gimmick, and the way you mount it on a Transformer changes where you go and how you get there.
What compositional SLMs are and why they matter
At the core, a compositional SLM is a language model that tries to predict both a sentence and its underlying constituency parse tree at the same time. The model is trained to generate a sequence of actions that builds the sentence and its tree in a left-to-right fashion. Think of a sentence as a Lego construction: you don’t just snap together bricks randomly; you build up little sub-structures (constituents) and then compose them into bigger pieces. A composition function is the clever brick-layer that fuses the representations of sub-constituents into a single representation for the parent constituent. That composed representation then informs the next steps of generation, either by feeding back into the same Transformer stack or by an external module that supplies the parent’s embedding.
The authors formalize a framework that subsumes two familiar modeling choices and several new twists. They consider two forms of parse trees: binary trees (Bi) and non-binary trees (Nb). They also compare two ways to linearize trees so that the model can generate them in a sequence: a top-down approach (pre-order) and a bottom-up approach (post-order). Finally, they contrast two flavors of composition: an internal function that operates inside the Transformer’s flow and an external one that sits alongside the Transformer as its own module. On top of this, they explore whether the model should “mask” sub-constituents after they’re composed—cutting the model off from some earlier information to create a learning bottleneck—or keep the information available. The paper’s framework allows any combination of these choices, giving rise to 16 distinct SLM variants that the team actually tested.
Put more simply: you can think of it as choosing how to scaffold a sentence while you’re building it, and how much of the scaffolding you reveal as you go. Each choice reshapes what the model learns about language structure, what it can generalize to, and how efficiently it can run in practice. And because these choices interact in nontrivial ways, the researchers treat the whole family of variants as a single ecosystem to study rather than isolated curiosities.
Inside the design space four axes shape how syntax helps
First, the binarization and the way we traverse the tree matter a lot. Non-binary trees (Nb) carry more natural linguistic structure, but they demand more elaborate action sequences to express, since a node may have many children. Binary trees (Bi), by contrast, reduce the branching into a standardized form. The paper shows that different tasks favor different couplings: bottom-up linearization pairs nicely with binary trees, while top-down linearization can be more compatible with non-binary trees. This isn’t just a nerdy bookkeeping detail; it changes how quickly a model can learn compositions and how efficiently it can generate long, structured sequences.
Second, how you compose matters just as much as where you compose. An internal composition function folds the sub-constituents’ representations directly into the Transformer’s hidden states. It’s elegant and unified, but it can incur a receptive-field limitation: the composition has to be built up step by step within the Transformer’s own layers. An external composition function uses a separate module to generate a single representation for the parent from its children, then feeds that back into the Transformer. In practice, the external approach can be faster at inference, because the composing work happens outside the main Transformer path, which can save compute when you’re generating a lot of text. The paper’s experiments show a meaningful gap in efficiency favoring the external approach in several configurations.
Third, whether to mask sub-constituents after their composition creates a training-time bottleneck that can force the model to learn the composition more deeply or to rely on surface cues. Masking (M) typically hurts immediate token prediction because information is blocked, but it can help the model learn more robust, composition-aware representations. Leaving sub-constituent information accessible (Nm) can boost language modeling accuracy, but it sometimes dampens the syntactic learning signal. The authors don’t declare masking as universally good or bad; instead, they show it interacts with the other design choices in nuanced ways depending on the tree type and the composition method.
Finally, there are the overarching choices that define the sixteen variants: binary vs non-binary trees, top-down vs bottom-up linearization, internal vs external composition, and masking vs not masking. Put together, these axes map a landscape where a model can specialize for different aims—whether it’s raw fluency, structural generalization, or the kind of long-form generation that needs to keep track of grammar across dozens of sentences.
What the experiments revealed and why it matters now
The study’s experimental program sits on a sturdy ground: a common Transformer backbone, a shared dataset (the BLLIP-LG corpus with silver constituency trees parsed for training), and a consistent evaluation suite that spans document-level language modeling, syntactic generalization, summarization, and dialogue. The authors compare their 16 SLM variants against two strong baselines: GPT-2-sized token models and a GPT-2-tree baseline that handles linearized trees without explicit composition. The goal isn’t to crown a single winner but to understand how the design choices interact with different tasks.
In the realm of document-level language modeling, the results were nuanced. The majority of SLM variants did not beat the plain GPT-2-token model in perplexity, and several performed worse than the tree-based GPT-2-tree baseline. This is telling: simply injecting syntax as an extra signal does not automatically improve fluency or predictability in long documents. Some variants, however, did better than GPT-2-tree, notably those that model non-binary trees with a bottom-up approach and that rely on a particular mix of internal composition and masking. The takeaway is subtle: explicit composition and well-chosen tree representations can help readability and generalization, but they don’t guarantee better word-by-word prediction across long texts.
Where the compositional SLMs really shine is in syntactic generalization. Across six syntactic phenomena designed to probe how models handle agreement, long-distance dependencies, licensing, garden-path effects, and more, several configurations outperformed both baselines by a comfortable margin. In particular, binary trees with bottom-up linearization and an external composition function (Bi-Up-Ex-Nm, for example) achieved strong syntactic generalization, sometimes rivalling or surpassing GPT-2-tree. The pattern persisted across other high-difficulty setups: when the composition function could express interactions among a varying number of sub-constituents (as is natural with binary trees and a robust external composer), syntactic judgments improved markedly. The result felt almost mercifully intuitive: when you give the model a clean, compositional storytelling device, it uses grammar more effectively to generalize rules it didn’t see in training.
In downstream tasks, the story remains mixed but hopeful. For summarization (XSum) and dialogue (DailyDialog), several compositional SLMs outperformed GPT-2-token by a meaningful margin, driven by the extra structural information the models had access to during generation. Yet GPT-2-tree—the non-compositional but syntax-aware baseline—often achieved the top scores, suggesting that for generation tasks, managing the right blend of syntax and generation power matters more than any single architectural trick. The headline is not that explicit composition is a universal win; it’s that, when tuned to the task, structure-aware models can beat strong baselines in ways that reflect deeper language understanding, especially for tasks that crave coherent long-form text and faithful grammatical behavior.
Efficiency is a practical angle that can’t be ignored in real-world systems. The researchers found that models using an external composition function were noticeably faster at inference than those doing internal composition, sometimes by a wide margin. They also observed that non-binary trees, while linguistically faithful, tend to demand more forward passes during beam search, which can erode practical speedups. In short, a careful pairing of binary trees with external composition offers a sweet spot: good syntactic generalization, reasonable downstream performance, and better throughput for generation tasks.
From a design standpoint, the study yields a pragmatic set of recommendations. If speed and scalability are priorities, favor binary trees with an external composition function and avoid heavy sub-constituent masking unless the task explicitly benefits from a robust syntactic bottleneck. If the goal is to maximize syntactic generalization, the authors show that certain Nb- or Bi-based configurations with the right combination of external composition and masking can outperform standard transformers on grammar-like tasks. And if you’re building systems that leverage parsing as an auxiliary signal for downstream generation, the results argue for a nuanced approach: the parser should be reliable enough to provide meaningful structure, but not so heavy as to crush efficiency in production use.
All of this is more than an empirical chart-topping exercise. It reframes how we think about “syntax in AI.” It’s not a single knob to turn; it’s a constellation of design decisions that shape what the model can learn about language structure, how it generalizes rules it never saw, and how you balance computational cost with linguistic fidelity. The authors’ core message is strengtheningly practical: explicit, well-chosen syntactic biases can unlock better generalization and better downstream performance, but only when they are aligned with the task and the scale of the system. That alignment is, in itself, a design problem—one that modern NLP research is increasingly adept at solving.
Why this matters for the near future of AI language tools
There’s a bigger story behind the numbers and trees. If you care about AI that can summarize long documents accurately, hold coherent conversations across dozens of turns, or reason about complex grammatical constraints in legal or technical prose, you want models that understand—not just memorize—structure. The compositional SLMs studied here provide a blueprint for building such models in a way that makes the grammar explicit, learnable, and testable. They show that the right kind of syntactic bias can translate into tangible gains on tasks that demand more than fluent word prediction: the system must respect the architecture of language to preserve meaning across longer spans and more complex discourses.
Another practical upshot is about efficiency, a constraint that often determines whether a technique ships in production. By moving some of the composition work outside the main Transformer, researchers can cut inference time without sacrificing too much on essential linguistic capabilities. This is particularly relevant as teams push toward real-time generation in chat, live summarization, and on-device AI where compute is precious. The study doesn’t pretend you can have your cake and eat it too—some configurations still lag behind plain token models in perplexity, and non-binary trees add complexity—but it provides a concrete menu of options researchers and engineers can customize for their use case.
Beyond practicality, the work also invites a cultural takeaway about how we think about language intelligence. Humans don’t produce language by memorizing a giant string of tokens; we compose meaning through hierarchical structures—phrases built from smaller units that combine into larger ones. The compositional SLMs are an attempt to mirror that cognitive logic inside machines, not to imitate human grammar for its own sake. The result is a more interpretable, potentially more controllable, and—crucially—more generalizable kind of AI language system. That’s a step toward systems that can grow with the complexity of human language rather than stagnate at surface-level fluency.
As with many academic projects, several caveats accompany the optimism. The study relies on a particular dataset and a GPT-2–sized backbone, and the authors are candid about limitations, including the use of unlabeled constituency trees and the computational cost of their comprehensive variant exploration. They also emphasize that more work is needed to extend the framework to other syntactic structures and larger corpora. Still, the core contribution stands: a unified, extensible framework for compositional SLMs, a rigorous experimental map of design choices, and actionable guidance for building the next generation of syntax-aware language systems.
The paper closes with a practical invitation: a GitHub repository that shares the code for building and evaluating these variants. It’s not a fantasy bench project; it’s a toolkit for researchers who want to probe how grammar can shape language in more controlled, testable ways. The authors’ collaboration between ShanghaiTech University and Ant Group—two institutions with deep expertise in dense, real-world NLP applications—speaks to a broader trend: the best ideas in AI often emerge where academic curiosity meets industrial-scale needs.
A human perspective on a technical landscape
What does it feel like to navigate this design space? It’s a bit like tuning a musical instrument you didn’t know you were playing. The instrument is a Transformer-based model, the strings are the constituents, and the bow is the way you organize and blend those strands into a coherent melody. If you tune too aggressively toward raw word-prediction at the expense of structure, the melody can feel flat. If you tune too aggressively toward syntax at the cost of fluent generation, the music becomes brittle. The sweet spot—where language is both fluent and faithful to its grammar—appears to be achieved not by a single magic toggle but by a thoughtful choreography of the four design axes the authors explore.
For readers who think in terms of applications, the results offer both reassurance and a nudge. Reassurance: syntax-aware models aren’t a silver bullet that will immediately outshine all other approaches on every task. A careful, task-aligned configuration can yield clear gains in syntactic generalization and certain generation tasks. Nudge: rather than chasing bigger models alone, we should invest in how we structure and compose linguistic information inside models. The architecture matters as much as the data, and the way we encode grammar into learning can change what the model can do with it.
Finally, this work is a reminder of the value of systematic, apples-to-apples comparisons when new ideas proliferate. The sixteen variants aren’t just a curiosity; they’re a controlled experiment in how different design decisions interact. In an field where claims of “syntax helps” or “structure hurts” can drift in the noise, a framework that lays out the variables clearly and tests them across multiple tasks is a rare and valuable thing. It doesn’t hand us a perfect model tomorrow, but it gives us a map for the next several steps—a map drawn not with grand promises, but with careful science and a willingness to test ideas against the hard facts of language use.
Lead institutions and people: The work was carried out by researchers from ShanghaiTech University, with collaboration from Ant Group. The paper identifies Yida Zhao, Hao Xve, Xiang Hu, and Kewei Tu as leads, reflecting a joint effort across academia and industry. The team also notes the contribution of the Shanghai Engineering Research Center of Intelligent Vision and Imaging and provides a GitHub link (https://github.com/zhaoyd1/compositional_SLMs) for researchers who want to reproduce or extend the work.
