Hi everyone!

I've been experimenting with an alternative Transformer architecture called GAP (Gated Attention Projection).

The main idea is to combine attention and MLP-style processing into a single gated block, instead of using the standard separate attention and feed-forward sublayers.

I tested it against a standard Transformer using 100 million scored training tokens per model from WikiText-103.

Some results:

GAP20 used 19.91% fewer parameters than the standard Transformer.

A smaller configuration can reduce parameters by 34.31%, although that version did not fair as well as the current model against the regular transformer, so i ended up giving 14% of my perameters savings to the GAP model for extra width, leaving the model at about 20% fewer perameters, which is why i called it in testing GAP20

The standard Transformer performed better on final language-model accuracy.

Compiling GAP20 with torch.compile improved training throughput by approximately 19.9% in one CPU benchmark compared with running GAP20 without compilation.

I've included the architecture code, configurable model sizes, speed benchmark and research paper.

This is an experimental architecture, not a claim that it outperforms Transformers overall. I'd be interested in feedback on the design, mathematics and possible improvements.

GitHub:

https://github.com/rickey1990/Gated-Attention-Projection