VAE and DRAW, Compared

DRAW generates an image the way a person sketches, building it up over several steps through a moving attention window, rather than producing the whole thing in one pass like a standard VAE. This project, done by a group of three, tested how much that recurrent, attentive process really buys.

Both models were implemented from scratch in PyTorch and compared on MNIST across five configurations: a vanilla VAE, DRAW without attention, and DRAW with its Gaussian filterbank attention at one, five and ten glimpses. Everything is seeded, and the hardware and library versions are recorded so the numbers rebuild.

Test NELBO, lower being better: the vanilla VAE reached 102.863. DRAW without attention reached 93.817, the best result of the set. DRAW with attention reached 134.485 at a single glimpse, 104.949 at five, and 99.737 at ten.

Read carelessly that says attention hurts, and it does not. The no-attention model carries 2.61 million parameters against 0.87 million for the attentive one, three times as many, and the best attention run was given twice the training epochs to reach its number. Compared at a matched budget, reading attention off the same curve at epoch 30, it sits at 105.5, so recurrence is doing most of the work and attention is buying parameter efficiency rather than raw likelihood, and needs a longer schedule before it pays.

That confound is stated in the report's own abstract rather than left for a reader to find, which is the part of this project I would actually defend. The whole comparison took about thirty minutes of compute and is written up as a short paper in ICLR format with training curves and samples per configuration.