You are reviewing the design of a heterogeneous inference runtime that splits a large language model between a general purpose CPU and a spatial dataflow accelerator. The accelerator is organised as a two dimensional array of compute tiles, each with its own local memory and a pair of DMA channels, connected by a packet switched interconnect whose routes are configured once when a bitstream is loaded. Data reaches the array through shim tiles at the boundary, which translate buffer descriptors into bursts against external memory. Because the routes are fixed at configuration time, the shape of a computation is baked into the bitstream, and a design that works well for one sequence length can be badly matched to another.

The model itself is a hybrid: three quarters of its decoder layers use a linear recurrent mixer with a fixed size state, and the remaining quarter use ordinary grouped query attention against a growing key value cache. The linear layers are cheap to extend token by token but expensive to snapshot, since their state must be copied in full before any speculative batch is attempted. The attention layers are the opposite: rolling back costs nothing more than moving a cursor, but every step must re-read the whole cache, so their cost grows with the length of the context rather than staying flat.

Explain, in terms a systems engineer would find useful, why the number of dispatches to the accelerator can dominate the wall clock even when the arithmetic being dispatched is a small fraction of the total, and describe how you would measure whether a change that reduces dispatch count but moves arithmetic back onto the host is actually a net improvement. Be concrete about what you would instrument, what baseline you would compare against, and which measurements would tempt you toward a wrong conclusion.
