Is a smaller language model always greener?

Which AI model should run your agent? Most comparisons score a model per call. An agent does not stop after one call. It keeps working until the task is done, and some attempts fail and have to be repeated.

Sketch of energy per completed task against task length. The small model starts lower but rises steeply, the larger model rises gently, and they cross at a crossover point.

So we measured something else: the energy, cost and time for each task that was completed successfully.

A small model is cheaper per call. But if it needs more steps, or fails more often, that saving can disappear. The best model can change as tasks get longer. The task length where it changes is a crossover point.

We built a simple decision model around these crossover points. It first removes models that are beaten on every measure. Only then does it ask for preferences, and only when a real trade-off is left.

We tested it on 15 open-weight models, from 0.6B to 27B parameters, on two multi-step agent tasks. GPU energy was measured directly. The crossovers appeared where the model predicted them.

Measured GPU energy per successfully completed task against task length for four of the 15 pilot models. — GPU energy per successfully completed task, order task, four of the 15 models. Source: the paper.

The paper is joint work with Maria Barbati: When Is Smaller Greener? Right-Sizing Language Models for Agentic AI, arXiv:2610.19345.

The code is on GitHub: github.com/sajidsiraj/whichone.

Leave a Comment Cancel Reply