My Blog

I’ve been using a lot of gpt-5.6-luna at work lately, mainly because it does a lot of work for me for not a lot of credits. Code/MR reviews are typically done in ~$0.20, simple tasks like updating dependencies to address security findings takes ~$1.00, while an architecting task might take ~$5.00 (depending on depth).

The main motivation for sticking to luna is due to the Intelligence Index vs. Cost per Intelligence Index Task section on Artificial Analysis. If you deselect all models (as in this link), you get the pareto line for this comparison that had hints for only models that lie on it. The Pareto line represents the data points that maximise on factor B given factor A, in this case, it shows for each tier of money spent, what is the model with the highest Intelligence Index. I understood this concept when I was researching Sharpe ratio and the optimal portfolio construction, which introduced me to the Efficient Frontier concept. There used to be an online tool that I obsessed over tweaking it constantly to see whether my portfolio lies on the efficient frontier line, so that I’m not taking more risk on a lower-returning asset.

Anyway, this AA chart shows a similar information for LLMs’ intelligence per money spent on completing a task. The chart conveniently draws a green box (I’ll call this the efficient quadrant) showing the region where models in this box are considered good as far as the metric is concerned (above 50% mark in intelligence, below 50% mark in cost). The efficient quadrant shows that for just a bit more money-budget, you get a significant improvement in model intelligence, while once you exit the box towards higher cost, there’s diminishing gains in intelligence (you’d need to pay a lot more for higher intelligence). At the time of writing, at the lower-end of the cost spectrum in the efficient quadrant, this is dominated by gpt-5.6-luna, all reasoning efforts, with glm-5.3-flash sneaking into at higher cost side of the box. Between the two models mentioned, I only have access to luna, so that’s what I picked.

This got me thinking: different users optimise for different aspects when using LLMs in their work. A model being called “frontier” in the current canonical sense (i.e. highest intelligence) is meaningless to most of us because it is cost-prohibitive. For cost-conscious users, I’d argue the more useful “frontier” would be what AA presented with their pareto efficiency frontier charts, i.e with caveats for cost and speed (to a solution). gpt-5.6-luna is the cost-optimised frontier model today. Some users might not care one bit about money, and only care about how fast a model can solve a given problem. In this case, the Intelligence Index vs. Time per Task chart would be of more use. In that chart, the more typical “frontier” models dominate the efficient quadrant box (because generally, higher intelligence models don’t need to reason so much for the same task?). The frontier line shows for a bit more time-budget, you get a massive increase in intelligence, beyond which there’re diminishing improvements to intelligence.