I've failed twice to get "Generate an SVG of a pelican riding a bicycle" to work with max, because in both cases it ran out of the 128,000 token budget while it was still reasoning about the problem.
I'm suspicious that "max" may be virtually useless if it's that easy to have it overthink to the point that it doesn't get to a response.
This is totally a thing I noticed myself about 3 months ago. Medium thinking effort is ideal for most tasks. At high and above, models tend to generate more output in the form of comments or code for the same problem with no real benefit. Its a self-feeding loop: more output becomes more input, which then becomes more output. High is the highest I go. If I need more intelligence, it's better to use a more powerful model with less thinking effort or break the problem into phases. Much better result.
RGS1811 2 hours ago [-]
"This is a classic test request..."
I know there's been discussion about whether pelicanmaxxing is happening, but this is at least evidence that Claude was explicitly exposed to this problem.
croemer 53 minutes ago [-]
Of course it was exposed - not sure it's explicit or not. Why wouldn't HackerNews comments be part of the training data? And Simon's blog and the many discussions about Pelicans? It'd be hard to miss. Doesn't mean Anthropic has made this an explicit goal in training.
Just want to say: you’re such a legend, please do not stop sharing your pelicans, it’s always fun to see how they change over the months :)
cubefox 2 hours ago [-]
The model recognizing the task doesn't mean it was benchmaxxed (RLVR-trained) to solve it. It might simply recognize it from pre-training on Internet text.
Someone1234 2 hours ago [-]
For people with any kind of budget, Opus 5.5's [Medium] actually can make sense dollar per intelligence/dollar per task wise. Heck, it puts some other models to shame. [Max]'s cost is completely unhinged.
My most exciting recent release is actually 5.6 Luna, not because it is the best on any index, but the dollar per work is insane value for money. I find myself more exciting by "value" than hypothetical ceilings because I'm just not in that budget category.
seabass-salmon 1 hours ago [-]
That was true for me four weeks ago, but 2-3 weeks ago Luna turned into drivel in essentially the same complexity of task. I feel it came back somewhat in recent days but does feel like it's being manipulated.
samuelknight 2 hours ago [-]
I have experienced this with open weight models too. "Max" is for benchmaxxing the intelligence metric and is not meant for use in productive work. Like drawing pelicans.
sidewndr46 1 hours ago [-]
I've asked Opus 5 Max for what I thought were easy tasks at work to be completed. It always fails after reaching a tool limit.
I asked Opus 5 High for the same task and requested it to minimize tool usage. It produced an answer in a few minutes that I was deploying to my target platform about 30 minutes later.
az226 2 hours ago [-]
How did you get the reasoning trace? Is it the actual one or the summarized one?
simonw 2 hours ago [-]
It's the summarized one returned by their API.
Piping the visible reasoning trace through their token counter API (I use https://tools.simonwillison.net/claude-token-counter for that) counts 27,888 tokens, so it's definitely a summary of the 128,000 actual token trace.
beardsciences 2 hours ago [-]
I am very interested in why it was able to overthink that much. In the 20-30mins of Max reasoning I've had so far, I'm not having the same issues (yet).
breckenedge 2 hours ago [-]
Do these evaluations get re run a few weeks after launch? I started doing that yesterday for our internal dataset and found Sol’s performance had regressed to be equal to Luna’s. Granted this was one run, but something I’m becoming more concerned about, the model providers want to quickly prove they’re the best, people switch to them, then they pull the rug.
____tom____ 4 minutes ago [-]
"somewhat expensive when comparing to other models of similar price"?
That says something about your selected range, and nothing about the model.
hglaser 2 hours ago [-]
Half the cost per task compared to Opus 5, comparing high effort to high effort. That's just really nice.
That is a lot. I thought Anthropic models would just do the opposite because they are greedy for money.
giancarlostoro 2 hours ago [-]
Greed is not what's driving these prices, its cost. They considered very much in the red.
user43928 2 hours ago [-]
Astra High is slightly cheaper at $1.73 vs $1.82 for Opus 5.5
onlyrealcuzzo 36 minutes ago [-]
The UI/UX seems impressively bad. DeepSWE's cost curve has a better, more obvious way to sort by only the top level of reasoning to avoid 80% of the graph just being the same 3-5 models at their 8 different reasoning levels...
It's also less clear what a lot of their metrics mean. Does Cost per Task include only things that can be verified to work and passed? As best I can tell, it does not.
I'm less concerned if one model's cost per task is $0.10 and another model's cost is $1.50 if the $0.10 task got it right 1% of the time and the $1.50 model got it right 66% of the time.
An equalized / weighted cost/time per task is much more valuable - being massively penalized for taking a lot of time and ultimately not passing when OTHER models did pass.
Many benchmarks start to plateau after high, this benchmarks better than Fable, and my initial tests show it working really well.
sharktheone 2 hours ago [-]
Interesting to see it now. I've used it a bunch before it came out and i pretty much didn't notice it. It might have been slightly better code quality, but still not great in that. I guess it just was slightly less frustrating to work with, but still AI...
giancarlostoro 2 hours ago [-]
I think we're hitting the ceiling of most models capabilities. We're getting to a point where too much training apparently creates models that hack people.
46 minutes ago [-]
bkishan 2 hours ago [-]
Definitely a quiet release. Perhaps pre-empting marketing for Astra public release?
meric_ 2 hours ago [-]
All anthropic launches are like this. They just post it and don't particularly put out the PR sprint that OpenAI does with videos, livestreams or whatever.
(Except for of course Mythos and whatnot when they want to push the whole "safety" thing)
firemelt 2 hours ago [-]
so its more intelligence than fable?
can anyone help me?
WhitneyLand 2 hours ago [-]
China who?
2 hours ago [-]
floki165 54 minutes ago [-]
[flagged]
justindotdev 2 hours ago [-]
[dead]
qsort 2 hours ago [-]
I am begging you on my knees to please stop posting this cringe.
The model is just out. It could be good, great even, I don't know. But I do know that this index has Opus 5, one of the worst releases of 26, ahead of Astra. What information are we supposed to deduce from number having gone up?
kzrdude 2 hours ago [-]
What's more valuable than a good benchmark? IMO a benchmark that has been run against very many competitors and versions. Collecting data has something going for it, and it's up to the readers to interpret and make the best use out of it.
nozzlegear 3 minutes ago [-]
[delayed]
losvedir 1 hours ago [-]
This index doesn't have "Astra" and "Opus 5". Every entry with corresponding data is a `(model, reasoning)` tuple.
So I'm unclear what you're actually saying and wondering if you've missed that. Are you saying that at every reasoning level it says Opus 5 beats Astra? I just compared Opus 5 high to Astra high and it has Astra as generally better than Opus.
ruszki 56 minutes ago [-]
I don't try to say that the parent commenter is right in any way, but the two models' "high" settings probably doesn't mean the same thing. So probably comparing only them is not useful.
Someone1234 1 hours ago [-]
You forgot to include whatever you're proposing instead.
"Trust me bro, Astra is better" isn't perhaps as useful as you seem to believe. I'm not even saying it is right or wrong, just that my opinion on this topic is still just one additional subjective data-point.
Only thing I wish with these benchmarks is that they would run repeat tests every couple of months. Then re-rank based on that too. We've seen a lot of performance fall-off after a couple of weeks with new releases.
esafak 2 hours ago [-]
That it's better in specific ways? What difference does it make when it came out? The benchmark results are not going to change unless they're messing with the model.
qsort 2 hours ago [-]
Yes, but crucially, in ways that are increasingly decoupled from any practical pattern of usage, considering that I wouldn't see how you can argue that Astra is worse than Opus 5.
One man's modus ponens is another's modus tollens I guess.
I've failed twice to get "Generate an SVG of a pelican riding a bicycle" to work with max, because in both cases it ran out of the 128,000 token budget while it was still reasoning about the problem.
I'm suspicious that "max" may be virtually useless if it's that easy to have it overthink to the point that it doesn't get to a response.
Transcript for one attempt here - expand the "Reasoning trace" bit to see it: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
I know there's been discussion about whether pelicanmaxxing is happening, but this is at least evidence that Claude was explicitly exposed to this problem.
My most exciting recent release is actually 5.6 Luna, not because it is the best on any index, but the dollar per work is insane value for money. I find myself more exciting by "value" than hypothetical ceilings because I'm just not in that budget category.
I asked Opus 5 High for the same task and requested it to minimize tool usage. It produced an answer in a few minutes that I was deploying to my target platform about 30 minutes later.
Piping the visible reasoning trace through their token counter API (I use https://tools.simonwillison.net/claude-token-counter for that) counts 27,888 tokens, so it's definitely a summary of the 128,000 actual token trace.
That says something about your selected range, and nothing about the model.
Edit: https://artificialanalysis.ai/models/claude-opus-5-5?models=...
It's also less clear what a lot of their metrics mean. Does Cost per Task include only things that can be verified to work and passed? As best I can tell, it does not.
I'm less concerned if one model's cost per task is $0.10 and another model's cost is $1.50 if the $0.10 task got it right 1% of the time and the $1.50 model got it right 66% of the time.
An equalized / weighted cost/time per task is much more valuable - being massively penalized for taking a lot of time and ultimately not passing when OTHER models did pass.
Not sure about how adaptive reasoning works though as they mention adaptive reasoning for every reasoning level
Many benchmarks start to plateau after high, this benchmarks better than Fable, and my initial tests show it working really well.
(Except for of course Mythos and whatnot when they want to push the whole "safety" thing)
can anyone help me?
The model is just out. It could be good, great even, I don't know. But I do know that this index has Opus 5, one of the worst releases of 26, ahead of Astra. What information are we supposed to deduce from number having gone up?
So I'm unclear what you're actually saying and wondering if you've missed that. Are you saying that at every reasoning level it says Opus 5 beats Astra? I just compared Opus 5 high to Astra high and it has Astra as generally better than Opus.
"Trust me bro, Astra is better" isn't perhaps as useful as you seem to believe. I'm not even saying it is right or wrong, just that my opinion on this topic is still just one additional subjective data-point.
Only thing I wish with these benchmarks is that they would run repeat tests every couple of months. Then re-rank based on that too. We've seen a lot of performance fall-off after a couple of weeks with new releases.
One man's modus ponens is another's modus tollens I guess.