Is the Gap Widening Between Anthropic and Open-Source Models?

Some developers have told me that the https://www.theinformation.com/newsletters/ai-agenda/rising-ai-costs-becoming-problem-investors">rising costs of frontier AI models from https://www.theinformation.com/articles/anthropic-flexes-pricing-power-customers-willingly-eat-cost">Anthropic and other firms could prompt them to shift to cheaper open-source AI. After all, when companies as sophisticated as https://www.theinformation.com/newsletters/applied-ai/uber-cto-shows-claude-code-can-blow-ai-budgets">Uber are accidentally blowing through their entire year’s AI budget in a matter of months, it makes sense to cut back by using a less capable open-source model to automate simpler tasks. (In fact, companies like Uber and Airbnb are doing exactly that!)

It’s not clear whether open-source AI is good enough to meet the challenge, though.

For instance, one executive at a major customer of OpenAI and Anthropic told me that they’ve been trying to use open-source models like Moonshot AI’s Kimi K2.6 and DeepSeek V4.

But while these models have performed well on benchmarks and are good at answering more surface-level questions in a variety of areas, they tend to struggle with follow-up questions or deeper lines of questioning, this executive said. 

For instance, you could imagine a model doing well on a popular brainteaser but then struggling if you tweak a few assumptions or details in the brainteaser.

Of course, this is just one developer’s experience, and usage of open-source models does seem to be growing overall, based on https://openrouter.ai/rankings">data from inference provider OpenRouter.