Last month, AI founders and investors discussed a “second era of scaling laws.” They noted that traditional methods of improving AI models were yielding lower returns. One promising new approach, referred to as “test-time scaling,” seems to be driving the success of OpenAI’s o3 model. This model has gained attention for its high performance. It excelled on benchmarks like ARC-AGI, where it outperformed other models.
The o3 model also achieved a remarkable 25% on a challenging math test, a feat no other AI had come close to. The AI community is excited about o3’s achievements. But, TechCrunch is cautious and awaits firsthand testing. Still, the announcement of o3 has made it clear that AI scaling progress is far from stagnating. Open AI’s Noam Brown, a co-creator of the o-series models, noted that the o3’s launch comes just three months after the o1’s debut.
This marks an unusually rapid leap in performance. He expressed confidence that this positive trajectory would continue. Anthropic co-founder Jack Clark also weighed in, calling the o3 model a sign that AI progress will accelerate in 2025. He suggested that combining test-time scaling with pre-training could boost AI performance. This is important. It hints that other companies, like Anthropic, may release similar models in 2025.
This follows Google’s recent advances in reasoning AI. The progress is clear. But, the debate on it shows that the AI community is entering a new phase of rapid evolution. There may be breakthroughs ahead. Open AI’s latest model, o3, has introduced a concept called test-time scaling. It uses more compute resources during the AI’s inference phase, after a user submits a prompt.
This could mean Open AI is using extra chips or more powerful hardware. Or, it is running these chips longer (10 to 15 minutes) to generate answers. The tech’s exact details are unclear. But, early tests suggest it could greatly boost the model’s performance. However, o3’s improved performance comes at a cost. Open AI is using a higher level of compute power, which results in a more expensive inference process.
As Open AI’s Clark noted in his blog, o3 may produce better answers. But, it’s more costly to run. This makes AI model pricing less predictable. Before, one could estimate the output generation cost by checking the model’s architecture. But with test-time scaling, it’s clear that higher computational resources lead to better results at a higher cost. One of o3’s most striking achievements is its ARC-AGI score.
This benchmark tests progress toward artificial general intelligence (AGI). While passing this test doesn’t indicate that an AI has reached AGI, it serves as a key metric of advancement. o3 far outperformed previous models, scoring 88% on one attempt. OpenAI’s best, o1, only managed 32%. This suggests that test-time scaling may be key to advancing AI. But, it makes the cost of running such models harder to predict.
Open AI’s o3 model has made great strides in AI research. But, its high resource use raises concerns about its practical, everyday use. François Chollet, creator of the ARC-AGI benchmark, says this: Open AI’s top version of o3 used over $10,000 worth of compute for a task that scored 88%. A more efficient version of o3 achieved only 12% lower performance, but at a much lower cost.
This shows a gap in resource use. The high-efficiency version costs much more than o1 models, which only use about $5 of compute per task. Chollet says o3 is a breakthrough in AI. It excels at adaptable, general tasks, nearing human performance in complex scenarios. However, he notes that this capability is very costly. It’s far more than hiring a human, who could do it for much less.
Chollet believes o3 shows AI’s potential to handle more general tasks. But, its high cost limits its use. The core issue is the high computational cost required for these breakthroughs. O3’s advancements are notable. But, it won’t be a daily-use tool like GPT-4 or Google Search. It demands too many resources for small tasks. Instead, it seems better for high-stakes, long-term projects.
These include strategic decision-making in academia, finance, and industry, where costs are justifiable. Some wealthy institutions may afford o3’s high compute costs, at least initially. This could include universities and large corporations. They want to use AI for complex problem-solving. However, as Chollet notes, o3 still struggles with basic tasks that humans handle with ease.
This shows it isn’t a true AGI (Artificial General Intelligence) yet. To improve the efficiency of such high-compute models, new AI inference chips could play a crucial role. Startups like Groq, Cerebras, and MatX are working on cheaper AI compute solutions. They could lower the costs of using models like o3. O3 shows promise in scaling AI’s performance through test-time compute. However, it raises questions about the sustainability of such models. They must overcome issues, like hallucinations, in large language models.
In conclusion, o3 is an exciting AI advance. But, its cost and limits suggest it won’t be widely available soon. As AI companies seek to boost efficiency, such models may become more viable for daily use.
