Gemini 4 performs well in tests, but bogs down in operation – Google employees complain

Google has begun rolling out its long-awaited flagship artificial intelligence model, the Gemini 4 Argon, but doubts are growing within the company about its performance qualities in key areas, including writing software code. However, Google management is optimistic.

Image source: blog.google

Google showed off the Gemini 4 Argon model to a small group of partners in the cybersecurity space and promised to expand its use after additional testing, starting with paying customers. According to the official statement, Gemini 4 showed the best results in multiple benchmark tests and surpassed OpenAI Astra in the security field. But sources say these indicators don’t tell the whole story. Bloomberg – In practice, it shows milder results and encounters difficulties in some programming tasks.

The search giant is actively developing AI models that can compete with products from OpenAI and Anthropic; the company originally planned to release Gemini 3.5 Pro in June, but later gave up. Google officials said there was no truth to claims that Gemini 4 programming performed poorly, quoting DeepMind CEO Koray Kavukcuoglu as saying: “I have complete trust in this team. I am confident that we will always be at the forefront of technology”.

Google employees are divided. Some people say that Anthropic Fable and OpenAI Astra are developing faster than Gemini; and Gemini 4, even with its best efforts, will still be inferior in many aspects. Others believe the upcoming version will be functionally comparable to products from leading AI labs. A Google employee familiar with the model’s development said there was “broad consensus” within the company that the company had tested the models thoroughly and dismissed claims that the models were weak at handling complex, real-world programming problems.

Google needs Gemini 4 to succeed. Variants of Gemini are used in nearly all of the company’s products, including Search, Maps, Gmail, and AI answers in the Chrome browser. Billions of people use these services — an audience reach that some of Google’s rivals can’t boast. OpenAI and Anthropic are no longer simply selling access to models; they are releasing full-fledged products, including AI agents for writing code. If Google doesn’t respond with cutting-edge models, competitors will have time to convince customers that their platforms are the future of search and software.

Google admits that the latest Pro models were released back in February, but the company has since seen growth in its AI offerings: Gemini for enterprise, chatbots for mass users, and AI comments in search—the latter two services have reached more than 1 billion users. In November last year, the Gemini 3 model outperformed all competitors; in May this year, the Gemini 3.5 family debuted; the company promised to release the flagship Gemini 3.5 Pro in June, but according to unofficial information, the project was cancelled.

The decision hurts the company’s position in the artificial intelligence market and could cost it dearly in terms of time and costs. Experts estimate that training such a system alone would cost as much as $400 million, and that doesn’t even take into account the salaries of very well-paid engineers. Some employees insist that Gemini 4 still has some issues. Her coding skills were mixed: for example, the model struggled in the front-end area responsible for the appearance and usability of apps and websites. In addition, this model is very large and expensive to operate, which may be detrimental to Google’s profitability.

Experts speculate that Google may be too obsessed with “benchmarking,” a popular industry term in which engineers optimize artificial intelligence models to achieve high test scores, and less focused on creating a product that does its job well. Sources familiar with the development of this model confirm that the Gemini 4 does not escape this trend.

One of the advantages of this model is its ability to process information beyond text, such as extracting metadata from videos. People familiar with the matter said Gemini 4 has a high level of security and network security and can communicate in a clear and natural way.

There is clear dissatisfaction with the situation within Google: Researchers point to a bloated bureaucracy trying to implement artificial intelligence across all of the company’s products; and constant changes in missions and priorities that undermine a consistent strategy. The company has lost many leading artificial intelligence researchers: legendary engineer Jeff Dean, Nobel Prize winner John Jumper, Gemini co-leader Noam Shazeer and DeepMind CEO Demis Hassabis have all been replaced by the more pragmatic Koray Kavukcuoglu.

According to the official version, Gemini 4 is designed to perform long-term and complex tasks in the fields of software development, finance, law, and cybersecurity. It can generate up to 1 million tokens at a time – approximately 750,000 words. Meanwhile, Google’s rivals are moving forward at a fast pace despite calls to slow down development; the most popular AI agent service among consumers is Meta✴ Muse – It can perform daily tasks, including online purchases and booking appointments for users.