Google, in collaboration with Artificial Analysis, has announced the launch of a new benchmark named MKA (Multimodal Knowledge Agent). This tool is designed to assess the capability of artificial intelligence in processing visually complex documents and performing tasks typical of intellectual work.
Evaluating Knowledge Beyond Text
Most existing benchmarks for testing intelligent agents can be solved solely through textual information, disregarding images or page structure. However, everyday professional activities often involve working with visual elements: analyzing architectural blueprints, engineering diagrams, and synthesizing data for presentations, reports, and spreadsheets. MKA was developed specifically to measure AI’s effectiveness in these scenarios.
Realistic Tasks Crafted by Experts
To ensure the benchmark’s realism, Google engaged experts from 10 industries and 83 visually intensive professions, including architects, biologists, underwriters, and data scientists. These specialists formulated a set of 200 tasks that reflect multi-step projects. Each task includes a realistic prompt, a set of multimodal input files, and a detailed evaluation criterion (rubric) with approximately 70 yes/no points. The evaluation considers factual accuracy (75%) and visual design (25%).
Technical Details and Evaluation
Each AI agent is tested under identical conditions—within the open-source agent harness Stirrup, utilizing the same tools, sandbox environment, and a constrained turn budget. The evaluation of results is performed by an automated system powered by a non-Google model. This system analyzes each file, visualizes pages, verifies formulas in tables, and cross-references results with the expert rubric. Notably, experts agreed with the automated system’s evaluations in 96% of cases.
To maintain objective results, Artificial Analysis conducts testing on a private set of 200 tasks, preventing their inclusion in the training data of models. A separate set of 10 public tasks has also been released on Hugging Face for familiarization with the format and for testing custom agents.
Research Findings
The technical report provides a detailed description of the methodology and results. It reveals significant differences in the performance of current models: on the same task, models used an average of 30 to 130 turns. The research confirms the critical importance of advanced image perception capabilities for successful task completion and demonstrates how models attempt, but often fail, to compensate for the lack of visual analysis.
Gemini 4 Carbon in MKA
The Gemini 4 Carbon model demonstrated the second-best quality score among the tested models and achieved an execution cost that was 6 times lower compared to the leader of the ranking.
Availability
Gemini 4 Carbon is now available via the Gemini API in Google AI Studio. An updated developer guide and documentation are provided to explore the model’s capabilities.
