How Tools Help And Complicate Visual Agents

Giving a vision-language model access to image search, image processing, or Python seems like a straightforward way to expand its capabilities. In practice, turning these tools into something genuinely useful for the model — and eventually for the user — comes with many caveats. In this talk, I will share lessons we learnt while developing visual agentic capabilities for the Yandex Alice AI model. In particular, I will cover the tricky parts of evaluating multimodal agents, what can go wrong when building agentic infrastructure for training, and what we learnt from both successful and unsuccessful SFT and reinforcement learning experiments. The attendees will learn what to expect — and what not to expect — from multimodal agents, and how to deal with models exploiting weaknesses in training and evaluation setups.
Format: TECHNICAL TALK
Location: ROOM B
Speakers: Boris Zimka
Level: Intermediate to Advanced
Audience: Registered participants