ToolSmith

An Automatic Tool-Generation Framework for Multimodal Search Agents

Rui Sun♠, Junhua Huang♠, Luca Zancato♣, Md Mofijul Islam♣†, Kai-Wei Chang♠†

UCLA NLP♠University of California, Los Angeles AWS♣AWS GenAI

†Equal advising

Findings of EMNLP 2026
Paper Code BibTeX OpenVLHarness
A fixed tool library fails on 'What event is associated with this bus?'; ToolSmith's generated Event In Image tool answers Manila hostage crisis
Overview

Hand-crafted tool libraries are frozen at deployment. ToolSmith lets a multimodal search agent grow its own tools from its failures.

Mined capability gaps become new, verified, reusable Python tools that lift accuracy across 13 benchmarks.

Fixed tool library

Eight base primitives. Asked which event a bus is linked to, the agent searches generic bus accidents and answers the wrong one.

ToolSmith library

The same primitives plus generated tools. Event In Image identifies the bus first, then finds the event: the Manila hostage crisis.

Abstract

Recent multimodal search agents use reasoning, vision, and live web search across many turns to answer questions beyond parametric knowledge by orchestrating a small library of hand-crafted search and visual tools. However, the library is laborious to create and maintain, and frozen when new query types, evidence formats, and failure modes are encountered in the wild. Human practitioners, by contrast, cope with insufficient tools by fashioning new ones from familiar parts and reusing them across tasks. Inspired by this, we propose ToolSmith, an automatic tool-generation framework that grows and reuses modules on top of a lean base tool library. We evaluate ToolSmith on three in-domain multimodal search benchmarks and ten out-of-domain benchmarks under a VLM-as-a-Judge protocol. ToolSmith consistently improves over the base library across deployed agents of varying scales, with larger backbones reaping greater gains, demonstrating the scalability of the approach. Our code and data will be publicly available.

Highlights

Generated Tools Lift Every Backbone

13
benchmarks: 3 in-domain and 10 out-of-domain
36/39
backbone × benchmark cells improved
3
inference backbones: Qwen3-VL-8B, Qwen3-VL-32B, GPT-5
+2.98
largest average gain (out-of-domain, Qwen3-VL-32B)
Average Accuracy, Base Tools vs. + ToolSmithGPT-5 as the tool generator · VLM-as-a-Judge
Base tools+ ToolSmith
Method

A Four-Phase Forge

ToolSmith runs a four-phase pipeline on top of an eight-primitive base library, then routes the new tools to the agent at inference. Click a phase, or let it play.

Show the full framework figure
ToolSmith framework: trajectory mining, reuse-or-grow, tool generation, two-gate verification, and inference with new tools
Results

Consistent Gains over a Strong Base-Tool Agent

Across 3 in-domain and 10 out-of-domain benchmarks, tools generated by ToolSmith (GPT-5 generator) consistently improve over an already strong base-tool agent.

Inference modelIn-domain avgOut-of-domain avg
Qwen3-VL-8B47.94→50.14+2.2026.57→27.90+1.33
Qwen3-VL-32B55.47→57.14+1.6731.27→34.25+2.98
GPT-551.80→52.84+1.0439.80→42.20+2.40

Each cell: base tools → base tools + ToolSmith. Gains appear in 36 of 39 backbone × benchmark cells.

Strongest generator wins

The tool-generation model should be as strong as possible.

Mine with the deployed model

The trajectory-mining model should match the inference model.

Quality vs. coverage

A cluster-count sweep exposes a quality–coverage trade-off.

Examples

Generated Tools in Action

Each case pairs a base-tool failure with the ToolSmith run on the same question. Click the figure to enlarge it.

Image
Citation

BibTeX

@inproceedings{sun-etal-2026-toolsmith,
    title = "{T}ool{S}mith: An Automatic Tool-Generation Framework for Multimodal Search Agents",
    author = "Sun, Rui  and
      Huang, Junhua  and
      Zancato, Luca  and
      Islam, Md Mofijul  and
      Chang, Kai-Wei",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EMNLP} 2026",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.findings-emnlp.1699/",
    doi = "10.18653/2026.findings-emnlp.1699",
    pages = "36305--36324"
}