Fixing Tool-Call Accuracy in vLLM

All Blog PostsOctober 20, 2026 · 8 min readJesse Donkervliet and Junhao LiSenior Software EngineersWhen I use an open-weight model, I see results that are nowhere close to what the quality benchmarks indicate.Software developer working with AI models these daysTool call accuracy is an important part of serving open-weight models. When your tool calls aren’t accurate, the model builds up on nonsensical context. This leads to low quality results, particularly for agentic workloads where multi-turn sessions are common.Sadly, many providers and widely-used inference engines don’t fully pass tool-call accuracy tests. For example, when we served Kimi K2.6 with vLLM version 0.28, we got the expected tool call schema only 92% of the time. In other words, 2 out of every 25 attempted tool calls failed.Fortunately, vLLM recently added support for constrained decoding, a technique to guarantee schema-adherence. This feature has then been released as a server-side flag in vLLM 0.31. By enabling it, you can get to 100% schema accuracy with the newer Kimi and DeepSeek models.In this blog post, we first describe tool call schema accuracy and why it’s important in serving agentic workloads. We then talk about constrained decoding and how it fixes the schema accuracy problem.Tool-call SchemasYou can enable an LLM to call an external tool in two steps: (1) inform the AI model about the existence of externally available tool calls and (2) provide the model with instructions on how to output structured tool-calls. For example, when connecting OpenCode to Kimi, OpenCode sends Kimi a JSON array describing available tools. One of the tools in this list is webfetch, which fetches Web page contents:[{ "type": "function", "function": { "name": "webfetch", "description": "- Fetches content from a specified URL\n- Takes a URL and optional format as input\n- Fetches the URL content, converts to requested format (markdown by default)\n- Returns the content in the specified format\n- Use this tool when you need to retrieve and analyze web content\n\nUsage notes:\n [content redacted for brevity]\n", "parameters": { "$schema": "https://json-schema.org/draft/2020-12/schema", "type": "object", "properties": { "url": { "type": "string", "description": "The URL to fetch content from" }, "format": { "type": "string", "enum": [ "text", "markdown", "html" ], "description": "The format to return the content in (text, markdown, or html). Defaults to markdown.", "default": "markdown" }, "timeout": { "type": "number", "description": "Optional timeout in seconds (max 120)" } }, "required": [ "url" ] } } }, ...] The inference engine then wraps this JSON description in LLM-specific tokens before passing it into the LLM, to match how the LLM was trained to recognize tool declarations. When serving Kimi, this looks something like:<|im_system|>tool_declare<|im_middle|>[{"type":"function","function":{...}, ...]<|im_end|>The JSON object describing the tool (or function, in the JSON above) includes the tool’s parameters, their types, which of the parameters are required, etc. This is known as the tool’s schema. For an agent harness (e.g., OpenCode) to successfully execute a tool call generated by the LLM, the call must follow the rules specified in its schema.Then, when a user asks to fetch a Web page via their agent harness, the LLM may output the following tokens:I'll fetch that page. <|tool_calls_section_begin|><|tool_call_begin|>functions.webfetch:0<|tool_call_argument_begin|>{"url": "https://ubicloud.com", "format": "markdown"}<|tool_call_end|><|tool_calls_section_end|>These tokens are parsed by the inference server’s parser, extracting the LLM-specific tokens and turning the call into a JSON object such as:{ "role": "assistant", "content": "I'll fetch that page.", "tool_calls": [ { "id": "functions.webfetch:0", "type": "function", "function": { "name": "webfetch", "arguments": "{\"url\": \"https://example.com\", \"format\": \"markdown\"}" } } ] }This is then sent back to your harness (OpenCode, in our example), which runs its webfetch tool and sends its output back to the model.The Importance of Schema AccuracyFor a tool-call to succeed, the call must adhere to the tool’s schema.The simple approach to tool calling is to let the model generate tool calls freely and trust that the model’s training is sufficient to generate correct calls. This is similar to how we trust the model to generate syntactically correct prose.The simple approach works most of the time because modern models are good at following structured patterns. But success “most of the time” implies that many tool calls can still fail, resulting in context window pollution due to error messages, lower model performance, and increased cost due to the agent having to retry the call one or more times.To avoid such experiences when using Kimi, Moonshot AI published their K2VV ToolCall benchmark. This benchmark lets providers verify that they serve Kimi with the same quality as the original, including its default behavior of only generating valid tool calls.K2VV ToolCall’s workload consists of 2000 requests selected to evaluate when the LLM generates tool calls and whether the generated tool calls are syntactically correct (i.e., adhere to the provided schema). The benchmark allows analyzing the model’s quality by quantifying two metrics: trigger similarity and schema accuracy. Trigger similarity considers when the service decides to trigger a tool call, and how similar its decisions are to Moonshot AI’s own service. Schema accuracy quantifies the fraction of tool-calls that adhere to the tool’s schema and can therefore be invoked successfully. For example, when tool-call accuracy is 50%, half of the tool calls generated by the model are malformed.Schema accuracy usually falls short due to the probabilistic nature of language models. A model generates tokens based on likelihood. It is not running a compiler or type checker. Even with a clear prompt, it can still produce malformed tool calls. This can be caused by, for example, a mismatch between the model’s training data and the current inference harness. Models are trained on many tool-calling datasets, each with its own tools, schemas, and conventions. As a result, a model may generate a tool or field name that it has seen during the training, even if it does not exist in the current production environment. For example, it may generate search_web when the available tool is web_search, or use city_name when the active schema expects location.Constrained DecodingAn LLM computes for each token the probability that it is the next in the sequence. It then selects the next token by performing a weighted sample. Constrained decoding avoids malformed tool calls by preventing models from sampling tokens that would result in a syntactically invalid tool call. When using constrained decoding, instead of letting the model sampling tokens freely and validating the result afterward, the inference engine guides generation step by step. At each step, only valid tokens are allowed.Which tokens are allowed is determined by the provided tool schema. vLLM implements this using structural tags from the XGrammar library. The snippet below shows an excerpt of how XGrammar implements constrained decoding in Kimi K2.6. It defines the valid structure of a tool call, including where the call begins, where the index appears, where the arguments begin, and where schema-constrained JSON must be generated.# Kimi K2.6 tool call syntax: # <|tool_call_begin|>{function.name}:{index} # <|tool_call_argument_begin|>{arguments} # <|tool_call_end|> TagFormat( begin=f"{TOOL_CALL_BEGIN_PREFIX}{name}{TOOL_CALL_SUFFIX}", content=SequenceFormat( elements=[ RegexFormat(pattern=r"\d+"), ConstStringFormat(value=TOOL_CALL_ARGUMENT_BEGIN), JSONSchemaFormat( json_schema=parameters, any_order=any_order, max_whitespace_cnt=max_whitespace_cnt, ), ] ), end=TOOL_CALL_END, )