Evaluation results
Evaluation results

Quality equivalent to Gemini 3.8 Flash, at 6.5× lower cost

Two workflows were evaluated on sample cases. Every response was scored programmatically, across 3 runs.

  • Quality · Hitachi model · by Dvar
    100%
    of merchant support responses passed every check
    Gemini 3.8 Flash 99.9% · Claude Sonnet 5.5 99.7%
  • Cost · Hitachi model · by Dvar
    ₹14,000
    per 10 lakh merchant support responses
    Gemini 3.8 Flash ₹94,000 · Claude Sonnet 5.5 ₹2,90,000
  • Disputes · Hitachi model · by Dvar
    100%
    of dispute recommendations passed every check
    Gemini 3.8 Flash 100% · Claude Sonnet 5.5 100%

Merchant support

324 queries: 27 query types in 12 languages
ModelAll checks passedQuery understoodCorrect next actionCorrect amountMerchant's languageConciseCost per 1,000 responsesMedian response time
Hitachi model · by Dvar100% (972 of 972)100%100%100%100%100%₹142.7 s
Gemini 3.8 Flash99.9% (971 of 972)100%100%100%99.9%100%₹94 · 6.5×3.1 s
Claude Sonnet 5.599.7% (969 of 972)99.8%99.8%99.8%99.7%99.8%₹285 · 20×2.7 s
Merchant support, by languageEnglishHinglishHindiMarathiGujaratiBengaliTamilTeluguKannadaMalayalamPunjabiOdia
Hitachi model · by Dvar100%100%100%100%100%100%100%100%100%100%100%100%
Gemini 3.8 Flash100%100%100%100%100%100%100%100%100%98.8%100%100%
Claude Sonnet 5.5100%98.8%100%98.8%98.8%100%100%100%100%100%100%100%

Dispute operations

48 disputes across five reasons, POS and ATM
ModelAll checks passedCorrect decisionCorrect refund amountCase reference citedConciseCost per 1,000 responsesMedian response time
Hitachi model · by Dvar100% (144 of 144)100%100%100%100%₹7.51.6 s
Gemini 3.8 Flash100% (144 of 144)100%100%100%100%₹82 · 11×3.1 s
Claude Sonnet 5.5100% (144 of 144)100%100%100%100%₹223 · 30×2.6 s

Methodology

  • Merchants, accounts, disputes and decision rules are sample data created for this evaluation. No Hitachi data was used.
  • Hitachi model · by Dvar is the highest-scoring open model on each workflow, served from Dvar's India-hosted API at list price. No custom training has been applied.
  • All models passed every dispute case, so that evaluation differentiates on cost and response time only.
  • The language check verifies the script and does not distinguish Hindi from Marathi. Evaluated 18 Nov 2026.

Impact of custom training

A trained 0.5B model reached 88.1% accuracy; Claude Sonnet 4.5 reached 81.2%.

Hinglish banking messages, at 64× lower cost and 65 ms response time. Public research by Dvar, not a bank deployment: dvarlabs.ai/banking77