Evaluation results
Quality equivalent to Gemini 3.8 Flash, at 6.5× lower cost
Two workflows were evaluated on sample cases. Every response was scored programmatically, across 3 runs.
- Quality · Hitachi model · by Dvar100%of merchant support responses passed every checkGemini 3.8 Flash 99.9% · Claude Sonnet 5.5 99.7%
- Cost · Hitachi model · by Dvar₹14,000per 10 lakh merchant support responsesGemini 3.8 Flash ₹94,000 · Claude Sonnet 5.5 ₹2,90,000
- Disputes · Hitachi model · by Dvar100%of dispute recommendations passed every checkGemini 3.8 Flash 100% · Claude Sonnet 5.5 100%
Merchant support
324 queries: 27 query types in 12 languages| Model | All checks passed | Query understood | Correct next action | Correct amount | Merchant's language | Concise | Cost per 1,000 responses | Median response time |
|---|---|---|---|---|---|---|---|---|
| Hitachi model · by Dvar | 100% (972 of 972) | 100% | 100% | 100% | 100% | 100% | ₹14 | 2.7 s |
| Gemini 3.8 Flash | 99.9% (971 of 972) | 100% | 100% | 100% | 99.9% | 100% | ₹94 · 6.5× | 3.1 s |
| Claude Sonnet 5.5 | 99.7% (969 of 972) | 99.8% | 99.8% | 99.8% | 99.7% | 99.8% | ₹285 · 20× | 2.7 s |
| Merchant support, by language | English | Hinglish | Hindi | Marathi | Gujarati | Bengali | Tamil | Telugu | Kannada | Malayalam | Punjabi | Odia |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Hitachi model · by Dvar | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% |
| Gemini 3.8 Flash | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 98.8% | 100% | 100% |
| Claude Sonnet 5.5 | 100% | 98.8% | 100% | 98.8% | 98.8% | 100% | 100% | 100% | 100% | 100% | 100% | 100% |
Dispute operations
48 disputes across five reasons, POS and ATM| Model | All checks passed | Correct decision | Correct refund amount | Case reference cited | Concise | Cost per 1,000 responses | Median response time |
|---|---|---|---|---|---|---|---|
| Hitachi model · by Dvar | 100% (144 of 144) | 100% | 100% | 100% | 100% | ₹7.5 | 1.6 s |
| Gemini 3.8 Flash | 100% (144 of 144) | 100% | 100% | 100% | 100% | ₹82 · 11× | 3.1 s |
| Claude Sonnet 5.5 | 100% (144 of 144) | 100% | 100% | 100% | 100% | ₹223 · 30× | 2.6 s |
Methodology
- Merchants, accounts, disputes and decision rules are sample data created for this evaluation. No Hitachi data was used.
- Hitachi model · by Dvar is the highest-scoring open model on each workflow, served from Dvar's India-hosted API at list price. No custom training has been applied.
- All models passed every dispute case, so that evaluation differentiates on cost and response time only.
- The language check verifies the script and does not distinguish Hindi from Marathi. Evaluated 18 Nov 2026.
Impact of custom training
A trained 0.5B model reached 88.1% accuracy; Claude Sonnet 4.5 reached 81.2%.
Hinglish banking messages, at 64× lower cost and 65 ms response time. Public research by Dvar, not a bank deployment: dvarlabs.ai/banking77