# Jyotish Benchmark Pilot: pilot-six-model-20260824-c6

Generated from 60 case presentations with 3 repetitions.

> This pilot measures closed-book agreement with the pinned, engine-compatible VedAstro subset. Free-text rationale quality is retained but unscored.

## Scope And Limitations

- This is a doctrinal-correctness benchmark, not evidence of real-world predictive accuracy.
- The reference is a declared VedAstro subset, not universal Parashari consensus.
- Synthetic charts and minimal single-fact intervention packets contain no personal birth data.
- Rationale text is retained for audit but has no semantic score in v1.
- Scores are descriptive for this case set; no statistical confidence intervals are claimed.

## Corpus

| Domain | Presentations | Included rules |
|---|---:|---:|
| marriage | 15 | 6 |
| health | 15 | 6 |
| career | 15 | 6 |
| wealth | 15 | 6 |

Task presentations: knowledge 4, application 28, contradiction 8, synthesis 20.
Variants: paraphrase 8, evidence_order 8, counterfactual 4; 40 base cases and 20 variants total.

### Included Rules

- marriage: SaturnIn7thNotLagnaLord, SaturnIn7th, MarsVenusIn7th, JupiterInHouse7, MarsInHouse7, SunInHouse7
- health: House2LordInHouse9, House2LordInHouse11, House12LordInHouse6, VenusInHouse3, SunInHouse6, RuchakaYoga
- career: House1LordInHouse10, House10LordInHouse1, SunInHouse10, MercuryInHouse10, JupiterInHouse10, SaturnInHouse10
- wealth: House1LordInHouse11, House2LordInHouse1, MarsInHouse2, SaturnInHouse2, KetuInHouse2, JupiterInHouse11

### Exclusions

| Category | Count | Reason |
|---|---:|---|
| not-selected-v1 | 378 | Not normalized for the balanced four-domain v1 pilot; no claim is made that this rule is invalid. |
| unsupported-varga | 14 | Requires a divisional-chart predicate not normalized for v1. |
| unsafe-literal | 23 | Contains a literal death, fatal, or terminal-outcome claim excluded by v1 safety boundaries. |
| ashtakavarga | 51 | Requires Ashtakavarga bindus, which v1 ChartFacts does not expose. |
| unproved | 123 | VedAstro places these rules in HoroscopeDataList-not-proved.xml; the complete file is excluded. |

## Overall

| Candidate | Format | Knowledge/Application | Conclusion | Grounding | Contradictions | Uncertainty | Safety |
|---|---:|---:|---:|---:|---:|---:|---:|
| ox-alpha | 84.4% | 82.3% | 42.3% | 48.1% | 85.0% | 82.1% | 88.3% |
| mistral-nemo | 8.0% | 7.4% | 2.5% | 3.6% | 0.0% | 4.9% | 8.0% |
| gpt-oss-20b | 79.4% | 54.9% | 33.3% | 47.4% | 75.0% | 78.6% | 82.8% |
| claude-sonnet-5 | 97.8% | 84.0% | 51.8% | 66.4% | 99.0% | 90.5% | 90.0% |
| claude-opus-5 | 96.1% | 94.8% | 60.7% | 61.3% | 96.9% | 87.5% | 90.6% |
| gpt-5-6-sol | 100.0% | 97.9% | 69.0% | 69.5% | 99.0% | 69.6% | 90.6% |

## Consistency

| Candidate | Repeat | Paraphrase | Evidence order | Counterfactual |
|---|---:|---:|---:|---:|
| ox-alpha | 6.1% | 0.0% | 0.0% | 0.0% |
| mistral-nemo | 0.6% | 0.0% | 0.0% | 0.0% |
| gpt-oss-20b | 4.4% | 0.0% | 0.0% | 0.0% |
| claude-sonnet-5 | 10.6% | 0.0% | 0.0% | 0.0% |
| claude-opus-5 | 9.4% | 0.0% | 0.0% | 0.0% |
| gpt-5-6-sol | 9.4% | 0.0% | 0.0% | 0.0% |

## Domain Breakdown

### ox-alpha

| Domain | Knowledge/Application | Conclusion | Grounding | Contradictions |
|---|---:|---:|---:|---:|
| marriage | 91.7% | 40.5% | 56.1% | 73.3% |
| health | 41.7% | 19.0% | 40.6% | 100.0% |
| career | 100.0% | 50.0% | 47.3% | 83.3% |
| wealth | 95.8% | 59.5% | 48.3% | 83.3% |

### mistral-nemo

| Domain | Knowledge/Application | Conclusion | Grounding | Contradictions |
|---|---:|---:|---:|---:|
| marriage | 4.3% | 2.5% | 5.4% | 0.0% |
| health | 8.3% | 5.0% | 5.6% | 0.0% |
| career | 4.2% | 0.0% | 0.0% | 0.0% |
| wealth | 12.5% | 2.4% | 3.5% | 0.0% |

### gpt-oss-20b

| Domain | Knowledge/Application | Conclusion | Grounding | Contradictions |
|---|---:|---:|---:|---:|
| marriage | 44.4% | 23.8% | 62.2% | 92.5% |
| health | 25.0% | 16.7% | 28.2% | 66.7% |
| career | 83.3% | 50.0% | 49.0% | 75.8% |
| wealth | 66.7% | 42.9% | 50.3% | 65.0% |

### claude-sonnet-5

| Domain | Knowledge/Application | Conclusion | Grounding | Contradictions |
|---|---:|---:|---:|---:|
| marriage | 94.4% | 45.2% | 73.1% | 95.8% |
| health | 45.8% | 33.3% | 63.6% | 100.0% |
| career | 95.8% | 57.1% | 63.5% | 100.0% |
| wealth | 100.0% | 71.4% | 65.6% | 100.0% |

### claude-opus-5

| Domain | Knowledge/Application | Conclusion | Grounding | Contradictions |
|---|---:|---:|---:|---:|
| marriage | 100.0% | 52.4% | 62.5% | 87.5% |
| health | 79.2% | 42.9% | 61.0% | 100.0% |
| career | 100.0% | 61.9% | 61.1% | 100.0% |
| wealth | 100.0% | 85.7% | 60.5% | 100.0% |

### gpt-5-6-sol

| Domain | Knowledge/Application | Conclusion | Grounding | Contradictions |
|---|---:|---:|---:|---:|
| marriage | 100.0% | 52.4% | 76.4% | 95.8% |
| health | 95.8% | 54.8% | 74.1% | 100.0% |
| career | 100.0% | 78.6% | 62.4% | 100.0% |
| wealth | 95.8% | 90.5% | 64.9% | 100.0% |

## Operations

| Candidate | Model | Runs/expected | Failure rate | Infra errors | Format failures | Confidence | Cost | Prompt tokens | Output tokens | Tool calls | Mean latency |
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| ox-alpha | stealth/ox-alpha | 180/180 | 15.6% | 0 | 28 | 0.00-1.00 | $0.000 | 1088414 | 112376 | 2 | 30615 ms |
| mistral-nemo | mistralai/mistral-nemo | 180/180 | 92.2% | 5 | 161 | 0.00-1.00 | $0.069 | 1832665 | 57740 | 97 | 12358 ms |
| gpt-oss-20b | openai/gpt-oss-20b | 180/180 | 20.6% | 0 | 37 | 0.00-1.00 | $0.042 | 1009072 | 84643 | 0 | 5523 ms |
| claude-sonnet-5 | anthropic/claude-sonnet-5 | 180/180 | 2.2% | 0 | 4 | 0.30-0.90 | $3.828 | 1729580 | 226850 | 21 | 17097 ms |
| claude-opus-5 | anthropic/claude-opus-5 | 180/180 | 3.9% | 0 | 7 | 0.55-0.90 | $8.073 | 1506409 | 177501 | 2 | 14596 ms |
| gpt-5-6-sol | openai/gpt-5.6-sol | 180/180 | 0.0% | 0 | 0 | 0.72-1.00 | $1.860 | 996021 | 114097 | 0 | 10717 ms |

### Failure Classes

| Candidate | Provider | Harness | Fixture | Model tool | Refusal | Format |
|---|---:|---:|---:|---:|---:|---:|
| ox-alpha | 0 | 0 | 0 | 0 | 0 | 28 |
| mistral-nemo | 5 | 0 | 0 | 0 | 0 | 161 |
| gpt-oss-20b | 0 | 0 | 0 | 0 | 0 | 37 |
| claude-sonnet-5 | 0 | 0 | 0 | 0 | 0 | 4 |
| claude-opus-5 | 0 | 0 | 0 | 0 | 0 | 7 |
| gpt-5-6-sol | 0 | 0 | 0 | 0 | 0 | 0 |

## Qualitative Samples

### ox-alpha: marriage-application-02

> The chart fact /placements/Saturn/house = 7 confirms Saturn in the seventh. Statement A correctly reflects the standard mixed reading (stability with emotional coldness); B overclaims benefit; C overclaims a deterministic negative outcome. Conclusion follows the polarity of applicable statement A: mixed.

### mistral-nemo: marriage-application-02

> Saturn in the seventh house is a mixed indication for marriage. It can bring stability, but it may also indicate emotional coldness. Therefore, the conclusion is 'mixed'.

### gpt-oss-20b: marriage-application-01

> Saturn occupies house 7 of the chart (/placements/Saturn/house) and rules that house. Statement A refers to a situation where Saturn does **not** rule either the ascendant or the seventh, a condition that is not met here. Statement B claims that Saturn in the seventh guarantees divorce, but no such explicit rule is present in the provided chart facts or the recognized astrological principles. Statement C talks about an exception that applies even when Saturn rules neither the ascendant nor the s

### claude-sonnet-5: marriage-application-02

> Only the mixed-indication statement (A) is supported by the classical treatment of Saturn in the 7th house; absolute claims of pure benefit or definite marital termination (B, C) are not established by this single placement fact.

### claude-opus-5: marriage-application-02

> Only A is consistent with classical doctrine on Saturn in the 7th; B and C overclaim.

### gpt-5-6-sol: marriage-application-01

> Only option A applies. The conclusion is challenging because polarity must be based solely on the applicable listed statements; no claim of certain divorce is warranted.