How answers are checked
Every answer passes the same rules before you see it, and each rule leaves something you can check. This page lists them, and then what Datagoat does not claim.
The rules
| Rule | What you can check |
|---|---|
| Learns only from recorded outcomes. A question names a yes/no column of your record; nothing else is read as the outcome. | The question's outcome_column; the Verdict's outcome |
Checked on rows it never saw. A fit holds rows back and measures itself on them: the latest rows when the record has time (out_of_time), a random share otherwise (holdout). Several rows of one case are held out together. |
quality.validation_scheme, quality.held_out |
Refuses rather than guesses. An answer needs both lifts at 1.5 or more; below that, it is refused, the same every time. With too few labeled rows (about 500) there is nothing to check, and the answer is not_yet. |
state, quality (Quality) |
| Shows its reasons. Every answered case lists up to four columns that moved its chance, each with the band its value fell in. | reasons, pattern, pattern_match |
| Deterministic. The same record, question and engine give the same numbers, byte for byte. | record.content_hash, core_hash; ask twice |
| Signed. Every answered question carries a Verdict signed with Ed25519, checkable with no call to Datagoat. | Verdicts |
| Closes the loop. Outcomes, actions and drift are recorded against the model that made the call. | Run it, Prove it |
What Datagoat does not claim
- Reasons are not causes. A reason or a pattern says what went together in your record. Changing it may not change the outcome; Did acting work? measures that, and even that comparison is observational.
- A chance is not a certainty. A
pof 0.7 means three in ten cases like this one did not have the outcome. - Past patterns are not future ones. A model describes the record it learned from. Watch drift and refit (Run it).
- The samples are synthetic. Numbers quoted from them show how Datagoat behaves, not how your data will.
- Held-out checks are on your record. They say how the model did on rows it never saw from the same source, not on a different business.
More: Known limits and Your data.