formsieve Docs Support
All docs pages

Calibration

Formsieve's thresholds are set for jev-1.13.0. This page explains what that means, how Formsieve's service reports the Jev version that answered, and what Formsieve does when it answers with a version it does not know. Last checked: 2026-09-26.

Calibrated probabilities and calibrated thresholds

Jev returns calibrated probabilities rather than a bare yes or no. TypeSafe notes that calibration "is measured across groups of predictions; it does not guarantee that an individual answer is correct", and recommends pinning the version you tuned your thresholds on. A probability is only comparable with a threshold for the model version the threshold was chosen on. A new Jev version can shift scores, so Formsieve records the model name every response reports and checks it against a list of versions its presets were set for (the calibration identities):

Reported model Where it comes from Calibration identity
typesafe/jev-1.13-20260917 Formsieve's service, which asks OpenRouter for typesafe/jev-1.13 jev-1.13.0
jev-1.13.0 TypeSafe directly (used only in Formsieve's own evaluation) jev-1.13.0

A Jev version is added to this list only in a Formsieve update, after our evaluation set has been run on it.

The version Formsieve's service asks for

Formsieve's service always asks for typesafe/jev-1.13: the version is pinned at the minor version (1.13), and the reply names the exact snapshot, so the status you see is Calibrated (as of 2026-09-26). You cannot choose the model in the plugin. When a new Jev version is evaluated, Thinking42 decides when the service moves to it, and a plugin update adds it to the calibrated list.

Test connection shows the status, and the dashboard shows the status of recent checks.

When a route answers with an unknown version

If a response reports a model that is not on the list (for example a future jev-1.14.0), the status becomes Unrecognized. Formsieve then:

  • keeps filtering with your current thresholds;
  • shows a notice on its own screens saying that the API answered with a model version it was not calibrated on;
  • lists every unrecognised model name under Settings → Advanced → Unrecognized model versions;
  • records the model and its status on every log row, and in the Formsieve note of each entry that gets one ("Model … (unrecognized) …").

The notice stays until you click Hide until another new model appears (in the notice, or Hide the notice until another new version appears under Settings → Advanced); it comes back when yet another unrecognised model answers. A later Formsieve update that adds the version to its calibrated list does not remove the notice or the list by itself.

If you prefer caution, switch on Strict calibration (Settings → Advanced): while the model is unrecognised, submissions that would have been blocked go to the review band instead, so they are delivered and tagged rather than marked as spam.

How the presets were chosen

The presets (review / block: Low 0.65 / 0.92, Medium 0.50 / 0.85, High 0.40 / 0.75) are starting points taken from TypeSafe's guidance and from public evaluation data for jev-1.13.0, which is English e-mail rather than form submissions. Formsieve's own evaluation set adds contact-form spam, legitimate enquiries that look commercial, very short messages, prompt-injection attempts and a small Spanish sample; it supports no accuracy claim until it has been run against the live service.

Before a release, the Medium block threshold must pass a certification on a frozen set of real, legitimate contact-form messages that is never used for tuning: it passes only if the one-sided 95% upper confidence bound on the false-positive rate (Clopper-Pearson) is at most 1.5%, which needs at least 315 legitimate messages. Prompt-injection examples must never be allowed, and the "hard" legitimate examples must never be blocked. The evaluation is to be re-run for every new Jev version and every change to the questions Formsieve asks. The first run is below (2026-09-27).

Results: run of 2026-09-27 (Formsieve 1.1.0, jev-1.13.0)

Measured on 2026-09-27 on a frozen set of 2,217 real submissions from the contact forms of two small-business websites in Miami (a pet-sitting service and a web design agency), with names, e-mail addresses and phone numbers removed first. The set was labelled before any score existed and never used for tuning. Route: TypeSafe's direct API, model jev-1.13.0.

  • Legitimate messages: 298 tested. At the Medium preset (block at 0.85), none was blocked and none was sent to review; the highest score any of them received was 0.25. The one-sided 95% upper confidence bound on the false-positive rate is 1.01%. 124 of the 298 had no message text (bookings), which the trivial-content pre-filter allows without an AI call; counting only the 174 that reached the AI, the bound is 1.71%, so the AI step on its own has not yet cleared the 1.5% gate above. That needs at least 25 more real legitimate messages with text, scored in a later, dated run.
  • Spam: 1,919 tested. At Medium, 1,716 were blocked (89.4%; 95% interval 87.9% to 90.8%) and 110 more were sent to review. By kind: sales pitches and vendor outreach 496 tested, 99.2% blocked; scams and phishing 90.6%; bot-filled gibberish 17.6% (the weak spot; those submissions carry no pitch and no link). F1 at Medium: 0.944.
  • Languages: the legitimate messages were English (276) and Spanish (24). No claim is made for Spanish or any other language: the Spanish sample is too small.
  • Labels: each submission was labelled "legitimate" or "spam" by two independent AI passes (99.7% agreement) and an adjudicator, and on 28 September 2026 the sites' owner reviewed a 64-entry sample (every blocked entry without links that could hide a short real message, the blocked entries mentioning the site's topic, and 40 legitimate entries) and confirmed every label. Two legitimate messages were left out because they contained a third party's details.
  • What this does not measure: the on-site checks (honeypot, timing, rate limit, blocklists) were not part of the run, and the route was TypeSafe direct rather than the Formsieve service. Both will be covered in the next run.

Formsieve makes no accuracy claim that has not been measured this way. The home page's "99%" is the sales-pitch figure above.

Non-English sites

When your first site language is not English, both thresholds move up by 0.05. TypeSafe says Jev handles other languages "but not equally well"; the shift is a precaution, not a measured correction, and costs some spam recall. Formsieve's interface is translation-ready; a copy already downloaded from formsieve.com includes Spanish (Spain), and WordPress.org copies get it once its language pack is published (as of 2026-10-08), but the translation is not an accuracy claim: we make no claim about filtering accuracy in any language other than English until it has been measured.