What a scheduled run checks
Three questions, asked again and again
Every run answers the same three questions about your application, and keeps the evidence for each answer so a change can be reviewed rather than argued about.
Continuous monitoring
Continuous monitoring for LLM apps and chatbots
Necio is a web app that tests your LLM application or chatbot on a schedule — daily, weekly or on demand — and reports what changed: behavioral drift, hallucinations, and the vulnerabilities an attacker would look for. No SDK, no code changes, and no traffic from your own users.
A one-time scan tells you how your chatbot behaved on the day you ran it. It cannot tell you what changed last Tuesday.
Drift
Behavioral drift: the same question, a different answer
The same question, asked again next week, gets a materially different answer. Necio compares each run against a stored baseline and reports what changed — a guardrail that stopped holding, a refusal that became compliance, a policy that quietly loosened.
Because the baseline is versioned, you can point at the run where the behavior changed and the model, prompt or configuration change that landed next to it. That is the difference between “the bot feels off” and a finding with a timestamp.
Hallucination
Hallucination: answers that are confidently wrong
Answers that invent facts, contradict the material they were given, or state confident detail that the application cannot know. Frontier models grade each answer against the objective and the supplied source, and unsupported claims are reported as findings with the answer that produced them.
This is graded, not keyword-matched. The grading models read the answer the way a reviewer would: did it stay on topic, did it stay inside what the application actually knows, did it stay polite and safe.
Vulnerabilities
Vulnerabilities: what an attacker would try first
Prompt injection, jailbreaks, system-prompt and data leakage, unsafe output handling, excessive agency, and tool or function misuse — tested against the OWASP Top 10 for LLM risk categories and mapped back to the category each finding belongs to.
Test prompts are written the way attackers write them — by openly available models, deliberately, rather than by a polite template library. Those models never touch your application; Necio sends every request itself, and every model in the pipeline runs under a zero-data-retention agreement.
The schedule
What actually happens on a scheduled run
1. Threat model once. You describe what the application does, who uses it, what data and tools it can reach. Necio builds the test objective from that description.
2. Run on a schedule. Daily, weekly, or on demand — a single prompt, a whole category batch, or a Maestro conversation where one model attacks and another defends. Scans run from the cloud, not from your users' sessions.
3. Grade every answer. Frontier models from different families score each response, so a single model's blind spot does not become your blind spot.
4. Compare with the last run. New findings, fixed findings, and changed findings are separated, with a report card score and a list of what to fix first.
5. Tell someone. Findings export as structured JSON events — Guardrail Failure, Prompt Injection Detected, Behavioral Drift, Policy Regression, Sensitive Data Disclosure, Unsafe Agent/Tool Behavior, Model Response Change — so they land in the SIEM your security team already watches.
Web app, not a script
Why a web app instead of an SDK or a notebook
Observability tools watch the traffic you already have. Necio actively probes the application on a schedule, with tests written to break it, and grades what comes back — so it finds the failure before a customer does, not after.
It also means the person who owns the chatbot can run it. Configure the target — website chat, Slack, Discord, Telegram, or an API endpoint — set the schedule, and read the report. The browser simulator and WebSocket testing cover applications that only exist behind a chat widget. No library in your build, no deploy, no instrumentation budget.
Results are kept as history: every run, every finding, every answer, with the evidence attached, so a reviewer six months later can see what was tested and what it said.
Coverage
What each finding looks like when it lands
Every run produces findings in the same shape, so they can be triaged, compared across runs, or forwarded to a SIEM without rework.
| Check | What triggers it | How it is reported |
|---|---|---|
| Behavioral drift | Answer changed against the baseline for the same test | Drift finding, with both answers side by side |
| Hallucination | Answer unsupported by the source or the objective | Graded finding with the unsupported claim quoted |
| Prompt injection | Instructions in user content override the system prompt | Prompt Injection Detected event |
| Data leakage | Secrets, system prompt or customer data appear in a response | Sensitive Data Disclosure event |
| Unsafe output | Model output is used in a context it should not reach | Guardrail Failure event |
| Agent and tool abuse | Tool calls or actions outside the granted scope | Unsafe Agent/Tool Behavior event |
| Policy regression | A behavior that previously passed now fails | Policy Regression event |
Necio tests against the OWASP Top 10 for LLM risk categories. It is not affiliated with, certified by, or endorsed by the OWASP Foundation.
Cost
Credits, not a subscription
Credit is bought in $5 increments and never expires. It is spent only when a test runs, so an idle month costs nothing and a weekly schedule costs what the weekly schedule actually uses. Scheduled runs stop cleanly when credit runs out: the report, the history and the findings stay available.
A mixed schedule — one full OWASP sweep a month, a drift check on your most sensitive prompts every week — is the usual pattern, and it is deliberately cheap to start.
Questions
Continuous monitoring, answered plainly
What does continuous monitoring mean for an LLM app?It means the same tests run again and again — daily or weekly — instead of once before launch. Each run is compared against a stored baseline, so an answer that changed after a model update, a prompt edit or a tool change shows up as a difference you can act on.
How is drift detected if LLM answers are never identical twice?Necio grades every answer against the objective you gave it and stores the score, the verdict and the evidence. Runs are compared on those graded outcomes rather than on exact wording, so normal variation does not raise an alert while a real change in behavior does.
Does it test for hallucination as well as security issues?Yes. Grading covers whether an answer stayed on topic, kept secrets, stayed polite and stayed supported by the material it was given. Answers that invent facts or contradict the supplied source are reported as hallucination findings alongside the security findings from the same run.
Do I need to install an SDK or change my application?No. Necio is a web app: you describe what your application does and who uses it, and Necio sends the test traffic itself — to a website chat widget, a Slack, Discord or Telegram bot, or an API endpoint. Your code does not change and no library is deployed into your application.
How often can tests run, and what happens when one fails?Schedules can run daily, weekly or on demand, over a single prompt or a whole category batch. Results land in a report card with the score, the findings and the remediation guidance, and findings can be forwarded to a SIEM as structured JSON events.
What does it cost?Usage credit, not a subscription. Credit is bought in $5 increments, never expires, and is only spent when a test actually runs. Ask us for $5 of free credit and run a real scan before you pay anything.
Start
Run one real scan before you decide
Tell us what your bot does and where it runs. We will add $5 of credit to your account and send a ten-minute quickstart, so the first thing you see is your own application's report card — not a demo.