LLM observability
Necio and LLM observability tools
LLM observability platforms such as Langfuse, Arize Phoenix and LangSmith record the traces your
application sends them: every prompt, response, tool call, token count and latency from real users. Necio works
from the other side. It sends its own test conversations to the deployed app on a schedule, grades every answer,
and reports drift and vulnerabilities, with no SDK and no access to your traces.
Two ways to watch an LLM app
Inside-out vs. outside-in
Observability platforms work from the inside. An SDK, an OpenTelemetry exporter or a proxy sits in your
application and records each real request: the prompt, the response, the tool calls, the tokens, the cost and
the latency. That record is what you need to debug a specific conversation or to see where money and time go.
Necio works from the outside. You describe what the application does and who uses it, and Necio sends its own
test conversations to the deployed app — a website chat widget, a Slack, Discord, Telegram or Microsoft Teams
bot, or an API endpoint — on a schedule. Every answer is graded and compared with the last run. Nothing is
installed in your application, and Necio never sees your users' traffic.
Infrastructure teams already know this split as APM versus synthetic monitoring: traces tell you what
happened to real requests, and scripted checks from outside tell you whether the service still works before
anyone complains. Necio is the second kind, for LLM behavior and security.
Side by side
What each approach sees
Both approaches are useful, and they answer different questions. Where Necio does not cover something, the
table says so.
| LLM observability platforms | Necio |
|---|
| Where the data comes from | Traces of your real traffic, sent by an SDK, OpenTelemetry or a proxy | Test conversations Necio sends to the deployed app |
| Setup | Instrument the application, or route model calls through a proxy | Describe the app and point Necio at its chat or API endpoint; no code changes |
| What it sees | Every real request: prompts, responses, tool calls, tokens, cost, latency | How the app answers a chosen set of risky and important questions, graded on every run |
| When a problem shows up | After real users trigger it, or when an evaluator scores live traffic | On the next scheduled run, whether or not a user has asked |
| Behavioral drift | Shifts in evaluation scores and topics across live traffic | Graded results compared with a stored baseline for the same tests |
| Prompt injection | Some platforms flag injection attempts that appear in your traffic | Sends injection, jailbreak and data-leak tests on purpose, mapped to the OWASP Top 10 for LLM |
| Cost, latency and tokens | Core feature | Not covered — use your observability tool |
| Debugging one bad conversation | Core feature: open the trace | Not covered for real traffic; each Necio finding keeps the test, the answer and the grade |
Together
Using Necio alongside Langfuse, Arize or LangSmith
The two fit together without any integration work.
1. Necio runs on a schedule. A daily or weekly run sends the same graded tests to your
deployed app.
2. A finding lands. A refusal turned into compliance, an answer started leaking an internal
detail, or a fact went wrong after a model update.
3. You open the trace. If your app is instrumented, Necio's test conversation was recorded
like any other conversation, so the trace for that time window can show the prompt, the retrieved context and
the tool calls behind the bad answer.
4. Your security team sees it too. Findings are forwarded to your SIEM as structured JSON
events.
Questions
Necio and observability, answered plainly
Is Necio an LLM observability tool?Not in the tracing sense. Observability platforms record the traces your application sends them through an SDK, OpenTelemetry or a proxy. Necio sits outside the application: it sends scheduled, graded test conversations to the deployed app and reports what changed. Think of it as synthetic monitoring for LLM behavior and security.
I already use Langfuse, Arize or LangSmith. Do I need Necio?They answer “what did my users' conversations do?” Necio answers “does the app still behave on the questions that matter, including the ones attackers ask?” Drift after a model update often shows up in traces only after real users have run into it; a scheduled Necio run reports it on the next run. The two are designed to run side by side.
Does Necio need an SDK, a proxy or access to my traces?No. You describe what the app does and point Necio at its website chat widget, Slack, Discord, Telegram or Microsoft Teams bot, or API endpoint. Nothing is installed in your application, and Necio never reads your traces or your users' traffic.
Datadog LLM Observability already flags prompt injection. Is that the same thing?No. It flags injection attempts that appear in your real traffic, which tells you someone tried. Necio sends injection and OWASP Top 10 for LLM tests on purpose, on a schedule, so you learn whether a guardrail still holds before an attacker finds out.
Start
Run one real scan before you decide
Tell us what your bot does and where it runs. We will add $5 of credit to your account and send a
ten-minute quickstart, so the first thing you see is your own application's report card — not a demo.