How it works
What the SDK does inside your app, what the agent does beside it, where traces live, and how Tracepanel keeps your data local.
Tracepanel has two parts. An SDK runs inside your app and records traces. The agent is a separate program that analyzes and stores them, and serves the console.
The SDK
The SDK is a library you add to your app. When your app starts, the SDK reads its settings and decides whether to turn on. If it is off, it stops there and changes nothing in your app.
When it is on, the SDK hooks into your web framework and into the libraries it supports. Each request opens a trace, and everything the request runs records into it. The trace collects these parts:
- SQL queries and transactions.
- The view, templates and cache calls.
- Calls to Redis, HTTP APIs, MongoDB and other systems.
- Sent emails.
- Log records.
- Errors, with their stack and the source of your frames.
When the response is ready, the SDK adds the route, the view and the status to the trace. It adds the user only if the view already loaded one, and it never queries for the user. Then a background thread sends the trace to the agent.
The SDK also adds headers to the response: Tracepanel-Id, Tracepanel-Url
and Server-Timing. The badge reads them. For an HTML page, the SDK also adds
the panel's script.
A background task run and a command each become a trace of their own. A task queued during a request shows in the request's call tree, and its run links back to that request.
In the Python SDK
The SDK is the tracepanel package, and it installs the agent with it. Where
a framework has a hook, the SDK uses it. Where there is none, it wraps a few
methods, such as the view call and template rendering. The trace lives in a
context variable, so async code records into it too. Tracepanel skips
servers, shells and workers among management commands. See
Integrations and ignore_commands in
Configuration.
In the JavaScript SDK
The SDK is the @tracepanel/js package, and it installs the agent with it.
It loads before your app and records through OpenTelemetry's
instrumentations, which come with it. See
Integrations.
The browser
A page can record itself in the browser too, with the JavaScript SDK. Each
view of the page and each action on it, such as a click that called the API,
becomes a trace with its calls, errors and console output. The page sends it
to the console on its own origin, at /__tracepanel__/api/ingest, and the
console passes it on to the agent. The console takes these traces as JSON
from this machine only, so a page on another site can't send to it. See
Pages in the browser.
The agent
tracepanel-agent is a separate program. Each project has one agent. The
first trace starts it, and it exits after 30 minutes unused. The agent keeps
running across the reloads of your dev server and across workers, so every
process of the project sends to the same agent. Several apps can share one
agent, each under its own service name.
The agent does the work your app doesn't:
- It finds the issues in each trace.
- It stores the traces.
- It serves the console, its API and the panel's script.
The badge shows a request's status and time at once. It shows the request's issues a moment later, once the agent has found them.
To run the agent in the foreground, with the console on http://127.0.0.1:8765/, start it from your project's directory:
tracepanel-agent --serve 127.0.0.1:8765tracepanel dev does the same, from the Python SDK or the JavaScript SDK.
See the commands of Python and
JavaScript.
Use it to watch the agent's output, or to read the traces after your app has stopped. If the agent can't run, the SDK prints the reason in the terminal. Requests still get their headers, but Tracepanel stores nothing.
OpenTelemetry
The agent also takes OpenTelemetry spans over OTLP/HTTP, on 127.0.0.1:4318.
That is where OpenTelemetry's exporters send spans unless told otherwise. A
service that exports its traces with any OpenTelemetry SDK shows in the
console with no Tracepanel SDK in it. The agent reads the spans by
OpenTelemetry's semantic conventions and finds the same issues in them. It
names the service's language and framework from the spans, such as Java and
Spring Boot, .NET and ASP.NET Core, Ruby and Rails, or Go and Gin.
--otlp HOST:PORT moves the address, and --otlp off turns it off. If the
port is in use, the agent says so and runs without it.
Traces across apps
A call from one app to another carries the W3C trace context. The agent links the caller's trace to the trace of the app it reached, whether each app uses a Tracepanel SDK or OpenTelemetry. The call tree shows the reached app's trace under the call, and the Services page draws an arrow from app to app.
The store
The agent keeps the traces in a SQLite file under .tracepanel/ in your
project, traces.v21.sqlite3. The number in the name is the store's format.
When a new version of the agent changes the format, it starts a new, empty
file, and the traces in the old one aren't carried over.
The agent keeps the newest 2000 traces and deletes older ones. One trace keeps at most 2000 spans and 1000 log records: its earliest spans, each with its parent, and its first records. The Request tab shows how many it didn't keep, under Not recorded. An issue keeps its count and the time it was first seen after its traces are deleted. The SDK's settings change that number and move the store. The trash button in the console deletes everything in it.
What runs in your app's process
Your app's process records the trace and sends it. It does no analysis and keeps no store. Collecting a trace adds little to a request, and a background thread sends the trace after the response is made. Tracepanel collects nothing when it is off.
Requests under /__tracepanel__/ go to the agent. Your app handles two of
them itself, because it has the database connections and the sessions:
EXPLAINof a query from the console. In the agent's own console, the agent sends it to the app over the connection the app sends traces on.- Switching the user from the panel, in a Django project.
Several projects on one machine
Each agent lists itself in a file in your user's cache directory, readable by you only. When your app calls another app on the same machine, and that app's project runs its own agent, the trace's Request tab links to the call's trace in the other project's console. The agents match the call by the W3C trace context the SDKs send with each outgoing HTTP call.
Clocks
The agent notes when each trace arrives. When the app's clock is more than 10
seconds ahead of the agent's, or a minute behind, the Request tab shows a
Clock note, since the trace's times may not line up with other apps or
with since:. A note for a clock behind can also mean the app sent the trace
late.
Privacy and safety
Tracepanel is a development tool, and its defaults keep it local.
It stays on your machine. The agent stores traces in .tracepanel/ in your
project and sends nothing anywhere. By default the console and the OTLP
intake listen on 127.0.0.1, and the agent takes traces from this machine
only.
Add the store to .gitignore. The traces hold development data, so keep
them out of your repository:
echo ".tracepanel/" >> .gitignoreIt masks secret-looking values. To help you debug, traces keep SQL
parameters for EXPLAIN, local variables in errors, email bodies and task
arguments. Tracepanel masks values in local variables, task arguments and log
records when their names look secret. That covers names with password,
secret, token, api_key, auth, cookie, session, csrf,
signature or private in them. Each kind of data has a setting that turns
it off.
It is off outside development. The SDK turns on only in development, such as in your framework's debug mode. Otherwise it installs nothing.
In the Python SDK
The settings capture_sql_params, capture_locals, capture_mail_body and
capture_task_args turn off each kind of data. Tracepanel turns on only with
DEBUG or the framework's debug mode, under tracepanel run, or with
TRACEPANEL_ENABLED=1. See Configuration.
In Django, switching the user needs DEBUG = True, even when you turned
Tracepanel on another way. It also needs django.contrib.auth and sessions.
The request must come from this machine or from an address in INTERNAL_IPS,
and from the page itself, as a same-origin call with Tracepanel's header. The
user_switch setting turns it off.