Integrations

Tracepanel for Go records through the standard library: net/http, database/sql and log/slog. A router, driver or library built on them shows in the trace with no setup of its own. Other libraries have an integration module each:

Add a module with go get, at the same version as the core module:

go get github.com/tracepanel/tracepanel-go/gintrace@v0.10.0

Everything joins a trace through its context. Pass the request's context down to the database, the logger and the HTTP client. Work done without it isn't recorded.

RequestsCopy Anchor Link

tracepanel.WrapHandler traces each request. It names the request by its route, such as GET /books/{id}, the pattern ServeMux matched. On a ServeMux, the handler that served the request is a view span, named after its function, such as (*App).showBook.

Every response carries Tracepanel-Id, Tracepanel-Url and Server-Timing, and an HTML page gets the badge. The page is held until the handler returns, unless the handler flushes it. Any other response, such as JSON, a file or a stream, goes out as it is written. A response the handler takes over, such as a WebSocket, is its own. http.ResponseController reaches the connection.

A panic in a handler is an error of the trace, with your frames and the source around them. The request answers 500.

A request that carries a W3C traceparent continues the caller's trace, and the agent links the two. See Traces across apps.

Other routersCopy Anchor Link

Another router on net/http takes tracepanel.Middleware. It can't tell Tracepanel the route, so name it with Config.Route, as here for chi:

tracepanel.Setup(tracepanel.Config{
	Route: func(r *http.Request) string {
		return chi.RouteContext(r.Context()).RoutePattern()
	},
})
router.Use(tracepanel.Middleware)

Without it, a request is named by its path, such as GET /books/42.

Fiber runs on fasthttp, not net/http, and isn't supported yet.

ginCopy Anchor Link

router := gin.Default()
router.Use(gintrace.Middleware())

Each request is named by its route, such as GET /books/:id, with its handler as a view span. Pass c.Request.Context() down, or c itself with the engine's ContextWithFallback on.

EchoCopy Anchor Link

e := echo.New()
e.Use(echotrace.Middleware()) // echo5trace.Middleware() for Echo v5

Each request is named by its route, such as GET /books/:id. On Echo v4, its handler is a view span. An error a handler returns is answered by Echo's error handler inside the trace. Pass c.Request().Context() down.

The middleware of gin and Echo sets Tracepanel up when it records, and the badge names the router, such as Go 1.27 · gin. To give it a Config, call tracepanel.Setup before it.

SQLCopy Anchor Link

Your app opens its database as before. tracepanel.Driver gives sql.Open the name of the same driver with its statements recorded:

db, err := sql.Open(tracepanel.Driver("pgx"), dsn)

It works with any driver of database/sql, such as pgx's stdlib, lib/pq, go-sql-driver/mysql or SQLite. Whatever runs on the *sql.DB is recorded: your own queries, sqlx, sqlc, GORM or Bun. Wherever a driver's name goes, the name it returns goes too:

db, err := sqlx.Open(tracepanel.Driver("pgx"), dsn)

db, err := gorm.Open(postgres.New(postgres.Config{
	DriverName: tracepanel.Driver("pgx"),
	DSN:        dsn,
}))

For GORM, use this or the GORM plugin, not both.

tracepanel.WrapConnector wraps a driver's connector, for sql.OpenDB:

db := sql.OpenDB(tracepanel.WrapConnector(stdlib.GetConnector(*config)))

Each statement keeps its parameters, the rows it returned and the line of your code that ran it. The agent finds N+1 queries, duplicates and slow SQL from them.

The console runs EXPLAIN of a recorded SELECT through your app, on the database the query came from, in a transaction it rolls back. With a second database, give it a name, so EXPLAIN runs on the right one:

replicaDriver := tracepanel.Driver("pgx", tracepanel.DBAlias("replica"))
replica, err := sql.Open(replicaDriver, replicaDSN)

Two more options describe a database. tracepanel.DBSystem names its kind, such as "postgresql", when the driver's name doesn't tell it. tracepanel.DBName is the name shown with its queries.

TransactionsCopy Anchor Link

Each transaction is a span around its statements. The rest of its work, such as an HTTP call, a log record or a span, goes under it too when it runs with the transaction's context:

tx, err := db.BeginTx(ctx, nil)
ctx = tracepanel.TxContext(ctx, tx)

The agent then finds an HTTP call made inside a transaction. Once the transaction ends, work done with that context goes under the request again.

pgxCopy Anchor Link

pgx's own interface, a pgx.Conn or a pgxpool.Pool, takes a tracer from pgxtrace:

config, err := pgxpool.ParseConfig(dsn)
config.ConnConfig.Tracer = pgxtrace.Tracer()
pool, err := pgxpool.NewWithConfig(ctx, config)

Each statement is recorded as tracepanel.Driver records it. A batch's queries are recorded as they are read, and a CopyFrom as one statement. pgxtrace.Tracer takes the options tracepanel.Driver takes, such as tracepanel.DBAlias for a second database, and the console runs EXPLAIN on the database a query came from.

A transaction's statements go under it. For the rest of its work, pgxtrace.TxContext puts it there too:

tx, err := pool.Begin(ctx)
ctx = pgxtrace.TxContext(ctx, tx)

GORMCopy Anchor Link

GORM v2 takes a plugin from gormtrace:

go get github.com/tracepanel/tracepanel-go/gormtrace@v0.10.0

db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{})
err = db.Use(gormtrace.Plugin())

The plugin records each statement as tracepanel.Driver records it, with the line of your code that ran it, and each transaction, GORM's own around a create, update or delete too. Run the queries with the request's context, as in db.WithContext(ctx).Find(&books). A query without it isn't recorded.

The rows a query returned are there for Find, First, Pluck, Count and Scan, as in db.Raw(...).Scan(&rows). GORM tells its logger how many rows Scan read, so the plugin wraps gorm.Config.Logger to read it. A Session given a logger of its own after db.Use goes without. Rows() and Row() have no count: your code reads those rows from the *sql.Rows that database/sql hands over. With tracepanel.Driver as the dialector's DriverName, in place of the plugin, the driver counts them as it hands each row over.

The database's kind comes from the dialector, such as postgres, mysql, sqlite, sqlserver or clickhouse. gormtrace.Plugin takes the options tracepanel.Driver takes, such as tracepanel.DBAlias for a second database, and the console runs EXPLAIN on the database a query came from.

A transaction's statements go under it. For the rest of its work, gormtrace.TxContext puts it there too:

err = db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
	ctx := gormtrace.TxContext(ctx, tx)
	// ...
})

Use the plugin or tracepanel.Driver as the dialector's DriverName, not both. A database opened through tracepanel.Driver or tracepanel.WrapConnector is recorded already, and the plugin leaves it to it.

RedisCopy Anchor Link

rdb := redis.NewClient(&redis.Options{Addr: "localhost:6379"})
redistrace.Instrument(rdb)

redistrace.Instrument records the commands of go-redis v9. Each command is a span with its key, the database and the server, so the agent finds a command repeated for each item (N+1) and duplicates. A pipeline or a MULTI/EXEC is one span. Cluster and ring clients work too, without the server. A missing key is an answer, not a failure. Run each command with the request's context, as in rdb.Get(ctx, key).

LogsCopy Anchor Link

A value you print to look at goes through tracepanel.Dump, which the call tree always shows. See Dump.

slogCopy Anchor Link

slog records logged with the request's context join its trace, with the line that logged them:

slog.InfoContext(ctx, "order paid", "order_id", order.ID)

Tracepanel puts itself in front of the default logger's handler, so your output stays as it was. Each attribute is a field under extra., such as extra.order_id, and a group's attributes are named by its path, such as extra.user.id. A value under a secret-looking name is masked. A logger of your own takes tracepanel.WrapLogHandler in front of its handler:

logger := slog.New(tracepanel.WrapLogHandler(slog.NewJSONHandler(os.Stdout, nil)))

A record logged without the context, as with slog.Info, isn't recorded.

zapCopy Anchor Link

zap's loggers carry no context. zaptrace.Context(ctx) is a field that names it, on a record or on a logger, and writes nothing. Wrap the logger's core with zaptrace.WrapCore:

logger = logger.WithOptions(zap.WrapCore(zaptrace.WrapCore))
logger.Info("order placed", zaptrace.Context(ctx), zap.Int("order", id))

zerologCopy Anchor Link

zerolog takes a writer and a hook, and needs both. The hook sees the record's context, and the writer its fields:

logger := zerolog.New(zerologtrace.Writer(os.Stdout)).Hook(zerologtrace.Hook{})
logger.Info().Ctx(ctx).Int("order", id).Msg("order placed")

A record of zap or zerolog keeps its level, its message, its fields with secret-looking ones masked, and the line that logged it when the logger adds the caller (zap.AddCaller(), zerolog's Caller()). Your output stays as it was.

Outgoing HTTP callsCopy Anchor Link

Calls through http.DefaultClient are recorded with their method, address and status. For a client of your own, wrap its transport:

client := &http.Client{Transport: tracepanel.WrapTransport(nil)}

WrapTransport(nil) wraps http.DefaultTransport. Pass another http.RoundTripper to wrap that one. Make each call with the request's context, through http.NewRequestWithContext. A call without it, such as http.Get(url), isn't recorded.

Each call carries a traceparent, so the app it reaches links its trace to the call. When the panel's Profile asked for the request, its calls carry X-Tracepanel-Profile, so the apps down the chain profile their part.

gRPCCopy Anchor Link

server := grpc.NewServer(grpctrace.ServerOptions()...)

options := append(grpctrace.DialOptions(), grpc.WithTransportCredentials(creds))
conn, err := grpc.NewClient(target, options...)

Each call a server serves is a trace of its own, named Service/Method. A server's fault, such as Internal, Unavailable or a panic, fails it. A client's error, such as NotFound, doesn't. Each call a client makes is a span of the trace in its context, and carries a traceparent in its metadata, so the server links its trace to it.

A call with the x-tracepanel-profile metadata key, 1 as the HTTP header has it, asks the server to profile its trace. A call made in a profiled trace asks the same, so the servers down the chain profile their part.

The interceptors are there one by one too, for a chain: UnaryServerInterceptor, StreamServerInterceptor, UnaryClientInterceptor and StreamClientInterceptor. A recovery interceptor goes before them.

KafkaCopy Anchor Link

kafkatrace works with segmentio/kafka-go. Wrap the writer to record what it sends. A consumer hands each message to kafkatrace.Process:

writer := kafkatrace.WrapWriter(&kafka.Writer{Addr: kafka.TCP(broker), Topic: "orders"})
err := writer.WriteMessages(ctx, kafka.Message{Value: order})

msg, err := reader.FetchMessage(ctx)
err = kafkatrace.Process(ctx, msg, handleOrder)

The messages a call sends to a topic are one span, such as publish orders, and each message carries a traceparent. Each message handled is a task trace of its own, named after its handler function, or the topic for a function literal. It links to the trace that sent it, even in another app with Tracepanel. An error the handler returns, or a panic, fails the trace.

A message sent in a profiled trace carries a tracepanel-profile header, as the other SDKs' messages do, and the trace that handles it is profiled too.

What a request carriedCopy Anchor Link

The panel's Request tab shows what each request carried: its headers, query and cookies, the body as far as your handler read it, and the response's headers. The SDK never reads a body your handler didn't. It keeps up to 16 KB of it: JSON and forms with their secrets masked, a multipart form's fields and its files by name and size, text as it is, and binary left out.

A value under a name that looks secret, such as Authorization, password, sessionid or api_key, is masked before it is recorded, and none of it leaves your machine. A name is read word by word, split at camelCase, snake_case, kebab-case and dots, as every Tracepanel SDK reads it: authToken, AUTH_HEADER and X-Api-Key are masked, Author, tokenizer, token_type and hasPassword aren't, and tokens holding numbers, a model's counts, is shown. The same rule masks a log record's fields and a dump's values. TRACEPANEL_CAPTURE_REQUEST=0 keeps none of it.

ProfilesCopy Anchor Link

The panel's Profile button, or a call carrying X-Tracepanel-Profile: 1, records the next requests with a profile, which the Profile tab shows: a flame graph of where their CPU time went, from Go's own profiler, runtime/pprof. The request's goroutine carries a profiler label, and so do the goroutines it starts, so a profile holds only that request's samples, however many requests run at once. A task it starts with tracepanel.Go is profiled too. TRACEPANEL_PROFILE=1, or Config.Profile, profiles every trace.

Your code can profile the rest of the trace it runs in:

ctx = tracepanel.Profile(ctx)

The calls it makes and the tasks it starts with that context from then on ask the same. A trace profiled already stays as it is.

A profiled trace says why on its root span, in profile.reason: header for an X-Tracepanel-Profile header, cookie for the panel's button, parent when a profiled caller asked through a traced call or message, or started the task, setting for TRACEPANEL_PROFILE, and code for tracepanel.Profile.

Go's profiler samples 100 times a second of CPU time, so a request that takes less than 10 ms of CPU may have no sample. Time spent waiting, such as for the database or a lock, isn't in the profile, and the trace's spans show it. An app that runs a CPU profile of its own, such as /debug/pprof/profile, gets no Tracepanel profiles meanwhile.

Go's profiler hands its samples over only once it stops, and on macOS stopping it takes up to 100 ms, when nothing is sampled. So one profile runs on from one profiled trace to the next, and is cut once none has run for 200 ms, or every 2 s under steady load. Back-to-back traces lose no samples, and a profile reaches the console a moment after its trace. A trace that runs while a cut stops the profiler says for how long, in tracepanel.profile_gap_ms. Samples the profiler lost are counted as dropped.

Every profile has the garbage collector's runs while the trace ran, and how long they stopped the program.

MemoryCopy Anchor Link

The panel's Profile → Memory, X-Tracepanel-Profile: memory, or TRACEPANEL_PROFILE_MEMORY=1 for every profiled trace, adds what the heap did meanwhile, from runtime/metrics: the bytes it allocated, its highest level above the start and its level at the end.

Go's heap is the whole process's, and nothing the runtime tells of it is per goroutine: its heap profile's samples carry no labels. So these are the heap's numbers while the trace ran, the trace's own when it ran alone, and marked shared when another trace ran meanwhile. Garbage counts until the collector frees it, and there are no allocation sites.

The App tabCopy Anchor Link

The console's App tab shows Go, the router, your module and the modules it depends on, and the build's settings that hold no secret. -ldflags is never among them.

Not there yetCopy Anchor Link

Some things other SDKs record aren't in the Go SDK yet:

• Tasks in flight. tracepanel.Go and kafkatrace.Process record each run once it ends, and the Tasks page has no Now for them.
• Tests. There is no test helper to fail a test on an N+1.
• Fiber, and other servers that don't run on net/http.