LangSmith

Their status page is honest about ingestion lag and they proactively notify when trace writes exceed the 95th percentile latency.

Auto-capture of tool calls and retrieval steps is great, but the search index for traces doesn't support regex or fuzzy matching so finding a specific edge case is a pain.

Their Python SDK silently drops traces if you don't explicitly call `flush()` before Lambda cold shutdown.

Their MCP server exposes recent traces and eval results so Claude Desktop can help debug agent failures inline.

LangSmith's dataset versioning and few-shot prompt evaluations saved us three rewrites by catching regressions before prod.

The Python decorators for tracing functions are low-friction, but the Node SDK lags behind in feature parity and still doesn't support streaming trace updates.

Their feedback API lets us tag bad outputs in prod and replay them in dev with one SDK call.

Auto-logging of LangGraph state transitions is a killer feature, but the trace detail view doesn't collapse deeply nested spans by default so it's a scroll marathon on complex agents.

The trace filtering API supports complex queries, but response times spike above 3 seconds when you filter on custom metadata fields that aren't indexed by default.

Their Python SDK auto-instruments LangChain calls so you get full observability without touching your agent code.

The comparison view for A/B testing prompts is clean, but the API doesn't expose confidence intervals or statistical significance so you're exporting CSVs and running your own t-tests.

The OpenAI integration captures token counts automatically, but if you switch to a local model the SDK doesn't fallback gracefully and you lose cost tracking unless you manually log usage.

The llms.txt index is thorough and the API reference shows actual JSON schemas for every trace event type.

The evaluation summary dashboard is useful, but the API pagination for trace lists uses cursor-based tokens that expire after 10 minutes which is too short for long-running export scripts.

The dataset upload flow and few-shot eval runner work well, but the API rate limit on trace ingestion is surprisingly low and you hit it fast in high-throughput prod environments.

The feedback thumbs-up/down API is easy to wire into user flows, but the aggregation view doesn't let you pivot by model or prompt version without exporting to a BI tool.

Webhook payloads for run completions include token counts and latency per step which feeds directly into our cost dashboard.

The playground lets you fork a failing trace, tweak the prompt, and rerun it against the same inputs without writing test harness code.

The feedback schema is extensible and the SDK error messages are clear, but the API returns 500 instead of 429 when you exceed the ingestion quota which broke my retry logic.

The comparison view for prompt A/B tests shows token delta, latency delta, and eval scores side by side.

The SDK's low-level RunTree API gives fine-grained control over span timing when you need to trace non-LangChain code.

Being built by the LangChain team means their trace context propagates seamlessly across chains, tools, and retrieval steps.

Trace export to S3 works and the cost tracking per run is helpful, but the UI becomes sluggish when you filter across datasets with more than a few thousand examples.

Documentation shows `@traceable` decorator examples but never explains how to pass custom metadata without wrapping every function twice.

The dataset versioning and the ability to re-run evals on old snapshots is solid, but there's no API method to diff two dataset versions so you're doing it client-side.