Pinned
Topics is now GA on all plans.
Continuously find the patterns worth investigating across your production traffic.
00:00
Braintrust
@braintrust
The observability layer for production AI.
Joined August 2023
- AI governance is entering a new phase. The EU AI Act enforces legal accountability for any company with EU customers, and ISO 42001 compliance is becoming a requirement in enterprise procurement. The teams best prepared for these governance frameworks are the teams that already
00:00 - Every AI team is building with different frameworks, different model providers, and different languages. Braintrust is designed for this reality. Instrument once via SDKs or OpenTelemetry, and get consistent traces, evals, and debugging across your entire stack.
- Braintrust reposted
"What you do is you prioritize the top few benchmarks and then you probably bullshit the rest." We talk a lot about how AI is making coding easier for non-technical folks, but don't hear much about how the most elite engineers are delegating their most technically complex work.
00:00 - Because LLMs are non-deterministic, a sudden change in your eval score could just be due to variance. A binomial test tells you the probability of getting a certain result by chance, so you can be confident that an eval score isn't random.
00:00 - How does your team rank when it comes to shipping quality AI products? Braintrust's AI quality assessment maps your current practices to the next useful step, whether you're still manually checking outputs or already running online scores in production.
00:00Take the assessment → - When production issues hit, engineers need to search logs and identify problems in real time. Brainstore delivers query times under one second, even across terabytes of AI observability data. It's 23.9x faster at full text search and 3.73x faster at loading spans compared to
- Traces tell you how customers are using your agents. Topics groups those interactions into patterns so you can uncover opportunities for improvement. Braintrust is hosting a workshop on using Topics to identify customer use cases from production data and turning those
