Conducting A/B Testing On Sites

Following user feedback is a Product Management virtue. Is there an actual way to implement it, between all the noise, bugs, and stakeholder requests? Well… Most teams claim they are customer-driven. Yet the moment you open Zendesk, App Store reviews, survey results, and Slack threads, you instantly remember why everyone quietly avoids this work. Feedback is everywhere, contradictory, emotional, duplicated, and nearly impossible to turn into decisions.  It is chaos disguised as “insights.” This is why the new Amplitude AI Feedback release caught my attention and made it all the easier to decide to partner with them on this update. It successfully connects what users say with what they actually do, in one workflow. No extra tools.  No extra tabs. You see their words, frustrations, and praise. You see their behavior. And AI transforms it into ranked themes, rising trends, top requests, and complaints. Noise turns into clarity. Opinions turn into patterns. Patterns turn into action. And because it is native inside Amplitude, it kills the biggest problem in feedback work: Fragmentation. Everything flows into analytics, session replay, and cohorts, creating a full loop from insight to fix. You can trace why an issue matters, how many users care, how it impacts behavior, and which actions you should take. Finally, a single source of truth for PMs, UX, CX, and marketing. I’m also genuinely impressed with the supported sources of feedback: App Store, Google Play, Zendesk, Intercom, Freshdesk, Salesforce Service, Gong, Trustpilot, G2, Reddit, Discord, and X. Slack arrives in Q1, and there will be more! If you ever felt overwhelmed by feedback, this is one of the first attempts I have seen that genuinely solves the operational pain, not just the reporting part. It launches… Today! Take a look: https://lnkd.in/dAJKeTez What was the most successful update you know that came from the product’s users? Let me know in the comments. #productmanagement #productmanager #userfeedback

Customer discovery via functional prototypes + PostHog is night & day better than the old school way of asking for feedback on Figma mockups. Here's why: I get to observe actual user behavior instead of asking the user to guess how they might use my product. My favorite example of why this matters comes from a Sony Walkman user study. They asked a bunch of people what they thought about a yellow walkman and they said "so sporty! not boring like the black one!". And yet, when they were given the opportunity to take a walkman home after the study, everyone picked the black one. We learned a lot more from user behavior than we did expressed preferences. Here's my setup for now observing user behavior from prototypes: 1. Create a functional prototype in your favorite prototyping tool (Bolt, Lovable, Reforge Build, Magic Patterns, Claude Code) 2. Ask the prototyping tool to integrate PostHog analytics 3. Ask the prototyping tool to instrument key user actions in PostHog Then you get all of these ways of observing actual behavior: - DAUs \ WAUs \ retention curves - I can actually see if people come back and use my prototype instead of taking their word for it - Action metrics dashboards - I can see what actions people are taking vs not - Post-usage survey - I can add a built-in pop-up survey to ask the user a question about the experience after they have engaged with the prototype - Session replays - I can see exactly where people are clicking and how they are using the product to identify usability issues - Heatmaps - I can see what part of my design is working across all sessions I'd never go back to testing with just a mockup after this.

Everyone obsesses over AI benchmarks. Smart people track what actually matters. I analyzed 200+ AI deployments to find the metrics that predict real-world success. The crowd obsesses with: ❌ MMLU scores (academic tests) ❌ Parameter counts (bigger = better myth) ❌ Training FLOPs (vanity metrics) ❌ Benchmark leaderboards (gaming contests) Smart people track: ✅ Token efficiency ratios ✅ Hallucination consistency patterns ✅ Real-world failure rates ✅ Cost per useful output The data is shocking: GPT-4: 92% MMLU score, 34% real-world task completion Claude-3: 88% MMLU score, 67% real-world task completion Why benchmarks lie: → Test contamination in training data → Optimized for specific question formats → Zero real-world complexity → Gaming beats genuine capability The 4 metrics that actually predict success: 1. Hallucination Consistency → Does it fail the same way twice? → Predictable failures > random excellence 2. Token Efficiency → Value delivered per token consumed → Concise accuracy > verbose mediocrity 3. Edge Case Handling → Performance on 1% outlier scenarios → Robustness > average performance 4. Human Preference Alignment → Do people actually choose its outputs? → Usage retention > initial impressions Real example: Company A: Chose model with highest MMLU score → 67% user abandonment in 30 days Company B: Chose model with best token efficiency → 89% user retention, 3x engagement The insight: Benchmarks measure what's easy to test. Reality measures what's hard to fake. What hidden metric have you discovered matters most?

Very excited to share a new paper that has been a long time in the making. This has been a fun collaboration with my co-authors Ruoxuan Xiong (Emory) and Alex Chin (my co-worker at Lyft and now Motif Analytics). Randomized experiments are the gold standard for measuring causal effects, but in marketplaces we are often testing policies that have many plausible spillovers that make it difficult to learn what we need by assigning treatment across users. Instead we randomize over time. This type of experiment seems simple to design, you are implementing a square wave (a type of oscillator) that determines what policy you are running based on time. When I was at Lyft, we had some heuristics for choosing switchback parameters but we rarely had bandwidth to understand their impact. It turns out to be a rich design space, and by choosing how and when you switch policies, you control the bias and variance of the estimates from your experiment. Intuitively, faster switching yields lower variance by increasing your sample size but increases bias because effects tend to persist over time (carryover effects). Your measurements from each time period are also correlated and have heteroskedastic errors due to seasonality (marketplaces tend to have strong daily and weekly cycles). Our approach is effectively a model-based design process where we use historical data to estimate the inputs to the experimental design process. The data allow us to make informed decisions about switching behavior that will yield the lowest error in our estimates. Carryover effects are the hardest quantity to estimate from historical data because on any individual test they are quite noisy, so pooling is necessary to gain some additional precision. We analyze a corpus of hundreds of switchback tests from Lyft's marketplace, and cluster them into an interpretable distribution over impulse responses. A broader point of this research is that all experimental designs lean on prior knowledge to improve the chances of a successful experiment -- even choosing a sample size for desired power in a standard A/B test. In switchback tests, there is an important bias-variance tradeoff we must manage. Without some means to estimate the covariance of errors and the likely size and shape of carryover effects, it is difficult to design an experiment that is likely to be successful.

✈️ Week 2: When A/B testing meets airline reality Airline pricing experiments start with a clear idea, but the reality underneath is far more complex. Classic A/B tests depend on a clean separation between treatment and control, yet airline markets are full of interactions that make this separation hard to maintain. In most industries A/B testing relies on a simple idea. Customers in treatment should not affect customers in control. This principle is known as the Stable Unit Treatment Value Assumption, or SUTVA. This assumption matters because it creates a clean comparison. If nothing spills over between groups, differences in outcomes can be attributed to the treatment itself. That is the foundation of valid causal inference. In airlines this clean separation breaks almost instantly. ──────────────── ⚠️ The usual assumptions fall apart because: • Shared seat inventory Customers draw from the same pool of seats. If one group books earlier, fare availability changes for everyone else. It matters because Revenue Management systems use the number of seats left as a core input in their forecasts and recommendations. • Network interactions across routes Changes on one OD (like AMS JFK) influence others. Connections, shared legs, and passenger substitution across nearby ODs create natural spillovers. • Competitive reactions A price test does not happen in a vacuum. Competitors respond, and both treatment and control see a changed market. • Booking curve timing effects Demand follows a time pattern. Early bookings in one group change the environment for the other later in the curve. Revenue Management forecasts are built on booking curves, so timing matters. These interactions mean that airline experiments rarely behave like clean, independent A/B tests. Standard assumptions simply do not hold. ──────────────── 🚀 So how do we run experiments anyway? We design experiments that reduce interference instead of pretending it does not exist. • Randomize at the flight or route level, not the customer level. • Use pods to keep spillovers within controlled clusters. • Use switchback designs where the entire system alternates between treatment and control. • Adjust statistical models to reflect the structure of interference. Airline pricing experiments require designs that match the reality of shared inventory, network spillovers, and competitive dynamics. In the coming weeks we will explore these issues in more detail and show how to design experiments that remain reliable even when these challenges are present. These are the same principles we apply in our work with airlines at ADC Consulting. #ADCConsulting #AirlinePricing #CausalInference #Experimentation

AI models in medical imaging often boast high accuracy, but are we measuring what really matters? 1️⃣ Many AI models are judged using metrics that do not match clinical goals, like relying on AUROC (area under the receiver operating characteristic curve, which shows how well the model separates classes) in imbalanced datasets where rare but critical findings are overlooked. 2️⃣ A single metric such as accuracy or Dice can be misleading. Multiple, task-specific metrics are essential for a robust evaluation. 3️⃣ In classification, AUROC can stay high even if a model misses rare cases. AUPRC (area under the precision-recall curve, which focuses on the model's performance on the positive class) is more useful when positives are rare. 4️⃣ For regression, MAE (mean absolute error, the average size of prediction errors) and RMSE (root mean squared error, which gives more weight to large errors) do not reflect how serious the errors are in real clinical settings. 5️⃣ In survival analysis, the C-index (concordance index, which measures how well predicted risks match actual outcomes) and time-dependent AUCs (area under the curve at specific time points) each reflect different things. Using the wrong one can mislead. 6️⃣ Detection models need precision-recall metrics like mAP (mean average precision, which combines detection quality and location accuracy) or FROC (free-response receiver operating characteristic, which shows sensitivity versus false positives per image). Accuracy is not useful here. 7️⃣ Segmentation metrics like Dice (which measures the overlap between predicted and true regions) and IoU (intersection over union, the overlap divided by the total area) can miss small but important errors. Visual review is often needed. 8️⃣ Calibration means checking if predicted risks match observed outcomes. ECE (expected calibration error, the average gap between predicted and actual risks) and the Brier score (the mean squared difference between predicted probability and actual outcome) help assess this. 9️⃣ Foundation models need extra checks: generalization (how well they perform across tasks), label efficiency (how few labeled examples they need), and alignment across inputs and outputs. Zero-shot means no examples were given before testing. Few-shot means only a few examples were used. 🔟 Metrics must fit the clinical context. A small error in one use case may be acceptable, but the same error could be dangerous in another. ✍🏻 Burak Kocak, Michail Klontzas, MD, PhD, Arnaldo Stanzione, Aymen Meddeb MD, EBIR, Aydin Demircioglu, Christian Bluethgen, Keno Bressem, Lorenzo Ugga, Nate Mercaldo, Oliver Diaz, Renato Cuocolo. Evaluation metrics in medical imaging AI: fundamentals, pitfalls, misapplications, and recommendations. European Journal of Radiology Artificial Intelligence. 2025. DOI: 10.1016/j.ejrai.2025.100030

Cracking a GenAI Interview? Be Ready to Talk LLM Quality & Evaluation First If you’re walking into a GenAI interview at an enterprise, expect one theme to dominate: “How do you prove your LLM actually works, stays safe, and scales?” Here’s a practical checklist of evaluation areas you must be know for sure: 1. Core Model Evaluation • Accuracy, Exact Match, F1 for structured tasks. • Semantic similarity scores (BERTScore, cosine). • Distributional quality (MAUVE, perplexity). 2. Generation Quality & Faithfulness • Hallucination detection via NLI/entailment. • Groundedness in RAG with RAGAS metrics. • Multi-judge scoring: pairwise preference, rubric-based evaluation. 3. RAG & Contextual Systems • Retrieval metrics: Recall@k, MRR, nDCG. • Context efficiency: % of tokens in window that actually matter. • Hybrid retrieval performance (vector + keyword). 4. Alignment & Safety • RLHF limits and failure modes. • Safety tests: toxicity, jailbreak success rate, PII leakage. • Human-in-the-loop QA for high-risk cases. 5. Agentic & Multi-Step Workflows • Tool-use accuracy and recovery from errors. • Success rate in completing tasks end-to-end. • Multi-agent orchestration challenges (deadlocks, cost spirals). 6. LLMOps (Enterprise Grade) • Deployment: FastAPI + Docker + K8s with rollback safety. • Monitoring: hallucination rate, latency, prompt drift, knowledge drift. • Drift detection: prompt drift, data drift, behavioral drift, safety drift. • Continuous feedback: synthetic test sets + human eval loops. 7. MCP (Model Context Protocol) • Why interoperability across tools matters. • How to design fallbacks if an MCP tool fails mid-workflow. 🔑 Interview Tip: Don’t just name metrics. Be ready to explain why they matter in production: • How do you detect hallucination at scale? • What do you monitor beyond tokens/sec? • How do you know when your RAG pipeline is drifting? 👉 If you can answer these clearly, you’re not just “LLM-ready.” You’re enterprise-ready.

I’m working on 5 Behavioural Science experiments across 𝟱𝟳𝟬𝗸+ 𝗽𝗲𝗼𝗽𝗹𝗲 today. An experiment, specifically a 𝗿𝗮𝗻𝗱𝗼𝗺𝗶𝘀𝗲𝗱 𝗰𝗼𝗻𝘁𝗿𝗼𝗹𝗹𝗲𝗱 𝗲𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁 (𝗥𝗖𝗘), is the gold standard for testing whether a drug 💊or a vaccine 💉works. Scientists use randomised controlled trials to 𝗶𝗻𝗳𝗲𝗿 whether a drug is 𝗰𝗮𝘂𝘀𝗶𝗻𝗴 𝘁𝗵𝗲 𝗵𝗲𝗮𝗹𝗶𝗻𝗴 𝗲𝗳𝗳𝗲𝗰𝘁 for which they designed it.  In my work as a behavioural strategist, I always prefer to test an intervention (aka treatment) 𝘁𝗼 𝗲𝘀𝘁𝗮𝗯𝗹𝗶𝘀𝗵 𝗰𝗮𝘂𝘀𝗮𝗹 𝗲𝘃𝗶𝗱𝗲𝗻𝗰𝗲 𝗯𝗲𝘁𝘄𝗲𝗲𝗻 𝘁𝗵𝗲 𝗶𝗻𝘁𝗲𝗿𝘃𝗲𝗻𝘁𝗶𝗼𝗻 𝗮𝗻𝗱 𝘁𝗵𝗲 𝘁𝗮𝗿𝗴𝗲𝘁 𝗰𝗵𝗮𝗻𝗴𝗲 𝗶𝗻 𝗯𝗲𝗵𝗮𝘃𝗶𝗼𝘂𝗿. For instance, if I’m adjusting the buying journey, I’d like to test whether the adjustment is causing the shift in buying behaviour before I throw funds into scaling the adjustment.🤔 I also want to avoid 𝗳𝗮𝗹𝘀𝗲 𝗽𝗼𝘀𝗶𝘁𝗶𝘃𝗲𝘀❌ i.e., conclude that the adjustment works when I just got lucky. Many loosely refer to RCEs as an ‘A/B test’ (or ‘A/B/n test’ if there is more than one intervention to test). I’m careful about using those terms because many of these tests have sloppily disregarded an RCE cornerstone – 𝗿𝗮𝗻𝗱𝗼𝗺𝗶𝘀𝗮𝘁𝗶𝗼𝗻. Why is randomisation important? Because without randomisation, we end up 𝗯𝗶𝗮𝘀𝗶𝗻𝗴 𝗼𝘂𝗿 𝗿𝗲𝘀𝘂𝗹𝘁𝘀.😮 There are 𝟯 𝗽𝗼𝗶𝗻𝘁𝘀 in an RCE relevant to randomisation that I’d like to highlight: selection, allocation, and intervention delivery. 1️⃣& 2️⃣𝗦𝗲𝗹𝗲𝗰𝘁𝗶𝗼𝗻 𝗮𝗻𝗱 𝗔𝗹𝗹𝗼𝗰𝗮𝘁𝗶𝗼𝗻. Randomly select individuals from your population to create a representative sample for the experiment and randomly allocate them to your treatment and control groups. Both the 𝘀𝗲𝗹𝗲𝗰𝘁𝗶𝗼𝗻 𝗮𝗻𝗱 𝗮𝗹𝗹𝗼𝗰𝗮𝘁𝗶𝗼𝗻 𝘀𝗵𝗼𝘂𝗹𝗱 𝗯𝗲 𝗿𝗮𝗻𝗱𝗼𝗺. Selecting and allocating by the first letter of the last name, by the order individuals are stored in a database, or by the city they live in aren’t random. Any time you follow a pattern, you throw randomisation out and bring bias in. 3️⃣𝗜𝗻𝘁𝗲𝗿𝘃𝗲𝗻𝘁𝗶𝗼𝗻 𝗗𝗲𝗹𝗶𝘃𝗲𝗿𝘆. Ideally, you should deliver the intervention and control (baseline) treatments to participants at the same time. But, sometimes, this is not possible. For instance, if you have to broadcast a treatment message to 𝟱𝟬𝟬𝗸 𝗽𝗮𝗿𝘁𝗶𝗰𝗶𝗽𝗮𝗻𝘁𝘀 and your messaging system only allows you 𝟭𝟬𝗸 𝗺𝗲𝘀𝘀𝗮𝗴𝗲𝘀 𝗽𝗲𝗿 𝗵𝗼𝘂𝗿, you must randomly sequence your broadcast. You don’t want a particular treatment group A to receive your message at 7 a.m., while treatment group F receives your message at 7 p.m., to avoid broadcast time from biasing your results (unless broadcast time is a treatment in itself). As Matteo Maria Galizzi, my mentor from The London School of Economics and Political Science (LSE) taught me, 𝗻𝗼 𝗮𝗺𝗼𝘂𝗻𝘁 𝗼𝗳 𝗱𝗮𝘁𝗮 𝘀𝗰𝗶𝗲𝗻𝗰𝗲 𝗰𝗮𝗻 𝗳𝗶𝘅 𝗮 𝗳𝗮𝘂𝗹𝘁𝘆 𝗲𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁𝗮𝗹 𝗱𝗲𝘀𝗶𝗴𝗻. #behavioraleconomics #behavioralscience  #behavioraldesign

*** Model Validation *** Model validation is critical in developing any predictive model—it’s where theory meets reality. At its core, model validation assesses how well a statistical or machine learning model performs on data it hasn’t seen before, helping to ensure that its predictions are accurate and reliable. This step is especially essential in high-stakes domains like finance, healthcare, or credit risk, where decisions based on flawed models can have significant consequences. **Precision** - **Definition**: This metric measures how many of the model's positive predictions were correct. - **Use Case**: Precision is crucial when false alarms are costly, such as in credit card fraud detection cases. **Recall (Sensitivity)** - **Definition**: Recall indicates how many actual positives the model successfully identified. - **Use Case**: It is imperative when failing to detect positives, as it can have serious consequences, such as cancer detection. **F1-Score** - **Definition**: The F1-Score combines precision and recall into a single metric, offering a balanced view of the model’s performance. - **Use Case**: This metric is ideal in scenarios where class imbalance can mislead accuracy, as is often true in fraud or rare event detection. **AUC (Area Under the ROC Curve)** - **Definition**: The AUC measures the model's ability to distinguish between classes across all decision thresholds. - **Range**: It ranges from 0.5 (indicating no better than random chance) to 1.0 (indicating perfect separation). - **Use Case**: AUC is particularly effective for comparing models regardless of the threshold used, especially for binary classifiers. These four metrics provide different perspectives, enabling you to build models that are not only accurate but also reliable and actionable. This rigorous validation process is especially critical when deploying systems in regulated or high-stakes environments, such as loan approvals or medical triage. However, a rigorous validation process doesn’t just test a model’s predictive power—it also illuminates its assumptions, robustness, and potential biases. Whether using cross-validation, out-of-sample testing, or benchmarking against industry standards, adequate validation provides the confidence to deploy models responsibly in the real world. --- B. Noted

User Acceptance Testing (UAT) is where the real users put the system to the test — and that’s when bugs often pop up like uninvited guests. 🎯 So how should a Business Analyst react? Here’s a practical, real-world approach👇 🔹 𝟏. 𝐒𝐭𝐚𝐲 𝐂𝐚𝐥𝐦, 𝐍𝐨𝐭 𝐃𝐞𝐟𝐞𝐧𝐬𝐢𝐯𝐞 Example: During UAT for a loan origination platform, a tester flagged that loan application forms were crashing on submit. 💡Instead of blaming dev or users, BA should listen carefully, replicate the issue, and documented the exact steps. 🔹 𝟐. 𝐋𝐨𝐠 𝐈𝐭 𝐂𝐥𝐞𝐚𝐫𝐥𝐲 𝐢𝐧 𝐭𝐡𝐞 𝐃𝐞𝐟𝐞𝐜𝐭 𝐓𝐫𝐚𝐜𝐤𝐢𝐧𝐠 𝐓𝐨𝐨𝐥 Use tools like JIRA or Azure DevOps. 💡Include: ✅ Clear description ✅ Steps to reproduce ✅ Screenshots/video ✅ Environment ✅ Severity and priority 🎯Tip: Categorize whether it’s a functional defect, UI issue, or data mapping error — devs love clarity! 🔹 𝟑. 𝐓𝐫𝐚𝐜𝐞 𝐈𝐭 𝐁𝐚𝐜𝐤 𝐭𝐨 𝐑𝐞𝐪𝐮𝐢𝐫𝐞𝐦𝐞𝐧𝐭𝐬 Was it a missed requirement, misunderstood user story, or a change that wasn’t captured? 💡Let's say, a “Export Report” button isn’t working. It turned out that the requirement wasn’t documented properly. BA should update the user story and collaborated with the Product Owner to include it in the next sprint. 🔹 𝟒. 𝐏𝐫𝐢𝐨𝐫𝐢𝐭𝐢𝐳𝐞 & 𝐂𝐨𝐦𝐦𝐮𝐧𝐢𝐜𝐚𝐭𝐞 Not all bugs are blockers. 📍BA can work with the QA Lead and Product Owner to determine which bugs were critical for go-live and which could go into post-launch patching. Clear communication = smoother releases. 🔹 𝟓. 𝐕𝐚𝐥𝐢𝐝𝐚𝐭𝐞 𝐅𝐢𝐱𝐞𝐬 𝐚𝐧𝐝 𝐂𝐥𝐨𝐬𝐞 𝐭𝐡𝐞 𝐋𝐨𝐨𝐩 Once the dev team resolves the bug, the BA ensures it meets the business need — not just technically fixed. 💡BA must always retest or sit with the UAT tester to verify resolution and update stakeholders. ✅ 𝐊𝐞𝐲 𝐓𝐚𝐤𝐞𝐚𝐰𝐚𝐲 𝐟𝐨𝐫 𝐁𝐀𝐬: Your job during UAT isn’t just to observe — it’s to bridge users and tech when bugs appear, ensuring issues are documented, fixed, and business value is preserved. Let’s normalize the fact that bugs are not failures — they are feedback. Handle them like a pro. 🧠💬 BA Helpline