Performance Optimization Techniques

Last week, I described four design patterns for AI agentic workflows that I believe will drive significant progress: Reflection, Tool use, Planning and Multi-agent collaboration. Instead of having an LLM generate its final output directly, an agentic workflow prompts the LLM multiple times, giving it opportunities to build step by step to higher-quality output. Here, I'd like to discuss Reflection. It's relatively quick to implement, and I've seen it lead to surprising performance gains. You may have had the experience of prompting ChatGPT/Claude/Gemini, receiving unsatisfactory output, delivering critical feedback to help the LLM improve its response, and then getting a better response. What if you automate the step of delivering critical feedback, so the model automatically criticizes its own output and improves its response? This is the crux of Reflection. Take the task of asking an LLM to write code. We can prompt it to generate the desired code directly to carry out some task X. Then, we can prompt it to reflect on its own output, perhaps as follows: Here’s code intended for task X: [previously generated code] Check the code carefully for correctness, style, and efficiency, and give constructive criticism for how to improve it. Sometimes this causes the LLM to spot problems and come up with constructive suggestions. Next, we can prompt the LLM with context including (i) the previously generated code and (ii) the constructive feedback, and ask it to use the feedback to rewrite the code. This can lead to a better response. Repeating the criticism/rewrite process might yield further improvements. This self-reflection process allows the LLM to spot gaps and improve its output on a variety of tasks including producing code, writing text, and answering questions. And we can go beyond self-reflection by giving the LLM tools that help evaluate its output; for example, running its code through a few unit tests to check whether it generates correct results on test cases or searching the web to double-check text output. Then it can reflect on any errors it found and come up with ideas for improvement. Further, we can implement Reflection using a multi-agent framework. I've found it convenient to create two agents, one prompted to generate good outputs and the other prompted to give constructive criticism of the first agent's output. The resulting discussion between the two agents leads to improved responses. Reflection is a relatively basic type of agentic workflow, but I've been delighted by how much it improved my applications’ results. If you’re interested in learning more about reflection, I recommend: - Self-Refine: Iterative Refinement with Self-Feedback, by Madaan et al. (2023) - Reflexion: Language Agents with Verbal Reinforcement Learning, by Shinn et al. (2023) - CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing, by Gou et al. (2024) [Original text: https://lnkd.in/g4bTuWtU ]

I just published a new tutorial article explaining how KV caching works in LLMs, both conceptually and in code, with a clean, from-scratch implementation. It's one of the key techniques for efficient LLM inference. While recovering from an injury and taking a break from more research-heavier writing in the last few weeks, I wanted to share this practical guide on a topic many readers asked about (and one I deliberately left out of the Build a Large Language Model From Scratch book due to its added complexity). In this tutorial, I walk through: 1. Why LLMs recompute attention weights inefficiently during generation 2. How a KV cache avoids that by storing key/value vectors for reuse 3. A side-by-side walkthrough of inference with and without caching 4. Step-by-step code changes to implement caching in a readable way 5. Performance comparison and key optimizations (like preallocation and sliding windows) Even with a tiny 124M parameter model, enabling KV caching led to a substantial speed-up in generation. 🔗 Full tutorial: https://lnkd.in/g-vYFVTa Happy reading, and as always, feel free to share feedback or questions!

𝗥𝗔𝗚 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿’𝘀 𝗦𝘁𝗮𝗰𝗸 — 𝗪𝗵𝗮𝘁 𝗬𝗼𝘂 𝗡𝗲𝗲𝗱 𝘁𝗼 𝗞𝗻𝗼𝘄 𝗕𝗲𝗳𝗼𝗿𝗲 𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 Building with Retrieval-Augmented Generation (RAG) isn't just about choosing the right LLM. It's about assembling an entire stack—one that's modular, scalable, and future-proof. This visual from Kalyan KS neatly categorizes the current RAG landscape into actionable layers: → 𝗟𝗟𝗠𝘀 (𝗢𝗽𝗲𝗻 𝘃𝘀 𝗖𝗹𝗼𝘀𝗲𝗱) Open models like LLaMA 3, Phi-4, and Mistral offer control and customization. Closed models (OpenAI, Claude, Gemini) bring powerful performance with less overhead. Your tradeoff: flexibility vs convenience. → 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸𝘀 LangChain, LlamaIndex, Haystack, and txtai are now essential for building orchestrated, multi-step AI workflows. These tools handle chaining, memory, routing, and tool-use logic behind the scenes. → 𝗩𝗲𝗰𝘁𝗼𝗿 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲𝘀 Chroma, Qdrant, Weaviate, Milvus, and others power the retrieval engine behind every RAG system. Low-latency search, hybrid scoring, and scalable indexing are key to relevance. → 𝗗𝗮𝘁𝗮 𝗘𝘅𝘁𝗿𝗮𝗰𝘁𝗶𝗼𝗻 (𝗪𝗲𝗯 + 𝗗𝗼𝗰𝘀) Whether you're crawling the web (Crawl4AI, FireCrawl) or parsing PDFs (LlamaParse, Docling), raw data access is non-negotiable. No context means no quality answers. → 𝗢𝗽𝗲𝗻 𝗟𝗟𝗠 𝗔𝗰𝗰𝗲𝘀𝘀 Platforms like Hugging Face, Ollama, Groq, and Together AI abstract away infra complexity and speed up experimentation across models. → 𝗧𝗲𝘅𝘁 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴𝘀 The quality of retrieval starts here. Open-source models (Nomic, SBERT, BGE) are gaining ground, but proprietary offerings (OpenAI, Google, Cohere) still dominate enterprise use. → 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 Tools like Ragas, Trulens, and Giskard bring much-needed observability—measuring hallucinations, relevance, grounding, and model behavior under pressure. 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆: RAG is not just an integration problem. It’s a design problem. Each layer of this stack requires deliberate choices that impact latency, quality, explainability, and cost. If you're serious about GenAI, it's time to think in terms of stacks—not just models. What does your RAG stack look like today?

As Product Managers it’s so easy to loose trust if features on the roadmap are not prioritised correctly. Here are 5 prioritization frameworks and when to actually use them: 1. RICE (Reach, Impact, Confidence, Effort) ✅ Use when: You have multiple ideas/features and want to prioritize based on expected impact. 📌 Best for: Growth experiments, new features, MVP ideas 💡Tip: Confidence % is often biased calibrate with data! 2. MoSCoW (Must have, Should have, Could have, Won’t have) ✅ Use when: You’re working with tight deadlines and multiple stakeholders. 📌 Best for: Sprint planning, product launches 💡Tip: Don’t let every stakeholder label everything as “Must have.” 3. Kano Model ✅ Use when: You want to balance delight with functionality. 📌 Best for: Customer-facing products 💡Tip: A feature that delights today might be expected tomorrow. 4. ICE (Impact, Confidence, Ease) ✅ Use when: You want a quicker version of RICE for fast decision-making. 📌 Best for: Rapid prototyping, early-stage prioritization 💡Tip: Use ICE when you don’t have a ton of data but still need to move. 5. Value vs. Effort Matrix ✅ Use when: You want to visualize trade-offs with stakeholders. 📌 Best for: Roadmap discussions, stakeholder alignment 💡Tip: Plot features on a 2×2: * Quick Wins (High value, low effort) * Strategic Bets (High value, high effort) * Time Wasters (Low value, high effort) * Fillers (Low value, low effort) So which one should you pick? Use RICE when you’re in a data-driven company. Use MoSCoW when time is tight and alignment is tough. Use ICE when you need speed > accuracy. Use Kano when delight matters. Use the Value/Effort Matrix when people keep asking, “Why this first?” 📌 Save this for your next prioritization war. 💬 Tried any of these at work? Drop your go-to framework in comments! #productmanager #job #PMjobs #learning #frameworks

How to Improve API Performance? If you’ve built APIs, you’ve probably faced issues like slow response times, high database load, or network inefficiencies. These problems can frustrate users and make your system unreliable. But the good news? There are proven techniques to make your APIs faster and more efficient. Let’s go through them: 1. Pagination ✅ - Instead of returning massive datasets in one go, break the response into pages. - Reduces response time and memory usage - Helps when dealing with large datasets - Keeps requests manageable for both server and client 2. Async Logging ✅ - Logging is important, but doing it synchronously can slow down your API. - Use asynchronous logging to avoid blocking the main process - Send logs to a buffer and flush periodically - Improves throughput and reduces latency 3. Caching ✅ - Why query the database for the same data repeatedly? - Store frequently accessed data in cache (e.g., Redis, Memcached) - If the data is available in cache → return instantly - If not → query the DB, update the cache, and return the result 4. Payload Compression ✅ - Large response sizes lead to slower APIs. - Compress data before sending it over the network (e.g., Gzip, Brotli) - Smaller payload = faster download & upload - Helps in bandwidth-constrained environments 5. Connection Pooling ✅ - Opening and closing database connections is costly. - Instead of creating a new connection for every request, reuse existing ones - Reduces latency and database load - Most ORMs & DB libraries support connection pooling If your API is slow, it’s likely because of one or more of these inefficiencies. Start by profiling performance and identifying bottlenecks Implement one optimization at a time, measure impact A fast API means happier users & better scalability. ✅

𝗘𝘅𝗽𝗹𝗮𝗶𝗻 𝗧𝗵𝗶𝘀: 𝗟𝗹𝗮𝗺𝗮 𝟯 𝗡𝗲𝗲𝗱𝘀 𝟮.𝟰𝗧𝗕. 𝗬𝗼𝘂𝗿 𝗚𝗣𝗨 𝗛𝗮𝘀 𝟴𝟬𝗚𝗕. 𝗜𝘁 𝗦𝘁𝗶𝗹𝗹 𝗧𝗿𝗮𝗶𝗻𝘀. Training Llama-3 405B needs ~2.4TB with BF16 + 8-bit Adam: • Weights: 810GB • Gradients: 810GB • Optimizer: 810GB (vs 3.24TB with standard Adam!) • Total: ~2.4TB (Illustrative budget—config-dependent; FP32 masters, ZeRO stage, and offload change totals) Your H100? 80GB. You'd need 30+ GPUs just to hold everything. 𝗧𝗵𝗿𝗲𝗲 𝗧𝗿𝗶𝗰𝗸𝘀 𝗧𝗵𝗮𝘁 𝗠𝗮𝗸𝗲 𝗜𝘁 𝗪𝗼𝗿𝗸 𝟭. 𝗗𝗮𝘁𝗮 𝗣𝗮𝗿𝗮𝗹𝗹𝗲𝗹: Split batch. Problem: Each GPU needs 2.4TB. Fix: ZeRO splits it across N GPUs. 𝟮. 𝗠𝗼𝗱𝗲𝗹 𝗣𝗮𝗿𝗮𝗹𝗹𝗲𝗹: Split layers. Problem: Sequential bottleneck. Fix: Pipeline batches. 𝟯. 𝗦𝗲𝗾𝘂𝗲𝗻𝗰𝗲 𝗣𝗮𝗿𝗮𝗹𝗹𝗲𝗹: Split tokens. This is the game changer. 8K tokens → 8 GPUs → 1K each. But attention needs every token to see all others. 𝗧𝗵𝗲 𝗠𝗮𝗴𝗶𝗰 𝗠𝗼𝗺𝗲𝗻𝘁: Instead of moving the 2.4TB model, GPUs only exchange attention keys/values (K,V). Each GPU: • Computes K,V for its 1K tokens (32MB) • Sends to others via all-to-all • Receives 7×32MB = 224MB total • Computes attention, deletes copies 𝟮𝟮𝟰𝗠𝗕 𝗺𝗼𝘃𝗲𝗱 𝗶𝗻𝘀𝘁𝗲𝗮𝗱 𝗼𝗳 𝟮.𝟰𝗧𝗕. That's 10,000x less. 𝗧𝗵𝗲 𝗥𝗲𝘀𝘂𝗹𝘁: Combine all three (ZeRO + tensor + pipeline + sequence parallel). Each GPU holds ~75GB instead of 2.4TB. This exact choreography powers ChatGPT, Claude, and every frontier model. Without it? 10K token limits. With it? Entire books in one context. Not magic. Just brilliant engineering making the impossible routine.

💥 Data Engineer Interview Killer: Handling 500GB Daily with PySpark Data pros — have you ever been asked this in an interview? 👉 “How would you efficiently process a 500 GB dataset in PySpark, and how would you size your cluster?” It’s one of my favorite questions — because it blends architecture, optimization, and cost awareness into one real-world scenario. Here’s how I’d break it down 👇 💡 The 5-Step Optimization Blueprint 1️⃣ Format First — The Foundation of Speed 🚀 Action: Convert raw data (CSV/JSON) into Parquet or Delta Lake right away. Why: Columnar storage, compression, and predicate pushdown drastically cut I/O. 👉 This single step often gives the biggest performance boost. 2️⃣ Partitioning Math — Define Your Parallelism 🧮 Each Spark task should process around 128 MB. Calculation: 500 GB × 1024 MB/GB ÷ 128 MB/partition ≈ 4,000 partitions ➡️ Spark now has ~4,000 tasks to parallelize — perfect for scaling efficiently. 3️⃣ Cluster Sizing — Predictable Execution 🧠 Let’s assume: 10 worker nodes 8 cores & 32 GB RAM per node Parallelism: 10 nodes × 8 cores = 80 cores total Each core handles ~2–3 tasks → ~240 tasks concurrently Total time: 4,000 ÷ 240 ≈ 17 waves of execution At ~1–2 min per wave → ~25–30 minutes total runtime That’s how you explain both scaling and efficiency in an interview. 4️⃣ Memory Management — Avoid the Spill 💾 Plan for roughly 3× data size during joins and shuffles. Estimate: (500 GB × 3) ÷ 10 nodes = 150 GB per node With only 32 GB per node, Spark will spill to disk — which is fine if SSD-backed. For critical workloads, upgrade to 64 GB nodes to keep processing smooth. 5️⃣ Performance Tweaks — Fine-Tuning ⚙️ spark.sql.shuffle.partitions = 400 spark.sql.adaptive.enabled = True ✅ Use Broadcast Joins for small lookup tables. ✅ Implement Incremental Loads (Delta Lake makes this easy). ✅ Avoid full reloads — only process what’s changed. 🧭 The Real Data Engineering Challenge Optimizing Spark isn’t about adding more compute — it’s about finding the sweet spot between performance, cost, and scalability. 🔥 Question for you: If you got this same question in an interview — how would you size your cluster or optimize it differently? 👇 I’ll be sharing my cost–benefit breakdown in the next post — how to choose between scaling up vs scaling out for real workloads. #PySpark #ApacheSpark #Databricks #BigData #DataEngineering #Optimization #InterviewPrep #Azure

Demystifying PLC Communication Protocols: The Backbone of Industrial Connectivity In today’s smart manufacturing environments, communication protocols are essential for enabling seamless interaction between PLCs, HMIs, sensors, and control systems. They define the language and rules devices use to share data across networks. Without these protocols, devices may be physically connected—but not truly communicating. Here’s a quick breakdown of common PLC communication protocols and where they shine: Common Protocols: Ethernet: High-speed, internet-capable, and supports up to 255 devices. Ideal for modern industrial networks requiring fast, real-time data transfer. Profibus: A robust fieldbus protocol supporting up to 127 devices over 15 km. Widely used in process automation. RS-232 & RS-485: Serial communication protocols for short-distance, point-to-point (RS-232) or multi-drop (RS-485) communication. Still useful in legacy systems. MPI & PPI: Proprietary Siemens protocols. MPI is used for multi-master communication; PPI for basic point-to-point links in smaller applications. ControlNet & DeviceNet: Allen-Bradley/ Rockwell protocols designed for deterministic data exchange (ControlNet) and device-level networking (DeviceNet). USB/PC Adapters: Often used for programming or initial setup. Not typically suited for large-scale communication. Key Factors to Consider: 1. Baud Rate: Speed of data transfer (e.g., Ethernet = 100 Mb/s, RS-232 = 19.2 kb/s) 2. Network Length: How far communication can reach (e.g., ControlNet = 30 km) 3. Number of Nodes: How many devices can connect (e.g., Profibus = 127) Tip: Use Ethernet for high-speed, scalable networks. Choose ControlNet when long-distance and real-time reliability are critical. #PLCProgramming #CommunicationProtocols #IndustrialAutomation #ControlSystems #SmartManufacturing #IIoT #Ethernet #Profibus #RS232 #RS485 #AutomationEngineer #FactoryAutomation

In the last three months alone, over ten papers outlining novel prompting techniques were published, boosting LLMs’ performance by a substantial margin. Two weeks ago, a groundbreaking paper from Microsoft demonstrated how a well-prompted GPT-4 outperforms Google’s Med-PaLM 2, a specialized medical model, solely through sophisticated prompting techniques. Yet, while our X and LinkedIn feeds buzz with ‘secret prompting tips’, a definitive, research-backed guide aggregating these advanced prompting strategies is hard to come by. This gap prevents LLM developers and everyday users from harnessing these novel frameworks to enhance performance and achieve more accurate results. https://lnkd.in/g7_6eP6y In this AI Tidbits Deep Dive, I outline six of the best and recent prompting methods: (1) EmotionPrompt - inspired by human psychology, this method utilizes emotional stimuli in prompts to gain performance enhancements (2) Optimization by PROmpting (OPRO) - a DeepMind innovation that refines prompts automatically, surpassing human-crafted ones. This paper discovered the “Take a deep breath” instruction that improved LLMs’ performance by 9%. (3) Chain-of-Verification (CoVe) - Meta's novel four-step prompting process that drastically reduces hallucinations and improves factual accuracy (4) System 2 Attention (S2A) - also from Meta, a prompting method that filters out irrelevant details prior to querying the LLM (5) Step-Back Prompting - encouraging LLMs to abstract queries for enhanced reasoning (6) Rephrase and Respond (RaR) - UCLA's method that lets LLMs rephrase queries for better comprehension and response accuracy Understanding the spectrum of available prompting strategies and how to apply them in your app can mean the difference between a production-ready app and a nascent project with untapped potential. Full blog post https://lnkd.in/g7_6eP6y

If you’re building LLM applications today, reasoning is where the real leverage lies. And yet, I see a lot of engineers still treating LLM outputs as a single-shot black box. LLMs can reason, but only if you give them the right scaffolding and the right post-training. Here’s a mental model I’ve been using to think about LLM reasoning methods (see chart below): ✅ Inference-time reasoning methods: These are techniques that can be applied at inference time, without needing to retrain your model: → Tree of Thoughts (ToT), search through reasoning paths → Chain of Thought (CoT) prompting, prompt models to generate intermediate reasoning steps → Reasoning + Acting, use tools or function calls during reasoning → Self-feedback, prompt the model to critique and refine its own output → Episodic Memory Agents, maintain a memory buffer to improve multi-step reasoning → Self-consistency, sample multiple reasoning paths and select the most consistent answer ✅ Training-time enhancements: Where things get really powerful is when you post-train your model to improve reasoning, using human annotation or policy optimization: → Use Preference pairs and Reward Models to tune for better reasoning (RFT, Proximal PO, KL Regularization) → Apply RLHF, PPO + KL, Rejection Sampling + SFT, Advantage Estimation, and other advanced techniques to guide the model’s policy → Leverage multiple paths, offline trajectories, and expert demonstrations to expose the model to rich reasoning signals during training Here are my 2 cents 🫰 If you want production-grade LLM reasoning, you’ll need both, → Smart inference-time scaffolds to boost reasoning without slowing latency too much → Carefully tuned post-training loops to align the model’s policy with high-quality reasoning patterns → We’re also seeing increasing use of Direct Preference Optimization (DPO) and reference-free grading to further improve reasoning quality and stability. I’m seeing more and more teams combine both strategies, and the gap between "vanilla prompting" and "optimized reasoning loops" is only getting wider. 〰️〰️〰️ Follow me (Aishwarya Srinivasan) for more AI insight and subscribe to my Substack to find more in-depth blogs and weekly updates in AI: https://lnkd.in/dpBNr6Jg