Optimizing Code for Cache Performance

If you have a two-element array in C++ where Thread 0 writes to index 0 and Thread 1 writes to index 1, the code is logically correct. There are no data races, and you don't need a mutex. However, you will likely hit a massive performance bottleneck known as false sharing. The Mechanism: Cache Line Ping-Pong CPUs don’t read or write to system memory in individual bytes; they load data in chunks called cache lines (usually 64 or 128 bytes). Because your two array elements are adjacent, they sit on the exact same cache line. 1. Thread 0 modifies element 0, marking the entire cache line as modified. 2. When Thread 1 tries to write to element 1, its CPU core sees its local copy is invalid. 3. Thread 1's core must stall while the updated line is invalidated and re-fetched via the high-latency CPU interconnect—only to immediately invalidate it for Thread 0. While hardware cache coherence guarantees your data stays correct, this constant bouncing destroys performance, wastes power, and saturates the interconnect. How Hardware Behaves The underlying bottleneck is the same on modern consumer CPUs, but the structural details differ: - x86_64 (Intel/AMD): Cache coherence protocols handle synchronization flawlessly for correctness, but execution speed drops severely due to the ping-ponging. - Apple Silicon (M-series): These chips use 128-byte cache lines (compared to the 64 bytes typical of Intel, AMD, or Snapdragon). Data structures must be spaced twice as far apart to avoid sharing a line. - Other Architectures: Cache line sizes and topologies vary wildly. Assuming a flat 64 bytes across the board will eventually backfire when porting code. The Fix: C++17 Alignment In C++17 and later, you can use std::hardware_destructive_interference_size from <new> to query the compiler for the target architecture's cache line size: #include <new> struct alignas(std::hardware_destructive_interference_size) AlignedInt { int value; }; // Element 0 and Element 1 now live on separate cache lines AlignedInt safe_array[2]; Note: This constant is a compiler guide—some platforms fallback to a default like 64, so it is a best practice rather than an absolute guarantee. Practical Takeaways - Work locally: Have threads perform heavy calculations using stack-allocated local variables, writing to the shared array just once at the end. - Pad shared structures: If different threads must frequently update adjacent fields in a shared structure, use explicit alignment to keep them separated.

Cache-Friendly Structs Last week, we conducted a pool, and cache efficiency was one of the most requested topics. I’m really glad this one came up because cache-friendly data structures are one of the most critical factors in modern high-performance systems — and yet, many developers still underestimate how much performance depends on memory layout rather than algorithm complexity. Modern CPUs, such as those designed by Intel, operate at extremely high speeds, but memory access remains relatively slow. To bridge this gap, processors use multiple levels of cache (L1, L2, L3). These caches store small portions of memory closer to the CPU, allowing much faster access compared to main RAM. At first glance, a struct may appear to be just a simple grouping of fields. However, the way fields are ordered and accessed has a direct impact on performance. Struct layout determines how efficiently the CPU cache can load and reuse data. When structs are designed properly, the CPU can fetch useful data in fewer cache lines, reducing latency and improving throughput. To demonstrate why cache-friendly structs matter in practice, consider the core principles that influence performance: Spatial locality — accessing data that is physically close in memory Temporal locality — reusing data that was recently accessed Cache line utilization — maximizing useful data per cache fetch Predictable memory access patterns — enabling hardware prefetching Without these principles, CPUs spend more time waiting on memory than executing instructions. This leads to cache misses, pipeline stalls, and significant performance degradation — especially in systems that process millions of objects per second. In practice, cache-friendly struct design provides something extremely valuable: efficiency. The CPU can load fewer cache lines, reuse more data, and execute instructions continuously without waiting on memory. This is essential in performance-critical environments such as trading systems, real-time engines, and large-scale simulations. One of the biggest strengths of cache-friendly design is that it improves performance without changing algorithms. Simply reorganizing fields can reduce memory stalls and dramatically increase throughput. Below, we included a simple example showing how struct layout directly affects cache efficiency. Even in its minimal form, it illustrates how memory organization impacts performance. And here’s the key takeaway: Cache-friendly structs succeed because they prioritize memory locality, predictability, and efficient cache utilization over convenience. In high-performance systems, memory layout is often more important than algorithm complexity. Struct design provides the foundation that allows modern CPUs to operate at their full potential. Have you ever improved performance significantly just by reorganizing struct fields? #Cpp #LowLatency #CacheFriendly #MemoryManagement #AlgorithmicTrading #EngineeringExcellence #SoftwareArchitecture

🚀 Cache Locality in C++: The Invisible Performance Killer In low-latency requirements, the CPU cache is your true data center. Main memory is hundreds of cycles away — a single cache miss can destroy your latency budget. 💡 A common layout (Array of Structs - AoS): struct Trade { double price; int quantity; char side; // 'B' or 'S' }; std::vector<Trade> trades; This is convenient, but not always cache-efficient. When you access price, the CPU fetches the entire cache line containing that field. If the struct is large or misaligned, unnecessary data (quantity, side) may get pulled in, and fewer useful price values fit in the cache line. ⚡ A cache-friendlier layout (Structure of Arrays - SoA): struct Trades { std::vector<double> prices; std::vector<int> quantities; std::vector<char> sides; }; Here, if your algorithm only touches prices, the cache lines are filled with exactly the data you need — no wasted bandwidth. 🔑 Takeaway: • AoS is convenient, but can waste cache capacity • SoA improves utilization when access patterns are predictable • In HFT, this translates directly into nanoseconds saved per iteration 👉 Next time you design a performance-critical loop, ask yourself: Am I feeding the CPU cache what it needs, or wasting bandwidth? 💭 I’m curious — what’s your favorite technique to get the most out of CPU caches in performance-critical systems? #Cplusplus #Performance #LowLatency #HighFrequencyTrading #SystemDesign

I just published a new tutorial article explaining how KV caching works in LLMs, both conceptually and in code, with a clean, from-scratch implementation. It's one of the key techniques for efficient LLM inference. While recovering from an injury and taking a break from more research-heavier writing in the last few weeks, I wanted to share this practical guide on a topic many readers asked about (and one I deliberately left out of the Build a Large Language Model From Scratch book due to its added complexity). In this tutorial, I walk through: 1. Why LLMs recompute attention weights inefficiently during generation 2. How a KV cache avoids that by storing key/value vectors for reuse 3. A side-by-side walkthrough of inference with and without caching 4. Step-by-step code changes to implement caching in a readable way 5. Performance comparison and key optimizations (like preallocation and sliding windows) Even with a tiny 124M parameter model, enabling KV caching led to a substantial speed-up in generation. 🔗 Full tutorial: https://lnkd.in/g-vYFVTa Happy reading, and as always, feel free to share feedback or questions!

You're in an AI Engineer Interview. Interviewer: How do you implement caching strategies for LLM applications? Here's how I'd approach 👇 Caching is one of the easiest ways to reduce latency and inference costs in LLM applications. The key is understanding what should be cached and at which layer. 1. Prompt Response Cache If the exact same prompt is submitted multiple times, store the generated response and return it directly instead of calling the LLM again. Best for: FAQs Customer support bots Internal knowledge assistants Benefit: Near-instant responses and significant cost reduction. 2. Semantic Cache Users rarely ask the same question using identical wording. Instead of exact string matching, store embeddings of previous queries and retrieve cached responses when a new query is semantically similar. Example: "What's your refund policy?" and "Can I get my money back?" can map to the same cached answer. Benefit: Higher cache hit rates. 3. RAG Retrieval Cache In Retrieval-Augmented Generation systems, retrieval often consumes a large portion of latency. Cache: Retrieved document chunks Vector search results Frequently accessed knowledge sources Benefit: Faster retrieval and reduced database load. 4. Embedding Cache Embedding generation can become expensive at scale. Store embeddings for: Documents Product descriptions Frequently searched queries Reuse them instead of recomputing. Benefit: Lower compute cost and faster indexing. 5. Multi-Level Cache Architecture A production-grade setup typically uses: L1: In-memory cache (Redis/Memcached) ⬇️ L2: Distributed cache ⬇️ L3: Database/Object Storage ⬇️ LLM Provider This minimizes expensive model calls while maintaining scalability. 6. Cache Invalidation & Freshness Caching introduces a new challenge: stale responses. Common strategies: Time-to-Live (TTL) Versioned knowledge bases Event-driven invalidation Cache refresh on document updates Production Mindset When designing caching for LLM applications, optimize for: ✅ Latency ✅ Cost Reduction ✅ Freshness ✅ Accuracy ✅ Scalability A well-designed caching layer can reduce LLM costs dramatically while improving user experience at the same time. #AI #GenAI #LLM #RAG #AIEngineering #MachineLearning #MLOps #PromptEngineering #ArtificialIntelligence #SystemDesign #DataScience Follow Sneha Vijaykumar for more...😊

Everyone and their grandma is worried about KV caches, so let's take a second to learn more about them and how you can optimize them. Transformers store two vectors for each token at every layer: the key (K) and value (V). Together, these form the KV cache. Using it taps the classic software engineering tradeoff — building solutions that save compute by using memory. The formula is simple: KV cache per layer = context length × hidden size × 2 × precision. The 2 here comes from the fact that we have to store both the K and V vectors. Example: Context length = 32,000 tokens Hidden size = 4,096 Precision = FP16 (2 bytes per number) That comes out to roughly 8 GB per layer. Multiply across 40 layers, and you’re staring at around 320 GB. This is why your memory bandwidth Yamchas your systems much sooner than FLOPs (more on this later). When it comes to Techniques to shrink or tame the cache, I like to group them by the level they operate at. Let's take a look. Structure-level fixes (how the cache is organized): Paged KV: Split the cache into fixed-size pages, like virtual memory. This makes it easier to reuse common prefixes and prevents memory fragmentation. Instead of duplicating a giant block, you reuse pages. Prefix caching: In chat systems, the opening prompt or system instructions repeat constantly. Cache them once and point subsequent requests at the same data. The challenge is normalization: tiny differences (like timestamps, whitespace, or IDs) can break cache hits unless you clean them first. Representation-level fixes (make each entry smaller): Multi-Query or Grouped-Query Attention (MQA/GQA): Normally, every head has its own K and V. With 32 or 64 heads, that means 32 or 64 copies of the same information. MQA shares one K/V across all heads, and GQA shares across groups of heads. This can reduce cache size by 8× or more. KV quantization: Store K/V in int8 or int4 instead of FP16. This cuts storage and bandwidth by half or more. But to work, the dequantization must be fused into the attention kernel — otherwise you waste the savings on overhead. Policy-level fixes (decide what to keep at all): Sliding window or forgetful attention: Limit attention to the last W tokens. For many use cases, W between 4k and 16k is plenty. This caps memory growth directly. But beware: tasks like code generation, legal reasoning, or math proofs can collapse if you drop long-range dependencies. This one of the difficulties of context management, especially when you consider that some tasks require you to hold entire documents in memory (at Irys, Legal AI, our users often upload a document and try to turn it into a template for other drafts; this requires keeping the entire doc and later the template in memory). Without KV optimization, context scaling stalls around 8k–16k tokens. With it, 100k+ contexts become realistic and commercially viable.

Every caching tutorial on the internet is lying to you. They show you this: """ if cache.has(key): return cache.get(key) else: data = db.query(key) cache.set(key, data) return data """ - Looks clean. Works in demos. - Destroys production systems. Here's what actually happens at scale: 𝗣𝗿𝗼𝗯𝗹𝗲𝗺 #𝟭: 𝗧𝗵𝘂𝗻𝗱𝗲𝗿𝗶𝗻𝗴 𝗛𝗲𝗿𝗱 Cache key expires. 10,000 requests hit simultaneously. All 10,000 miss cache. All 10,000 slam your database. Database dies. Cascade failure. Fix: Distributed locks + cache stampede prevention. Only ONE request rebuilds. Others wait or get stale data. 𝗣𝗿𝗼𝗯𝗹𝗲𝗺 #𝟮: 𝗖𝗮𝗰𝗵𝗲 𝗣𝗲𝗻𝗲𝘁𝗿𝗮𝘁𝗶𝗼𝗻 𝗔𝘁𝘁𝗮𝗰𝗸 Attacker queries keys that don't exist. user_9999999999, user_9999999998... Cache always misses. Every request hits database. Free DDoS using your own infrastructure. Fix: Bloom filters. Cache negative results. Rate limiting per key pattern. 𝗣𝗿𝗼𝗯𝗹𝗲𝗺 #𝟯: 𝗛𝗼𝘁 𝗞𝗲𝘆 𝗠𝗲𝗹𝘁𝗱𝗼𝘄𝗻 One celebrity posts. Millions request the same cache key. Single Redis node handles ALL traffic. That node melts. Game over. Fix: Key replication with suffixes (key_1, key_2... key_N). Client-side random distribution. 𝗧𝗵𝗲 𝗿𝗲𝗮𝗹 𝗹𝗲𝘀𝘀𝗼𝗻: Caching isn't a performance optimization. It's a distributed systems problem disguised as a simple key-value lookup. The moment you add a cache, you've added: -> Consistency challenges -> Failure modes -> Cold start problems -> Memory pressure decisions -> Eviction policy trade-offs " The best cache is the one you understood deeply before deploying.

🔁 How I Optimized a PySpark Job That Was Taking 2 Hours – Now It Finishes in Just 10 Minutes! Performance tuning is a data engineer’s secret superpower 💪 — and I recently had the chance to use it on a PySpark job that was painfully slow. 📍 The Problem: A daily ETL job processing 100M+ rows was taking over 2 hours to complete. It was clogging up the pipeline and delaying downstream processes. ⚙️ What I Did to Optimize It: ✅ 1. Caching I cached intermediate DataFrames that were reused multiple times. This reduced repeated computations and I/O. df.cache() ✅ 2. Partitioning Input data was poorly partitioned. I used repartition() based on a high-cardinality column, which balanced the load across executors. df = df.repartition("customer_id") ✅ 3. Broadcast Joins Switched a skewed join to use broadcast join for a smaller dimension table (30K rows). It prevented massive data shuffling. df = fact_df.join(broadcast(dim_df), "key") ✅ 4. Predicate Pushdown Filtered early in the pipeline instead of after joins. This significantly reduced the volume of data being shuffled. df = df.filter(col("status") == "active") 📈 Result: Runtime reduced from 2 hours → 10 minutes 🚀 Cluster cost dropped by 70% Downstream jobs now start early — smoother scheduling! 💡Takeaway: PySpark is powerful, but without optimization, it can also be painfully slow. Understanding how Spark executes under the hood makes all the difference. Have you had a similar experience optimizing PySpark or Spark jobs? Let’s exchange tips in the comments 👇 #PySpark #DataEngineering #ApacheSpark #BigData #ETL #PerformanceOptimization #SparkSQL #TechLeadership #DataPipeline #BroadcastJoin #PredicatePushdown #Partitioning #Caching

I, once spent two days 𝗺𝗮𝗸𝗶𝗻𝗴 𝗼𝘂𝗿 𝗾𝘂𝗲𝗿𝗶𝗲𝘀 𝘀𝗹𝗼𝘄𝗲𝗿 A PM came to me: "The dashboard is slow. Can you fix it?" I looked at our Databricks cluster. Ten analysts running queries against a 500GB table. Some queries took 45 seconds. Not terrible, but not great I thought: "Easy, I'll cache the table in memory" One line of code: spark.sql("CACHE TABLE daily_sales_summary") Ran the cache operation. Took 8 minutes to load the entire table into cluster memory. Then I re-ran the slow queries They went from 𝟰𝟱 𝘀𝗲𝗰𝗼𝗻𝗱𝘀 𝘁𝗼 𝟰 𝗺𝗶𝗻𝘂𝘁𝗲𝘀 I made queries 𝟱𝘅 𝘀𝗹𝗼𝘄𝗲𝗿 Here's what I didn't understand: If Spark, now, doesn’t have to load data from disk, why does it take 4 minutes (5x slower)? Most analysts were only querying recent data — last 30 days, specific regions, certain product categories. Their queries were scanning maybe 20GB, not 500GB When the table wasn't cached, Spark would read only the relevant Parquet files from object storage (thanks to partition pruning). Fast, focused reads of just the data they needed When I cached the entire table:  • The cache loaded all 500GB into memory (one-time 8-minute cost)  • Every query now filtered through 500GB of in-memory data  • Instead of reading 20GB from disk with partition pruning, we were scanning the full cached dataset I uncached the table. Queries went back to 45 seconds Then I did what I should have done earlier:  • Analyzed the actual query patterns  • Found that 80% of queries filtered on 𝘰𝘳𝘥𝘦𝘳_𝘥𝘢𝘵𝘦 and 𝘳𝘦𝘨𝘪𝘰𝘯  • Cached only the last 90 days of the top 5 regions  • 50GB cached instead of 500GB Queries on recent, high-traffic data dropped to 12 seconds. Everything else stayed at 45 seconds. Good enough The learning wasn't: "don't cache" It was: "measure, then optimize" Caching is a trade-off between:  • Memory cost vs compute cost  • Read patterns vs data volume  • Cache hit rate vs cache maintenance overhead In my experience, caching helps when:  • The same small-ish dataset is queried repeatedly  • You have memory to spare  • The cache actually gets reused before eviction Caching will perform even worse when:  • Queries already benefit from partition pruning on disk  • Cache eviction overhead exceeds read savings  • You're caching data that's rarely accessed How do you decide what to cache? Do you measure cache hit rates, or do you cache based on gut feel? --------------------------------------------- Found value? Repost, share w/ your network Follow Afaque Ahmad for more stories, experiences, and experiments in Data Engineering :)

I love learning new, counter-intuitive lessons about performance engineering. I was optimizing a skip list (fun holiday project!) but when I removed some unused fields performance got... worse? Turns out those "unused" fields were functioning as padding. This is a problem called "false sharing." CPUs fetch data from main memory into cache in 64-byte chunks called cache lines. Whether you request 1, 32 or 64 bytes, neighboring data hitches a free ride into your CPU. This is normally good! You generally access multiple fields so you may as well fetch them together. The problem is with concurrent code. When one core writes to a cache line, it invalidates that line in every other core's cache. Those cores must re-fetch it before they can access any data on that line I wrote a quick benchmark in rust: N threads each increment their own counter. With all counters packed into the same cache line, performance was ~20x worse at 8 threads compared to padding each counter to 64 bytes so they each get their own cache line. A disclaimer: don't reach for this unless you're writing truly high-performance code. It's easy to mess up and just waste memory.