Dynamic Load Scheduling Algorithms

The Flexible Job Shop Scheduling Problem (FJSP) represents a critical advancement in industrial optimization, extending the classical Job Shop Scheduling Problem (JSSP) by introducing a dual-decision layer. While JSSP requires determining the sequence of operations on pre-assigned machines, FJSP adds the complexity of 'machine assignment', where each operation can be processed by any machine from a compatible set. This flexibility is essential for modern smart manufacturing, as it allows production systems to adapt to machine breakdowns and varying workloads, directly impacting operational efficiency and resource utilization in high-stakes environments. Historically, FJSP has been tackled using traditional exact methods like Integer Programming and meta-heuristics such as Genetic Algorithms (GA) or Taboo Search. More recently, Deep Reinforcement Learning (DRL) has emerged as a dominant approach, utilizing GNNs and Transformers to learn scheduling policies that can generate solutions in real-time. These neural net based methods treat the scheduling environment as a dynamic graph or sequence, attempting to map complex shop floor states to optimal dispatching rules. Despite their potential, current automated solvers face significant bottlenecks. The primary challenge lies in the 'curse of dimensionality' and sequence length. As the number of jobs and machines increases, the scheduling sequence grows quadratically, causing standard Transformers to suffer from extreme computational overhead due to their O(L^2) complexity. Furthermore, GNN-based methods often struggle to capture long-range dependencies between operations scheduled far apart in time, leading to sub-optimal machine assignments and increased makespan. To address the shortcomings highlighted above, the authors of [1] introduce M-CA (Mamba-CrossAttention), a novel architecture that replaces the standard self-attention mechanism with Selective State Space Modeling (Mamba). Mamba offers linear scaling O(L) with respect to sequence length, allowing the model to process much larger scheduling horizons efficiently. The M-CA framework specifically utilizes a 'Mamba-based Encoder' to capture global temporal dependencies and a 'Cross-Attention Decoder' to focus on the immediate machine-operation compatibility. This hybrid approach is superior because it maintains the high-fidelity global context of the entire factory state while drastically reducing the memory footprint and inference time required by traditional Transformers. Experiments show M-CA consistently outperforms state-of-the-art DRL baselines, Transformer-based models, and traditional heuristics across problem scales, achieving lower makespans and up to 5× faster inference. Mamba’s superior 'forgetting and remembering' mechanism drives scalability and robust performance by filtering out irrelevant scheduling noise to focus on critical constraints. The link to the paper [1] is posted in the comments.

Mixtures of Experts have been used in LLMs for quite a few years now, but DeepSeek's MoE implementation introduces several modeling tricks that are worth taking a closer look. 1 - Hybrid routing strategy. DeepSeek uses a hybrid of soft routing and hard routing. In soft routing we compute the weighted sum over all expert outputs, whereas in hard routing we limit the sum to the top k experts with the highest routing scores. In the hybrid version, we have a combination of shared experts and a pool of routed experts, of which only the top k are activated for each input token. The output of the MoE layer is a weighted sum over the shared and routed experts, where the shared experts’ weights are 1 and the routed experts’ weights are the router scores. (Unlike standard MoE implementations, DeepSeek uses a per-expert Sigmoid instead of a Softmax to normalize the router scores. This decouples the expert's router scores from each other, which is important for the next trick, dynamic load balancing.) 2 - Dynamic load balancing. Load balancing — making sure all experts and hence all GPUs inside the training cluster receive the same number of tokens during training — has been one of the most difficult challenges in sparse MoEs. So far, the status quo has been to introduce either load balancing losses (e.g. Switch Transformer) or customized compilers (e.g. MegaBlocks). DeepSeek’s MoE demonstrated for the first time a third solution, namely dynamic load balancing. The trick is to add a bias term b to each expert’s router scores prior to taking the top-k. If an expert is “overloaded” (i.e. receiving more tokens than the total number of tokens divided by the total number of experts), we reduce that expert’s bias by 𝛾, resulting in a smaller chance of the expert being selected by the router. In contrast, if the expert is underloaded, we increase the expert’s bias by 𝛾, increasing the chance of the expert being selected. As training progresses, eventually expert loads are mathematically guaranteed to reach perfect balance. 3 - Sequence-wise balancing. Unlike other MoE models, DeepSeekMoE adds a novel auxiliary loss term that ensures expert balance not just across the entire batch but even more fine-grained across each individual token sequence inside the batch. For example, given a sequence of 100 tokens and a pool of 4 routed experts with k=1, ideally we want each expert to be activated for 25/100 tokens. As usual, more on this in my blog: https://lnkd.in/gwxzX7ud

We have a new blog post on Cluster Launch Control (CLC) for dynamic tile scheduling on NVIDIA Blackwell GPUs! In GPU GEMM kernels, the output matrix is partitioned into tiles, each of which is assigned to a CTA (or cluster of CTAs) for computation. The problem of "tile scheduling" is to determine how to best distribute these work tiles across available processors. Traditional approaches, such as single-tile or static persistent scheduling, each come with their own tradeoffs around load balancing and occupancy. CLC is a hardware-supported Blackwell feature that combines the efficiency of persistent scheduling with fully dynamic work and processor assignment. The blog goes through the CLC optimization in concrete and practical terms. We explain CLC's concept of work stealing and how this is implemented in terms of the two PTX-level instructions try_cancel and query_cancel. We then walk through a CuTe DSL GEMM kernel leveraging CLC, where a dedicated scheduler warp per cluster steals work tiles dynamically and a CLC pipeline propagates results to the remaining warps. Blog: https://lnkd.in/gtcJGKsm