Training Delivery Models

Our friends at Thinking Machines Lab just gave the Keru.ai team early access to Tinker and it changes everything around how we personalize AI agents. First, some context. John McRaven and I were on a call with the head of data analytics at a large financial institution. He said: "The funny thing about every single product demo I've seen is that the AI never actually learns." He was right. Most companies do fine-tuning. Train once on labeled examples, freeze it, same output for everyone forever. Real learning requires reinforcement learning. You capture what users prefer in real-time, update the model per-user, keep improving it. Analyst A wants concise? Their model learns that. Analyst B wants deep analysis? Their model learns that. They can't switch platforms, they'd lose all that personalization. The problem is the infrastructure: You need logprobs access (most APIs don't expose them). Custom loss functions (generic APIs only have cross-entropy). Real-time training pipelines (can't batch). Per-analyst model management (versioning nightmare). Fault tolerance at scale (one failure cascades everywhere). Build it yourself? 6 months. $500K. 2-3 engineers. OpenAI's API doesn't support it. AWS SageMaker? 3+ months of setup for the same problems. We were looking at half a year delay or accepting a frozen model. Then we got early access to Tinker. Instead of building distributed training systems, you literally just write: await training_client.forward_backward_async(user_preference_data) await training_client.optim_step_async(learning_rate=5e-5) Tinker handles logprobs, versioning, real-time orchestration, reliability. You define what to optimize for. It works. It took me 6 runs to get what I wanted, and it's simple (see screenshot). Now when our customers use our platform, their votes actually matter. The model learns from them. Their model gets better every time they use it. That's defensible. That's why they don't leave. Infrastructure was the blocker. Thinking Machines Lab removed it. Thanks to the Tinker team - you built the thing that let us actually do what we wanted.

AI models just got a new skill: organising their own thinking. Microsoft just dropped their new paper: the Era of Agentic Organization. The core idea: as LLMs get better at reasoning, our current techniques don’t scale. Sequential thinking is too slow. Parallel thinking is wasteful. Neither approach lets the model adapt mid-reasoning. So Microsoft Research built AsyncThink, a new reasoning paradigm where one model plays two roles at once: - an Organizer that decides how the reasoning should be structured - multiple Workers that solve sub-problems in parallel The interesting part: the model learns via RL how to manage its own thinking. It figures out: - when to fork into parallel sub-tasks - which branches are worth exploring - when to merge everything back together Impact: - 28% lower latency than parallel thinking - Higher accuracy on math tasks - And crazy generalization -- models trained only on simple countdown games suddenly solving Sudoku and complex math through learned asynchronous reasoning. As companies lean into AI agents across workflows, this becomes a big deal. Systems need to organize their reasoning like project managers: splitting work, coordinating processes, and adjusting on the fly. How AsyncThink works (simple breakdown): 1. Model receives a problem. 2. Organizer decides if it should fork into sub-problems. 3. Workers solve each sub-problem concurrently. 4. Organizer monitors progress and decides which branches to continue or drop. 5. Partial results get merged through a Join step. 6. Final reasoning is produced from the coordinated structure. It’s basically the first step toward “self-organizing agents” that can scale across an entire organization’s workflows. Understand the paper in 5 mins with this visual breakdown made with Miskies AI: https://lnkd.in/gJnwYMi7 If you want more breakdowns like this, I share tutorials on building and improving AI apps + agents in my newsletter: AI Agent Engineering → https://lnkd.in/gaJTcZBR ♻️ Share it with anyone who wants to understand how next-gen agentic reasoning actually works :) #AI #LLMs #AIAgents

This summer, I worked on speeding up the OpenAI GPT API with Asynchronous Batch Calls.✨ I obtained a response per task<.25 seconds!📈 I needed to find out more about the speed capabilities of OpenAI's latest models, including GPT-4, GPT-4o, and the new GPT-4o-mini. Here's what you can expect in my free open-source notebook: 📚 Key Highlights: - Benchmarking GPT-4 Models: Is GPT-4o-mini really faster than GPT-4o? Dive into detailed benchmarks and find out how these models perform in advanced production environments. - Asynchronous Batch Processing: Learn how to handle 144 OpenAI API requests in split seconds per task by using efficient asynchronous batch processing techniques. - Hands-On Learning: Step-by-step instructions, code snippets, and explanations designed to help you implement these techniques in your own projects. 🏃♂️ Performance Insights: - GPT-4: 30.95 seconds (Avg. time per task: 0.2149 seconds) - GPT-4o: 26.75 seconds (Avg. time per task: 0.1858 seconds) - GPT-4o-mini: 21.54 seconds (Avg. time per task: 0.1496 seconds) 🔧 Technical Breakdown: - Understand the concepts and philosophy of asynchronous batch processing. - Detailed code walkthroughs to ensure you grasp each step and can replicate the results. 🔗 Check out the attached 230+ page notebook with code + results! 🚀 You can also access my GitHub repository, Chapter15, for the source code of this free, open-source notebook on my resource page: https://lnkd.in/eJqbmzpG 📢 Join the Conversation: - Share your thoughts and experiences with these models. - Let's discuss the future potential of Massive Multitask Language Understanding (MMLU) models. Let's push the boundaries of what's possible together! #AI #MachineLearning #OpenAI #GPT4 #AsynchronousProcessing #BatchProcessing #AICommunity #EducationalResource #Benchmarking #NLP

When Reasoning Meets Reinforcement Learning: Mistral AI's Magistral Breaks New Ground 👉 Why This Matters Large language models often stumble on complex reasoning tasks requiring multi-step logic. Traditional approaches rely heavily on supervised fine-tuning with human demonstrations, creating bottlenecks in scaling and adaptability. Mistral AI's new "Magistral" research demonstrates how pure reinforcement learning (RL) can unlock new reasoning capabilities while preserving core model strengths. 👉 What They Achieved Magistral introduces two models trained with a novel RL pipeline: - Magistral Medium: Achieves 50% improvement on AIME-24 math benchmarks over its predecessor - Magistral Small (open-source): A 24B parameter model competitive with larger RL-trained counterparts Key innovations: 1. Pure RL Training: No reliance on pre-existing reasoning traces or distillation 2. Multilingual Consistency: Forces models to reason "in the user’s language" via reward shaping 3. Infrastructure Scalability: Asynchronous system updates generators mid-generation for efficiency 4. Capability Preservation: Maintains multimodal understanding and function calling despite text-only RL 👉 How It Works The framework uses Group Relative Policy Optimization (GRPO) with three critical tweaks: - Dynamic Length Adaptation: Gradually increases response length thresholds to encourage deeper reasoning - Language Alignment Rewards: Penalizes code-switching by verifying consistency between problem language and solution - Minimalist Trust Regions: Relaxes policy update constraints to explore rare-but-correct reasoning paths Notably, RL training improved performance even in unseen domains (e.g., models trained solely on math tasks showed coding gains) and preserved multimodal skills inherited from base models like Mistral Medium 3. 👉 Practical Implications - Open-Source Release: Magistral Small (Apache 2.0 licensed) provides a reproducible baseline for RL research - Efficiency Gains: The asynchronous architecture processes 4K+ concurrent generations with minimal idle time - Cold-Start Potential: Smaller models achieved 70%+ AIME-24 accuracy "without" prior distillation The team also shared valuable negative results, including: - Proportional rewards for partial code correctness underperformed binary scoring - Entropy bonuses caused instability compared to adjusted clipping thresholds Why It’s Notable This work challenges assumptions that RL requires massive scale to be effective. By combining infrastructure innovation with careful reward design, Mistral demonstrates how RL can evolve language models’ "reasoning mechanics" rather than just their outputs. The open release of Magistral Small invites broader exploration of lightweight, RL-driven reasoning systems. Explore Magistral Small on Hugging Face | Full Paper Details or AI practitioners: How might pure RL pipelines reshape your approach to model specialization?

Big news in agentic AI! Researchers from Ant Research and Tsinghua University have just unveiled a major leap in training intelligent agents, tackling one of the biggest bottlenecks: scaling Reinforcement Learning for complex, multi-step tasks. Their new paper, "Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL," introduces ASearcher. Traditionally, training RL agents for search hits limitations with short turn limits and efficiency issues due to varying trajectory lengths. This restricts agents from truly exploring complex search paths. ASearcher breaks through these barriers with AReaL (Asynchronous Reinforcement Learning). This innovative framework decouples trajectory collection from model training, ensuring near-100% GPU utilization and enabling incredibly efficient, long-horizon RL. They also developed a brilliant Data Synthesis Agent that autonomously generates high-quality, challenging QA pairs, making the training data richer and more effective. The results are impressive! ASearcher-Web-QwQ achieves state-of-the-art performance, with substantial gains over existing 32B open-source agents. Crucially, it demonstrates "extreme long-horizon search," completing tool calls exceeding 40 turns and generating over 150,000 tokens during training. This entire project, including models, datasets, and code, is fully open-source and readily available on the Hugging Face Hub. This commitment to open science empowers everyone to build and train their own advanced search agents. A massive thanks to the team for pushing the boundaries of agentic AI and for sharing their work so openly. We encourage researchers to explore ASearcher and consider the Hugging Face Hub for sharing their own groundbreaking AI artifacts. Paper: https://lnkd.in/ekUmyqyH Models: https://lnkd.in/etMtgFaK Datasets: https://lnkd.in/e4XEkkdB Code: https://lnkd.in/exuKDJcW

Happy to share our recent work on making Federated Learning (FL) much more efficient over wireless networks. Federated Learning is a powerful way to train AI models collaboratively without sharing raw data — perfect for privacy and security. But there’s a big problem: in wireless environments, devices need to send huge amounts of information back and forth during training. This overloads the network, slows everything down, and makes FL impractical in many real-world settings. We developed two new methods — one based on zero-order optimization and another on first-order optimization — that cut communication costs dramatically. Instead of transmitting long gradient vectors with thousands or millions of numbers, each device sends just two tiny scalar values per round. Even better: We integrate the wireless channel effects directly into the learning process — no need to estimate or correct them. Our methods work even if devices are asynchronous (not perfectly synchronized). We provide theoretical guarantees of convergence, even in challenging non-convex settings. This could make FL far more practical for large-scale AI training at the network edge, from IoT devices to mobile phones, where bandwidth and latency are critical. It’s a step toward scalable, privacy-preserving AI that works in the real world. Big thanks to my co-authors Mohamad Assaad and Zeinab Nehme for their brilliant contributions! 🔗 Preprint: https://lnkd.in/gr7ZUWVS

Frontier agents are this good partly because the model was trained inside the very harness it ships with. NVIDIA's new paper "Polar: Agentic RL on Any Harness at Scale" brings that recipe to the open: it turns coding harnesses like Codex, Claude Code, Qwen Code or Pi into RL training environments without touching their internals. The core idea: every agent, however complex or closed, talks to a model through an API, so they put a proxy there. The harness runs exactly like in production while the proxy records prompts, sampled token ids and logprobs. Trajectories get rebuilt outside, token faithful, so gradients hit the exact tokens the policy sampled. The gains are consistent across all four harnesses. Same Qwen3.5-4B, plain GRPO, evaluated on SWE-Bench Verified: Codex 3.8 → 26.4 (+22.6) Claude Code 29.8 → 34.6 (+4.8) Qwen Code 34.6 → 35.2 (+0.6) Pi 34.2 → 40.4 (+6.2) The biggest gains appear on unfamiliar execution paths, Codex being the clearest case. The takeaway: you are not just training a model, you are training the model + harness system. Two engineering pieces make it work at scale. Async worker pools isolate container boots (CPU), agent execution (GPU) and long tail test runs, so slow runtimes never block the GPUs. And prefix merging stitches hundreds of captured API calls back into contiguous traces: 5.4x faster trainer updates and rollout GPUs at 88% utilization. It also doubles as an SFT data factory: 504 test verified agent traces from a 122B teacher, multi-turn conversations averaging 104 messages each, coming to the Hub under Apache 2.0 (release pending review). Paper authors: Binfeng Xu, Hao Zhang, Shaokun Zhang, Songyang Han, Mingjie Liu, Jian Hu, Shizhe Diao, Zhenghui Jin, Yunheng(Jackie) Zou, Michael Demoret, Jan Kautz and Yi Dong. > Paper: https://lnkd.in/eFJdzvrr > Code: https://lnkd.in/eTVerJGb > Training data: https://lnkd.in/eN5hFE37 We have written about some of these ideas lately: > TITO, on token fidelity: https://lnkd.in/e6sdygnM > The trainer-generator mismatch: https://lnkd.in/eiB-7DNd

What’s old is new again in ML infrastructure. Google's latest paper DiLoCo just brought LLM training back to Parameter Server era! For a lot of ML training workloads such as LLM, CV, perhaps only except for recommendation systems, we have gone a long way shifting away from the asynchronous Parameter Server architecture to synchronous training using Collective Communication (Uber's Horovod was an early framework for this), aka SPMD. Recently, with the extreme scale of modern AI supercomputers, lock-step synchronization has a fatal flaw: a single straggling chip or hardware failure stalls the entire cluster. The new DiLoCo paper fixes this by combining small group data-parallel sync training (learners) and cross-cluster async updates with CPU-only nodes. If a few learners report in, syncer updates that fragment of the model and keeps moving. This isolates hardware failures and stragglers. The rest of the cluster just keeps training. It is a great reminder that in systems engineering, architectures are rarely obsolete forever—they just go dormant until the scale of our bottlenecks shifts. https://lnkd.in/gszpivGd

🚀 𝗢𝗽𝗲𝗻𝗖𝗹𝗮𝘄-𝗥𝗟: 𝗧𝗿𝗮𝗶𝗻 𝗔𝗻𝘆 𝗔𝗴𝗲𝗻𝘁 𝗦𝗶𝗺𝗽𝗹𝘆 𝗯𝘆 𝗧𝗮𝗹𝗸𝗶𝗻𝗴 A new paper from Princeton introduces 𝗢𝗽𝗲𝗻𝗖𝗹𝗮𝘄-𝗥𝗟, a very interesting direction for agent training. The idea is simple: instead of relying only on curated datasets or offline RL pipelines, an agent can learn directly from real interactions. That is a big shift. In many current agent systems, inference and training still live in separate worlds. The model serves users, logs interactions, and only later those logs may become training data. OpenClaw-RL challenges that pattern by treating interaction streams as a live source of supervision. Every agent interaction generates a next-state signal: a user reply, tool output, terminal result, GUI update, or completion state. Rather than discarding those signals, OpenClaw-RL turns them into RL supervision. The framework extracts two kinds of signals:  • Evaluative signals — feedback about how well the action worked.  • Directive signals — hints about what the action should have been, recovered through hindsight-guided policy distillation. This is what makes the paper compelling. Conversations, tool traces, terminal sessions, GUI actions, and software engineering tasks become one learning loop. Another strong idea is the asynchronous design:  • The agent keeps serving live requests.  • The PRM evaluates interactions in parallel.  • The trainer updates the policy over time. So instead of waiting for large offline retraining cycles, the system moves toward continuous improvement from live usage. The paper shows this approach across categories: 💬 personal conversational agents 🖥 terminal agents 🧑💻 software engineering agents 🧰 tool-calling agents 🖱 GUI agents For personal agents, this is especially exciting. It suggests assistants could improve by observing corrections, follow-up questions, clarifications, and preferences. Everyday usage becomes training signal. For general-purpose agents, it opens the door to scalable RL across many environments without requiring a handcrafted dataset and reward loop for every domain. What I like most is the broader paradigm shift: We may be moving from dataset-centric agent training to interaction-centric agent training. That matters for production AI. Real systems live in dynamic environments. Users change intent. Tools fail. Context evolves. Static datasets are always behind reality. Interaction-native learning systems can adapt faster. Of course, this also raises important questions around safety, reward quality, drift control, and evaluation for continuously evolving agents. Overall, 𝗢𝗽𝗲𝗻𝗖𝗹𝗮𝘄-𝗥𝗟 feels like one of the more interesting steps toward agents that do not just respond, but actually get better with real use. 📄 Paper (arXiv): https://lnkd.in/gNeYSE7K #AI #LLM #AgenticAI #ReinforcementLearning #MachineLearning #OpenSourceAI #AIAgents 🚀

Have you ever wondered why data transfer between CPU and GPU becomes a silent bottleneck in LLM training pipelines? 🤔 When training large language models in #PyTorch, we often focus on GPU compute, attention efficiency, and memory optimization, but overlook a crucial detail: how data is staged before it even reaches the GPU. Enter Pinned Memory (Page-Locked Memory) in PyTorch. 🧠 At a high level, pinned memory is host memory that is locked in RAM and cannot be swapped out by the OS. This seemingly small detail unlocks a major performance benefit: 🚀 It enables faster and asynchronous GPU transfers via DMA (Direct Memory Access) Instead of the typical pipeline: CPU memory → pageable memory → GPU memory (slower, blocking in many cases) Pinned memory allows: CPU (pinned RAM) → GPU memory (overlapped + faster transfer) Why does this matter for LLM training? 📌 Faster data loading pipelines = higher GPU utilization 📌 Reduced input pipeline stalls during training steps 📌 Better overlap between data transfer and compute 📌 Crucial when using large datasets + high-throughput training setups In PyTorch, this is commonly enabled via: pin_memory=True in DataLoader .pin_memory() for tensors But the real gain shows up when combined with: ⚙️ multi-worker DataLoader pipelines ⚙️ asynchronous GPU copies (non_blocking=True) ⚙️ mixed precision training ⚙️ large-scale distributed setups In modern LLM training, performance is not just about GPU FLOPs; it is about end-to-end pipeline efficiency. Pinned memory is one of those low-level optimizations that quietly removes CPU→GPU friction. What other “hidden bottlenecks” have you run into in LLM training pipelines? 👇