Host:
Amit Prakash - Co-Founder & CTO of ThoughtSpot, former engineer at Google and Microsoft
Dheeraj Pandey - Co-Founder and CEO of DevRev, and former CEO of Nutanix
Guest:
Alex Dimakis - Professor at UC Berkeley and co-founder of Bespoke Labs
Summary:
In Episode 10 of the DevRev Podcast, host Dheeraj Pandey, co-host Amit Agarwal, and special guest Alex Dimakis, Professor at UC Berkeley and co-founder of Bespoke Labs, dive deep into the cutting-edge landscape of AI, machine learning, and reasoning models. They explore the evolution of foundation models, the growing importance of post-training techniques like supervised fine-tuning (SFT) and reinforcement learning, and how enterprises can harness modular AI systems for specialized tasks. With discussions ranging from synthetic data creation to the philosophy of monolithic vs. modular AI systems, this episode is a masterclass in bridging academic insights with practical enterprise applications.
Key Takeaways:
The Evolution of Foundation Models:
Foundation models are incredibly powerful, but their one-size-fits-all approach has limitations. Specialized agents trained on domain-specific data may hold the key to solving complex enterprise challenges. The ongoing debate between monolithic general-purpose AI and modular systems with specialized small models highlights the trade-offs between breadth and depth.Post-Training is the Future:
Supervised fine-tuning (SFT) and reinforcement learning (RL) are essential for adapting foundation models to domain-specific needs. The creation of synthetic data, particularly question-and-answer pairs grounded in internal knowledge, is a critical component for successful post-training.Challenges in AI Training:
Catastrophic forgetting, where new knowledge overwrites prior learning, remains a major challenge in fine-tuning. Current post-training tools and pipelines are still in their infancy, presenting an opportunity for innovation in this space.RAG vs. Reasoning Models:
Retrieval-augmented generation (RAG) is effective for shallow, fact-based tasks, but complex reasoning requires chain-of-thought techniques. Models tuned for step-by-step problem-solving are better suited for deeper logical workflows.Synthetic Data is Key to Post-Training:
Generating high-quality synthetic data—crafted as questions and answers grounded in internal data—enables models to adopt domain-specific reasoning capabilities. Tools and frameworks for automating synthetic data creation are pivotal to post-training success.Building Modular AI Systems:
Modular AI systems, composed of specialized small models, can complement foundation models by handling tasks with greater predictability, efficiency, and scalability. This approach mirrors how human teams operate, with generalists delegating specific tasks to specialists.The Business Opportunity in Modular AI:
Combining general-purpose foundation models with specialized systems unlocks a massive $100 billion enterprise AI opportunity. This is particularly valuable in domains like financial reporting, customer engagement, and other complex business processes.
In-Depth Insights:
1. The Rise of Specialized Agents
Dheeraj opened the episode with a thought-provoking question: Is the future of AI monolithic (general-purpose foundation models) or modular (a collection of specialized agents)? Alex introduced the concept of “compound AI systems” inspired by Unix’s modularity, where specialized tools work together to accomplish tasks more efficiently. While general-purpose models like GPT-4 excel at breadth, they often lack the depth and precision needed for domain-specific tasks.
2. Post-Training and the Role of Synthetic Data
Alex highlighted the importance of post-training techniques like supervised fine-tuning (SFT) to teach large language models new skills. However, the challenge lies in creating high-quality training data. Synthetic data generation, which involves creating question-and-answer pairs based on internal documents, emerged as a crucial strategy.
He shared insights from Bespoke Labs' work on building post-training pipelines:
Using GPT to generate synthetic Q&A data from internal documents.
Combining diverse personas, scenarios, and multi-turn conversations to create rich training datasets.
The surprising efficiency of SFT: only a few thousand carefully curated examples can yield dramatic improvements in reasoning capabilities.
3. Catastrophic Forgetting and Continual Learning
Amit raised concerns about catastrophic forgetting — where fine-tuning a model erases prior knowledge. Alex discussed strategies to mitigate this, such as interleaving original training data during fine-tuning. He also noted that reinforcement learning techniques, where models are guided step-by-step, can minimize the amount of new data required while preserving prior learning.
4. Reasoning Models: The New Frontier
Alex and Amit explored the limitations of current foundation models in reasoning tasks. Retrieval-augmented generation (RAG) works well for retrieving facts, but tasks like writing SQL queries or solving complex math problems demand deeper reasoning. Here’s where chain-of-thought (CoT) techniques shine — models are trained to generate step-by-step reasoning, enabling them to solve complex problems.
Alex shared an example from Berkeley’s SkyT1 model, which achieved reasoning capabilities by fine-tuning on just 17,000 examples of synthetic math problems. This shows the potential for even small, focused datasets to produce remarkable results.
5. Balancing Generalist and Specialist Models
One of the key debates was how to integrate generalist foundation models with specialist small models. Dheeraj drew an analogy to startups scaling their leadership teams: generalists (like founders) eventually delegate to specialists (like executives) as complexity grows. Similarly, modular AI systems could allow foundation models to delegate specialized tasks to smaller, more efficient models.
Alex outlined the architecture for such systems:
A hub-and-spoke model where a foundation model acts as the orchestrator, delegating tasks to specialized agents.
Specialized agents can handle domain-specific functions, such as financial reporting or customer support, with higher accuracy and lower latency.
Host Biographies
Dheeraj Pandey
Co-founder and CEO of DevRev, and former CEO of Nutanix. Dheeraj has led multiple tech ventures and is passionate about AI, design, and the future of product-led growth.
Amit Prakash
Co-founder and CTO at ThoughtSpot, previously at Google and Microsoft. Amit has an extensive background in analytics and machine learning, holding a Ph.D. from UT Austin and a B.Tech from IIT Kanpur.
Guest Information:
Alex Dimakis is a professor at UC Berkeley, an expert in machine learning, and the co-founder of Bespoke Labs, a company focused on building AI systems tailored to enterprise needs. Alex has spent over a decade researching machine learning foundations, including time at USC and UT Austin. Learn more about Bespoke Labs at Bespoke Labs.
Episode Breakdown
{00:00:00} Setting the Stage:
Dheeraj and Amit kick off the episode after a hiatus, introducing their guest Alex and the day’s topic: the evolution of AI, modular architectures, and reasoning in LLMs.
{00:01:00} Guest Introduction:
Alex, a professor and AI researcher, shares his journey from UT Austin to UC Berkeley, his experience with foundational machine learning projects, and starting Bespoke Labs.
{00:03:00} Evolution of AI and Deep Learning:
Alex recounts his shift from classical machine learning to deep learning at UT Austin during the rise of AlexNet, outlining major milestones like NSF’s IFML initiative.
{00:04:00} Generalist vs. Specialist Models:
The group debates the trade-offs between large foundational models (generalists) and smaller, task-specific models (specialists), exploring potential applications of both in enterprise.
{00:08:00} Modular AI Architectures:
Alex introduces the concept of modular architectures versus monolithic AI systems, drawing parallels with Unix philosophy and modern microservices in computing.
{00:12:00} Context, RAG, and Specialized Models:
They discuss Retrieval-Augmented Generation (RAG) as a tool to integrate enterprise context into LLMs and where it might fall short for deeper reasoning tasks.
{00:19:00} Reasoning and Chain of Thought:
The conversation shifts to reasoning in AI models, the role of chain-of-thought prompts, and emerging techniques for enabling more sophisticated decision-making.
{00:27:00} Post-Training and Fine-Tuning:
Alex breaks down the process of supervised fine-tuning (SFT), the challenges of creating effective post-training data, and how RL techniques can refine LLMs for specialized tasks.
{00:34:00} Synthetic Data for Post-Training:
The group delves into synthetic data generation, how it aids in creating question-and-answer datasets, and its importance for embedding organizational knowledge into AI models.
{00:40:00} AI Memory and Catastrophic Forgetting:
Amit brings up challenges like catastrophic forgetting and new approaches, including Sakana and Google’s recent advances in long-term AI memory.
{00:46:00} Analogies to Biology and Human Learning:
They compare the learning dynamics of AI models with human learning, discussing “plasticity,” long-term memory, and the role of built-in biases in both systems.
{00:52:00} Practical Applications in Business AI:
Dheeraj and Amit explore how these AI advancements can be applied to enterprise workflows, highlighting examples like ITSM, financial reporting, and business decision-making.
{01:01:00} Challenges in AI Scalability and Workflows:
The hosts and Alex discuss the hurdles in building scalable, autonomous AI systems for business use, including managing ambiguity in workflows and tool orchestration.
{01:09:00} Future Directions in AI Development:
The conversation turns to the future of post-training stacks, improving tools for fine-tuning, and enabling companies to better leverage their proprietary data in AI solutions.
{01:12:00} Closing Reflections:
The group wraps up by reflecting on the importance of collaboration between AI researchers and engineers to solve practical challenges, emphasizing the value of bespoke solutions in the enterprise AI space.
References and Resources
Bespoke Labs
Bespoke Labs specializes in building AI systems tailored for enterprise needs, focusing on post-training pipelines, synthetic data generation, and specialized models. Their tools and expertise help companies unlock the power of AI by connecting general-purpose models with domain-specific applications.
Learn more about Bespoke LabsBerkeley SkyT1 Model
The Berkeley SkyT1 Model showcases the potential of reasoning models trained on synthetic data. With a focus on math and coding tasks, it demonstrates how even modest datasets with step-by-step solutions can enable models to achieve high reasoning capabilities.
Discover more about Berkeley’s SkyT1 ModelDataComp Dataset
DataComp is an open dataset initiative aimed at democratizing large-scale AI training. It provides a curated dataset designed to improve the pre-training of foundation models while offering insights into data selection and preparation for machine learning applications.
Explore the DataComp DatasetRetrieval-Augmented Generation (RAG)
Retrieval-augmented generation (RAG), a method pioneered by Meta AI, combines pre-trained language models with external retrieval systems to improve performance on fact-based tasks. This approach is particularly useful for integrating enterprise-specific knowledge into AI systems.
Read Meta’s research on RAGSynthetic Data Generation: The “Self-Instruct” Paper
The “Self-Instruct” paper demonstrates a simple yet effective method for generating synthetic question-and-answer datasets. By using large language models to create diverse prompts, scenarios, and personas, this approach accelerates post-training pipelines for specialized AI systems.
Read the Self-Instruct Paper on Arxiv
Conclusion:
The future of AI lies in the synergy between foundation models and specialized agents. As enterprises seek to leverage AI for increasingly complex tasks, tools like supervised fine-tuning, synthetic data generation, and reasoning models will be game changers. Whether you’re building workflows, financial systems, or customer engagement platforms, the combination of generalist intelligence and specialist skills is key to unlocking new opportunities.
Got questions? Leave a comment or reach out to Alex Dimakis and the team at Bespoke Labs to learn more about building domain-specific AI systems.
Share this post