Understanding AI's Confrontation with Real-World Issues

Sachin KunduSachin Kundu
21 min read

Table of contents

Let's be honest, the last decade in AI has been a wild ride. Deep learning has been like a brilliant PhD student who aced every exam they ever took. It devoured the internet, and in return, gave us god-like powers in the digital world. It writes our emails, generates photorealistic images of astronauts riding unicorns, and can beat the world's best players at games so complex we haven't even figured out the rules. We threw a party, and the champagne was flowing.

But now, we're trying to get that same brilliant student to do something a little more... practical. We're asking it to make toast. And suddenly, it’s staring at the toaster like it's an alien artifact, holding the bread like a foreign object, and has a 50/50 chance of setting off the smoke alarm.

This is the core dilemma of embodied AI. The very techniques that achieved god-tier status on the clean, predictable grid of the internet are proving clumsy and brittle in the messy, high-dimensional, and physics-bound real world. The data of the physical world simply doesn't conform to the clean manifold hypothesis as nicely as curated image datasets do. The party's over, the cleanup has begun, and we're realizing our guest of honor might have been a little overhyped for this new environment.

A Robot's Comedy of Errors

If you've spent any time in a robotics lab, you've seen the highlight reel. You have a multi-million dollar humanoid that walks perfectly, but put a stray pencil in its path and it collapses with the grace of a dropped filing cabinet. In technical jargon, this is a classic out-of-distribution (OOD) generalization problem. The pencil represents a tiny deviation from its training data, but it’s enough to cause a catastrophic failure.

These models are also comically sample-inefficient. A human toddler might learn to open a cabinet after one or two tries by forming a causal model of how hinges and latches work. A state-of-the-art reinforcement learning agent might need tens of thousands of simulated attempts to learn the same task, essentially memorizing a huge set of state-action pairs without any real understanding.

This points to the deepest problem: a fundamental lack of common sense and intuitive physics. As AI pioneer Yann LeCun famously put it in his cake analogy, unsupervised or self-supervised learning is the cake, supervised learning is the icing, and reinforcement learning is the cherry on top. Our current embodied agents are trying to survive on a diet of cherries and icing, while missing the foundational understanding of the world—the cake—that we humans get from observation. They can correlate pixels to actions, but they don't know that an unsupported object will fall.

The 'Bigger Hammer' Fallacy

The default response from the deep learning playbook is simple: if it’s not working, you’re not using a big enough hammer. This is the era of scaling laws, where foundational papers from labs like OpenAI and DeepMind have shown that performance on many benchmarks predictably improves with more compute, more data, and larger models.

But the physical world laughs at our bigger hammers.

Imagine trying to solve a Rubik's Cube. The "scaling" approach is like building a machine to try every single one of the 43 quintillion possible combinations. The scaling laws for this machine would be perfect, but the process is absurdly inefficient. The intelligent approach is to understand the cube—to learn the algorithms and group theory behind the moves.

For robotics, pure scaling is that brute-force machine. The amount of data required to cover every possible physical interaction is effectively infinite. While scaling helps, relying on it alone may be a path of steeply diminishing returns.

Daydream Believers: Giving Robots an Inner Life

So if the bigger hammer isn't the answer, what is? The most compelling path forward isn't just a single idea, but a full-fledged research program that reimagines how machines learn about the world. To understand it, we have to go back to a 2022 manifesto from Yann LeCun, aptly titled "A Path Towards Autonomous Machine Intelligence."

The paper laid out a blueprint for AI that could reason, plan, and understand the world with human-like common sense. The core problem it identified was that trying to predict the future in all its pixel-perfect detail is computationally explosive and, frankly, wasteful. A self-driving car doesn't need to predict the exact rustle of every leaf on a tree; it needs to predict that a ball rolling into the street will likely be followed by a child.

The solution proposed was a revolutionary shift in perspective: the Joint-Embedding Predictive Architecture (JEPA). The core idea is simple but profound: what if, instead of trying to paint a perfect picture of the future, the AI just learned to predict the gist of it? In technical terms, it learns to predict the abstract representation of a future state, not the raw data itself.

This was the theory, but the real test came in 2023 with I-JEPA, the first concrete implementation. Researchers at Meta AI created a system that was shown parts of an image and tasked with predicting the abstract representation of the missing pieces. It wasn't asked to "inpaint" the missing pixels, but to understand the high-level features of what should be there. The results were stunning: I-JEPA was not only highly efficient but excelled at learning useful representations with far less labeled data, proving the core hypothesis was sound.

With still images conquered, the logical next frontier was video. Later, V-JEPA extended the same principle to the temporal domain. Instead of predicting a missing patch in a photo, it learned to predict the abstract "idea" of what would happen next in a video. This is a crucial step closer to a true world model—a system that learns the intuitive "plot" of the physical world, not just its scenery.

And this brings us right back to our clumsy robot trying to make toast. The ultimate goal of this research program is embodied intelligence. The latest iterations, like V-JEPA 2 and ACT-JEPA, are now applying this predictive engine to robotics and sequential decision-making. By learning to predict the abstract consequences of actions, an agent can plan, reason, and act in the physical world with a model that understands cause and effect. Instead of just mapping pixels to motor commands, the robot can "imagine" a sequence of abstract outcomes to achieve a goal, making it vastly more flexible and sample-efficient.

From Daydreams to Diamonds: The Story of the Dreamer Agents

While the JEPA family was exploring one path to building common sense, another equally epic saga was unfolding at Google and DeepMind. This story, the "Dreamer" lineage, starts with the same foundational question: why should an AI learn by constantly bumping into walls in the real world when it could just imagine bumping into them first?

The tale begins in 2018 with the landmark "World Models" paper by Ha & Schmidhuber. It was a brilliantly simple proof of concept. The agent's brain had three parts: a Vision model (a VAE) that compressed what it saw into a compact summary, a Memory model (an RNN) that predicted how that summary would change over time, and a tiny Controller that made decisions using only the cheap, internal predictions. It was a resounding success, showing an agent could learn to navigate a simulated world almost entirely in its own "daydream."

The community took notice, and the idea evolved. The 2019 follow up, PlaNet, gave the agent a major engine upgrade. It introduced the Recurrent State Space Model (RSSM), a far more sophisticated way to model the world's dynamics, especially from messy, high dimensional pixels. Think of the original World Model as a basic physics simulator; PlaNet was the upgrade to a high fidelity, next generation game engine for the AI's imagination.

But here’s where the story takes a fascinating turn with DreamerV1. The team asked a new question: what if we don't just use the dream to plan our next move, but use it to learn our entire behavior? By backpropagating value gradients through imagined trajectories, the agent could learn an effective policy without ever taking a step in the "real" world. It was like a pilot learning to fly entirely in a hyper realistic flight simulator, mastering their craft through pure imagination. This innovation made learning incredibly sample efficient.

With this new power, it was time for a trial by fire. DreamerV2 (2021) took on the ultimate video game challenge: the Atari 55 benchmark. This was the home turf of model free agents, which had dominated for years. The key innovation was using discrete latent representations, essentially teaching the model to think in terms of clean "concepts" or "categories" instead of fuzzy, continuous values. The result? DreamerV2 became the first model based agent to achieve human level performance, proving that learning from a world model wasn't just efficient; it was a world class competitor.

This brings us to the final evolution (for now): DreamerV3 (2023). This is where the idea truly matures from a specialist into a generalist. The team created a single, "master key" algorithm with fixed settings that could solve over 150 diverse tasks, from controlling a robot arm with visual feedback to the holy grail of AI challenges: autonomously collecting a diamond in Minecraft. This wasn't just about winning a game anymore; it was about creating a robust, general purpose agent that could adapt to wildly different worlds.

This entire saga, from a simple "what if" to a generalist agent that can master Minecraft, tells a clear story: building and dreaming within an internal world model is one of the most powerful and scalable paths we have toward more general artificial intelligence.

The Thinker and the Doer: Two Paths to the Imagination Engine

After our two deep dives into the JEPA and Dreamer sagas, you might be thinking: are these two research programs rivals? Or are they just two sides of the same coin? The answer, like most things in science, is a bit of both. They are two brilliant teams of explorers trying to reach the same summit—true machine intelligence—but they've chosen different paths and packed different gear.

Their shared belief is the foundation of this entire article: the fastest path to smart, adaptable AI is to build a predictive internal model of the world. They both agree that agents need to learn by imagining, not just by doing. But how and why they imagine reveals a fascinating philosophical split.

Path 1: JEPA, The Worldly Philosopher

Think of the JEPA program as a team of worldly philosophers or linguists. Their primary obsession is understanding. They believe that if you can teach a machine to form rich, abstract concepts about the world, then reasoning and intelligent action will naturally follow.

JEPA learns by observation, like a linguist trying to decipher a new language. Its goal is to build a universal dictionary of concepts (a "joint embedding space"). It learns that "a ball" is a concept, "rolling" is a concept, and that they often go together, all without ever needing to generate a perfect pixel-by-pixel video of a rolling ball. Its approach is non-generative; it’s focused on defining the world, not recreating it.

Path 2: Dreamer, The Pragmatic Pilot

The Dreamer team, on the other hand, is full of pragmatic pilots and engineers. Their primary obsession is acting. They want to solve a mission, whether it's landing a lunar module or collecting a diamond in Minecraft. To do this, they build the most hyper realistic flight simulator they can (their "world model").

The entire purpose of their dream world is to be a safe, fast, and cheap place to practice. Dreamer learns by running millions of simulations within its imagination to discover the best possible strategy for a given task. Its world model is a means to an end: becoming an expert doer through model based reinforcement learning. It's a master of practice.

The Tale of the Tape

Dimension

JEPA (The Thinker)

Dreamer (The Doer)

Primary Goal

Learn a universal understanding of the world.

Learn to act efficiently to solve specific tasks.

Core Method

Predict abstract concepts to build a rich library of knowledge.

Predict future states to create a practice simulator.

The "Dream" Is For

Understanding the fundamental rules and concepts of the world.

Rehearsing actions and policies to master a mission.

Killer App

Creating highly efficient and generalizable representations from raw data.

Solving complex reinforcement learning tasks with record breaking sample efficiency.

The Paths Converge

Here’s the beautiful part: these two paths are beginning to merge. The philosophers at JEPA are realizing their rich world understanding is incredibly useful for teaching agents how to act and plan. Meanwhile, the pilots of Dreamer are finding that building richer, more abstract concepts into their simulators makes them more robust and general.

The Thinker is learning how to do, and the Doer is learning how to think more deeply. Both are proving, from different directions, that the next leap in AI will come from machines that don't just process the world, but imagine it.

Spock Meets Kirk: The Neuro-Symbolic Revolution

So far, we’ve explored two incredible research programs, JEPA and Dreamer, that are teaching machines to imagine. But both are still fundamentally rooted in the world of neural networks. There's a third, radical path gaining momentum, one that looks at the problem and says: why force our neural networks to learn everything from scratch when we already have a perfectly good system for logic and reason?

For decades, AI research was split into two rival camps. On one side were the Symbolists, the architects of "good old-fashioned AI" who believed intelligence was all about logic, rules, and structured knowledge. On the other were the Connectionists, who championed neural networks and the idea that intelligence could emerge from data. For the last decade, the Connectionists have been throwing a massive victory party.

But the last few years have seen a truce, and now, a powerful alliance. Researchers are realizing that the future isn't a choice between logic and learning; it's about combining them. This is the neuro-symbolic revolution.

Think of it as finally putting Spock and Captain Kirk on the same bridge. The neural network is Kirk: the intuitive, pattern-matching genius who can make sense of a chaotic, messy world from raw data. The symbolic system is Spock: the rigorous logician who operates on facts, rules, and pure reason.

How Does This Relate to JEPA and Dreamer?

This is the fascinating part. Our imaginative Dreamer and JEPA agents are like Captain Kirk trying to develop his own Spock-like logic through pure observation. By predicting the world, they are implicitly learning its rules. A neuro-symbolic agent, however, is like Kirk having Spock standing right there on the bridge to consult.

Both are building a "world model," but the approach is different:

  • JEPA/Dreamer: Learns an intuitive world model from the ground up.

  • Neuro-Symbolic AI: Builds a hybrid world model, where the neural net handles perception and the symbolic system manages the explicit rules, physics, and causal reasoning. It has a cheat sheet grounded in logic.

The Neuro-Symbolic Superpowers

This hybrid approach unlocks several "superpowers" that are notoriously difficult for pure deep learning systems, offering a compelling alternative to just building bigger and bigger models.

  • The "Why" Button (Explainability): Because part of its "brain" operates on clear logic, a neuro-symbolic agent can explain its decisions. You can audit its reasoning, a feat nearly impossible for a pure "black box" neural network.

  • Built-in Guardrails (Safety & Trust): You can program hard constraints, ethical rules, and physical laws directly into the symbolic side. It provides a framework to ensure the AI "plays by the rules" in high-stakes environments.

  • Learning on a Dime (Data Efficiency): By leveraging logical rules, these systems can generalize from far less data. If it knows the rule for gravity, it doesn't need to see a million objects fall to understand what will happen to the next one.

In the grand quest for artificial general intelligence, the neuro-symbolic path offers a compelling vision. It’s not just about building a bigger brain, but a better structured one—one that combines the intuitive genius of Kirk with the rigorous, trustworthy logic of Spock. It might just be the most direct route to an AI we can not only use, but truly understand.

From Robot Boot Camp to Kindergarten: The Rise of the Lifelong Learner

So far, we've explored the incredible new "brains" being designed for AI agents—the imaginative engines of JEPA and Dreamer, and the logical-intuitive hybrid of neuro-symbolic AI. But a brilliant brain is only half the story. The other half is the learning strategy. For decades, we've trained robots like they're in a one-shot "boot camp"—drilling them on a static dataset until they pass a test.

The last decade has seen a revolution against this rigid approach, inspired by a simple but profound model: a human child. This has given rise to a new kind of agent: the curious, developmental, lifelong learner.

The Spark: Curiosity and a Curriculum (2015-2018)

The story begins with a simple question from psychology: what if we gave robots a sense of curiosity? Pioneering work, especially from researchers like Pierre-Yves Oudeyer, introduced intrinsic motivation. Instead of only chasing external rewards (like "success"), the robot gets a little dopamine hit from novelty, surprise, or simply making progress on a skill. It becomes an explorer, driven to find things that challenge its understanding of the world.

At the same time, Yoshua Bengio's hugely influential idea of Curriculum Learning took hold. The concept is straight out of primary school: don't start with calculus. Start with simple addition and build from there. By structuring training from easy to hard, robots could learn complex skills far more efficiently.

Curing Robot Amnesia (2019-2022)

This new, curious student immediately ran into a huge problem: catastrophic forgetting. When a traditional neural network learned a new skill, it would often completely erase its memory of the old one. It was a severe case of robot amnesia.

The solution was continual learning. Researchers developed new frameworks that allowed a robot to accumulate knowledge over a lifetime. It could learn to open a door today, and then learn to stack a cup tomorrow without forgetting how doors work. This was the key to creating robots that could adapt and grow, rather than being static, one-trick ponies.

The Robot Becomes Its Own Teacher (2023–2025)

This is where it all comes together in the state-of-the-art. The robot is no longer just a curious student following a pre-made curriculum. It becomes its own teacher.

Using techniques like Intrinsically Motivated Goal Exploration Processes (IMGEPs), the robot sets its own goals. It analyzes its own progress and asks, "What's the most interesting thing I could try to learn right now?" This leads to a self-organized curriculum, where the robot creates a personalized lesson plan perfectly tailored to maximize its own growth. It's the ultimate autonomous learner.

The Grand Unification: How It All Clicks Together

So how does this curious, lifelong learner relate to JEPA, Dreamer, and our Spock-and-Kirk hybrid? It's the operating system they all run on.

  • Fuel for the Imagination Engine: Curiosity is the fuel for world models like JEPA and Dreamer. The "surprise" a robot feels when its prediction is wrong is precisely the learning signal that forces it to build a better, more accurate internal model of the world.

  • A Persistent Mind: Continual learning ensures that the knowledge captured by a neuro-symbolic system or the world model of a Dreamer agent isn't lost. It provides the long term memory needed for true intelligence.

  • The Ultimate Synthesis: The most advanced robots today are a beautiful synthesis of all these ideas. They are developmental learners (the learning strategy) with a hybrid neuro-symbolic brain (the architecture) that uses a predictive world model (the cognitive tool) to imagine the world.

This shift from a "boot camp" to a "kindergarten" model is profound. It's how we get from robots that can perform a single, pre-trained task to autonomous agents that can learn, adapt, and grow for a lifetime, getting ever closer to the flexible, general intelligence we see in the natural world.

The Ultimate Reality Check: Forcing Our AI to Obey the Law (of Physics)

We’ve explored the incredible new minds we're building for AI: imaginative dreamers, logical reasoners, and curious, child-like learners. But there’s a final, crucial piece of the puzzle. What happens when a brilliant AI, trained on petabytes of data, devises a plan that is elegant, efficient, and completely violates the laws of physics?

This isn’t a theoretical problem. Without a grounding in reality, a robot might try to move its arm through a solid table, or a predictive model might design a new molecule that is energetically impossible. The final frontier, then, is not just making AI smarter, but making it obey the rules of the universe. This is the domain of Physics-Informed Machine Learning (PIML).

The Eureka Moment: When Physics Becomes the Teacher

The revolution began in earnest around 2017 with a groundbreaking idea known as Physics-Informed Neural Networks (PINNs). The concept was as elegant as it was powerful. When training a neural network, you normally grade it on one thing: how close its answer is to the data. PINNs added a second grade: how well its answer conforms to a known law of physics, like a partial differential equation (PDE) describing fluid dynamics or heat transfer.

The loss function, the very thing the AI tries to minimize, was now a blend of data error and a "physics residual." In simple terms, it's like telling the AI, "Your answer might look right, but it violates conservation of energy, so it's wrong. Go back and try again until your solution is not only accurate but physically plausible." This simple twist enabled neural networks to solve complex science and engineering problems with far less data, because the physics filled in the gaps.

The Great Convergence: How Physics Grounds Imagination

PIML didn't stay in its own lane for long. Its core idea, injecting hard constraints into learning, was too powerful to ignore. It quickly began to fuse with the other paradigms we've discussed, acting as the ultimate grounding force.

  • The Physics Teacher Meets the Dreamer: Remember our JEPA and Dreamer agents, learning to imagine the world? PIML gives their imaginary worlds a set of non-negotiable rules. By infusing their internal simulators with real physics, the agents' "dreams" become far more realistic. This is a game-changer for robotics, dramatically speeding up sim-to-real transfer and ensuring a robot doesn't learn impossible habits in its imagination.

  • A Cousin of Spock: PIML is a close cousin to Neuro-Symbolic AI. While neuro-symbolic methods inject logical rules ("If A, then B"), PIML injects physical rules (like F=ma). Both attack the same weakness of pure deep learning by embedding explicit human knowledge. The convergence is already happening, with state-of-the-art models combining PINNs with symbolic regression to produce answers that are not only physically correct but also human-readable equations.

The Full Picture: A New Architecture for Intelligence

The most advanced AI systems of the near future won't be just one of these things; they will be a modular hybrid, a beautiful synthesis of all these ideas. Picture tomorrow's autonomous robot:

  • It uses a JEPA-like architecture for raw perception, turning sensor data into rich, abstract concepts.

  • It feeds those concepts into a Dreamer-like world model, imagining possible futures to plan its actions.

  • Its actions are governed by a Neuro-Symbolic reasoning layer that handles logic, strategy, and ethical constraints.

  • And the entire system is wrapped in a PIML core, ensuring that every perception, prediction, and action is consistent with the fundamental laws of physics.

This is the path forward. The convergence of these fields—the imaginative, the logical, the developmental, and the physical—is creating a new generation of robust, efficient, and trustworthy machine intelligence. It's how we finally build an AI that doesn't just live on the internet, but can act safely and intelligently in our world.

The Definitive Further Reading List

Here is the complete, curated list of key papers covering all the major topics in the article.

1. The JEPA (Joint Embedding Predictive Architecture) Series

  • LeCun, Y. (2022): "A Path Towards Autonomous Machine Intelligence"

  • Assran, M., et al. (2023): "Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture" (I-JEPA)

  • MA, Jean-Baptiste, et al. (2024): "V-JEPA: Video Joint-Embedding Predictive Architecture"

    • Role: The crucial extension of JEPA to the video domain, enabling the learning of temporal and dynamic representations.

    • Link: https://arxiv.org/abs/2401.08343

2. The Dreamer Series by DeepMind/Google

  • Ha, D., & Schmidhuber, J. (2018): "World Models"

  • Hafner, D., et al. (2019): "Learning Latent Dynamics for Planning from Pixels" (PlaNet)

    • Role: The critical bridge between "World Models" and Dreamer, introducing the RSSM engine that powers the entire Dreamer series.

    • Link: https://arxiv.org/abs/1811.04551

  • Hafner, D., et al. (2020): "Dream to Control: Learning Behaviors by Latent Imagination" (DreamerV1)

  • Hafner, D., et al. (2021): "Mastering Atari with Discrete World Models" (DreamerV2)

  • Hafner, D., et al. (2023): "Mastering Diverse Domains through World Models" (DreamerV3)

3. Curriculum Learning and Intrinsic Motivation

4. Neuro-Symbolic AI

  • Garcez, A. S. d'Avila, & Lamb, L. C. (2020): "Neurosymbolic AI: The 3rd Wave"

  • Colelough, J., & Regli, W. (2025): "Neuro-Symbolic AI in 2024: A Systematic Review"

5. Physics-Informed Machine Learning (PIML)

  • Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2017): "Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations"

  • Ramesh, A., & Ravindran, B. (2022): "Physics-Informed Model-Based Reinforcement Learning"

0
Subscribe to my newsletter

Read articles from Sachin Kundu directly inside your inbox. Subscribe to the newsletter, and don't miss out.

roboticsAI

Written by

Sachin Kundu
Sachin Kundu