A Concern for implementing LLMs in the context of Human-Robot Interaction
Conceptual Illustration, not empirical performance data - Mehrdad Farimani, Nov. 2026, Sweden
There is a pattern I’ve been observing through my own interactions with generative AI and LLMs, robotics project implementations, and conversations with other experts in the field.
The first output is often surprisingly good, particularly when generating images, text, code, or even CAD concepts. But as we try to refine it through additional prompts, things sometimes start to go in the wrong direction. A correction might improve one detail but negatively affect something that was already right. After several iterations, we might end up with something worse than the original, despite having provided more information and clearer instructions.
Now, I can already imagine some tech-bros commenting: “Ah, you just need to buy my course to learn how to prompt properly!”
Of course, bro, prompt engineering matters. The way we structure instructions, manage context, and define constraints can significantly affect the results. But I don’t think we should dismiss this issue as simply poor prompting, especially when we’re talking about systems that are eventually expected to interact naturally with humans, through speech in the case of speech-to-action in robotics.
This brings up a fundamental question: Does giving an AI system more instructions necessarily bring us closer to the intended result?
We usually approach these systems assuming that refinement leads to improvement, makes sense! We look at the first result, think it’s good enough, but naturally assume it can get better. So we keep prompting, expecting each correction to bring us closer to what we have in mind. However, the result doesn’t always follow that trajectory. Sometimes it improves, sometimes it plateaus, and sometimes it actually gets worse.
There is some research supporting this concern. A 2025 study titled LLMs Get Lost in Multi-Turn Conversation reported an average 39% performance drop across six generation tasks when instructions were provided gradually rather than all at once. Research on iterative image editing has also identified issues with accumulating errors and preserving consistency across successive modifications.
This doesn’t mean every additional prompt makes the result worse. But it does suggest that generating a good initial result and reliably improving that result are two different capabilities.
Now, what happens when we bring this into robotics?
With the growing integration of LLMs and Vision-Language-Action (VLA) models into robotic systems, natural language is becoming an increasingly attractive interface for communicating with robots.
But here’s something important about human behavior: we rarely give someone all the instructions at the beginning of a task.
Instead, we guide them gradually. We observe what they’re doing, correct them, change our minds, and sometimes introduce new constraints halfway through the process. We expect them to understand that a new instruction is usually a refinement of the previous one, not a replacement for everything we have already agreed on.
Imagine asking a household robot to clean the kitchen. Initially, you ask it to clean the counter. Then you tell it not to move the coffee machine. A few minutes later, you ask it to clean behind the machine as well. The robot now needs to interpret the new instruction without violating the previous constraint.
Or consider an industrial robot performing a manipulation task. An operator might ask it to adjust its grip, change the approach angle, or increase its speed. Each correction is intended to improve the operation without compromising the parts of the task that were already working correctly.
These are fairly simple examples, but imagine hundreds of interactions over days or weeks, with different users, changing environments, and sometimes contradictory instructions.
The challenge is no longer just understanding an individual command. It’s maintaining continuity of intent across multiple interactions, preserving constraints, and distinguishing between what should change and what should remain unchanged.
Of course, a robotic system isn’t simply an LLM connected to a set of actuators. Perception, memory, state estimation, planning, and closed-loop control can help keep the system grounded in the physical world. But if we increasingly rely on generative models to interpret human intent and make higher-level decisions, their behavior across repeated interactions becomes an important part of the overall system reliability.
The expectation gap
I think there’s another interesting aspect to this problem, which is the difference between how AI systems actually improve and how we expect them to improve.
Across model generations, we’re generally seeing improvements in capability. Each new model tends to start from a stronger baseline than the previous one. But within a particular model, repeated prompting doesn’t necessarily follow the same upward trajectory.
We might start with a strong result, then experience a gradual decline or fluctuations as more corrections are introduced, only to get a better starting point when the next generation of models arrives.
Meanwhile, as users, we expect almost the opposite. We assume that every additional instruction should improve the outcome. That’s precisely why we keep refining the prompt. We rarely think that the first result might already be the best one we’re going to get from that particular interaction.
This creates what I’d call an expectation gap (dashed lines in the graph) between the improvements we anticipate and the actual trajectory of the system’s performance.
It’s worth emphasizing that this is a conceptual pattern rather than a universal law of LLM behavior. Some iterative workflows perform very well, particularly when the system has explicit evaluation criteria, persistent state, and mechanisms for verifying changes.
But for human-robot interaction, this gap deserves more attention.
In conversational AI, a failed correction might mean rewriting a paragraph or regenerating an image. In robotics, the same kind of inconsistency could result in an unexpected physical action, a failed task, or potentially a safety issue.
Before making LLMs and VLAs central to human-robot interaction, I think we need to evaluate more than their ability to understand and execute an initial instruction. We should also investigate how reliably they preserve previous constraints, handle incremental corrections, and maintain the user’s original intent across repeated interactions.
After all, successful human-robot collaboration isn’t just about getting the first instruction right. It’s about making sure the fifth or tenth correction brings us closer to the goal, rather than further away from it.
Ironically, this text was proofread by AI; the same pattern was experienced :)
