MIT and NYU Engineers Introduce FLEX to Help Robots Infer Intent from Natural Language
MIT and NYU researchers have introduced FLEX, a platform that uses LLMs like GPT-4o to help robots deduce implicit rules and common sense from conversational commands.
Engineers from MIT and New York University have created a new artificial intelligence learning platform designed to help robots understand the implicit meaning behind human instructions. Named FLEX (flexible feature-editing pipeline for robot motion-planning), the system allows robots to process conversational natural language and infer broader common-sense rules rather than merely executing literal commands. The research will be presented at the annual Conference on Robot Learning.
In standard robotics setups, translating human preferences into trajectory planning requires complex mathematical inputs or explicit physical directives. When users supply feedback in natural language, traditional systems typically add every mentioned item to a list of obstacles. Over time, this list becomes bloated and inefficient as environmental conditions shift. FLEX addresses this limitation by using a large language model to extract high-level intent from brief user inputs.
The three-step FLEX framework
The FLEX pipeline operates in a three-step cycle that completes in about five seconds:
Add: When a user gives a command—such as warning a robotic arm carrying a bowling ball, "Don't drop it on my mug!"—the robot adds the specified object to its trajectory avoidance list. This step reflects standard natural language motion-planning methods.
Generalize: FLEX passes the updated feature list to a large language model, specifically OpenAI's GPT-4o. By leveraging vast amounts of web text data, the model identifies underlying patterns. For example, if a user warns the robot away from a laptop while carrying coffee, the LLM recognizes that liquids damage electronics, inferring that the robot should avoid all nearby electronic devices like smartphones. Similarly, a command to avoid a mug prompts the robot to generalize to all fragile items, while avoiding paper prompts it to avoid flammable objects.
Mask: The platform uses a standard computer vision system to check the physical environment. If previously added objects are no longer present, FLEX removes them from the feature list to prevent redundant calculations and trajectory constraints.
Nathan Dennler, a postdoctoral researcher at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) and member of the Collaborative Learning and Autonomy Research (CLEAR) Lab, noted that people naturally understand higher-level intentions behind speech. Explaining those expectations to a machine usually requires significant effort, but FLEX provides a streamlined interface for robots to translate conversational phrases into broader rules.
Simulated performance and testing
The research team, led by Andreea Bobu, assistant professor of aeronautics and astronautics at MIT and head of the CLEAR Lab, tested FLEX across three simulated scenarios using a virtual robotic arm. Co-authors on the paper include Audrey Lee and Ilia Sucholutsky of NYU.
During testing, volunteers interacted with the virtual robot by typing simple prompts across three tasks: serving coffee, moving a candle, and moving a bowling ball. Commands included phrases like "Don't spill on my laptop," "Stay away from the stack of paper," and "Don't drop on the mug."
Across all trials, FLEX successfully followed the immediate directive while inferring broader categories: avoiding all electronics, staying clear of flammable items, and protecting fragile objects. Bobu emphasized that because home environments vary significantly, consumer robots cannot arrive pre-configured with every user's personal preferences. FLEX enables robots to learn those boundaries from just one or two natural language sentences.
The study received support from the MIT Generative AI Impact Consortium and the Social and Ethical Responsibilities of Computing program within the MIT Schwarzman College of Computing.
What it means for developers
For developers working at the intersection of AI and physical hardware, FLEX demonstrates how multimodal systems and LLMs can move beyond conversational interfaces into operational control loops. Rather than using language models purely for static task planning, the pipeline integrates real-time semantic generalization into trajectory evaluation algorithms.
Key takeaways for technical teams include:
- Semantic abstraction over hardcoded constraints: By routing physical constraint generation through LLMs like GPT-4o, systems can auto-generate contextual rules without engineers manually defining every edge case or obstacle category.
- Dynamic memory management: The inclusion of the "Mask" phase provides a practical blueprint for managing state in robotic systems, preventing memory bloat by reconciling high-level LLM outputs with real-time computer vision data.
- Multi-model integration: Building scalable applications often requires orchestrating specialized models alongside primary LLMs. Developers interested in exploring LLMs like GPT-4o, Claude, or DeepSeek for robotics and logic pipelines can try top AI models cheaply through one API at https://apixoai.online.
As natural language becomes a primary control layer for autonomous systems, architectures like FLEX demonstrate that common-sense reasoning can significantly reduce the training and programming overhead required to deploy robots in unstructured environments.
Source: AI-assisted learning platform helps robots reason beyond literal commands — MIT News AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
