Hybrid motion planning for Robotic grasping via Reinforcement Learning and NLP

Achieving reliable robotic grasping in cluttered
environments continues to pose a significant challenge for
autonomous systems, particularly when traditional motion
planning algorithms like MoveIt fail due to inverse kinematics
(IK) limitations or workspace constraints. This project presents a
hybrid manipulation framework that integrates classical IK
based planning using MoveIt with a policy learned via Proximal
Policy Optimization (PPO). The system allows a UR16e robotic
manipulator to interpret high-level natural language voice
commands, convert them into structured robotic goals using a
large language model (LLM), and execute grasping actions using
either inverse kinematics or a learning-based fallback strategy.
The perception module processes scene-level semantics and
bounding box detections to derive object IDs and 3D grasp
targets. A centralized control policy manages the planning
pipeline, selecting between MoveIt and the PPO policy depending
on planning feasibility. The full system is developed in NVIDIA
Isaac Sim and evaluated across environments of varying
complexity. Experimental results demonstrate that the PPO
enhanced planner improves success rates in failure-prone
configurations where IK-based methods struggle, providing a
viable fallback under constraints. This work demonstrates the
synergy of language-guided planning and hybrid control for
robust, user-driven robotic grasping.