Neuro-LIFT: A Neuromorphic, LLM-based Interactive Framework for Autonomous Drone FlighT at the Edge

This repository contains code associated with Neuro-LIFT: A Neuromorphic, LLM-based Interactive Framework for Autonomous Drone FlighT at the Edge.This code has most recently been tested with Python 3.10 and PyTorch 2.5.1

Introduction

The integration of human-intuitive interactions into autonomous systems has been limited. Traditional Natural Language Processing (NLP) systems struggle with context and intent understanding, severely restricting human-robot interaction. Recent advancements in Large Language Models (LLMs) have transformed this dynamic, allowing for intuitive and highlevel communication through speech and text, and bridging the gap between human commands and robotic actions. Additionally, autonomous navigation has emerged as a central focus in robotics research, with artificial intelligence (AI) increasingly being leveraged to enhance these systems. However, existing AI-based navigation algorithms face significant challenges in latency-critical tasks where rapid decision-making is critical. Traditional frame-based vision systems, while effective for highlevel decision-making, suffer from high energy consumption and latency, limiting their applicability in real-time scenarios. Neuromorphic vision systems, combining event-based cameras and spiking neural networks (SNNs), offer a promising alternative by enabling energy-efficient, low-latency navigation. Despite their potential, real-world implementations of these systems, particularly on physical platforms such as drones, remain scarce. In this work, we present Neuro-LIFT, a realtime neuromorphic navigation framework implemented on a Parrot Bebop2 quadrotor. Leveraging an LLM for natural language processing, Neuro-LIFT translates human speech into high-level planning commands which are then autonomously executed using event-based neuromorphic vision and physicsdriven planning. Our framework demonstrates its capabilities in navigating in a dynamic environment, avoiding obstacles, and adapting to human instructions in real-time. Demonstration images of Neuro-LIFT navigating through a moving ring in an indoor setting is provided, showcasing the system’s interactive, collaborative potential in autonomous robotics.

Installation

Clone this repository using:

git clone [email protected]:amoghj98/neuroLIFT.git

Create a conda environment using the provided yaml file as follows:

conda env create -n $ENV_NAME --file neurolift.yaml

Activate the environment using:

conda activate $ENV_NAME

Model Finetuning

Run file `llm_finetuning.py` using:

python llm_finetuning.py

Citations

If you find this work useful in your research, pleas consider citing: Amogh Joshi, Sourav Sanyal and Kaushik Roy, "Neuro-LIFT: A Neuromorphic, LLM-based Interactive Framework for Autonomous Drone FlighT at the Edge", arXiv preprint, 2025

@misc{joshi2025neuroliftneuromorphicllmbasedinteractive,
      title={Neuro-LIFT: A Neuromorphic, LLM-based Interactive Framework for Autonomous Drone FlighT at the Edge}, 
      author={Amogh Joshi and Sourav Sanyal and Kaushik Roy},
      year={2025},
      eprint={2501.19259},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2501.19259}, 
}

and Amogh Joshi, Sourav Sanyal and Kaushik Roy, "Real-Time Neuromorphic Navigation: Integrating Event-Based Vision and Physics-Driven Planning on a Parrot Bebop2 Quadrotor", arXiv preprint, 2024

@misc{joshi2024realtimeneuromorphicnavigationintegrating,
      title={Real-Time Neuromorphic Navigation: Integrating Event-Based Vision and Physics-Driven Planning on a Parrot Bebop2 Quadrotor}, 
      author={Amogh Joshi and Sourav Sanyal and Kaushik Roy},
      year={2024},
      eprint={2407.00931},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2407.00931}, 
}

Authors

Amogh Joshi, Sourav Sanyal and Kaushik Roy