| UAV | |
AITL-WING-HITL: Telemanipulation of autonomous drones using Digital Twins of aerial traffic interfaced With WING
The system is designed to intervene in scenarios, where an autonomous AI agent – the ‘‘new driver’’ – encounters conditions too complex or unorthodox for autonomy alone to handle. We present here the first part of the paper. The concluding part will be publsihed in July issue |
 |
|
 |
|
 |
|
 |
|
 |
|
 |
|
 |
|
 |
|
 |
|
 |
|
 |
|
Abstract
‘‘By touching an instrument placed in the southern gallery, a miniature Spanish cruiser anchored in the fountain lake on the lower floor, 90 ft away, was blown into the air. There was no connection between the transmitter and the vessel in the lake,’’ reported The New York Times in 1898. Though perceived as magic by many at the time, Nikola Tesla had once again pushed the boundaries of technology, becoming the forefather of modern remotely controlled wireless drones. His invention – an early radiocontrolled vessel operated via radio waves – was hailed as ‘‘the first of a race of robots, mechanical men which will do the laborious work of the human race,’’ and marked a leap far ahead of its time. Since then, human–robot interaction and the telemanipulation of drones have advanced substantially, especially in tandem with the evolution of autonomy. In this context, a scalable, agent-based platform – termed AITL-WING-HITL – was developed for the telemanipulation of Autonomous Unmanned Aerial Vehicles (A-UAVs) from the perspective of a human–multirobot architecture, leveraging Digital Twins (DTs) of integrated aerial traffic. This system is designed to intervene in scenarios, where an autonomous AI agent – the ‘‘new driver’’ – encounters conditions too complex or unorthodox for autonomy alone to handle. The platform incorporates a patented, force-sensitive, precision control device known as the WING – an immersive tool tested with participants using measurable parameters to validate the system’s effectiveness and identify areas for further improvement. The findings indicate that AITL (Autonomy-In-The-Loop) A-UAV agents and HITL (Human-In-The-Loop) Human Telemanipulator (HTM) agents can cooperate to achieve high-performance, synergistic task execution through a sociocognitive interaction model embedded in the platform. By addressing current challenges and outlining directions for future exploration, this research not only provides an overview of the evolving landscape of drone telemanipulation but also serves as a roadmap for ongoing and future efforts in this critical field.
1. Introduction
‘‘By touching an instrument placed in the southern gallery, a miniature Spanish cruiser anchored in the fountain lake on the lower floor, 90 ft away, was blown into the air. There was no connection between the transmitter and the vessel in the lake’’, reported The New York Times in 1898 [1]. Although many perceived it as magic, Nikola Tesla, once again pushing the boundaries of innovation, became the progenitor of modern-day remotely controlled wireless drones by inventing the first radio-controlled vessel using radio waves — an invention well ahead of its time [2]. This small radiotransmitting control box enabled Tesla to adjust the vessel’s speed and direction by manipulating on-board mechanisms [3]. When a New York Times reporter described the invention as a device ‘‘that could carry dynamite as a weapon of war’’, Tesla responded: ‘‘You do not see there a wireless torpedo; you see there the first of a race of robots, mechanical men – a borrowed mind (the machine’s own mind) – which will do the laborious work of the human race’’ [4]. His vision foreshadowed today’s multi-agent autonomous robotic systems, capable of remote operation via distinct frequencies. Tesla’s U.S. patent (US613809 A) referred to the invention as a ‘‘distant operator’’ – what we now call a teleoperator [3]. Since then, remote human–robot interaction and telemanipulation have evolved significantly alongside advances in autonomy. Many large-scale and complex tasks now require remote operation supplemented by extended human intelligence and experience. Drones are increasingly deployed by organisations and companies to execute various missions – such as logistics deliveries [5] – and operate in autonomous modes Beyond Visual Line of Sight (BVLOS) along preplanned routes. These missions must be conducted safely, avoiding collisions with other aerial vehicles within integrated Unmanned Aerial Vehicle (UAV) Traffic Management (UTM) and Air Traffic Management (ATM) systems, collectively referred to as the UTM+ATM framework.
In all future visions involving networks of fully autonomous self-driving ground [6] or aerial vehicles [7], some form of remote human intervention is anticipated. Specifically, roles such as ‘‘Human-onthe-Loop’’ (HOTL) and ‘‘Human-inthe-Loop’’ (HITL) for telemonitoring and telemanipulation are expected to foster a desirable level of trust in autonomous vehicles (AVs) as they operate within highly dynamic urban or aerial environments [8]. Ensuring the safe and cost-efficient routing of Autonomous UAVs (A-UAVs) within the integrated UTM+ATM system requires strategic human oversight: HOTL for fully autonomous UAVs (FA-UAVs) and HITL for semi-autonomous UAVs (SA-UAVs), enabling humans to monitor and intervene when necessary (Fig. 15) – particularly when the Artificial Intelligence (AI) agent, acting as the new ‘‘driver’’, encounters unorthodox scenarios beyond the scope of its autonomy. As automation continues to evolve, the nature of human–machine interaction has fundamentally shifted beyond the traditional Humans-Are-BetterAt/Machines-Are-Better-At (HABA/ MABA) paradigm, in which humans and machines are often seen as competitors [9]. Instead, the Human-Agent-Robot Teamwork (HART) frame-work emphasises collaborative augmentation, where humans and machines co-work by leveraging their complementary strengths to enable practical and profitable applications [10]. This paper examines co-work within this HART-centric framework from the perspective of two intelligent, self-reliant entities: humans equipped with extended intelligence and experience, and A-UAVs operating with a high degree of autonomy. These two entities are critically interdependent for the successful management of rare and complex challenges. Throughout the paper, the ‘‘human teleoperator’’ (i.e. master) and the ‘‘in-vehicle teleoperator’’ (i.e. slave) are referred to as the ‘‘Human Telemanipulator’’ (HTM), acting as the HITL agent, and the ‘‘A-UAV’’ (FA-UAVs and SAUAVs, i.e. AITL agents), respectively.
Manipulating A-UAVs BVLOS will remain a central topic in the coming years, particularly as autonomous systems increasingly take on indispensable roles in real-world applications. In alignment with this trajectory, the University of Lancashire is actively developing bespoke A-UAVs and AITL systems 3 through various funded projects,4 aiming to tackle a wide range of complex operational tasks using A-UAVs. To support this mission, the University has partnered with PilotAware Ltd. and Worthington Sharpe Ltd. to ensure safe aerial operations, mitigating collision risks while enabling the execution of advanced aerial tasks. The core research question of this study is framed as follows: Can A-UAVs be effectively and efficiently manipulated through the collaborative operation of AITL and HITL agents? To explore this question, the following components were developed and integrated: (i) A Collision Management (CM) system, termed DACM (Drone Aware Collision Management) (Section 3.1.4), which creates Digital Twins (DTs) of aerial traffic through real-time communication with the decentralised PilotAware ATOM-GRID Network (Section 3.1.3); (ii) A patented, advanced force-sensitive control device, referred to as the WING, with 6DoF (Degree of Freedom) (Section 3.1.1), designed as an immersive precisioncontrol interface for remotely operated vehicles (ROVs); (iii) A scalable agentbased platform – AITL-WING-HITL – that integrates these subsystems (Fig. 2) into a cohesive architecture, enabling HTMs to monitor and comanipulate multiple A-UAVs within nonsegregated aerial traffic environments, allowing human omnipresence.
While numerous novel robotic applications are anticipated, a range of emerging challenges must also be addressed — chief among them is ensuring that human–robot interaction remains safe, reliable, and effective [11]. One key question arises: Can the WING device facilitate effective interaction between HTMs and A-UAVs to accomplish tasks safely within the AITLWING-HITL platform, especially given the potential limitations in human attention and perception when managing multiple A-UAVs simultaneously? To the best of our knowledge, this study represents the first comprehensive investigation into the use of a 6DoF advanced control device for manipulating multiple A-UAVs by a single HTM through digital replicas of aerial traffic, employing human–robot dialogue across several intervention modes (see Section 3.2.1). In addressing this research gap, the primary contributions of this paper are as follows:
(i) This research analyses the simultaneous manipulation of heterogeneous A-UAVs – autonomous aerial robots – by immersing HTMs within a newly integrated multi-agent architecture, termed AITL-WING-HITL.
(ii) The study explores the cooperative interaction between two intelligent entities – HTMs (HITL agents) and AITL AI agents – in performing complex tasks safely, accurately, and efficiently via various intervention modes (Fig. 15).
This integration effectively bridges ‘‘the remote extension of human intelligence and expertise’’ with ‘‘vehicle autonomy’’.
The structure of the paper is outlined in Fig. 1 (see Table 1 for the abbreviations used in this paper).
2. Related works
A multitude of state-of-the-art studies have been conducted on the human telemanipulation of robots since the first teleoperator was built in the mid-1940s by Geortz [12]. Telerobotics, particularly bilateral robotic teleoperation activities between humans and robots, have generally been analysed based on the master and slave concept in which the slave side is mapped as ‘‘teleoperator’’ and the master side is mapped as ‘‘human operator’’ [8]. The supervisory role of human control in the remote manipulation of robots was first examined by Ferrell [13] in 1967. Most of the studies in the literature pay particular attention to HITL interactions, usually based on the semi-autonomous mode for human and single-machine hybrid activities to accomplish assigned tasks together, where semi-autonomous robotic systems are mainly designed with reliance on delay-tolerant human assistance [14]. The possibility of having multiple robots controlled by one operator (user) seems to be an additional crucial requirement in collaboration/cooperation [15]. Analysis of human–multirobot or human–multiagent cooperation and collaboration was immensely analysed in the 1990s as in [16–19], in particular, from the perspective of HITL decentralised systems, where the human needs to monitor the system all the time. The research of Nakauchi et al. [16] was the first in this direction in 1992. Laengle et al. [17] in 1997 had envisioned that ‘‘On one hand, robots have difficulties coping with unpredictable variations and uncertainties in their environment, which are normally no problem for humans. On the other hand, humans cannot always accomplish a task alone because of their limited force, slow execution speed, limited working area, limited execution time, and low precision. By combining the complementary capabilities of the robot with those of the human within a team, new perspectives for mastering complex systems could be opened’’. The seeds of HOTL robot manipulation were placed by Suzuki et al. [17] in 1995 while analysing the cooperation between the human operator and the decentralised multi-agent robotic system. The research emphasises several principles of HOTL as ‘‘(1) the human operator usually does not control each agent in the system,(2) all information of the system is not concentrated in the human operator, (3) and the human operator needs not always watch over the system’’, ‘‘the human operator must know the status of the system when an urgent event occurs in the system and human intervention is required, or he is asked for some cooperation from agents’’. In a similar domain, this subject was extensively focused on during the 2000s while a mass number of robotic applications were taking their indispensable places. Fong et al. [20–23] closely scrutinised this subject and developed a framework in 2003 [21]. In the framework, how a few humans can teleoperate multiple teams of robots to accomplish the targeted task – that is, to assemble large orbital structures – was analysed. Later, human– robot teaming was focused on by Nourbakhsh et al. in 2005 [24] in real-world problem-solving applications, e.g. for urban search and rescue. The research on human–multirobot interaction in many disciplines has increased exponentially to solve realworld problems from the perspectives of HITL and HOTL since these preliminary works. Different types of immersive devices have been developed to interface with remote objects. Today’s telemanipulation systems allow the interaction with environments at a distance and can also scale human force and motion to achieve stronger, bigger, or smaller action capabilities [25]. The study by Mithal et al. [26] suggests that joysticks are harder to control than mice. Martins et al. [27] concluded that participants’ (with motor disabilities) feedback was more positive regarding 3D optical mice compared to the use of a conventional keyboard+mouse combination for navigation in 3D environments. The WING was tested in controlling 3D objects within computeraided design (CAD) applications by Turner et al. [28], and by Sandoval et al. [29] in two short papers and found to be efficient in manipulating 3D virtual objects. Sandoval et al. [30], while testing several input devices with varying DoF, concluded that the more DoF available, the more enhanced the control over objects.
Human–drone interaction has been scrutinised by the research community to build easy-to-use interaction interfaces (e.g. natural interaction interfaces [31,32], touch-based [33]). The realworld telemanipulation initiatives with A-UAVs are expanding within the industry with the use of BVLOS, in particular, within the military industry. The aerial industry is partnering with leading telecommunication and technology companies that are experimenting with remote telemanipulation of A-UAVs in stochastic environments. Advanced multi-agent platforms, as well as hefty devices to interface with these platforms, are required to extend human intelligence and experience for tele coworking with multiple A-UAVs to accomplish tasks as desired. In this sense, existing works on drone manipulation need to be extended to mitigate the aforementioned concerns. To this end, to the best of the observed knowledge, having received limited attention in the literature, telemanipulating A-UAVs through different intervention modes has not been analysed in-depth in the sense of establishing a holistic HITL/HOTL approach for improving co-work between humans and multiple A-UAVs, which makes this paper unique, aiming at closing this gap.
3. Methodology
The integration of A-UAVs into non-segregated UTM+ATM aerospace environments necessitates unified platforms that support collective operations to ensure missions are carried out both efficiently and safely. Such platforms are essential for maintaining coordinated control of multi-DoF aerial vehicles while preserving safe separation distances. During the autonomous execution of multiple UAV missions – often orchestrated by different entities – unexpected events may arise, and individual missions cannot be treated in isolation, as they share the same aerospace infrastructure. This study aims to develop an integrated, collective framework that enables HITL and AITL agents to collaboratively manage uncertainty and execute autonomous missions effectively. The proposed approach supports global BVLOS operations by incorporating a comprehensive State and Situation Awareness (SSA) tracking system. The core components of the proposed platform – AITL-WING-HITL (Fig. 16) – for the manipulation of A-UAVs are illustrated in Fig. 2. At its foundation, the platform integrates the PilotAware system (Supplement II, Fig. 1) and the DACM system (Supplement III, Fig. 2) to generate DTs of aerial traffic. These primary components, along with supporting systems, are detailed in Section 3.1, prior to the full presentation of the developed platform in Section 3.2.
3.1. Background
Recent advances in Cyber-Physical Systems (CPSs) within the concepts of Internet of Everything (IoE) and Automation of Everything (AoE) [34] teleport us to teleoperate remote objects using DTs. In this treatise, the components of the designed platform which are depicted in Fig. 2 are explained in Sections 3.1.1, 3.1.2, 3.1.3, and 3.1.4 respectively and then the platform, as a holistic framework, is examined in 3.2.
3.1.1. WING device
The WING, a mouse-like, precision control device, was developed by Worthington Sharpe Ltd. in order to use the human motor system effectively. It is presented in Figs. 3, 4 with its functions. It, by extending the abilities of a mouse, combines the functions of a free-moving isotonic mouse and a free-flowing isometric joystick, leading to unprecedented freedom of movement from a desk-based control system. It enables movements in X, Y and Z directions as well as a complete precision roll (φ), pitch (θ) and yaw (ψ) joystick functions as delineated in Fig. 5.
The WING aims to create command and control solutions for ROVs and robotic equipment. The lower part contains the laser mouse sensor; the upper part attaches via a precision slide and pivot mechanism to provide pitch, roll and z-height movement. Further control is then added through the under-slung yaw-bar positioned towards the front, giving a total of 6DoF. It, with advanced sensory mechanisms allowing precision control, was designed to address the needs of users who require the speed and accuracy of a high-end mouse for pointing and selection tasks, along with full 3D control. Additionally, it offers a wide range of buttons and a speed wheel that can be further programmed both to perform numerous functions and to map certain actions as we employ in this study. Readers are referred to the video5 to observe how it functions with its particular features. Its particular functionalities regarding the primary objectives of this research can be summarised as follows.
i. Multiple A-UAVs can be hooked and manipulated separately by the same hand. ii. A-UAVs can be directed using pre-defined navigation waypoints, and their trajectories can be adjusted during the mission.
iii. Integrated 4-axis joystick functionality allows for performing various actions synchronously, such as simultaneous mission planner operation, camera gimbal control, flight path adjustments, or full manual piloting.
iv. Joystick functions can be integrated into APM Planner 2.0, Ardupilot Mission Planer, DJI Ground Station, and any independent Ground Control Station (GCS) that supports standard joystick inputs (Fig. -3).
v. Familiar computer mouse shape, speed, and accuracy reduce the learning curve, allowing the operator to focus on the task promptly during telemanipulation.
The min and max values of the rotational orientations — roll (φ), pitch (θ) and yaw (ψ)) are shown in Table 3. The WING’s motions with discrete microstructure values can be mapped into the manoeuvres of A-UAVs. The Yaw states of the WING detected by the developed application in this research are shown in Table 2, and their graphical representations are depicted in Fig. 6 for Yaw Port and Fig. 7 for Yaw Starboard. The acquired sequences and the time are incorporated to visualise how the velocity varies over the duration of the force applied and released. The readers are referred to Supplement I for all the translated input forces (Fig. 4) into sub-movement microstructure states.
3.1.2. Vehicle autonomy, AITL, HITL, HOTL, and HOOTL
The exponential growth of interest and research in UAVs is strongly pushing for the emergence of autonomous flying robots [35]. 10 control levels of UAV swarms from fully supervisorcontrolled to fully autonomous mode between human and machine were analysed in [36]. Drone Industry Insights (DRONEII) categorises the autonomy with 5 levels [37] based on degrees of independence, namely, 1: low automation (i.e. the UAV has control of at least one vital function, with a pilot in control); 2: partial automation (i.e. the UAV can take over heading and altitude under certain conditions with a pilot still responsible for safe operation); 3: conditional automation (i.e. the UAV can perform all functions and a pilot acts as a fallback system); 4: high automation (i.e. the UAV has back-up systems, so if one fails, the platform is still operational while a pilot is out of the loop); 5: full automation (i.e. the UAV can plan its actions using advanced AI learning techniques) with little to no human intervention in the control loop. As the level of autonomy increases, UAVs can operate in more complex environments and execute more complex tasks with less prior knowledge and fewer operator interactions [38].
AI agents are now capable of intentionally constraining, complementing, or substituting human involvement in the execution of routine tasks [39]. With Human-out-of-the-Loop (HOOTL) systems, AI agents are expected to execute assigned tasks without human intervention. Humans and machines may need to work hand-in-hand to complete numerous tasks safely, accurately, efficiently, and ethically, in particular, for safety-critical jobs requiring a high level of precision. HITL interaction, requiring active human involvement, is employed for the systems by which a task may not be achieved either by a human or a machine. The concept of HITL dates back to the early 1940s, when humans primarily served as error detectors, monitoring systems and intervening when necessary [40]. HITL, i.e. hands-on, system allows humans to monitor the actions continuously, interact with the system and manipulate decisions that do not seem to be an optimal outcome or situations that involve nondeterministic behaviours of AI. SA-UAVs require humans (HITL) and onboard AI (AITL) to interact with each other continuously, where uncertainty is possible to emerge. In other words, there is always a person or people watching the HITL system actively, in which SA-UAVS are mainly involved, to intervene to fix any problem that is expected to occur. On the other hand, there is no need to employ a person to monitor FA–UAVS. However, it‘s still best to have a person as HOTL passively for remote monitoring to easily step in when alerted by the AI agent and manipulate when autonomy – AITL –encounters uncertainty about what to do next. HOTL interaction turns into HITL when the human agent is alerted. However, HITL is a phase that continues until an alerted uncertainty is handled, and the system turns into the HOTL interaction after the job has been completed. In 2025, the London Office of Technology and Innovation (LOTI) launched a research project to explore the role of HITL, prompted by ongoing questions about what constitutes meaningful human involvement and how effective the designated individual is in mitigating the ethical risks associated with AI [41]. Generally speaking, HOTL/HITL systems are utilised to ensure the expected automated behaviour of A-UAVs and consequently to ensure aerial traffic safety as well as groundbased safety. HOTL systems, while operating with little to no human intervention and not requiring to be alerted all the time, still allow HTMs to perform other tasks while the autonomy level of decentralised agents is relied on to accomplish assigned missions/tasks. HTMs monitor A-UAVs during the progress of the assigned missions within the HITL system. HITL, where a HTM is supposed to be on alert all the time to fix any emerging problem that the vehicle cannot cope with, is commonly used in controlling level-3 and level-4 autonomy. A level-5 autonomy can perform under all predefined circumstances and conditions. In this regard, this paper focuses on both HOTL and HITL with A-UAVs, where the HTM is expected to intervene remotely in rare difficulties that the vehicle cannot tackle under exceptional conditions.
Autonomy has changed the relationship between humans and robots extensively. They are becoming more harmonious while taking part in specific task executions as the level of autonomy increases with further intelligence. This relationship is elaborated in Section 3.2.1 with various intervention modes. It must be noted that the HOTL/HITL manipulation through the intervention modes is performed if necessary, and AITL is expected to take over the HTM completely as soon as possible to realise the main objective of autonomy – self-reliance and self-governing (see Fig. 8 for the EC device and Table 4 for a data instance example acquired from the EC device).
3.1.3. PilotAware ATOM-GRID Network
The PilotAware ATOM-GRID Network (i.e. currently over 320 UK ATOM stations) is elaborated in Supplement II. The network detects aircraft that transmit any of the various Electronic Conspicuity (EC) signals used in Europe. These include ADS-B (Automatic Dependent SurveillanceBroadcast), CAP 1391 and Mode-S signals on the ICAO (International Civil Aviation Organisation) aviation band and Flarm, PilotAware, Fanet + and OGN (Open Glider Network) trackers that use the EASA (European Union Aviation Safety Agency) SRD 860 Band. Aircraft that transmit using the SafeSky mobile app are also detected.
3.1.4. DACM (Drone Aware Collision Management) system
The DACM system was developed to monitor aerial traffic and trigger autonomous cooperative manoeuvres that consider the aerodynamic, flight dynamics and particular features (e.g. physics, kinematics, manoeuvrability) of A-UAVs as mobile aerial agents within a multi-disciplinary joint project [42]. This system is illustrated in Supplement III (Fig. 2). The interface of this system, equipped with simulation capabilities and low-cost equipment, provides us with a full range of testing capabilities in both realworld applications of A-UAVs and co-simulation environments. The DACM – that is tightly coupled with the PilotAware ATOMGRID Network (Section 3.1.3) – collects instant operational manoeuvre information from all aerial vehicles via simultaneous low-bandwidth data streams from each of the aerial vehicles. The low-bandwidth dual-path data communication is also mainly used in this research for conveying HTM command messages (e.g. new_heading = heading + 15) and necessary telemetry data (e.g. battery level of A-UAVs). The DACM – a digital replica of aerial traffic – integrates all near-real-time streamed data within its DT counterpart for convenient monitoring with more immersive telepresence for HTMs. Autonomous actuation for A-UAVs is performed to avoid emerging hazards in this immersive interface. The DACM supports HTMs to perceive the remote aerial traffic with effective SSA. The main features are outlined as follows.
1) The DACM, equipped with a novel CM methodology within an autonomy control framework using geometry formation of flights, enables BVLOS operations with collision-free trajectories.
2) The system, with (i) perception, (ii) decision-making, and (iii) automated actuation phases, can perceive the surrounding dynamic environment precisely using a global coverage strategy requiring no sophisticated sensors and react efficiently to multiple non-linear collision risks at a time with minimum trajectory deviations, requiring no prior training.
3) The DACM in both air-only and air/ground modes not only implements Collision Avoidance (CA) for FA-UAVs, but also manages Pilot-Controlled UAVs (PC-UAVs) in improving operational safety, where it might be difficult for pilots to detect surrounding close-range/high-speed flights (see Fig. 13 for the geometric illustration of CA manoeuvres).
The modules of the DACM, allowing autonomous navigation for A-UAVs with appropriate actuation mechanisms by coordinating the actions of multiple heterogeneous A-UAVs, are introduced in Supplement III, and the methodology using these modules is depicted in Supplement III (Fig. 2). In addition to autonomous actuation mechanisms to avoid any collision risk, the DACM system alerts the HOTL/HITL HTM if any flight is departing from its pre-planned trajectory, where the two control inputs (i.e. pre-planned waypoints and active waypoints) do not agree, considering that a probable failure is causing this unexpected manoeuvre under no collision risk using the health assessment and failure detection module.
The main goal of CM is to keep flights away from hazardous regions with conflict-free solutions at the very early stage by keeping each flight well clear of other traffic, considering their current trajectories and preplanned routes. Currently, A-UAVs do not have a system that will warn an operator of an impending collision with other airborne vehicles [43] using broader local coverage. In this module, SSA through horizontal and vertical planes is provided with the active flights and early collision risk assessment is processed, considering their trajectory predictions. From a linear formulation technical viewpoint, the flights’ time-invariant connectivity of previous waypoints, their trajectory headings and their Probable Collision Travel Cone (P CT Zone) and Imminent Collision Travel Zones (ICT Zones) are used to geometry regarding the velocities, heading and altitudes. The system turns into the HITL state, where there is a crossing point indicating an impending collision risk. The system, second, warns for the probable collision risks (within Probable-collision-Risk zone (PcRz)) as illustrated in Fig. 11. Third, it alerts for imminent collision risks (within Imminent-collision-Risk zone (IcRz)) as illustrated in Fig. 12 along with autonomous deconflicting manoeuvres (Fig. 14).
3.2. AITL-WING-HITL platform
Fatigue, lack of concentration, and poor displays all contribute to reduced performance in telemanipulating remote objects, and humans have difficulty building mental models of remote environments [20]. Therefore, these shortcomings must be compensated for with the establishment of easy-to-use interfaces, enabling robust interaction between AITL AI agents and remote human cognitive intelligence. Most importantly, two intelligent nodes – A-UAV and skilled HTM – have to be tightly coupled with human-oriented and vehicle-centric interfaces by exploiting the smartness at the vehicle end. The remote environment of A-UAVs should be observed by HTMs using constant dynamic SSA to generate proper commands at the appropriate time.
The AITL-WING-HITL platform aims to provide a tightly coupled interface between two intelligent agents — HOTL/ HITL HTM and onboard aerial AITL AI agent from a multi-agent manipulation perspective. It encompasses a variety of human–robot co-work technologies that enable various interaction modes (elaborated in Section 3.2.1) and guarantee seamless transitions between these modes whenever they are needed. Technically speaking, the platform provides the HTM with a high degree of immersiveness ability with the established DTs of aerial traffic to enable the HTM to both perfectly perceive the remote aerial traffic with a full sense of telepresence (e.g. transparency) and track the results of their manipulated discrete microstructure actuation (i.e. teleimpedance) with backfeeding by incorporating a large set of integrated components as shown in Fig. 2. These integrated components mainly allow the user to interact with precision (e.g. correct aerial information with the PilotAware network (Supplement II, Fig. 1)), real-time user tracking of aerial traffic supported by a large set of proactive AI implementations (e.g. early warning, CA) with the DACM (Supplement III (Fig. 2)) and 6DoF in manipulation with the WING (Figs. 4, 5) as well as its discrete microstructure movement (Figs. 6, 7, Table 2). First, we would like to examine the co-work aspects of two intelligent entities theoretically through different intervention modes (Fig. 15) in Section 3.2.1 before delving into the merits of AITL-WING-HITL in Section 3.2.2.
3.2.1. Intervention modes within AITL-WING-HITL
A-UAVs operate autonomously and need to receive assistance whenever necessary. Time-varying delays are inevitable during the co-work interaction regarding the communication time delay problem, decision-making delays regarding the Quality of Experience (QoE) of HTMs and the evaluation of the voluminous sensor data and actuation time delays; Moreover, human remote intervention is vulnerable to human decision errors regarding incomplete environmental information based on the current state and imminent states of the vehicle concerning the other traffic participants, obstacles and hazards [8]. Even with a high level of SSA, human telemanipulation is strictly dependent on the cognitive and perceptual capabilities, spatial orientation abilities and motor skills of HTMs to avoid risky manoeuvres [8]. To this end, it is important to use the intelligence at the remote site, and it is advantageous to co-work during the execution of the complex tasks that cannot be handled by autonomy.
If A-UAVs are capable of performing on their own, human intervention is not needed, allowing HOTL systems, particularly with FA–UAVs. Otherwise, a remote extension of human intelligence and expertise with HITL interaction may be necessary either for better task performance to avoid disastrous situations during possibly hazardous operations or for executing very complex tasks beyond the capabilities of autonomy. The WING, with 6DoF, allows a HTM, with superior motor capabilities and perception/interpretation/implementation skills, to have partial or total control of A-UAVs to achieve their tasks safely and efficiently. With the help of the WING, HITL HTM inputs range with discrete microstructure values as depicted in Table 2, considering the intervention modes ranging from ‘‘no-control (supervisory) mode’’ to ‘‘full-control (master–slave) teleoperation mode’’ as delineated in Fig. 15. These intervention modes are elaborated in our previous study in [44]. These modes are summarised below in their dedicated subsections. The WING allows a HTM to switch between AITL and HITL states as well as between the interaction modes. The AITL-WINGHITL platform provides HTMs and AI agents with a high level of transparency through these interaction modes to work together with a high degree of efficacy. The WING, interfaced with AITL-WINGHITL, coordinates the tasks/subtasks by effectively modulating the HTM’s hand tele-impedance in a smooth, goaloriented manner based on the required behaviours of the intervention mode in action and the task to be accomplished. The HTM with the HOTL interaction mode can change the state to HITL to monitor the aerial traffic at any time and take action in one of the intervention modes to ascertain aerial safety or to accomplish any particular tasks. Then, the HITL state can be switched to the HOTL state after any uncertainty is managed successfully. The following intervention modes are analysed within the AITLWING-HITL framework when continued assistance is critical for A-UAVs.
No-control (supervisory) mode (Fig. 15A): it is a time where the AI can empirically be proved to be on such a high degree of superiority that the human’s input may affect the task performance negatively. The involvement of HTMs is aimed to be minimised in this mode. The HTM, as a task-specific HITL, encourages the A-UAV in a nondecisive mood to take either one of the options determined by the vehicle itself or a different predetermined option.
Co-activity (shared-control) mode (Fig. 15B): Some tasks may not be achieved by allocating sub-task responsibilities distinctively between the HTM and A-UAVs, which may lead to compromising task performance. Combining human and robot skills via intelligent interfaces seems very appealing; in this manner, establishing principled coactivity methods to seamlessly blend the control between the human and the robot to enable the combined system to surpass both the robot and human performance with reduced human effort is a prime goal for robotics [45]. This mode assists both intelligent agents – HITL HTM and AITL AI – in achieving the common goal safely.
Collaboration (joint-control or traded control) mode (Fig. 15C): Al-located/ assigned series of fine-grained subtasks need to be performed individually either by the AITL or HITL agent. Neither the HTM is in full control of the A-UAV nor the A-UAV is in full control of itself, where various specific subtasks are allocated between the HTM and A-UAV beforehand. The control is traded back and forth to execute fine-granular sub-tasks, enabling joint problem-solving. Partial control can be mainly used for tightly coupled coordination between collocated HTM and A-UAVs to achieve joint task performance, wherever difficulty that cannot be coped with by the autonomy is alerted. Instead of a supervisor dictating to a subordinate, the human and the robot engage in dialogue to exchange ideas and resolve differences in the collaborative mode [23]. This mode aims to produce a better outcome than either the human or A-UAV performs alone.
Full-control (master–slave) mode (Fig. 15D): Complete task may need to be performed by the HTM alone under extreme conditions in this mode. The HTM, as a leading agent, takes over the control and leads the A-UAV as a follower agent. Different from the collaboration mode, the A-UAV, piloted remotely by the HTM, obeys the manoeuvres performed by the HTM under any circumstances.
Cooperation (all togetherness) mode Swarms of A-UAVs sometimes need to interact with each other to accomplish a single task faster than a single A-UAV or to solve difficult tasks that are beyond a single A-UAV’s capability, where each A-UAV assists in the accomplishment of the desired goal, considering its specific capabilities, e.g. carrying a very heavy payload together with a robotic arm or a suspended cable grasped by a fixed gripper, search and rescue by assigning different target Region of Interest (RoI) to multiple A-UAVs.
The main distinctive features of the intervention modes are summarised in Table 5. A merge of these intervention modes can be employed by switching from one mode to another for specific parts of a complicated task to cope with highly difficult challenges. Switching decisions between modes can be determined based on the characteristics and dominance of the modes as presented in Table 6. For instance, the switching decision for ‘‘from no-control to coactivity and then to collaboration and then to full-control and then again back to no-control’’ is controlled and carried out by A-UAV, HTM, HTM and ‘‘HTM & A-UAV together’’ respectively.
3.2.2. Modules, techniques, & interface of AITL-WING-HITL
The main components of the AITLWING-HITL platform (Fig. 2) are already explained in the related sections of this paper above in Sections Section 3.1.1, 3.1.2, 3.1.3, and 3.1.4. Therefore, we will be focusing on the particular characteristics of the platform in this section to reveal the specific features that are gained from the mould of the aforementioned components. A-UAVs have diverse characteristics, and they are designed with particular features and onboard equipment concerning the missions they are expected to perform [7]. In this regard, the manipulation of different types of A-UAVs requires particular control parameters representing the particular features of A-UAVs. The HTM can manipulate the speed, altitude, and heading of A-UAVs, considering the other aerial vehicles in the environment both to avoid instant collisions and not to cause new collision risks with the generated manoeuvres as illustrated in Fig. 16E.1. Commands generated through the WING are transformed to the hooked A-UAVs using the off-theshelf drone controlling systems such as UgCS, DJI (Fig. 16E.2.) as elaborated in our previous research in [42].
Interface of the AITL-WING-HITL platform: The AITL-WING-HITL application provides an improved, easy-to-use Graphical User Interface (GUI) for manipulating A-UAVs in the specified aerial zone using the WING, leading to fine-granular force, direction, and amplitude. The interfaces of intelligent systems are supposed to be considered to reduce the cognitive load to the possible minimum complexity, paving the way for effective human intervention with a high degree of perception. In this direction, the interface of AITL-WING-HITL was designed to be as simple as possible by avoiding HTMs with unnecessary overwhelming information. Additionally, it, as displayed in Fig. 17, provides HTMs with effective perception capabilities which allow HTMs to focus on the particular sections of the interface with which they are supposed to interact. For instance, the colour of an A-UAV’s heading line turns black when that UAV is suggested to the HTM to be hooked and manipulated during a collision risk. The circles of P CT Zone and I CT Zone turn black to show that the A-UAV is hooked by the HTM for manipulation. The primary interface elements are (i) DTs of aerial traffic, (ii) the WING that is coupled between HTMs and A-UAVs within DTs, (iii) the interactive interface control elements and (iv) informative elements. Either the HITL agent or the AITL agent determines the intervention mode and transitions between these modes as outlined in Table 6. Either with a button click on the WING or with a selection from the interactive interface control elements, the HTM can switch between the interaction states (i.e. HOTL and HITL) and the intervention modes, namely, no-control (supervisory), co-activity (shared-control), collaboration (joint-control), full-control (master-slave), and cooperation (all togetherness) (Fig. 15).
Hooking with WING (Fig. 16C.1, D.III, D.IV): HTMs may need to intervene in the current actions of A-UAVs for better task performance or to avoid hazards. The flights that indicate a collision risk are highlighted in the interface, and the system turns into the HITL state from the HOTL state. A HTM can hook one of these flights using the immersive device to manipulate its actions. The interface may suggest that a specific A-UAV shall be hooked and manipulated during a collision risk by turning its heading line black.
More specifically, the system can recommend a specific A-UAV to be hooked if there is a collision risk between that A-UAV and Manned Aerial Vehicles (MAVs), where there is no possibility to hook MAVs. Similarly, fixed-wing drones are recommended by the system against rotary-wing drones, while the battery constraint feature of rotary-wing drones is evident. In the same manner, the fixed-wing drone with a stronger battery level is advised over the other fixed-wing drone with a lower battery level, whereas rotary-wing drones with stronger battery levels are proposed against other rotary-wing drones with lower battery levels. Moreover, drones with an air-based mission against drones with a ground-based mission are selected by the system for hooking, where drones with air-based missions are not required to turn to their leaving point at the original trajectory. Likewise, the slower drone can be recommended against the faster one, while the manoeuvring may be easier with the slower drone. The circles of the P CT Zone and ICT Zone turn black to denote that the A-UAV is hooked by the HTM for manipulation.
The hooking of a specific A-UAV takes place when the WING-Button-1-clicked point is within the ICT Zone of the A-UAV. The HTM remains in the manipulation mode until the hooking is released using the WING-Button-2-clicked point in the same way. The ICT Zone, P CT Zone, and the heading turn black (Fig. 17) to signify the hooked A-UAVs during the manipulation phase. The HTM can hook and manipulate multiple A-UAVs simultaneously. While an A-UAV is implementing the last command in the hooked state, e.g. heading to a new direction with a new altitude, the HTM can hook other A-UAVs to manipulate their actions. In this way, multiple A-UAVs can be hooked, which means that they no longer implement their preplanned waypoints, velocity, and altitude while the WING is actively manipulating another drone.
The actuation per A-UAV is performed using one of the intervention modes (Fig. 15) explained in Section 3.2.1. The liberated A-UAVs from the hooking state return to their original trajectory using the shortest path to implement their pre-planned missions.
Manipulation of A-UAVs with WING (Fig. 16E.1): The mobile Internet with low-latency and high-bandwidth capabilities, enabling access to information on an anywhere-anytime basis, has democratised our life in many aspects, in particular, providing fast remote operation abilities [8]. Telepresence of HTMs within aerial traffic is provided by the AITL-WING-HITL platform. The WING offers an intuitive way of controlling multiple degrees of freedom (i.e. multiple axes) simultaneously, aiming at using the human motor system effectively with a higher information capacity. One hand is used for manipulating A-UAVs; the other hand is free to operate aircraft systems through the keyboard or the control panel of the AITL-WING-HITL platform.
The mapping of the values acquired from the WING to drone manoeuvres is shown in Algorithm 1. This mapping actuates the hooked A-UAVs appropriately as illustrated in Fig. 16E.1. The user can choose the proper intervention mode within the interface and can translate the inputs of the WING into a particular transfer function that manipulates A-UAVs accordingly. HTMs are immersed with A-UAVs through force/torque applied or released to the rotational orientations: roll (φ), pitch (θ), and yaw (ψ) channels (Fig. 4). The buttons on the WING are employed to switch between several states and actuation functions. The use of the buttons in hooking and releasing A-UAVs is explained earlier.
The SSA of aerial traffic is constantly generated using the global SSA strategy implemented by the PilotAware system (Section 3.1.3). The movements of aerial vehicles are computed by the DACM (Section 3.1.4). The DACM triggers alarms and warnings when the AITL agent experiences a difficult situation or there is a hazard, such as impending collision risks, beforehand for the HTM to take action and manipulate A-UAVs remotely. The HTM within the UTM+ATM system is authorised to intervene in the autonomous actions of A-UAVs. The HTM can adjust the position (longitudinal and lateral), altitude, angular velocity and orientation of A-UAVs concerning their environments with a quick response using the WING. It is worth emphasising that drones can be manipulated with desired navigational actions, considering the dynamic characteristics of the aerial environment and the particular drone physics-based rules defined in the system, such as manoeuvrability, velocity, and acceleration. In this sense, interactions between HTMs and A-UAVs differ depending on the capabilities of A-UAVs.
A-UAV changes its trajectory as the HTM applies the desired force on the WING. The A-UAV follows the latest action taken by the HTM if the HTM releases the force completely. In other words, the A-UAV follows the heading with specified altitude and velocity directed by the HTM, even though the WING leaves the drone for manipulating other A-UAVs for the sake of simplicity, assuming that the HTM is not required to hold the WING actions at a specific trajectory for a specific drone. In this way, multiple A-UAVs can be hooked simultaneously. The A-UAV follows similar actions continuously, where the HTM applies the maximum allowed force on the device. For instance, the heading can change to the left or right of the original direction by 15°, which results in a circular trajectory. The relationship between the inputs and outputs is mapped by interactions between desired outcomes concerning the particular manoeuvrability capabilities of each A-UAV. The number of actions in the specified direction (i.e. roll (φ), pitch (θ) and yaw (ψ)) executed by an A-UAV, leading to the change of rotation angles of the (X, Y, and Z) axis, is dependent on the DoF, the number of sequences executed by the HTM and their values and timestamps (e.g. Table 2, Figs. 6, 7). The hooked drone has to co-work with the HTM by following the actions directed by the HTM to increase its task performance and avoid hazardous situations. The platform reacts to any imminent collision risk. An imminent collision risk can be caused if any required desired action is not actuated by the HTM within the early warning zone (Fig. 10) and the P-Rz (Fig. 1), despite the system collision warnings and alarms, which may lead to making drones meet in the L-Rz (Fig. 12). An imminent collision risk can also be caused by the instant actions of the HTM during the implementation of any intervention modes. The WING has a significant degree of inertia, which may cause the HTM to overshoot their desired location or move too fast. To avoid unexpected manoeuvres based on the fast-changing microstructure of the WING with a sequence of multiple sub-movements, the directions of the selected intervention mode within 6DoF are taken into consideration along with thecapabilities of the drones in manipulation. This provides the user with the means to limit movement with more resistance.
.
A HTM can be cautioned with an alert by the AITL AI agent about the emerging situation. Telemanipulation requests and intervention requirements can be prioritised by the HTM or the system, and several of these can be commanded to take an idle safe position or trajectory (e.g. a circular trajectory as mentioned above) without endangering the vehicle itself and surrounding traffic by hooking multiple A-UAVs simultaneously. They can be handled later in priority scheduling order or hazard states (i.e. red, yellow, green) by a HTM, enabling task allocation. This property enables a single HTM with limited sensorimotor capabilities to manage a large area with heavy aerial traffic.
Liberation of A-UAVs and their return to the original trajectory (Fig. 16C.I, D.III, D.IV): A specific drone can be released from hooking with the WING Button-2 clicked point is within the ICT Zone of the A-UAV. The circles denoting ICT Zone, P-CT Zone, and the heading line turn into their original colour from black (Fig. 17) after the A-UAV is liberated to indicate that they are performing their tasks independently with the AITL agent themselves. A-UAVs orient themselves in Euclidean space by means of the preplanned trajectory after being released from the hook, considering their physical manoeuvring features and the instant SSA environmental inputs.
The paper is originally published in Robotics and Autonomous Systems, Volume 202, 2026.
The paper is republished with authors’ permission. https://doi.org/10.1016/j. robot.2026.105486.
© 2026 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license (http:// creativecommons.org/licenses/by/4.0/).
To be concluded in next issue.
(No Ratings Yet)
Loading...
Leave your response!