Publications

2022

1. ASME IDETC/CIE

   A Computational Framework for the Evaluation of Resilience in Deep Space Habitat Systems

   ASME IDETC/CIE 2022

   Abs

   Resilience is a vital consideration for designing and operating a deep space habitat system. The numerous hazards that may affect a deep space habitat and its crew during its lifecycle need to be considered early in the design. Trade-off studies are the typical method used to assess the cost and value of different design choices. Here we develop a modular dynamic computational framework intended for rapid simulation and evaluation of the resilience of different system configurations. The framework uses a system-level phenomeno-logical Markov model of the habitat systems, enabling us to assess multiple habitat configurations and evaluate their performance in the presence of several hazards and user-defined control policies. System fault detection and repairs are modeled. External disturbances, including meteorites impact, temperature fluctuations, and dust, are modeled based on the lunar environment, envisioning a deep space habitat design. We use a reflexive health management subsystem that prioritizes recovery actions based on the robotic agent’s availability to close the loop. In addition to performance, a resilience metric is included to quantify the system’s resilience over the design lifecycle. We illustrate the use of the framework for supporting early-stage design decisions of a habitat system. Our case study focuses on designing the power generation system considering cost and energy efficiency.

2021

1. SIGGRAPH ASIA

   DroneStick: Flying Joystick as a Novel Type of Interface

   Tsykunov, Evgeny, Fedoseev, Aleksey, Dorzhieva, Ekaterina, Agishev, Ruslan, Ibrahimov, Roman, Vasquez, Derek, Labazanova, Luiza, and Tsetserukou, Dzmitry

   ACM SIGGRAPH Asia 2021

   Abs PDF

   DroneStick is a novel hands-free method for smooth interaction between a human and a robotic system via one of its agents, without training and any additional handheld or wearable device or infrastructure. A flying joystick (DroneStick), being a part of a multi-robot system, is composed of a flying drone and coiled wire with a vibration motor. By pulling on the coiled wire, the operator commands certain motions of the follower robotic system. The DroneStick system does not require the user to carry any equipment before or after performing the required interaction. DroneStick provides useful feedback to the operator in the form of force transferred through the wire, translation/rotation of the flying joystick, and motor vibrations at the fingertips. Feedback allows users to interact with different forms of robotic systems intuitively. A potential application can enhance an automated ‘last mile’ delivery when a recipient needs to guide a delivery drone/robot gently to a spot where a parcel has to be dropped.

2. SIGGRAPH

   SwarmPlay: A Swarm of Nano-Quadcopters Playing Tic-tac-toe Board Game against a Human

   Karmanova, Ekaterina, Serpiva, Valerii, Perminov, Stepan, Ibrahimov, Roman, Fedoseev, Aleksey, and Tsetserukou, Dzmitry

   ACM SIGGRAPH Emerging Technologies 2021

   Abs PDF

   We present a new paradigm of games, ie SwarmPlay, where each playing component is presented by an individual drone that has its own mobility and swarm intelligence to win against a human player. The motivation behind the research is to make the games with machines tangible and interactive. Although some research on the robotic players for board games already exists, eg, chess, the SwarmPlay technology has the potential to offer much more engagement and interaction with a human as it proposes a multi-agent swarm instead of a single interactive robot. The proposed system consists of a robotic swarm, a workstation, a computer vision (CV), and Game Theory-based algorithms. A novel game algorithm was developed to provide a natural game experience to the user. The preliminary user study revealed that participants were highly engaged in the game with drones (69% put a maximum score on the Likert scale) and found it less artificial compared to the regular computer-based systems (77% put maximum score). The affection of the user’s game perception from its outcome was analyzed and put under discussion. User study revealed that SwarmPlay has the potential to be implemented in a wider range of games, significantly improving human-drone interactivity

2020

1. IEEE HAPTICS

   SwarmCloak: Landing of Two Micro-Quadrotors on Human Hands Using Wearable Tactile Interface Driven by Light Intensity

   Tsykunov, Evgeny, Agishev, Ruslan, Ibrahimov, Roman, Moriyama, Taha, Labazanova, Luiza, Kajimoto, Hiroyuki, and Tsetserukou, Dzmitry

   In 2020 IEEE Haptics Symposium (HAPTICS) 2020

   Abs PDF

   For the human operator, it is often easier and faster to catch a small size quadrotor right in the midair instead of landing it on a surface. However, interaction strategies for such cases have not yet been considered properly, especially when more than one drone has to be landed at the same time. In this paper, we propose a novel interaction strategy to land multiple robots on the human hands using vibrotactile feedback. We developed a wearable tactile display that is activated by the intensity of light emitted from an LED ring on the bottom of the quadcopter. We conducted experiments, where participants were asked to adjust the position of the palm to land one or two vertically-descending drones with different landing speeds, by having only visual feedback, only tactile feedback or visual-tactile feedback. We conducted statistical analysis of the drone landing positions, landing pad and human head trajectories. Two-way ANOVA showed a statistically significant difference between the feedback conditions. Experimental analysis proved that with an increasing number of drones, tactile feedback plays a more important role in accurate hand positioning and operator’s convenience. The most precise landing of one and two drones was achieved with the combination of tactile and visual feedback.

2. IEEE RO-MAN

   DroneLight: Drone Draws in the Air using Long Exposure Light Painting and ML

   Ibrahimov, Roman, Zherdev, Nikolay, and Tsetserukou, Dzmitry

   In 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN) 2020

   Abs PDF

   We propose a novel human-drone interaction paradigm where a user directly interacts with a drone to light-paint predefined patterns or letters through hand gestures. The user wears a glove which is equipped with an IMU sensor to draw letters or patterns in the midair. The developed ML algorithm detects the drawn pattern and the drone light-paints each pattern in midair in the real time. The proposed classification model correctly predicts all of the input gestures. The DroneLight system can be applied in drone shows, advertisements, distant communication through text or pattern, rescue, and etc. To our knowledge, it would be the world’s first human-centric robotic system that people can use to send messages based on light-painting over distant locations (drone-based instant messaging). Another unique application of the system would be the development of vision-driven rescue system that reads light-painting by person who is in distress and triggers rescue alarm.

2019

1. IEEE ICAR

   AeroVR: Virtual reality-based teleoperation with tactile feedback for aerial manipulation

   Yashin, Grigoriy A, Trinitatova, Daria, Agishev, Ruslan T, Ibrahimov, Roman, and Tsetserukou, Dzmitry

   In 2019 19th International Conference on Advanced Robotics (ICAR) 2019

   Abs PDF

   — Drone application for aerial manipulation is tested in such areas as industrial maintenance, supporting the rescuers in emergencies, and e-commerce. Most of such applications require teleoperation. The operator receives visual feedback from the camera installed on a robot arm or drone. As aerial manipulation requires delicate and precise motion of robot arm, the camera data delay, narrow field of view, and blurred image caused by drone dynamics can lead the UAV to crash. The paper focuses on the development of a novel teleoperation system for aerial manipulation using Virtual Reality (VR). The controlled system consists of UAV with a 4-DoF robotic arm and embedded sensors. VR application presents the digital twin of drone and remote environment to the user through a headmounted display (HMD). The operator controls the position of the robotic arm and gripper with VR trackers worn on the arm and tracking glove with vibrotactile feedback. Control data is translated directly from VR to the real robot in realtime. The experimental results showed a stable and robust teleoperation mediated by the VR scene. The proposed system can considerably improve the quality of aerial manipulations.

2. SIGGRAPH ASIA

   SwarmCloak: Landing of a Swarm of Nano-Quadrotors on Human Arms

   Tsykunov, Evgeny, Agishev, Ruslan, Ibrahimov, Roman, Labazanova, Luiza, Moriyama, Taha, Kajimoto, Hiroyuki, and Tsetserukou, Dzmitry

   2019

   Abs PDF

   We propose a novel system SwarmCloak for landing of a fleet of four flying robots on the human arms using light-sensitive landing pads with vibrotactile feedback. We developed two types of wearable tactile displays with vibromotors which are activated by the light emitted from the LED array at the bottom of quadcopters. In a user study, participants were asked to adjust the position of the arms to land up to two drones, having only visual feedback, only tactile feedback or visual-tactile feedback. The experiment revealed that when the number of drones increases, tactile feedback plays a more important role in accurate landing and operator’s convenience. An important finding is that the best landing performance is achieved with the combination of tactile and visual feedback. The proposed technology could have a strong impact on the human-swarm interaction, providing a new level of intuitiveness and engagement into the swarm deployment just right from the skin surface.

3. ACM VRST

   SlingDrone: Mixed Reality System for Pointing and Interaction Using a Single Drone

   Tsykunov, Evgeny, Ibrahimov, Roman, Vasquez, Derek, and Tsetserukou, Dzmitry

   In 25th ACM Symposium on Virtual Reality Software and Technology 2019

   Abs PDF

   We propose SlingDrone, a novel Mixed Reality interaction paradigm that utilizes a micro-quadrotor as both pointing controller and interactive robot with a slingshot motion type. The drone attempts to hover at a given position while the human pulls it in desired direction using a hand grip and a leash. Based on the displacement, a virtual trajectory is defined. To allow for intuitive and simple control, we use virtual reality (VR) technology to trace the path of the drone based on the displacement input. The user receives force feedback propagated through the leash. Force feedback from SlingDrone coupled with visualized trajectory in VR creates an intuitive and user friendly pointing device. When the drone is released, it follows the trajectory that was shown in VR. Onboard payload (e.g. magnetic gripper) can perform various scenarios for real interaction with the surroundings, e.g. manipulation or sensing. Unlike HTC Vive controller, SlingDrone does not require handheld devices, thus it can be used as a standalone pointing technology in VR.

4. IEEE RO-MAN

   DronePick: Object picking and delivery teleoperation with the drone controlled by a wearable tactile display

   Ibrahimov, Roman, Tsykunov, Evgeny, Shirokun, Vladimir, Somov, Andrey, and Tsetserukou, Dzmitry

   In 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN) 2019

   Abs PDF

   We report on the teleoperation system DronePick which provides remote object picking and delivery by a humancontrolled quadcopter. The main novelty of the proposed system is that the human user continuously gets the visual and haptic feedback for accurate teleoperation. DronePick consists of a quadcopter equipped with a magnetic grabber, a tactile glove with finger motion tracking sensor, hand tracking system, and the Virtual Reality (VR) application. The human operator teleoperates the quadcopter by changing the position of the hand. The proposed vibrotactile patterns representing the location of the remote object relative to the quadcopter are delivered to the glove. It helps the operator to determine when the quadcopter is right above the object. When the “pick” command is sent by clasping the hand in the glove, the quadcopter decreases its altitude and the magnetic grabber attaches the target object. The whole scenario is in parallel simulated in VR. The air flow from the quadcopter and the relative positions of VR objects help the operator to determine the exact position of the delivered object to be picked. The experiments showed that the vibrotactile patterns were recognized by the users at the high recognition rates: the average 99% recognition rate and the average 2.36s recognition time. The real-life implementation of DronePick featuring object picking and delivering to the human was developed and tested.

5. IEEE ToH

   SwarmTouch: Guiding a swarm of micro-quadrotors with impedance control using a wearable tactile interface

   Tsykunov, Evgeny, Agishev, Ruslan, Ibrahimov, Roman, Labazanova, Luiza, Tleugazy, Akerke, and Tsetserukou, Dzmitry

   IEEE transactions on haptics 2019

   Abs PDF

   To achieve a smooth and safe guiding of a drone formation by a human operator, we propose a novel interaction strategy for a human-swarm communication which combines impedance control and vibrotactile feedback. The presented approach takes into account the human hand velocity and changes the formation shape and dynamics accordingly using impedance interlinks simulated between quadrotors, which helps to achieve a natural swarm behavior. Several tactile patterns representing static and dynamic parameters of the swarm are proposed. The user feels the state of the swarm at the fingertips and receives valuable information to improve the controllability of the complex formation. A user study revealed the patterns with high recognition rates. A flight experiment demonstrated the possibility to accurately navigate the formation in a cluttered environment using only tactile feedback. Subjects stated that tactile sensation allows guiding the drone formation through obstacles and makes the human-swarm communication more interactive. The proposed technology can potentially have a strong impact on the human-swarm interaction, providing a higher level of awareness during the swarm navigation.