“This is the first time that an astronaut in space managed to control a robotic system on the ground in such an immersive, intuitive manner,” comments Aaron Pereira of DLR.

“Our 6 degree of freedom control interface incorporates force feedback so that the astronaut can experience just what the rover feels, even down to the weight and cohesion of the rocks it touches. What this does is help compensate for any limitations of bandwidth, poor lighting or signal delay to give a real sense of immersion – meaning the astronaut feels as though they are there at the scene.”

Let the robots take the strain

Robotics engineer Thomas Krueger, heading ESA’s Human Robot Interaction Laboratory, adds: “Robots can be given limited autonomy in known, structured environments, but for systems carrying out exploratory tasks such as sample collection in unknown, unstructured environments some kind of ‘human-in-the-loop’ oversight becomes essential. But direct control has not been feasible due to the inherent problem of signal delay – with transmission times constrained by the speed of light.

“So we have been working towards the concept of humans staying safely and comfortably in orbit around the Moon, Mars or other planetary bodies, but being close enough for direct oversight of rovers on the surface – combining the human strengths of flexibility and improvisation with a robust, dexterous robot on the spot to carry out their commands precisely.”

A team from ESA’s HRI Lab and DLR’s Robotics and Mechatronics Center collaborated on a series of progressively more complex tests, first on Earth then extending into orbit.

“In the end we needed to perform feasibility experiments from space because past research shows that weightlessness can degrade human performance during force and motion tasks,” adds Thomas. “This and other unique environmental factors meant that Earth-based simulations would not be sufficient.”

Their efforts culminated in the first part of the Analog-1 experiment in late 2019. Astronaut Luca Parmitano aboard the ISS operated the gripper-equipped ESA Interact rover in a mock lunar environment inside a hangar in Valkenburg, the Netherlands to survey rocks and collect samples. The two-hour space-to-ground test was a success, overcoming a two-way signal delay averaging more than 0.8 seconds and a data packet loss rate of 1% plus.

Solving the time factor

“Even though the ISS is in orbit just 400 km overhead, its signals are relayed to Earth via geostationary telecom satellites, then to Europe from Texas via a transatlantic cable,” explains Aaron.

“Our team at DLR had to design a control algorithm that could function on a stable basis despite this time lag. Because there is a delay in the force feedback received by the operator, they might continue to move the robot further even after it has hit a rock. This could lead to the robot going out of sync with its controller, potentially vibrating like crazy, perhaps even damaging itself.

“To prevent that happening, we use a concept called ‘passivity’ – we look at the overall amount of energy an operator puts in, and on the remote side make sure the robot never gives out  any more energy than that, and vice-versa. Like when you push a child on a swing, they are never going to go any higher than the first swing – and with friction and so on will swing gradually lower and lower.