RobotX Research Program

There is an unprecedented hype around humanoid robots. Companies around the globe make tremendous investments to build advanced robotic systems and aim to incorporate the substantial research progress that has been seen in quadrupedal locomotion over the last decade. While building our own humanoid systems at ETH is out of reach due to the limited budgets available, our standing in the community gives us a unique possibility to position ourselves as the provider of the locomotion synthesis and control pipeline and to demonstrate this through early access to state of the art robots.

Within the scope of the Advanced Humanoid Locomotion (AHL) project, we aim to empower bipedal robots with robust walking skills to overcome terrains that humans and animals encounter in daily life. Such environments may be structured s.a. flat ground, stairs, and slopes, or can be highly irregular, e.g., dense vegetation, rocky ground, and detritus. Traversing such terrains requires sensory feedback from both vision and touch, which, when jointly processed, allows robots to understand topological properties and environmental uncertainties.

We plan to exploit the knowledge we have gathered with the controls of quadrupedal robots and target to establish a new baseline within the humanoid locomotion community. To this end, Robotic Systems Lab (RSL) will build upon our reinforcement learning (RL)-based sim-to-real pipeline to generate robust and versatile locomotion control policies. Specifically, we investigate the use of transformer-based terrain interpretation for optimal foothold selection during perceptive locomotion. In parallel, Computational Robotics Lab (CRL) investigates using model-predictive control (MPC)-guided RL and imitation learning as alternative approaches to control such a complex robotic system. Both teams will compare and merge the two approaches to generate versatile yet robust whole-body locomotion skills. The emerging controller will lay the foundation for future research of loco-manipulation (i.e., the combination of locomotion and manipulation) and more advanced bipedal skill synthesis, such as learning parkour.

Using engineered living muscle tissues to generate motion, bio-actuators have the potential for sustainable production, self-healing, self-assembly, and pathophysiological study of motion in living beings. However, as their architecture does not fully mimic native muscle, bio-actuators provide limited performance. We lack a principled approach to physically model muscle constructs starting from the myofiber level, that could guide bottom-up tissue engineering and increase biofabrication’s efficiency.

We aim to: (1) systematically bioprint bundles of myofibers; (2) characterize their contraction responses to electrical stimulation; (3) formulate a physical model predicting the geometricvariability and contractile behavior; and (4) biofabricate muscles with a biomimetic microarchitecture.

We will deposit high-resolution structures of myoblast-laden hydrogel via extrusion-based bioprinting and generate a myofiber performance library to formulate a physics model explaining the engineered muscle’s behavior. Equation discovery coupled with differentiable simulation will explain the contractile behavior of differently-sized myobundles. A physics-informed machine learning approach will improve the model’s performance and scale it up to solve the behavior of more complex tissue architectures. To validate the model’s generalizability, we will compare the performance of various myobundles’ architectures against our model, and finally realize a proof-of-concept bio-actuator with a ierarchical tissue organization and predicted performance.

Our biophysical machine learning-driven approach to predictable contractile tissue fabrication will enable the understanding of myofibers’ biophysics and scaling-up performant bio-actuators. This interdisciplinary project offers the first opportunity to apply equation discovery to living materials and engineer bio-actuators from the myofiber level with predicted contractions for highperformance operations.

This project aims to explore localization within 3D scene graphs, with the objective of enabling autonomous agents, such as the Boston Dynamics Spot and ANYmal, to self-localize within a preconstructed map of a changing environment. This graph characterizes object instances at the leaf nodes using various modalities (point cloud, image, and attributes), while the connecting edges denote interobject relationships (e.g., "nearby"). Interior nodes categorize groups into hierarchy levels, such as rooms or buildings. This representation supports the integration of multiple modalities in localization, ensuring both lightweight operation and efficiency while providing a natural way to handle changing environments. The lightweight nature stems from the ability to distill complex objects into comprehensive descriptors, capturing all modalities cohesively. This method stands out for its efficiency by eliminating the reliance on large point clouds or image datasets. Moreover, it allows for representing dynamic objects similarly to static ones. The project is structured into three work packages (WPs).

The first WP creates localization algorithms to process input from various modalities providing coarse and fine localization given a pre-existing map. This WP involves enhancing the framework to leverage sequences, aiming to improve localization robustness. The second WP is dedicated to change detection by comparing queries against the established map. The third WP develops algorithms for 3D scene graph construction capturing dynamic object properties. This approach ensures that generated graphs are optimized for cross-modal localization. This phase will also include data collection and proceed simultaneously with the previous WPs.

Every one of us interacts with numerous objects daily without giving the interaction much thought. Yet robot manipulators still struggle with actions beyond simple pick-and-place operations. Why is that? While there are many reasons, including limitations in hardware, sensing, and planning, an often overlooked one is going beyond shape to the ability to capture and represent physical properties.

Humans have a remarkably good intuition about an object's physical properties and how it will interact with the environment. And despite not knowing the actual physical quantities, we can make educated guesses about the physical interactions of objects. Current robotic systems are still far from exhibiting a comparable level of physical intuition. Despite recent advances in manipulation and perception, most approaches depend purely on shape information that can be passively captured through vision alone, neglecting other intrinsic properties such as friction and mass distribution.

In this project, we will develop a learning-based method that captures the essence of both visual and physical properties by embedding them in a single shared latent space. The robot will collect vision and force-torque observations through interactions with previously unknown objects. Leveraging state-of-the-art unsupervised learning approaches, we side-step the need for explicit labels for the observed properties, and focus on learning a latent representation that incorporates multiple relevant physical and shape properties for interaction. We will demonstrate the power of the learned latent space by demonstrating it on a stacking task with complex, non-uniform objects for which successful completion requires an understanding of both shape and physical properties.

Many of the items we interact with in our daily lives are soft and squishy. While research on the manipulation of rigid objects by robots is advancing at a rapid pace, the situation is not quite so for deformable objects. The problem involves the development of advanced optimal control and learning methods that would allow robots to handle soft objects. The goal is to equip robots with human-like
abilities, such as folding clothes and passing soft parcels with precision, flexibility, and adaptability. Advancements in this field are expected to have far-reaching implications for various industries, including manufacturing, healthcare, and logistics, as well as contribute to the overall progress of the field of robotics.

With this proposal, we seek to focus on and investigate the decision-making process involved in activities such as folding clothes. For example, the robot must identify the type of cloth and its properties, such as its material, texture, and size. Based on this information, the robot must determine the best folding strategy, taking into account the cloth's tendency to stretch or wrinkle. Next, the robot must use its sensors to detect the position and orientation of the cloth, as well as any obstacles that may interfere with the folding process. The robot must then choose the appropriate movement, such as picking up, manipulating, and folding the cloth, using its actuators. The entire process requires the integration of perception, decision-making, and motor control systems, as well as the ability to adapt to changing conditions, such as unexpected obstacles or changes in the cloth's shape.

Mobile robots with mounted manipulators allow for automating daily tasks of increasing complexity in environments designed for humans. However, versatile manipulation with human-like dexterity and autonomy is still beyond current capabilities.

To develop human-like manipulation skills, we propose a system design for a versatile, cost-efficient, agile, and compliant robotic hand with accurate position and tactile sensing. This hand will be integrated with an existing robotic arm on an ANYmal quadrupedal robot to solve real-world challenges in assisting humans such as opening doors, pressing buttons, picking up objects from the floor, and performing robot-to-human handover tasks.

To solve complex manipulation tasks with this robotic hand we propose a data-driven, deep-learning-based dexterous grasp planning framework. By using available crowd-sourced point-of-view datasets (3670h video) of humans performing daily tasks for pre-training our robot transformer, we will reduce by an order of magnitude the need for time-intensive robot demonstrations for training.

We will be the first to equip quadrupedal robots with the abilities for versatile and shape invariant grasping using different grasp types, dexterous in-hand manipulation and re-orientation of objects, and grasp compliance for human interaction, which have previously not been possible with the twoand three-fingered robotic grippers used today.

This robotic hand and grasp planning framework will serve as a versatile platform for future research in machine learning for robotic manipulation and could significantly increase the capabilities of mobile robots deployed in unstructured environments. The technology will enable applications such as care and service robots, inspection robots, and rescue robots.