Abstract

Overview of PseudoTouch

PseudoTouch infers the tactile signal from a visual input. Task-agnostic visual-tactile pairs from (a) unsupervised random data collection are used to train (c) the PseudoTouch model, which we apply to (i) object recognition and (ii) grasp success prediction.

Tactile sensing is vital for human dexterous manipulation, however, it has not been widely used in robotics. Compact, low-cost sensing platforms can facilitate a change, but unlike their popular optical counterparts, they are difficult to deploy in high-fidelity tasks due to their low signal dimensionality and lack of a simulation model. To overcome these challenges, we introduce PseudoTouch which links high-dimensional structural information to low-dimensional sensor signals. It does so by learning a low-dimensional visual-tactile embedding, wherein we encode a depth patch from which we decode the tactile signal. We collect and train PseudoTouch on a dataset comprising aligned tactile and visual data pairs obtained through random touching of eight basic geometric shapes. We demonstrate the utility of our trained PseudoTouch model in two downstream tasks: object recognition and grasp stability prediction. In the object recognition task, we evaluate the learned embedding's performance on a set of five basic geometric shapes and five household objects. Using PseudoTouch, we achieve an object recognition accuracy of 84% after just ten touches, surpassing a proprioception baseline. For the grasp stability task, we use ACRONYM labels to train and evaluate a grasp success predictor using PseudoTouch's predictions derived from virtual depth information. Our approach yields a 32% absolute improvement in accuracy compared to the baseline relying on partial point cloud data.

Technical Approach

Compact magnetic tactile sensors such as ReSkin are cheap and easy to integrate, but they produce only a low-dimensional signal and, unlike optical tactile sensors, cannot be simulated with standard rendering techniques. PseudoTouch addresses both issues by learning to predict the tactile reading a sensor would produce from a small depth image of the touched surface. Once trained, the model acts as a "virtual" touch sensor that can be queried on real or simulated depth data for different downstream tasks.

PseudoTouch Model

PseudoTouch is a function \(PT: z_d \mapsto \tilde{\tau}\) that takes a depth patch \(z_d \in \mathbb{R}^{17 \times 17}\) of the object surface and predicts the tactile reading \(\tilde{\tau} \in \mathbb{R}^{15}\) that touching this surface would produce. We implement it as a light-weight convolutional encoder, followed by average pooling and a single-layer MLP. We deliberately keep the model capacity low to avoid overfitting on our small dataset, given the large imbalance between input and output size.

Data Collection

We collect aligned visual-tactile pairs with a Franka Emika robot that carries a wrist-mounted RealSense D405 camera and a 3D-printed finger holding a ReSkin sensor. The procedure is largely automated:

  1. Scan the fixed object with the wrist camera and reconstruct a surface mesh.
  2. Sample candidate touch points and normals from the mesh, rejecting overly flat approach angles and obstructed approach corridors.
  3. Solve for a camera pose observing the point along its normal, and for a pre-touch pose of the sensor.
  4. Record a depth image of the target area from the camera pose.
  5. Record an ambient sensor reading, then approach the surface along the normal until the reading deviates; the dataset item is the difference to the ambient reading.
  6. Project the sensor's contact area into the depth image to extract a 17 mm × 17 mm depth patch.

In total, we collect 1600 samples from eight primitive objects of a shape-sorting toy, which cover edges of different curvature and points of different sharpness. Because our downstream tasks rely on simulated depth, we close the gap between real and simulated observations by pairing each tactile reading with both the real depth patch and a depth patch rendered from the object mesh at the same pose. Both are normalized to a 3 mm depth range, excluding depth information the sensor cannot realistically capture.

Downstream Task 1: Online Object Recognition

Given a set of candidate objects \(O\) with known 3D models, the robot identifies the object it is touching by performing a sequence of touches and updating its belief with a directed graphical model. For each candidate object \(o\), PseudoTouch predicts the expected reading \(\tilde{\tau}_o\) at the measured contact location \(l\) from a simulated depth patch, which yields the touch likelihood \(p(\tau \mid o, l) = \exp(-\lVert \tilde{\tau}_o - \tau \rVert)\). A proprioceptive likelihood \(p(l \mid o)\) scores how well the contact location fits the object's surface. Starting from a uniform prior, the posterior after \(N\) touches is

\(\displaystyle p(o \mid x_1, \ldots, x_N) \propto p(o) \prod_{n} p(\tau_n \mid l_n, o)\, p(l_n \mid o)\).

New touches are planned by sampling an object hypothesis from the current posterior and sampling a touch location on it.

Downstream Task 2: Grasp Stability Prediction

We train a small MLP that predicts grasp success from the tactile readings of the left and right fingers. Instead of collecting real grasps with tactile sensors, we generate the training data entirely in simulation: for annotated grasps from the ACRONYM dataset, we render depth patches with virtual orthographic cameras at the fingertips and use PseudoTouch to infer the corresponding tactile signals. This turns an existing grasp dataset into a large-scale tactile grasp dataset without simulating a tactile sensor or executing real grasps.

Experiments

Object Recognition

The robot identifies an object out of a set of five candidates after ten touches. We evaluate on five of the primitive training shapes and on five unseen everyday objects (puncher, apple, bulb, PC mouse, tin box), comparing proprioception (P), touch (T), and their combination (P+T). We additionally compare to a user study in which five participants solved the same task blindly with a single finger.

Primitive Objects Everyday Objects
PTP+T PTP+T
Human Baseline 34%60%71% 51%77%86%
PseudoTouch (N = 5) 12%42%42% 60%84%64%
PseudoTouch (N = 150) 24%47%47% 53%62%60%

Object recognition success rates. PseudoTouch is evaluated with informed object sampling (N = 5) and with random sampling from all recorded touches (N = 150). Touch consistently outperforms proprioception and follows the same trends as the human baseline, reaching 84% accuracy on everyday objects that were not part of the training data.

Grasp Stability Prediction

We train the tactile grasp success predictor on roughly 8500 ShapeNetSem objects with ACRONYM grasp labels and compare it to a point-cloud-based predictor (GQP) and to a combination of both. To test the transfer to the real world, we mount a ReSkin sensor on each gripper finger and record 76 grasps proposed by ContactGraspNet on seven household objects.

Modality Sim Accuracy Real Accuracy
GQP (Point Cloud) 64%47%
PseudoTouch (Touch) + GQP (Point Cloud) 73%36%
PseudoTouch (Touch) 76%79%

Grasp success prediction accuracy. The predictor trained purely on PseudoTouch's simulated tactile readings outperforms the point cloud baseline in simulation and transfers to the real robot without fine-tuning, improving accuracy by 32 percentage points over the baseline.

Video

Code

The code, trained models, and dataset will be released soon. This work will be released under the GPLv3 license. For any commercial purpose, please contact the authors.

Publications

If you find our work useful, please consider citing our full paper:

Adrian Röfer*, Nick Heppert*, Abdallah Ayad, Eugenio Chisari, Abhinav Valada
PseudoTouch: Efficiently Imaging the Surface Feel of Objects for Robotic Manipulation
IEEE International Conference on Robotics and Automation (ICRA), 2025.

(PDF) (BibTeX)

This work is based on Abdallah Ayad's Master Thesis:

Abdallah Ayad, Adrian Röfer, Nick Heppert, Abhinav Valada
Imagine2Touch: Predictive Tactile Sensing for Robotic Manipulation using Efficient Low-Dimensional Signals.
ViTac 2024: Robot Embodiment through Visuo-Tactile Perception, ICRA Workshop, 2024.

(PDF) (BibTeX)

Authors

Adrian Röfer

Adrian Röfer*

University of Freiburg

Nick Heppert

Nick Heppert*

University of Freiburg, Zuse School ELIZA

Abdallah Ayad

Abdallah Ayad

University of Freiburg

Eugenio Chisari

Eugenio Chisari

University of Freiburg

Abhinav Valada

Abhinav Valada

University of Freiburg

*equal contribution

Acknowledgment

This work was funded by the Carl Zeiss Foundation with the ReScaLe project and the BrainLinks-BrainTools center of the University of Freiburg. Nick Heppert is supported by the Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA) through the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Education and Research.