Learning a simulation-based visual policy for real-world peg in unseen holes.

This paper proposes a learning-based visual peg-in-hole that enables training with several shapes in simulation and adapting to arbitrary unseen shapes in the real world with minimal sim-to-real cost. The core idea is to decouple the generalization of the sensory-motor policy from the design of a fast-adaptable perception module and a simulated generic policy module. The framework consists of a segmentation network (SN), a virtual sensor network (VSN), and a controller network (CN).