Deformable Objects Perception is Just a Few Clicks Away – Dense Annotations from Sparse Inputs

Deformable Objects (DOs), e.g. clothes, garments, cables, wires, and ropes, are pervasive in our everyday environment. Despite their importance and widespread presence, many limitations exist when deploying robotic systems to interact with DOs. One source of challenges arises from their complex perception. Deep learning algorithms can address these issues; however, extensive training data is usually required. This paper introduces a method for efficiently labeling DOs in images at the pixel level, starting from sparse annotations of key points. The method allows for the generation of a real-world dataset of DO images for segmentation purposes with minimal human effort. The approach comprises three main steps. First, a set of images is collected by a camera-equipped robotic arm. Second, a user performs sparse annotation via key points on just one image from the collected set. Third, the initial sparse annotations are converted into dense labels ready for segmentation tasks by leveraging a foundation model in zero-shot settings. Validation of the method on three different sets of DOs, comprising cloth and rope-like objects, showcases its practicality and efficiency. Consequently, the proposed method lays the groundwork for easy DO labeling and the seamless integration of deep learning perception of DOs into robotic agents.