Static image or scene segmentation can never be perfect. We can't be sure where objects are unless we see them move relative to each other. In this project we investigate using motion as a cue to segment objects. We can make use of passive sensing (when other people move objects for us) or active vision (where the robot causes objects to move), and both long-term (between different visits to a scene) and short-term (between consecutive video frames) motion, to aid segmentation.
We investigate using scene flow (a 3-D version of optical flow) estimation to get at short-term motion. Long-term motion we estimate using background subtraction and dense feature matching. Either way we can fit rigid motions on top of these motion estimates in order to segment objects that move uniquely.
a long version of our ICRA video showing three example scenes
Scene flow on other scenes: