SeGAN: Segmenting and Generating the Invisible
Objects often occlude each other in scenes; Inferring their appearance beyond their visible parts plays an important role in scene understanding, depth estimation, object interaction and manipulation. In this paper, we study the challenging problem of completing the appearance of occluded objects. Doing so requires knowing which pixels to paint (segmenting the invisible parts of objects) and what color to paint them (generating the invisible parts). Our proposed novel solution, SeGAN, jointly optimizes for both segmentation and generation of the invisible parts of objects. Our experimental results show that: (a) SeGAN can learn to generate the appearance of the occluded parts of objects; (b) SeGAN outperforms state-of-the-art segmentation baselines for the invisible parts of objects; (c) trained on synthetic photo realistic images, SeGAN can reliably segment natural images; (d) by reasoning about occluder-occludee relations, our method can infer depth layering.
Humans have strong ability to make inferences about the appearance of the invisible and occluded parts of scenes [1, 34] . For example, when we look at the scene depicted in Figure 1 we can make predictions about what is behind the coffee table, and can even complete the sofa based on the visible parts of the sofa, the coffee table, and what we know in general about sofas and coffee tables and how they occlude each other. Devising algorithms to infer the appearance of what is behind an object exhibits several challenges. Predicting the appearance of the occluded regions of objects requires reasoning over multiple intertwined cues. Recognizing if an object is occluded or not is the first challenge to begin with. Second, knowing what pixels to color requires extending the boundaries of objects from their visible regions to invisible parts which requires some form of knowledge about the shapes of objects. The complex relations between the appearance of objects and the change in viewpoint and occlusion patterns form the third challenge. Deformable objects can even make the problem ill-defined. Fourth, it is challenging to provide large-scale, accurate, Figure 1 . Our goal is to jointly segment and paint the invisible regions of objects. For instance, we predict how the sofa looks like when the occluders (cushions, laptop and coffee table) are removed. The input to our model is an image and a mask for the visible regions of objects (shown in blue).
and reliable training data to train models for occlusion reasoning.
In this paper, we study the problem of generating the invisible parts of objects. Doing so requires segmenting the invisible part of the object and then generating the appearance of (painting) it. Inspired by the principles of learningthe-easy-thing-first, we propose SeGAN, a novel model that combines segmentation and generation and jointly optimizes for both of them. More specifically, given an input image and a segmentation for the visible regions of an object, our proposed GAN-based model learns to predict a segmentation for the occluded regions and generate the appearance by painting the invisible parts. Using segmentation masks of the invisible part as our intermediate step enables our network to learn about what pixels to paint before painting them. The generator network then paints the selected pixels. By jointly learning segmentation and generation networks SeGAN learns about the interdependencies between objects, their occlusion patterns, the shape and appearance of object segments. This allows us to address the first three challenges.
The key remaining challenge is training data; where can we find large-scale and accurate training data for what is behind the visible part of images? We argue that the proposed solution for Amodal segmentation in  is not suitable for our approach. Human judgements for predictions about the invisible parts of objects is subjective. Also, superimposing segments of images over other images  would result in unnatural occlusion boundaries. In this paper, we propose to use photo-realistic synthetic data to learn how to extend segmentation masks from the visible parts of objects to the invisible regions and how to generate the appearance of the invisible part. Doing so allows us to obtain large-scale and accurate training data for the invisible regions of objects in images.
Our experiments show that SeGAN can, in fact, segment and generate the invisible regions of objects. Our results also show that our proposed segmentation network can learn to segment the occluded regions of objects and outperforms various state of the art segmentation baselines. We also show that our segmentation network can reliably segment the invisible parts of objects in natural images, when trained on our photo-realistic training set. By reasoning about occlusion patterns, our model can also make predictions about occluder-occludee relationships resulting in depth ordering inferences. Note that SeGAN is category-agnostic and does not require semantic category information.
2. Related Works
There is a large body of work on object detection [14, 13, 18, 41, 42, 45] , semantic segmentation [31, 2, 3, 28, 36, 53, 7, 35, 30, 24] and instance segmentation [39, 26, 8, 6, 40, 51, 52 ] using deep learning. These methods are designed for the visible regions of objects and they are not able to capture occlusions or provide a depth ordering for objects in an image. In contrast, our goal is to reconstruct occluded regions.
Occlusion reasoning has been studied in the literature extensively.  propose a CRF for segmenting partially occluded objects.  infer occlusion edges of polygons that represent objects.  make DPM more robust to occlusion by inferring whether a cell inside the object bounding box belongs to the object or not.  use scene priors to infer the label for the occluded background regions.  pro-pose a layered object detection and segmentation method, where the goal is to infer depth ordering for the detected objects.  propose an occlusion model for object instance detection based on 3D interaction of objects. [15, 12] propose methods for detection and pose estimation of occluded people.  learn occluder-occludee patterns to improve object detectors.  synthesize scenes by retrieving segments from training images, which requires reasoning about depth layers in the scene.  provide a semantic label for each pixel in an image along with the occlusion ordering for objects.  use top-down information to tackle occlusions in multi-instance segmentation. We differ from all of these methods in that we complete the segmentation mask for the occluded objects and generate the appearance for the occluded regions of each object instance. Also, we show transfer from synthetic to natural images.
The problem of bounding box completion has been tackled by  , where the goal is to find the full extent of the object bounding box. Amodal segmentation methods have been proposed by [55, 27] , where they aim to provide a complete mask for occluded objects. The annotations that  provide is mainly based on the subjective judgment of the annotators (since the occluded parts of objects are not visible). In contrast, we modify our scenes by removing occluders and obtain an accurate groundtruth mask and texture for the occluded objects. The groundtruth annotation of  is obtained by pasting an object over an arbitrary image. Our argument is that occlusion relationships are not arbitrary and follow certain characteristics, and the way that we collect our occlusion data enables us to better model the occlusion relationships. Also, in contrast to these methods, we generate the appearance for the occluded regions.
Conditional Generative Adversarial Networks (cGANs)  have been used for different applications such as prediction of future frames  , style transfer  , colorizing and synthesizing images from edge maps  , etc. Image inpainting using cGANs and DCGANs has been explored by  and  . In this paper, we combine cGANs with a convolutional network to segment and paint the occluded regions of objects simultaneously. Our problem is different from inpainting since our goal is to paint regions outside the input mask.
Recently,  proposed a regression-based approach to synthesize images from a given semantic segmentation layout. Our method differ from  since their goal is not to reconstruct occluded regions. Also, our method performs both segmentation and painting and it is category-agnostic.
Our goal is to segment and paint the occluded regions of objects. The inputs to our model are a segmentation mask for the visible (non-occluded) regions of an object and an RGB image. The output is an RGB image where the oc- Figure 2 . Model architecture. Our network has three parts: segmentor, generator, and discriminator. The input to our model is an RGB image and a mask for the visible region of an object which is obtained automatically by  . The output is an RGB image that shows the appearance and segmentation for the full object (visible and reconstructed invisible regions). The segmentor part outputs an intermediate mask (the mask shown in the middle) that represents the full object, which is passed to the generator part of the network.
cluded regions of that object have been reconstructed. The segmentation masks for visible regions can be obtained automatically from any instance segmentation method (e.g.,  ).
We introduce SeGAN that infers the mask for the occluded regions and paints those regions in a joint fashion. Our model has two main parts: (1) segmentation and 2painting. The segmentation part provides a mask for the occluded and non-occluded regions of objects, which is fed into the painting part of the model. The painting part generates the appearance for the occluded region of the object. These two parts of the network are trained jointly. The architecture of the model is shown in Figure 2 .
The segmentation part of the network is a CNN that takes a four-channel tensor as input, where three channels correspond to the RGB image, and there is a single channel for the segmentation mask of the visible region of an object. The mask for the visible region is obtained automatically (refer to Section 5 for details). The idea is to use the information from visible regions to segment and paint the invisible regions. We modify ResNet-18  to generate a mask image as output (the output of the last convolutional layer). Then, the mask output is fed into an ROI pooling layer. The ROI pooling layer is followed by a fully connected layer with the output size of 3364 (58 × 58), and refer to its output by o. An upsampling layer converts o to 256 × 256. We denote the output of the upsampling layer by O. Our final result is more accurate when we use upsampling.
The painting part of the network generates the invisible or occluded regions of the object. This part is a conditional generative adversarial network (cGAN)  , which consists of a generator and a discriminator.
The input to the generator, M , is computed as follows:
M (I, O, V ) = I V + R (O − V ) + B (J − O − V ), (1)
where is element-wise multiplication, I and V are the input RGB image and input binary visible mask (SV), J is an all-one matrix of size 256×256, and R and B are 256×256 images, where their first and third channels are 1s, respectively, and the rest of their channels are 0s. All of the binary masks in the above equation are repeated three times to form a 3 channel image. Basically, in the generator's input, the mask for the invisible region (which is provided by the segmentation part of the network) is red, and the region outside the mask is blue.
We adopt Unet  for the generator network, which is an encoder-decoder with skip connections from encoders to the corresponding layers in the decoder. The discriminator network includes four convolutional layers, followed by one sigmoid layer. The architecture for this part is similar to that of the Pix2Pix network  .
The loss function for our model is a combination of the losses for segmentation and painting. For segmentation, we define a customized loss function using binary cross entropy loss that is computed on the prediction of the network and the groundtruth for the full object binary mask (referred to as SF). In Section 4, we explain how we obtain accurate groundtruth for the occluded regions. Ideally, the segmentation part should learn 1) not to change the mask for the pixels in SV (mask for visible regions) and 2) to predict the mask for the pixels in SI (mask for invisible regions) cor-rectly. The binary cross entropy loss is defined as:
EQUATION (2): Not extracted; please refer to original document.
where S is a subset of pixels (e.g., pixels of the visible region), g ij and o ij are pixels at location (i, j) of the groundtruth SF and predicted mask, respectively, and n = |S|.
Our loss function for segmentation is defined as:
EQUATION (3): Not extracted; please refer to original document.
where SF is the set of pixels in the image patch not in SF , or in other words the pixels that do not belong to either visible or invisible parts of the object. A sigmoid function is applied to the predicted output so we obtain a real number between 0 and 1. The intuition for defining this objective is to differentiate among making mistakes in segmenting the visible region, invisible region and the background. The loss function for painting is defined as follows:
EQUATION (4): Not extracted; please refer to original document.
where G and D are the generator and the discriminator networks, respectively, x is the input and z is a random Gaussian noise vector, which is mainly used for regularizing the generator network. Previous approaches found L1 and L2 distance losses to be helpful for GANs [37, 22] , thus the final loss function for the adversarial part is defined as:
EQUATION (5): Not extracted; please refer to original document.
The loss function for our SeGAN end-to-end model, L full , is defined as:
EQUATION (6): Not extracted; please refer to original document.
In this paper, we introduce DYCE, a dataset of synthetic occluded objects. This is a synthetic dataset with photo-realistic images and natural configuration of objects in scenes. All of the images of this dataset are taken in indoor scenes. The annotations for each image contain the segmentation mask for the visible and invisible regions of objects. The images are obtained by taking snapshots from our 3D synthetic scenes. A few examples of images and their annotations are shown in Figure 3 .
There are two advantages of a synthetic 3D dataset. First, we can obtain a 2D dataset of the desired size, and there is no restriction over the number of training samples we can Figure 3 . Example images of the dataset. The first column shows the images captured from 3D synthetic scenes. The second column shows the segmentation mask for the visible regions. Each instance is encoded by a different color. The third column shows the invisible regions. For example, in the second row, the cushions occlude the sofa. Therefore, the regions behind the cushions have grey color in the third column, which means that those pixels belong to the grey sofa in the second column.
generate. Second, we can move the camera to any location to capture interesting patterns of occlusion. We use the scenes of  to generate our dataset.
4.1. Generating 2D Images From 3D Scenes
For generating the images, we change the location and the viewpoint of the camera in order to get a variety of images. For each scene, we generate 500 images from different viewpoints of the camera. We restrict the areas that the camera can be located. We move the camera in locations that the head of a person can be located in order to obtain common patterns of occlusion that people observe. We also restrict the orientation of the camera such that the camera points to objects in the scenes. Otherwise, the dataset will contain many images with no objects (for example, images depicting a portion of a wall).
The procedure for generating the segmentation mask for the visible and invisible regions of objects is as follows. For each object, we generate an image with all other objects removed. Then, we compare this image with the original image, where no object is removed from the scene. The pixels that are the same in both images are the visible pixels of this particular object. To obtain the mask for the invisible region, we subtract the mask of the visible region from the mask of the full object.
The number of the synthetic scenes that we use is 11, where we use 7 scenes for training and validation, and 4 scenes for testing. Overall there are 5 living rooms and 6 kitchens, where 2 living rooms and 2 kitchen are used for testing. On average, each scene contains 60 objects and the number of visible objects per image is 17.5 (by visible we mean having at least 10 visible pixels). There is no common object instance in train and test scenes.
Our model performs segmentation and painting jointly. Hence, in this section, we evaluate our model from these two perspectives. In addition, we show results of generalization to natural images. Finally, we present our evaluation for the depth layering task. Our training and test sets include 41924 and 27617 objects depicted in 3500 and 2000 images, respectively.
5.1. Implementation Details
All input images and their masks are resized to 500×500. We used bilinear interpolation for resizing. Thus, the segmentation mask might contain values in the interval (0, 1). To obtain bounding boxes for the ROI pooling layer we expand the box around the input SV masks by a random ratio between 10-30% from each side. Note that we ignore the portions that lie outside the image. We compute the segmentation loss on groundtruth segmentation masks of size 58 × 58 (for each object, we crop the image using the expanded bounding box and scale the cropped image to 58 × 58). Then, we upsample the predicted mask to 256 × 256 using a bilinear upsampling layer and use the 256 × 256 mask as the input to the painter network. We do not train the upsampling layer. The generator outputs a three channel 256 × 256 image, which includes the RGB values for the full object (invisible and visible regions).
We use the following coefficients in the loss function: λ bg = 1, λ SV = 5, λ SI = 3, λ L1 = 100, and λ L * = 0.1. These values are obtained using a validation set. Also, to help the network to converge, we first train the segmentation network and the generator network jointly and then train the whole network end to end.
Segmetation & Painting. We evaluate our model, SeGAN, in two settings. First, we use the output of the Multipath network  , which is a state-of-the-art model for generating the segmentation mask for objects as our input mask for the visible regions (SV masks). Secondly, to factor out the effects of SV segmentation approach from our results, we also show the results using the groundtruth mask as the input for the visible region of the object.
After obtaining segmentation masks from Multipath, we find the segmentation mask that corresponds to the visible region of the groundtruth training object. The segmentation mask that has the largest intersection over union with the visible region of the groundtruth mask is selected as the input mask during training. For evaluation, we consider all masks generated by Multipath. We evaluate our model using three metrics for segmentation and two for painting. For segmentation, we evaluate how well we predict (1) the mask for the occluded regions (SI), (2) the mask for non-occluded regions (SV), and (3) the mask for the full object (SF=SV ∪ SI). The intuition for evaluating the mask for visible regions is to check whether our approach distorts the input mask when the object is not occluded. For all of these settings, we compute intersection over union between the predicted mask of the model and the groundtruth mask. For evaluating painting, we use L1 and L2 distance of the predicted output and the ground truth image. Table 1 summarizes our results for segmentation. First, our method (referred to as 'SeGAN w/ predicted SV') significantly outperforms Multipath for the task of predicting masks for the full object (SF) and invisible regions (SI). It is interesting to see that our method improves the segmentation of the visible regions (SV) as well. We have two variations of the Multipath network as our baselines: one trained only on natural images data (trained on MS COCO dataset  ) and one trained on the combination of natural images data and our synthetic images (trained to predict occluded and visible regions). Another baseline is IBBE  (an amodal segmentation method), which is fine-tuned on our synthetic data. We also compare with Pix2Pix  , where it receives the same inputs as our model and generates the appearance for the full object. Using groundtruth masks (SeGAN w/ GT SV) shows that our method will perform even better if it receives more accurate masks for the visible regions as the input. Figure 4 shows qualitative segmentation results of our method.