Despite of many efforts for handling various holes, it has been not sufficiently resolved and the instability and normalization issues exists due to the presence of the invalid pixels. We proposed the weighted convolution that balances the valid and invalid pixels throughout the networks to help the network efficiently cope with various hole shapes. In our convolution layer, the mask is utilized to store the validity of the features by using the real-valued mask. A weighted scheme for the normalization layers is also proposed to adaptively operate along with the weighted convolution. By balancing upon the invalid pixels caused by the holes and zero-paddings, the network can be trained more robust to the hole shapes. The experimental results verified that our method achieved improvements over the state-of-the-art inpainting methods.