Shadow detection is a challenging and essential task for interpreting the scene. Regardless of encouraging findings from current Deep Learning (DL) approaches used for shadow detection, the methods are also dealing with inconsistent situations where the visual representation of non-shadow and shadow regions is equivalent. In this article, a DL based approach is introduced for image pixel-level shadow detection. The proposed CNN-based approach, pattern conserver convolutional neural network (PCPerturNet) profits from a new design where shadow features are defined utilizing an effective skip-connection mapping arrangement. To make PCPerturNet robust from the change in brightness and contrast, several perturbed instances are generated by using a fuzzy-logic based method to train the system. Also, five types of augmentations are applied to images during training to make the system robust from the change in scale, orientation and flip. PCPerturNet derives and conserves shadow patterns in manifold layers and uses those layers progressively in several units to produce the shadow mask. The output of the proposed method is tested on two freely accessible databases and one self-created database where the accuracy rate obtained is 96.4%, 96.8%, and 89.4% which indicates that the proposed method outperforms the other shadow detection approaches used in the literature. © 2021 The Authors. IET Image Processing published by John Wiley & Sons Ltd on behalf of The Institution of Engineering and Technology