Semantic segmentation is pixel-wise labeling of the image. Recently deep convolutional neural network (DCNN) providing progressive results in semantic segmentation. However, in remote sensing multispectral imagery very limited work has been done due to lack of training dataset. In this paper, a Res-Seg-net model is proposed for the semantic segmentation which is motivated by the existing Resnet and Segnet models. This model consists of encoder-decoder parts in which residual mapping is followed. For validation and testing of the proposed model, the RIT-18 dataset of multispectral imagery is used. The comparison results of the experiment on a multispectral imagery dataset have demonstrated the effectiveness of the proposed model. © 2020 IEEE.