 This paper proposes using a convolutional neural network, CNN, to classify individual pixels from multispectral ortho-imagery and a digital surface model, DSM. The CNN was trained on a small city with five different categories, vegetation, ground, roads, buildings, and water. The CNN achieved a high level of accuracy when compared to other per pixel classification works on other land areas with similar category choices. The results of the full classification and segmentation on selected segments of the map showed that CNNs are a viable tool for solving both the segmentation and object recognition tasks for remote sensing data. This article was authored by Martin Langfist, Andrei Kizilev, Marjana Lyrizé, and others.