ing at the problems in the semantic segmentation of remote sensing images, such as missed detection of multi-scale targets and rough segmentation boundary, we propose a method of building change detection for aerial images based on an attention pyramid network. The method adopts an encoding-decoding configuration. In the encoding phase, we utilize ResNet101 as the basic network to extract the features and apply dilated convolutions to improve the receptive field in partial residual modules. Meanwhile, the pyramid pooling structure is selected as the last layer of the encoding network to extract multi-scale features of the images. In the decoding phase, the attention mechanism is employed in lateral connection to highlight significant features, and the procedure of top-down dense connection is used to calculate the feature pyramid and then to fuse the features with different resolutions at different phases. Furthermore, the verification experiments are performed on the dataset of building change detection, and the results indicate that our method has good adaptability to different-size-building change detection and has certain advantages in comparison with the classical semantic segmentation networks.