The main challenge presents in bitemporal building change detection (BCD) in remote sensing (RS) is to detect the relevant changes that are related to the buildings, while ignoring changes induced by other types of land cover as well as varied environmental condition during the sensing process. In this paper, we propose a new BCD model with a double encoder architecture. The Gabor wavelet-based encoder which aims to highlight the characteristic of buildings on RS imagery i.e., the comparatively more regular and repetitive texture than other objects on RS images. This Gabor Encoder is used in addition to the convolutional neural-network-based encoder that extracts other meaningful and highlevel information from the images. Moreover, we also propose Feature Conjunction Module to efficiently combine the extracted features by characterizing possible types of changes. Comparative results with Stateof-the-art models on 3 different BCD datasets (LEVIR-CD, S2Looking, and WHU-CD) confirm that the proposed model outperforms current BCD methods in producing a highly accurate change map of buildings. Our code is available on https://github.com/Ayana-Inria/AYANet.