build: Move setting of SRC_DIR to the only place it is used