The original Church of the Nativity was designed in 1832 by several architects including the renowned Alexander Jackson Davis who also designed Federal Hall in Lower Manhattan, among other famous structures. Built for the Second Avenue Presbyterian Church, in 1842 it was sold to the newly formed Nativity of Our Lord parish and became the Church of the Nativity. The imposing Greek Revival building was demolished in 1970 following a fire.

The second church building was designed by the firm of Genovese and Maddelene, and is imposing in its own way. The Modernist or specifically Brutalist church is constructed of brick and cinder blocks, with blocky massing and simplified religious details. It has been described by some as "starkly institutional." The church closed in July of 2015 and was deconsecrated two years later. Today it stands, graffiti covered. In early May of this year, a town hall meeting was convened to discuss potential future opportunities for former religious properties in the neighborhood.