Jacob Schieffelin gave the land for St. Mary's Episcopal Church. The parish was formed to minister to the poor in 1823 by fashionable St. Michael's Church in Bloomingdale at the north end of the present-day Upper West Side, where Schieffelin served as a vestryman. By the 1830s, St. Mary's had the distinction of being the first "free" Episcopal church in the nation, charging no annual fee for pew rental.

The rectory of St. Mary's Church at present-day 521 West 126th Street was built about 1840 by architect Robert Oughton. It survived a turn-of-the-century fire that destroyed the church. The church was quickly replaced-and enlarged-by the Schieffelin family (newly enriched through an alliance with the Vanderbilts). With a long history of serving the poor and the working class, it would survive in a changing Harlem where other churches failed.