Although it is often forgotten, Chelsea played a significant role in New York's African American community and its history, particularly in the second half of the 19th Century. As early as the 1840’s Pierre Toussaint a freed Haitian slave, now in the process of canonization, played a significant role in establishing St Vincent DePauls Roman Catholic on W 23 St for French speaking New Yorkers. This Church started an early school for African Americans in the city. On Lamartine Place, the block of West 29th St between 8th and 9th Aves, the Hopper-Gibbons a Quaker family that played a crucial role in Abolitionism and many progressive and charitable causes, used their house as a stop in the Underground Railroad. The house, still stands is to date the only verifiable Underground Railroad safe House in Manhattan. Particularly from the 1880s to 1900's (and lingering onto WWI), New York's African American community having been displaced from southern Greenwich Village settled in the West 20s and 30s. Thus the areas “Negro Tenderloin” was to the citys largest African-American settlement by the 1890s until once again the community was forced to move because of a brutal race riot in 1900 and displacement due to the demolition of a core area to build Penn Station. Faced with the prospects of bad living conditions, a violent and dangerous environment and displacement, St. Philips Episcopal Church, the city's oldest African American congregation which at the time was located on West 25th St played a central role in fomenting the move to Harlem by a significant portion of the cities African population. Several other of the cities older and most prominent black churches and institutions were in the neighborhood in the latter 19th Century including Bethel AME Church on W25 St and Mt Oliver and Shiloh on West 26th Street and the Colored Mission and the Sons of New York Club on West 30th St.

St. Vincent de Paul ca. 1932 French-speaking Catholics from both France and its New World colonies were among the earliest Catholic settlers in New York and were some of the first parishioners of St. Peter's, New York's original parish. The French population tended to assimilate quickly into the larger society; however, bolstered by continued immigration, in 1841, they organized their own parish, St. Vincent de Paul, on Canal Street, in lower Manhattan, which later relocated to West 23rd Street.

From the 1870's-90's, approximately 20,000 French immigrants lived in South Village and SoHo area. By the end of the 19th century, most of that colony had relocated to the Chelsea area.