503 Fifth Avenue, ca. 1918.

The first structure on this corner of Fifth Avenue and 42nd Street was a four-story residence completed in 1860. The brick and brownstone mansion with arched windows and a mansard roof became the home of Montagnie Ward, a drug merchant. The next owner, Levi P. Morton, a banker who would go on to national and international political notoriety, purchased the home in 1871. Morton proposed demolishing the house for a seven-story commercial building in the late 1870s. While that was not to come to pass, the home did have its mansard lopped off, two floors added, and was doubled in size by an extension to the full depth of the lot on 42nd Street. (Prior to conversion into a commercial building, a young Edith Jones, later Wharton, was introduced into society at the Morton Mansion.)

By the late 19th century the top floors of the building functioned as a hotel, while the ground level housed a showroom for one of the premier carriage makers in the city. Despite out-of-town positions such as Ambassador to France and American Vice President, Morton held on to the building. (It was quite the income generator for Morton, with considerable rents and money generated from the billboards that covered the exterior by the 1910s.) Morton died in 1920 but his heirs retained ownership through 1980.

The building changed hands between large commercial development firms a few times in the 1980s while being emptied of tenants. A history-cum-eulogy of the building in 1988 by NY Time writer Christoper Gray suggested an unknown yet imminent timeline for the building's disappearance. The building and a few of its neighbors came down in 1990, but the lot remained empty for more than a decade. In 2004 a 27-story glass and concrete commercial structure was erected.