Having almost lost his entire fortune in the Panic of 1907, Andrew Freeman, a self-made millionaire, came up with a peculiar charity: to create a retirement home for wealthy individuals who had lost their money. Unnerved by the experience, he left some of his substantial fortune upon his death in 1915 to create the eponymous home and it opened its doors in 1924.

The four-story building in French and Italian Renaissance styles cost a million dollars to build and was only to serve individuals who were in "good circumstances" financially. The recently impoverished were supposed to adhere to a set of manners and formal dress. In exchange, these residents were allowed to live rent-free in the plush Palazzo and were attended to by servants.

Two wings were added in 1928 and 1931 just in time to accommodate the suddenly destitute stockbrokers of the Great Depression.

Eventually, during the 1960s, the trust that supported the home, much like its denizens, ran out of money and the Home's residents were forced to pay rent. .

It was eventually bought by the mid-Bronx Senior Citizen's Council and turned into a retirement home for the destitute (regardless of previous circumstances). The Andrew J. Freedman Home was named a New York City Landmark in 1992 and now serves as an event and art space as well as a day care center.