Gertrude Vanderbilt Whitney was a talented sculptor and collector. Inspired by an early trip to France, Whitney was left with a long-lasting love of art. She went on to study at the Art Students League of New York. Exhibiting work under an alternate name, Whitney feared that her family name would disqualify her from a career as a serious artist. If she took commissions, she would be criticized for taking from artists in need of work. If she worked for free, she would be seen as undercutting the art market. Whitney opened a studio and apartment at this location in Greenwich Village in 1907.

With her personal wealth, Whitney began seriously collecting the work of avante-garde artists that were otherwise cast aside by the traditional academy. The "Whitney Studio" was established in her home in 1914 as an exhibition space. She later expanded it to incorporate housing and studio space for working artists.

By 1929, Whitney had collected around 700 pieces of art with the help of her assistant, Juliana R. Force. However, when she offered the majority of the collection to the Metropolitan Museum of Art as a gift, it was declined. The Met's rejection of the works and MoMA's preference towards exhibiting modern European works inspired Whitney to create her own museum, solely dedicated to American art. Architect Noel L. Miller connected rowhouses at 8, 10, and 12 West 8th Street to form the first Whitney Museum. Juliana R. Force served as the museum's director.

In 1954, the museum moved to a location adjacent to MoMA on 54th Street. In 1958, a fire spread through MoMA's upper floors, forcing both museums to evacuate. The fire was supposedly started by construction workers who were installing air conditioning on the second floor, who had been smoking near flammable dropcloths. Luckily, the floor was largely empty, and only 6 paintings were lost. The fire killed one staff member and injured over two dozen visitors, museum employees, and firefighters.

Despite the Whitney's trevails, it survived and moved to its most iconic location. Designed in the Brutalist style by Marcel Breuer in 1966, the building was the third location for the Whitney Museum of American Art. The modernist building with gray granite and reverse ziggurat facade was (and still is) a stark contrast to the surrounding 19th-century brownstones, brick row houses, and postwar apartment buildings. The facade is made of 1,500 granite slabs with each weighing between 500 and 600 pounds. Ada Louise Huxtable, a New York Times architecture critic, said of the design, “the taste for its disconcertingly top-heavy, inverted pyramidal mass grows on one slowly, like a taste for olives or warm beer.”

Despite this massive expansion, the museum struggled with space. The Whitney Museum operated multiple branches, including one on 55 Water Street, and used a few corporate office lobbies as exhibition space. This included the Phillip Morris International Lobby, in their Park Avenue location, and space in Champion International in Stamford, Connecticut.

Finally, in 2015, the Whitney Museum of American Art moved into a new building in the Meat Packing District. The Metropolitan Museum of Art began leasing the old structure for "the Met Breuer" with exhibitions that focus on both modern and contemporary art. The Whitney's new home is designed by Renzo Piano.