Serving more as an outpost than a home, the Duryea House was one of the few early colonial structures left in Greenpoint when it was razed in the 1920s. The farmhouse sat at the bank of Newtown Creek and served as both a defensive outpost and later a "pivot house" for control of the Meeker Avenue Bridge in its over 200-year lifespan.

The wooden farmhouse was likely built by Humphery Clay around 1681. He operated a ferry that docked near the home. The structure had thick outer walls, adorned with three V-shaped openings for shotguns and canons to be aimed through. If the outer wall was penetrated, inhabitants could retreat to a second wall, which was thick enough to support the second floor of the house. This wall was embedded with another series of holes for defense. In the back half of the ground floor, a central staircase led to rooms on the upper floor which included an ammunition magazine. Another of the rooms was a sort of crude holding cell, with an iron ring for chaining unlucky prisoners. The original inhabitants of the house were slaveholders, as slavery was legal in this area until 1827. For a long time, the house didn't even have a porch. To enter, a ladder was lowered and then raised to prevent American Indians from across the creek

The home was acquired by Joost Durie in 1749, a member of the Duryea family. The family left the home in 1828, though the house kept the name Duryea. In the early days of the house, a penny bridge was used to cross the Newtown Creek. By the 1800s, the original bridge burned down and a pivot bridge was constructed. The inhabitant of the home was in charge of turning the bridge, and thus the home took the name "Pivot House." Unfortunately, the structure fell into disrepair by the 20th century. Repairs were estimated to cost $1,500, a huge sum at the time. The owners of the home decided to tear it down, and Greenpoint lost one of its oldest pieces of colonial history.