The building known as Hampton House was built for Nathaniel Rogers in about 1840. Rogers was born in Bridgehampton, the son of John Topping Rogers and Sarah Brown. John Topping was the son of John Rogers, a patriot who served under Col. Smith in the Revolutionary War and who signed the Woolworth Agreement in 1787.

Nathaniel was initially involved with the ship building industry, though later, due to a knee injury, he returned to Bridgehampton.

Soon after his return he began to study painting under Mr. Wood in New York City, and eventually became one of the foremost miniature painters in New York. By the late 1830’s, suffering from ill health, Rogers decided to return to Bridgehampton where he built Hampton House. The house was impressive for its day, a full three stories with a fourth-story cupola, an Ionic two-story porch across the front, and elegant roof top balustrades.

Following Rogers’ death in 1844, the house was occupied by his widow Caroline Denison. Following her death, Capt. James Hunting purchased the house and lived there until he built a new house in the 1870’s. Following his ownership, the De Bost family lived in the house, and then in 1885 a man named Storms purchased the house and opened it as “Hampton House,” a hotel that was equipped with a bar. Hampton House did not flourish under his ownership and after his death, Frank Hopping and Capt. John N. Hedges purchased the building, renovated it, and re-opened it as a first-class hotel and boarding house with the same name.

The house was badly damaged in the 1938 hurricane, during which it lost its main roof balustrade and cupola. In 1969 New York State prepared a book entitled Long Island Landmarks, in which the State featured the most important structures on the Island. In this book Hampton House was illustrated and described as: