The house was built by James Brown, a wealthy plantation owner from Madison County, Mississippi. The house was started in late 1848 and was completed in 1852.
In 1877, Jefferson Davis was looking for a quiet retreat to write his books and papers. He fell in love with Beauvoir and bought it from a family friend for $5,500, to be paid in three payments. He made the first payment and six months later, the owner died and he found out he was her sole heir and eventually inherited the house along with other property.