Rutherford B. Hayes' uncle and guardian, Sardis Birchard, constructed the original portion of the home between 1859 and 1863 as a summer home he could share with his nephew and young family. Construction took five years because materials and labor were difficult to obtain during the American Civil War. The two story brick home had eight bedrooms and a wrap-around verandah. Rutherford B. Hayes particularly loved the verandah. In 1873 he wrote in his diary, "The best part of the present house is the veranda. But I would enlarge it. I want a veranda with a house attached!" Hayes spent the next 20 years planning additions and improvements to his home and estate, much as Thomas Jefferson had with his beloved Monticello.
Hayes moved his family into the home in 1873 for two years before leaving to serve as Governor of Ohio and then President of the United States. In 1880 President Hayes prepared for his return to Fremont from the White House by building a substantial addition and remodeling the interior. The addition included a library to house his 12,000 books, a large reception room, three bedrooms, and indoor plumbing. The most spectacular improvement was a four-story walnut and butternut staircase leading to a rooftop lantern offering a 360-degree view of Spiegel Grove (the Hayes estate).