Charles Chiswell obtained 9,976 acres on what was then the western part of New Kent County in 1717. He named his country home Scotchtown and imported a colony of Scots to build the house and outbuildings, including a mill. However, an outbreak of yellow fever killed many of the Scots and caused the remainder to flee, leaving behind only traces of the Scottish castle Chiswell had hoped to construct.
Hanover native Patrick Henry brought Scotchtown and 960 acres at auction in 1771. Already a leader in the pre-Revolutionary Virginia politics of the time, Henry moved there with his wife, Sarah, and six children. It is believed that he saw clients in the main house, which has changed little over the centuries. Scotchtown today contains three pieces of furniture - a map table, tea caddy and baby cradle - that belonged to Henry.