Originally designed in 1835 as a home for Samuel Polk Walker, President James K. Polk's nephew, in 1837 it became the residence of the Reverend Mr. Franklin Gillette Smith. Rev. Mr. Smith had come to Tennessee to serve as president of The Columbia Female Institute. In 1851, he left his post and founded The Columbia Athenaeum School.
The Rectory boasts the original seven-metal chandelier and "flashed glass" front door side panels, containing gold made in Europe. At its height, the library contained more than 16,000 volumes, with the natural science department holding more than 6,000 specimens, a few of which still remain. The school offered courses in art, music, history, science, tennis, gymnastics, bowling, croquet, as well as instructions in the Bible and etiquette.
The 22-acre estate passes through the family until 1973 when it became part of the Association for the Preservation of Tennessee Antiquities. Today, costumed docents are available for the 30-45 minute guided tours, as well as planned fashion shows which include instructions on parlor games.