Baker Mansion was originally home to ironmaster Elias Baker and his family. Baker purchased the nearby Alleghany Furnace in 1836 in partnership with his cousin Roland Diller. Elias brought his wife, Hetty, and their two sons, David Woods and Sylvester, from Lancaster County to what was described as a "tolerable good mansion house" near the furnace. Shortly after they arrived, a daughter, Anna, was born. A fourth child, Margaretta, was born in 1839, but she died of diphtheria at the age of two.
In 1844, Elias Baker bought out his cousin's share in the furnace. That same year, he contracted with Baltimore architect Robert Cary Long, Jr. to design him a new home. Work got underway on the mansion in 1845 but many problems and delays retarded its completion until 1849. The total cost was about $15,000. The cost overruns, coupled with falling prices for iron, pushed Mr. Baker to the brink of financial ruin before the home was finished. Still, Baker, ever the determined businessman, pressed ahead and achieved his dream house. He would enjoy it for fifteen years until his death in 1864. Hetty Baker remained a widow until her death in 1900.