Baker Mansion was originally home to ironmaster Elias Baker and his family. Baker purchased the nearby Alleghany Furnace in 1836 in partnership with his cousin Roland Diller. Elias brought his wife, Hetty, and their two sons, David Woods and Sylvester, from Lancaster County to what was described as a "tolerable good mansion house" near the furnace. Shortly after they arrived, a daughter, Anna, was born. A fourth child, Margaretta, was born in 1839, but she died of diphtheria at the age of two.
In 1844, Elias Baker bought out his cousin's share in the furnace. That same year, he contracted with Baltimore architect Robert Cary Long, Jr. to design him a new home. Work got underway on the mansion in 1845 but many problems and delays retarded its completion until 1849. The total cost was about $15,000. The cost overruns, coupled with falling prices for iron, pushed Mr. Baker to the brink of financial ruin before the home was finished. Still, Baker, ever the determined businessman, pressed ahead and achieved his dream house. He would enjoy it for fifteen years until his death in 1864. Hetty Baker remained a widow until her death in 1900.
David Woods Baker married Sarah Tuthill in 1851 and they had one daughter, Louise, in 1852. However, Woods was killed in a steamboat boiler explosion just two and a half weeks after the birth of his daughter. Louise later married Ernst Beckman and returned with him to live in his native Sweden. Sylvester and Anna both remained single and lived in the mansion until their deaths. After Anna died in 1914, the mansion was closed until 1922, when the Blair County Historical Society leased the building and opened it as a museum. With the community's support for a major fundraising campaign, the society was able to purchase the house in 1941. Since then, the mansion has served as the Society's headquarters and has become beloved community landmark.
In 2001, the Society completed an exterior rehabilitation, returning the building to its original appearance. Work has begun on interior rehabilitation. The project is financed in part from a state Redevelopment Assistance Capital grant and from local matching funds the Society is working to raise.