The house was built by Casparus F. Pruyn, the business agent of Stephen Van Rensselaer III and his son, William P. Van Rensselaer. Rensselaerswyck was a manor laid out on both banks of the Hudson River in 1630 and confirmed to the Van Rensselaer family in a perpetuity in 1685. The manor passed down through the eldest sons until the inheritance practice was forbidden in New York state after the Revolutionary War. Stephen Van Rensselaer, the last patroon to inherit the entire estate, died in 1839 and divided the land holdings amongst his heirs. His two oldest sons, Stephen and William received the West and East Manor respectively.
Pruyn's residence in Colonie was built either by Pruyn or his father-in-law, Robert Dunbar, who had previously served as the Van Rensselaer's agent and had owned the property before Pruyn. After serving as Stephen Van Rensselaer's business agent in Colonie, Casparus Pruyn was hired by William Van Rensselaer in 1839 to manage his business affairs on the East Manor. Pruyn moved to the city of Rensselaer and built a substantial brick residence with an attached office (listed on the National Register as the Patroon Agent's House and Office, 1979). The Colonie house then served as Pruyn's country estate. Between 1839 and 1844, when he resigned, Pruyn was a central figure representing the Van Rensselaer interests in the "anti-rent movement." Letters which survive indicate that he felt himself better able than his employer to handle the tenant opposition and to collect rents. Retiring to Colonie, Pruyn died there in 1847.
There are ten outbuildings including the Buhrmaster Barn reconstructed on the grounds in 1987 and the Verdoy Schoolhouse which arrived in 1996. Other structures include a smokehouse, a potting shed, a woodshed/privy building, a carriage house, wellhouse and restroom facility.