Beginning in 1872, the mansion was the home of Robert Garrett, president of the Baltimore & Ohio Railroad, and his wife Mary Frick Garrett. After Robert Garrett's death, Mrs. Garrett married Dr. Henry Barton Jacobs and the couple added the ballroom by purchasing the neighboring house and converting it into an space for entertaining.
The mansion is exceptional for many things, including that it can boast of being the product of two renowned architects, Stanford White and John Russell Pope. The building, and particularly the ballroom, was the location of many society balls and Gilded Age parties that included dinner, dancing, live music, theater, and other festivities often for hundreds of people.
After Mrs. Garrett passed in 1936 and Dr. Jacobs in 1939, the Mansion had a series of different owners. In 1961, The Engineer's Club leased the building from the City of Baltimore. The city had acquired the property as part of an urban renewal plan to demolish the south side of Mt. Vernon Place and make way for an expansion of the Walters Art Gallery. Fortunately, these plans never received the funding required and in 1962 the Engineer's Club purchased it outright beginning a dedicated effort to preserve and maintain the historic structure.