The historic Ivy Hotel got its start in the late nineteenth century when a prominent Baltimore banker named John Gilman commissioned a mansion in Mount Vernon for the princely sum of $40,000. Gilman died before the building's completion in 1889, but his widow lived there for several years before selling it to William and Harriet Painter. William Painter was the head of Crown Cork and Seal company and his invention of the bottle cap made him one of the city’s leading businessmen.
After the deaths of Mr. and Mrs. Painter, the mansion went through several other owners, including Robert Garrett, grandson of the president of the Baltimore & Ohio Railroad and the gold medalist in both discus and shot put at the first modern Olympics in Athens in 1896. Mr. Garrett eventually donated the building to the Playground Athletic League, which he chaired, and in 1939 the PAL donated it to Baltimore City for use as offices for the Department of Recreation and Parks. In 1985, Mayor William Donald Schaefer had the city purchase two adjacent rowhouses, undertook a complete historic renovation project, and turned the building into a city owned hotel: the Inn at Government House.
In 2015, the Azola Companies, Ziger/Snead Architects completed a restoration turning the building into a boutique historic hotel, complete with parquet floors, pocket doors, stained glass, and a grand staircase.