Edward Inman was heir to a large cotton brokerage fortune amassed after the Civil War. He was an Atlanta businessman with interests in real estate, transportation, and banking. His wife Emily was involved in philanthropy, politics, and society. In 1924, they hired the architectural firm of Hentz, Reid, and Adler to design a house for their property in Buckhead, a residential suburb of Atlanta.
Noted architect Philip Trammel Shutze designed Swan House and its gardens, as well as many other important buildings in the city. He adapted Italian and English classical styles to accommodate 20th-century living for Swan House, which many consider his finest residential work.
The Inmans moved into their new home in 1928, a year before the Great Depression began. Just three years later, Edward Inman died suddenly at age 49. Emily asked her oldest son Hugh, his wife Mildred, and their two small children to live with her. Grandchildren Sam and Mimi grew up in the home and moved out after they were married.
Many household staff members were African American men and women.
In 1933, after several decades of working for the Inmans, Elizabeth “Lizzie” McDuffie left to work in the President Franklin D. Roosevelt White House as a third-floor maid. There she advocated for racial equality, was a political activist, and helped form a worker’s union for government employees.
Emily lived in Swan House until her passing at age 84 in 1965. In 1966, the Atlanta Historical Society purchased the home and most of its original furnishings, ranging from 18th-century antiques to 20th-century objects. It opened to the public in 1967 as a house museum and headquarters of the Atlanta Historical Society.
In 2004, the Atlanta History Center completed a $5.4 million restoration of Swan House and its furnishings. It is listed on the National Register of Historic Places.