Dolley Madison was beloved throughout Washington, and the women of the city called her "Lady Madison." She gained the warm gratitude of many Washingtonians and Americans throughout the country during the War of 1812. The British, upon invading the capital, set fire to the city, and the White House was one of the buildings destroyed in the conflagration. During these frightful hours, Dolley reportedly left many of her own belongings in order to save a great, life-size portrait of George Washington from the flames.
In the years of President Madison's retirement, he and Dolley were surrounded by relatives from both their families. Dolley was a great hostess at Montpelier, as well, sometimes arranging dinners for over ninety people at one time. After Madison passed away in 1836, Dolley, who was sixty-nine years old at the time, moved back to Washington, D.C., and continued on as the capital's grande dame. She lived to be eighty-one, dying in 1849.