James Madison left Washington, D.C., at the age of sixty-six to retire to a very comfortable life at his Montpelier home in Orange County, Virginia. His health was good, and he entered upon his new life as a farmer and master of a bustling estate. Although his only immediate family members were Mrs. Madison and Madison's aged mother living at Montpelier, the estate was visited often by friends. Mrs. Madison was as enthusiastic a hostess at Montpelier as she had been at the White House, and the estate became a hub of social activity.

As a retired president, and one who had been instrumental in the founding of the American Republic, James Madison was one of the most widely respected elder statesmen in the 1820s and 1830s. He was able to watch the progress of national politics through three presidential administrations. His former cabinet secretary James Monroe was re-elected as President in 1820; John Quincy Adams, son of Madison's old rival John Adams, was elected in 1824; and Andrew Jackson, the hero of the Battle of New Orleans in the War of 1812 and the leading voice of a new era of populist, democratic politics, was elected in 1828 and 1832.

National politics during these years were troubled by the rise of sectionalism, the geographical hostility between North and South, which anticipated the Civil War era to come in mid-century. The Missouri Compromise of 1820 was one of the great political events of the period; it marked a clear division between free states and slave states along the latitudinal line 36º30' in the lands acquired from the Louisiana Purchase of 1803. Although Madison disliked the institution of slavery, he let it be known to President James Monroe and to the nation that he strongly disapproved of a political policy which forbade slavery above a certain geographical line. He saw the effect of the compromise as further partisan divisions between North and South.

As an elder statesman, Madison was an outspoken advocate of the Union against the states' rights proponents. States' rights proponents believed that all states in the union had the sovereign right to reject national laws and, if necessary, to withdraw from the union. In his opposition to such claims, Madison courted charges of hypocrisy from those who recalled his important role in the drafting of the Virginia Resolutions of 1798. The former president fended off such charges of hypocrisy whenever he could, and they did not deter him from speaking for the cause of America's perpetual union. Further threats to this union arose with the rise to prominence of South Carolina's John Calhoun and his doctrine of Nullification, whereby a single state could veto any federal law and make the law null and void for the whole nation. Calhoun also defended the right of secession for any state, and many Southerners embraced his ideas. The debate over Nullification petered out, however, after President Andrew Jackson made it clear to the nation how much he desired the union to be preserved at all costs.

Madison became actively involved in Virginia politics as a delegate to the Virginia Constitutional Convention of 1829–1830. As a delegate, he distanced himself from his involvement in the Virginia Resolutions of 1798, defending Federalism and union against the rising tide of sectional, states' rights sentiment which affected many members of the state assembly.

The former president was active in other arenas of Virginia life–chiefly education and the debate over slavery. His old friend Thomas Jefferson spent much of the end of his life founding the University of Virginia, of which he was rector until his death on July four, 1826. Madison helped Jefferson with the university, and took his place as rector. The job was administrative in nature. As far as the slavery debate was concerned, Madison was a founding member of the American Colonization Society, an anti-slavery society. The society promoted the idea that all American slaves should be emancipated from their bondage, and that upon emancipation they should be moved away to a settlement of their own somewhere out west, under the jurisdiction of the national government. The plan was popular among many opponents of slavery who were convinced that the assimilation of freed black slaves with the reigning white society would be virtually impossible.

In his last years, Madison became very frail. By 1831, he was the last of the signers of the U.S. Constitution still alive. Though he was very old and frail at this time, his mind remained very sharp, and he continued to comment on politics and write and dictate numerous letters and notes until the time of his death. That day came on June twenty-eight, 1836, when one of his nieces came to him with his breakfast, and he could not eat the food. She asked him, "What is the matter, Uncle James?" He replied kindly, "Nothing more than a change of mind, my dear," dying just a moment later. Before he died, he wrote a message to the nation, "Advice to my Country", sealed it, and ordered it to be opened after he had gone. The message read, "The advice nearest to my heart and deepest in my convictions is, that the Union of the States be cherished and perpetuated."