At age thirty-nine, Polk now stood for reelection as a veteran congressmen, with more than a decade of public service under his belt. He had made a name for himself with his even, fair and polite tactics and his opinion was respected by friends and foes alike.

The 1835 congressional election in Tennessee took on special national significance. Three incumbents–Davy Crockett, John Bell and James K. Polk–all had national aspirations, and with President Jackson's term about to wrap up, it was hard to say what would come the following year during the presidential election. The political winds in Tennessee, Jackson's home state, would have much to say about how the election shaped up nationally. Crockett's stunning defeat was read as a sign of waning anti-Jacksonian feelings and similarly Polk's overwhelming victory appeared to be a strong endorsement of Jackson. Bell won a victory for the anti-Jacksonians by being reelected.

As Congress reassembled, Jackson decided that Polk should be Speaker of the House. When Old Hickory explained that he wanted Polk to be the party's candidate for the post, the party was left with little choice but to comply. The next day, in the opening voting, John Bell lost handily and Polk became the speaker. Jackson's forces had won again. The factional effort to nominate Hugh Lawson White for president in place of Martin Van Buren cooled and Alabama even withdrew its prior nomination for White. Bell, though, remained a threat to Polk and the two men skirmished throughout Polk's time as speaker.

Despite a massive Democratic majority, Polk's session as speaker is remembered as one of the messiest and stormiest times in Congress. Jackson's enemies (and, by extension, Polk's enemies) had rallied together to block every move by the administration. Bell and other congressmen constantly tried to undermine Polk's authority by questioning decisions and raising obscure parliamentary challenges. However, Polk's attention to detail paid off and he knew the rules of debate cold, meaning that despite more parliamentary challenges in his term than had ever been offered in Congress before, he was only overruled by the House once–on an issue where he had ruled in favor of an opponent.

Polk's enemies then tried to upend him using the slavery issue. They presented a petition outlawing slavery in the District of Columbia. Other petitions followed, each disrupting the House and causing impassioned arguments on both sides of the issue. He settled on a controversial way to prevent further disruptions: Any and all petitions relating to abolition were referred to a special committee rigged by Polk to ensure that none of the petitions ever arrived on the floor again. While solving the problem and allowing the House to continue its work, the move angered both pro- and anti-slavery factions.

The slavery issue had been growing in contention for a long time. Abolitionist newspapers rallied supporters in northern cities, and the "underground railroad" helped escaped slaves make it to Canada and freedom. Southern plantation owners argued that slavery was necessary for the production of cotton and they held strong to the decades-old custom. For his part, Polk feared what would happen if the factional divisions were allowed to fester and grow. He saw the preservation of the Union as paramount in his work and since the congressional petitions threatened it, he wanted to rid them from his work.

In the final debate of the 1836 congressional session, the Congress tried to determine what to do with the nearly forty million dollar surplus that had accumulated in the treasury. Henry Clay suggested divvying up the money between the states, but Polk noticed that the wording of the bill left the money as a loan not a gift–making the states responsible to repay the money if Congress so requested. He saw the move as another assault on states' rights and so advocated for a tax cut and a reduction in tariffs. But in an election year, the Congress could not resist the million-dollar gift to each state, and Clay's Distribution Bill passed.

Polk's long hours and endless quarrels as speaker had begun to take a toll on the middle-aged politician. He lost weight, suffered from insomnia, and began to become susceptible to frequent illnesses again–just as he had during his sickly childhood. He and Sarah had moved into a larger suite on Pennsylvania Avenue with several justices of the Supreme Court. But during the summers, they still headed home to Columbia. Sarah noticed an almost immediate turnaround in her husband's health that summer and she wrote that she wished the summer could have lasted longer.

On election day 1836, Martin Van Buren won the presidency–although White carried Jackson's home state of Tennessee, which Jackson viewed as a personal slight. Jackson began to reorganize the home state to bring it back into the Democratic column for subsequent elections. Van Buren assumed the presidency during a trying time. The economic boom had held to massive inflation; Jackson had tried to arrest the rising currency by decreeing that all government purchases should be paid for in gold and silver. The move, though, barely stemmed the tide of land speculation. According to the Distribution Act, the government had to pay the states just under ten million dollars on January 1, 1837–but when the treasury asked for the money from its private banks, the banks admitted they lacked sufficient funds to grant the request. Most of the banks' money had been tied up in loans to finance new towns, new railroads and new building sprees. Hard money quickly vanished from circulation as the land buyers gathered it to buy more land from the government–thus depriving the government of the ability to use the same hard money to pay its bills. Panic began to spread. In February, riots hit New York City. Trade all but stopped nationwide. Unemployment spread. Companies filed for bankruptcy. Van Buren called a special session of Congress, but it did not start until September. By that time, ninety percent of Eastern factories had ceased operation.

The full heat of the economic collapse was felt by the congressional candidates in the election of 1837. Both the Whigs and the Democrats blamed the other for the problems. The Democrats lost many of their seats. Polk won again in Tennessee, but he only managed to eek out a reelection as speaker by three votes. It would be a difficult time ahead.