Although now happily reunited with his family, Grant faced a foreboding future in 1854. He had given up the military, his only career, and had repeatedly proved himself a failure at business. Grant's father offered him a job in the family tannery business in Galena, Illinois–as long as Grant lived with his brother while Julia and the children remained in Kentucky at the Grant family home. Grant balked. His two years on the West coast had taught him that he needed Julia's calming influence around him, to keep his drinking and smoking from becoming a problem.
Grant and his family returned to Julia's family in St. Louis, where Grant got the use of a brother-in-law's farm and land. Grant set about building a house, which he named Hardscrabble. Hardscrabble was the only thing Grant ever built, and it stood as a giant disappointment to his wife. She much preferred a simple frame house to the masculine masterpiece Grant created of rough-hewn logs and giant timbers. As Julia tended house, raised the children, and tended to the barnyard, Grant raised potatoes and tried to develop a taste for farming.
Despite initial promise, the farm began to suffer. While the Grants tried to display a picture of frontier independence, Grant and his wife had trouble escaping the fact that they were living on her father's land, working his fields, and tending crops he controlled. Grant also proved that in addition to being a failure as a businessman, he was a failure of a farmer. Though he made money on potatoes in 1855, the following year he suffered while building Hardscrabble. By that winter, he began trying to sell firewood on the side, which soon became a larger and larger part of the family income, as the farm returns continued to dwindle. Grant would lug the wood into St. Louis and sell it on street corners, allowing him to seek out old Army friends in the hope of getting a big break. In February 1857, he unsuccessfully asked his father for five hundred dollars. His ailing farm and a general depression in 1857 forced Grant to pawn his gold pocket watch at Christmas to afford presents for his family.
One untouchable aspect of the estate, however, was the Dent's ownership of twelve family slaves. Most of the slaves belonged to the family, though Grant owned one–William Jones–and Julia owned four. By all accounts the slaves were treated extremely well–one even said that Grant was the best master he had ever had–and Julia at least considered them valuable family servants. She thought it in bad taste to sell such valuable members of the household. In fact, Grant drew criticism from other slaveholders in the area for paying his men too generously. When Grant left for St. Louis two years later, in 1859, he freed Jones–no small act in light of the financial troubles that Grant faced and the fact that Jones would have fetched over $1,500 at that point.
On February 6, 1858, Julia gave birth to the couple's final child, Jesse Grant–named, of course, for the boy's grandfather, Grant's father. That same year, Grant finally abandoned the farming idea after he took seriously ill, likely with the same form of tuberculosis that was slowly killing his brother. Instead, Grant became a rent collector for the business Julia's cousin ran in St. Louis. Unfortunately, if ever there was a man not cut out to be a rent collector, it was Grant. By summer of 1859, he longed to quit the firm. His friends decided to run him for county engineer, but he lost the job due to political machinations. Grant had supported James Buchanan in the 1856 election at the urging of his fiercely Democratic in-laws. Thus, in the election for county engineer, the two Democrats on the selecting panel supported him while the three Free-Soilers opposed him. In December 1859, the same night Grant lost the election to be engineer, Julia and her husband agreed that their lives were not working out in St. Louis. Grant reluctantly agreed to return to his father in Kentucky and ask for help.
Grant received only a tepid homecoming in Covington, Kentucky–his family's new home. Nonetheless, his father agreed to let him enter the family's tanning business. Julia and the four kids packed up and moved to Galena, Illinois, as Grant began his job as a clerk in the family store. For the rest of Grant's life, Galena would stand as the closest thing he had to a home, and the friends he made there would travel with him to the battlefields of the Civil War and eventually to the White House itself. Grant remained sad, restless, and frustrated in his time at the store; people from Galena remember the "empty" expression he wore almost every day he lived there. He did not even have a horse to call his own.