The Remains of the Day is told in the first-person narration of an English butler named Stevens. In July 1956, Stevens decides to take a six- day road trip to the West Country of England—a region to the west of Darlington Hall, the house in which Stevens resides and has worked as a butler for thirty-four years. Though the house was previously owned by the now-deceased Lord Darlington, by 1956, it has come under the ownership of Mr. Farraday, an American gentleman. Stevens likes Mr. Farraday, but fails to interact well with him socially: Stevens is a circumspect, serious person and is not comfortable joking around in the manner Mr. Farraday prefers. Stevens terms this skill of casual conversation "bantering"; several times throughout the novel Stevens proclaims his desire to improve his bantering skill so that he can better please his current employer.

The purpose of Stevens's road trip is to visit Miss Kenton, the former housekeeper of Darlington Hall who left twenty years earlier to get married. Stevens has received a letter from Miss Kenton, and believes that her letter hints that her marriage is failing and that she might like to return to her post as housekeeper. Ever since World War II has ended, it has been difficult to find enough people to staff large manor houses such as Darlington Hall.

Much of the narrative is comprised of Stevens's memories of his work as a butler during and just after World War II. He describes the large, elaborate dinner parties and elegant, prominent personages who come to dine and stay at Darlington Hall in those times. It is gradually revealed—largely through other characters' interactions with Stevens, rather than his own admissions—that Lord Darlington, due to his mistaken impression of the German agenda prior to World War II, sympathized with the Nazis. Darlington even arranged and hosted dinner parties between the German and British heads of state to help both sides come to a peaceful understanding. Stevens always maintains that Lord Darlington was a perfect gentleman, and that it is a shame his reputation has been soiled simply because he misunderstood the Nazis' true aims.

During the trip Stevens also recounts stories of his contemporaries—butlers in other houses with whom he struck up friendships. Stevens's most notable relationship by far, however, is his long-term working relationship with Miss Kenton. Though Stevens never says so outright, it appears that he harbors repressed romantic feelings for Miss Kenton. Despite the fact that the two frequently disagree over various household affairs when they work together, the disagreements are childish in nature and mainly serve to illustrate the fact that the two care for each other. At the end of the novel, Miss Kenton admits to Stevens that her life may have turned out better if she had married him. After hearing these words, Stevens is extremely upset. However, he does not tell Miss Kenton—whose married name is Mrs. Benn—how he feels. Stevens and Miss Kenton part, and Stevens returns to Darlington Hall, his only new resolve being to perfect the art of bantering to please his new employer.

As Salman Rushdie comments, The Remains of the Day is "a story both beautiful and cruel." It is a story primarily about regret: throughout his life, Stevens puts his absolute trust and devotion in a man who makes drastic mistakes. In the totality of his professional commitment, Stevens fails to pursue the one woman with whom he could have had a fulfilling and loving relationship. His prim mask of formality cuts him off from intimacy, companionship, and understanding.