Stevens next writes from a seaside town in Weymouth, where he goes after he visits Miss Kenton. He is sitting on a pier watching all of the colored lights come on in the evening. He arrived at Weymouth the afternoon of the day before, and has stayed another day so that he might spend a little leisure time away from driving.
Miss Kenton actually surprises Stevens by coming to meet him at the hotel where he was staying in Little Compton. She has aged, but very gracefully, and he is extremely pleased to see her again. It strikes Stevens that Miss Kenton seems to have lost the spark that used to make her so lively; when her face is in repose, he thinks that its expression is sad.
Stevens and Miss Kenton fill each other in on their lives over the last twenty years. Although Stevens had thought that Miss Kenton's letter indicated that she had left her husband, she tells him she is in fact moving back in with her husband. Miss Kenton urges Stevens, on his return trip, to visit her daughter Catherine, who is expecting a child in the fall. Stevens tells Miss Kenton what Darlington Hall is like now with the reduced staff and Mr. Farraday as the employer. Stevens tells Miss Kenton the sad news that Reginald Cardinal was killed in World War II, in Belgium. Miss Kenton inquires about the unsuccessful libel action that Lord Darlington took against a newspaper that made claims that he was a Nazi sympathizer and a traitor to England. Stevens says that Lord Darlington lost the libel suit, and after his good name was ruined, he practically became an invalid.
The meeting goes on for two hours before Miss Kenton says she must return home. Stevens drives her to a bus stop a little way outside the village. While they are waiting at the bus station, Stevens asks Miss Kenton a question that he says has been troubling him for some time: he asks if she is being mistreated in some way, as her letters often seem unhappy. Miss Kenton says that her husband does not mistreat her in any way at all. Stevens says he does not understand why, then, she is unhappy. She tells him that for a long time, she did not love her husband, but that after having a daughter and going through the war together, she has grown to love him. However, there are times when she thinks she has made a great mistake with her life. She even says, "For instance, I get to thinking about a life I may have had with you, Mr. Stevens." But then she says that it is of no use to dwell on what might have been.
For the first time in the novel, Stevens appears to realize how much he loves Miss Kenton. Upon hearing her words about the possibility of a life they might have had together, he says that his "heart is breaking." He does not speak for a moment, but when he does, he only says that Miss Kenton is right: one cannot dwell on the past. He says that she must do all she can to ensure many happy years ahead with her husband and her grandchildren. Then the bus comes, and Miss Kenton leaves. Stevens sees that her eyes have filled with tears.
A man comes up and sits next to Stevens on the bench on the pier, and begins talking to him. During the conversation, the man reveals that he was once a butler at a small house. Stevens says that he is the head butler at Darlington Hall, and the man is very impressed. Stevens tells the man about how Darlington Hall was in the old days. Then Stevens tells the man he gave what he had to give to Lord Darlington; even though he is trying hard to please his new employer, he feels that he is making more and more errors. The man next to him offers Stevens a handkerchief—our only clue that Stevens is crying.
Stevens says that Lord Darlington at least made his own mistakes, but says that he himself cannot even claim that, because he trusted Lord Darlington so completely. Stevens does not think that there is much dignity in such an action—not even being able to say he has made his own mistakes. The man seated next to Stevens tells him not to look back so much because it will only make him unhappier. Then he says that the evening is the best part of the day for most folks. Stevens agrees, and apologizes for crying. He decides to make the best of "what remains of my day." The first thing he will work on upon his return to Darlington Hall is bantering: he hopes, when Mr. Farraday comes back, that he will be able "to pleasantly surprise him."
The final section of The Remains of the Day is incredibly sad, as Stevens never tells Miss Kenton that he loves her because he feels that it is too late. Listening to her talk about her husband and her daughter has made him realize how much time has passed, and how much opportunity lost. Stevens does ask Miss Kenton if she has ever thought of working again; she replies that she has, but now that she is going to have a grandchild, she wants to be nearby. Though Miss Kenton's words crush Stevens's last hope of her ever returning to Darlington Hall, he, of course, never even says to her that he was hoping she would do so. Stevens's last and largest hope has now been shattered, compounding the other losses and regrets that seem to have characterized much of his life.
The meeting is the climax of the novel. Even though Stevens relates his meeting with Miss Kenton at the end of the story, he tells it after the fact, a day afterward. The intervening falling action—what would constitute Day Five—is not presented in the narrative; we are left to imagine Stevens wandering around on the day after his meeting with Miss Kenton, having ultimately failed in both expressing his feelings and attaining any deep intimacy with another person.
It is clear that Miss Kenton has married the wrong man. Stevens notes that her passionate nature seems to have dissipated, and that her expression often seems to be one of sadness. When Miss Kenton voices regret at not spending her life with Stevens, it makes him realize how much better it would have been for both of them if they had been the ones to marry. It is at this point that Stevens tells us that his heart is breaking—an astounding revelation from a character who gives virtually no evidence of any emotion at all during the course of the novel.
Stevens finally breaks down during the evening when he is sitting on the pier, reaching at last the realization that he has deluded himself throughout his entire life. He finally questions aloud the use of being loyal to someone who used bad judgment, and finally sees how it may be foolish to completely accept someone else's judgment in place of one's own. Indeed, Stevens suddenly realizes that such blind loyalty may not be very dignified after all. It is in this part of the novel that Stevens's role—his mask as a perfect, poised butler —crumbles, and his real self—an sad, disillusioned man—takes over the story.
The man next to Stevens cheers him up by telling him not to look back or focus on regret and lost opportunity so much. Finally, Stevens comforts himself with the thought that there is dignity in the fact that he willingly sacrificed other things in life in order to devote himself to Lord Darlington. Small as this comfort may be, it seems enough for Stevens, who then tells us about his plans to improve his skills at bantering in an attempt to better serve his new employer. It is not clear, in the end, the extent to which Stevens realizes he has deceived himself. After all, as he never has known anything outside of his own limited existence, it may be difficult, if not impossible, for him to fully appreciate what he has missed, just as someone who is born blind would never miss seeing color. Indeed, despite its slightly optimistic ending, The Remains of the Day remains, on the whole, a tragic story of regret and missed opportunity.