Herzog has a narrative plot, but most of its important action takes place in the mind of Moses Herzog, its protagonist. Moses is a middle-aged college professor living temporarily in his country home in the Berkshires. Moses has made a habit of writing letters, which he never sends, to family, friends, acquaintances, scholars, writers, and the dead. These letters make up much of the novel.

Moses decides to visit his friends at Martha's Vineyard, mostly because he wants to escape his lover, Ramona. Almost immediately up arriving at the Vineyard, however, Moses decides to return to New York, where he writes letters compulsively. Moses spends the next night with Ramona. The following morning, he decides he must act somehow, and determines to fight his ex-wife, Madeleine, for custody of their daughter, June. June's babysitter wrote a letter accusing Valentine Gersbach, Moses' best friend and Madeleine's lover, of treating June badly. According to the babysitter, Valentine locked the child in the car while he and Madeleine argued.

Moses calls his lawyer, Simkin, and arranges a meeting in the courthouse. While he waits for Simkin, Moses witnesses several cases ranging from prostitution to a mother's murder of her daughter. The next day, Moses impulsively flies to Chicago to visit his daughter. In Chicago, he goes to his childhood home, where his widowed stepmother, Tante Taube, still lives. There, he goes to the desk of his late father, Jonah Herzog, and takes Jonah's old gun and some Russian rubles. Moses considers murdering his ex-wife and her lover with the gun. After spying on Madeleine and Valentine through a window of their house, however, Moses realizes that he will not kill them. He goes to Phoebe Gersbach, Valentine's wife, and asks for her help in gaining custody of June, but Phoebe refuses to help him.

Moses spends the night with his good friend Lucas Asphalter. Through Lucas, he arranges to meet with his daughter. The next morning he takes June to the aquarium. As they leave the aquarium, Moses gets into a car accident. June is not hurt, but Moses is knocked unconscious. He wakes up to find himself at the feet of the police. He is charged with possession of a weapon and taken to jail. His brother Will bails him out. Will is worried for Moses. Later, the brothers meet in the Berkshires. Will suggests that Moses allow himself to be taken to a hospital for observed rest under the care of a psychiatrist. Moses had once had the same idea himself, but now he rejects it and remains in the Berkshires. He arranges a night with Ramona, who comes to visit him. By the end of the novel, Moses has found contentment in his country home and the pleasant weather. He feels he does not need to write any more letters.

Other important facts arise, not in the course of the main plot, but through Moses' letters and memory. We learn that before Moses married Madeleine, he was married to a woman named Daisy, with whom he had had a son named Marco. We also learn that Moses was raised in a Jewish immigrant family in LaRoux, Canada, and that his father Johan failed in many business ventures and eventually became a bootlegger. Moses also recounts tales of his brothers and sister (Will, Shura, and Helen). Moses constantly mentions his efforts as a writer. He published one book entitled Romanticism and Christianity, which gained critical acclaim as time passed, despite an initially chilly critical response. His never completed the intended second volume of the book.