The story begins with the narrator recalling a time long ago, from when he was a child. He sits in his childhood home and evokes the fateful day when an ibis landed in a tree in the yard. He then begins a tale from when he was six years old. In it, a younger brother is born in his household. The brother, named William Armstrong, is born physically disabled, and his mother, father, and doctor all expect him to die. The only one in the family who believes the boy will live is Aunt Nicey. Aunt Nicey notes that the boy was born in a “caul,” which is a bit of membrane around the head of the newborn. She claims this substance to be made from “Jesus’ nightgown,” and that the boy will live. The father, however, goes so far as to commission a small casket from the carpenter to prepare for the baby’s imminent death. 

But the boy survives his early years. The older brother (the narrator of the story) wants a younger brother with whom he can run around and play. But his mother warns him that his brother might not be “all there.” She believes the child will be cognitively disabled. But the boy turns out to be mentally fit, and in time he learns to speak and communicate with his family. The older brother nicknames the child “Doodle,” since the boy crawls backward as a baby, in the same manner as a doodlebug, the larva of a certain type of insect. Despite his survival and ability to crawl, Doodle still cannot walk and likely never will.

The older brother, referred to only as “Brother” by Doodle, is tasked with pulling Doodle around the house and property in a wagon crafted specially for the disabled boy. This burden annoys Brother, and he races Doodle around, attempting to frighten him. Sometimes he turns the cart over, tossing Doodle out onto the ground. Doodle is physically very fragile, but his brother ignores his frailty. Eventually Doodle’s sweet personality wins over his older brother. “Brother” takes Doodle to his favorite place, Old Woman Swamp, and they spend long days at the swamp together, making necklaces and crowns from wildflowers.

One day, Brother shows Doodle the small casket that the family purchased when he was born. The casket is in the barn loft, is covered in rat poison, and has become a nesting place for a family of owls. Brother claims that the coffin belongs to Doodle, but Doodle rejects the idea. In an act of cruelty, Brother makes Doodle touch the casket, which scares the young boy and causes an owl to fly from the nest right in front of them. Doodle is so frightened he cannot stop pleading for Brother not to leave him behind in the barn loft, even after they have descended the ladder and returned out into the sunshine.

Brother and Doodle spend many hours in Old Woman Swamp, playing underneath a tall pine tree. When Doodle is five years old, Brother decides to teach Doodle to walk, in the hopes of having a healthy little brother who won’t have to be carted everywhere he goes. This seems impossible to Doodle, but Brother convinces him to try anyway. They try to stand Doodle up with little success and Doodle often despairs. Brother describes a scene for Doodle in which they are both old men, and Doodle is still stuck in the cart. The idea of this sad potential future convinces Doodle to keep trying. 

The brothers finally get Doodle walking after many weeks of practice in Old Woman Swamp. The rest of the family does not know about this project, and on Doodle’s sixth birthday, the brothers surprise the family with a demonstration. Doodle walks from his cart to his chair at the dining table. Everyone cries and hugs Doodle. Brother swells with pride and begins to believe that anything is possible for Doodle.

Brother then embarks on a rigorous program to teach Doodle to swim, row a boat, run, and climb on the vines of the swamp, just like other boys can. Their progress is interrupted by a drought followed by a massive hurricane. The hurricane destroys trees in the yard as well as the crops in the fields. Doodle and Brother witness their father cursing the weather for destroying his crops. The incident does not impede them much, however, and the boys continue to work on Doodle’s physical abilities, despite his obvious limitations. They make very little progress toward Brother’s goals. Brother hopes to get Doodle swimming and climbing by the end of the summer, when his younger brother will begin to attend school.

One Saturday at lunch, at the close of summer, the family hears a strange sound from outside. It is the cry of a scarlet ibis, a nonnative and unfamiliar red bird with long legs and drooping wings. The ibis sits in a tree in the yard and is obviously sick. As the family watches, the bird falls from the tree and dies on the ground in front of them. Doodle is very upset, and he insists on burying the delicate red bird. The boys’ father looks up the bird in a book and says that it was probably blown far north out of its natural habitat by a storm. The bird did not have much chance of surviving this far out of its range. The family watches through a window as Doodle struggles to bury the bird in the flower garden. Doodle is very serious about the burial, and somber after he accomplishes the task. Aunt Nicey states that dead birds are bad luck.

That afternoon, Brother brings Doodle down to the creek for swimming, but Doodle is tired and they end up rowing a boat instead. The boys are hoping to complete Doodle’s training before the school year begins, but he is obviously behind schedule. A storm approaches as Doodle struggles with the oars. The boys finally make it back to the landing, but the storm is right behind them. A tree in their path is shattered to pieces by a bolt of lightning, and the rain begins to beat down on them. Doodle is frightened and tries to keep up, but Brother is only aggravated at Doodle’s lack of ability. Tired of Doodle’s limitations, he begins to outrun his brother, despite Doodle’s pleading cries not to leave him behind. Brother leaves Doodle behind by running as fast as he can. Then he turns to wait, but Doodle does not come. He goes back along the path and finds Doodle crouched under a tree. As Brother tries to rouse him, he sees that Doodle has blood running from his mouth and will not answer him. Doodle’s efforts to catch up to Brother have killed him, and Brother cries over Doodle’s broken body in the pouring rain.