In animal conflicts, such as territory or food source defense, the success of one animal's chosen strategy depends on the strategies of the other individuals involved. Because of this, animal conflicts can be modeled after simple games, such as "Rock, Paper, Scissors," in which the outcome of the game depends on which strategies each player chooses. The benefit or detriment to each player is measured quantitatively, usually as the number of offspring (fitness), or net energy gain in calories.
In a two player game where both players have a choice of two strategies, we can calculate which is the best strategy, or the optimal frequency with which two play both strategies. In the four variations of the game pictured below, we have player 1 and player 2, each choosing between strategies A and B. By convention, the payoffs are shown for the player on the left side of the matrix. If both players choose strategy A, the payoff is Q. If both players choose strategy B, the payoff is T. If player 1 chooses strategy A and his opponent chooses strategy B, player 1's payoff is R. If player 1 chooses strategy B and his opponent chooses strategy A, player 1's payoff is S. Notice that in this last circumstance, player 2's payoff would be R, not S. Also note that though the name's for the payoffs remain the same, the value of those payoffs differe across the four variations.
In the situation described on the left side of , payoff Q is better than payoff S, and payoff R is better than payoff T. In this case, player 1 should always choose strategy A, because no matter which strategy his opponent chooses, strategy A will provide a better payoff. This situation is a pure evolutionarily stable strategy (ESS). All players will choose strategy A, and evolution will select for only those players who do choose strategy A.
In the situation described by the right side of the figure, because S>Q and T>R, player 1 should always choose strategy B for the same reasons listed above. This is also a pure ESS.
In part c, the situation becomes a bit more difficult. Since Q>S and T>R, there is no clear strategy that will always be best. Player 1's success will depend on which strategy his opponent chooses. This is known as an unstable mixed ESS. The best strategy for player 1 is to always do what his opponent does. If both players choose strategy A, player 1 gets the better payoff of Q. If they both choose strategy B, player 1 receives the better payoff of T. Evolution will result in the frequency of played strategies moving toward what the majority of the population was already doing.
In part d, where S>Q and R>T, the best strategy for player 1 is to always do the opposite of what the opponent does. This is a stable mixed ESS. It is this last situation in which we are most interested, because the other three will always eventually result in one strategy being played constantly. A stable mixed ESS is the only situation in which two strategies can be maintained. The two strategies in a game can be played by different individuals who always play the same strategy (a polymorphic population), or both strategies may be played by any individual in a population (polymorphic individuals).