On the evolution of behavioral complexity in individuals and populations

A wide range of ecological and evolutionary models predict variety in phenotype or behavior when a population is at equilibrium. This heterogeneity can be realized in different ways. For example, it can be realized through a complex population of individuals exhibiting different simple behaviors, or through a simple population of individuals exhibiting complex, varying behaviors. In some theoretical frameworks these different realizations are treated as equivalent, but natural selection distinguishes between these two alternatives in subtle ways. By investigating an increasingly complex series of models, from a simple fluctuating selection model up to a finite population hawk/dove game, we explore the selective pressures which discriminate between pure strategists, mixed at the population level, and individual mixed strategists. Our analysis reveals some important limitations to the ESS framework often employed to investigate the evolution of complex behavior.