Mutant invasion in stochastic individual-based models with cyclic resident dynamics

When a resident community oscillates, can a rare mutant invade, and with which probability? We study a multitype birth-and-death process with interactions, in which population sizes are of the order of a carrying capacity $K$, and whose large-population limit, as $K\to\infty$, has a hyperbolically stable limit cycle for the $D-1$ resident types. We introduce a single individual of a new type. We prove that, as long as the mutant population is smaller than $\varepsilon K$ for a small $\varepsilon>0$, the resident population sizes divided by $K$ stay close to a moving point of the cycle, whose asymptotic phase moves by at most $O(\varepsilon)$ per period. We deduce that, for large $K$, the mutant dies out quickly if its growth rate averaged over one period, $f(\bar{\mathbf n})$, is negative, and that it invades with positive probability, in a time close to $\ln K/f(\bar{\mathbf n})$, if $f(\bar{\mathbf n})$ is positive. This justifies a criterion used heuristically in ecology for decades. The invasion probability, unlike $f(\bar{\mathbf n})$, depends on the phase of the cycle at which the mutant arises. We give it explicitly through Kendall's formula, and show that its average over the cycle, weighted by the birth rate of the mutant, equals $f(\bar{\mathbf n})$. In predator-prey, competitive and mutualistic examples, the invasion probability varies by a factor between $2$ and $600$ along the cycle, and individual-based simulations agree quantitatively with the theory. After invasion, the oscillations may vanish, persist on a cycle of higher dimension, or give way to chaotic coexistence.