We study the stochastic dynamics of evolutionary games, and focus on the so-called 'stochastic slowdown' effect, previously observed in (Altrock et. al, 2010) for simple evolutionary dynamics. Slowdown here refers to the fact that a beneficial mutation may take longer to fixate than a neutral one. More precisely, the fixation time conditioned on the mutant taking over can show a maximum at intermediate selection strength. We show that this phenomenon is present in the prisoner's dilemma, and also discuss counterintuitive slowdown and speedup in coexistence games. In order to establish the microscopic origins of these phenomena, we calculate the average sojourn times. This allows us to identify the transient states which contribute most to the slowdown effect, and enables us to provide an understanding of slowdown in the takeover of a small group of cooperators by defectors: Defection spreads quickly initially, but the final steps to takeover can be delayed significantly. The analysis of coexistence games reveals even more intricate behavior. In small populations, the conditional average fixation time can show multiple extrema as a function of the selection strength, e.g., slowdown, speedup, and slowdown again. We classify two-player games with respect to the possibility to observe non-monotonic behavior of the conditional average fixation time as a function of selection strength.