For the first time, elastic pion-nucleon scattering is analyzed in the framework of chiral perturbation theory up to fourth order with explicit $\Delta$ degrees of freedom. The analysis is performed within the heavy-baryon expansion as well as in a covariant approach based on an extended on-mass-shell renormalization scheme... The renormalization of low-energy constants in both chiral approaches is discussed in detail and the explicit expressions to cancel both power-counting breaking as well as decoupling breaking terms are given. The low-energy constants from the $2\pi \bar{N}N$ interaction as well as additional constants from the $\Delta$ sector are reliably constrained by fits to experimental data. The traditional $K$-matrix unitarization is employed in the near threshold region, whereas a complex mass approach is used to extend the applicability of the theory up to the $\Delta$ pole region. Additionally, we estimate a theoretical error based on the truncation of the chiral series as employed in recent analyses of nuclear forces as well as pion-nucleon scattering. The obtained results provide a clear evidence that the explicit inclusion of $\Delta$ degrees of freedom is fundamental to describe pion-nucleon physics at threshold. The resulting predictions for the subthreshold and threshold parameters as well as phase shifts are in excellent agreement with the ones determined by the recent Roy-Steiner analysis of pion-nucleon scattering. (read more)