The semiclassical general formula for the probability of radiation of twisted photons by ultrarelativistic scalar and Dirac particles moving in the electromagnetic field of a general form is derived. This formula is the analog of the Baier-Katkov formula for the probability of radiation of one plane wave photon with the quantum recoil taken into account... The derived formula is used to describe the radiation of twisted photons by charged particles in undulators and laser waves. Thus, the general theory of undulator radiation of twisted photons and radiation of twisted photons in the nonlinear Compton process is developed with account for the quantum recoil. The explicit formulas for the probability to record a twisted photon are obtained in these cases. In particular, we found that the quantum recoil and spin degrees of freedom increase the radiation probability of twisted photons in comparison with the formula for scalar particles without recoil. In the range of applicability of the semiclassical formula, the selection rules for undulator radiation established in the purely classical framework are not violated. The manifestation of the blossoming out rose effect in the nonlinear Compton process in a strong laser wave with circular polarization and in the wiggler radiation is revealed. Several examples are studied: the radiation of MeV twisted photons by $180$ GeV electrons in the wiggler; the radiation of twisted photons by $256$ MeV electrons in strong electromagnetic waves produced by the CO$_2$ and Ti:Sa lasers; and the radiation of MeV twisted photons by $51.1$ MeV electrons in the electromagnetic wave generated by the FEL with photon energy $1$ keV. (read more)