We consider a macroscopic quantum system such as a qubit, interacting with a bath of fermions as in the Fr\"ohlich polaron model. The interaction Hamiltonian is thus linear in the macroscopic system variable, and bilinear in the fermions... Using the recently developed extension of Feynman-Vernon theory to non-harmonic baths we evaluate quadratic and the quartic terms in the influence action. We find that for this model the quartic term vanish by symmetry arguments. Although the influence of the bath on the system is of the same form as from bosonic harmonic oscillators up to effects to sixth order in the system-bath interaction, the temperature dependence is nevertheless rather different, unless rather contrived models are considered. (read more)