We consider the S-duality transformation of gauge invariant operators and states in $\mathcal{N}$ $=4$ SYM theory. The transformation is realized through an operator $ S $ which is the $ SL(2,Z) $ canonical transformation in loop space with the gauge invariant electric and the magnetic flux operators composing the canonical variables... Based on $ S $, S-duals for all of the physical operators and states can be defined. The criterion for the theory to be S-duality invariant is that the superconformal charges and their S-duals differ by a $ U(1)_{Y} $ phase. The verification can be done by checking the S transformation for supersymmetry and special supersymmetry variations of the loop operators. The fact that supercharges preserved by BPS Wilson operators and the S-dual BPS 't Hooft operators differ by a $4d $ chiral rotation could in some sense serve as a proof. (read more)