The computation of the radiative transfer equation is expensive mainly due to two stiff terms: the transport term and the collision operator. The stiffness in the former comes from the fact that particles (such as photons) travel at the speed of light, while that in the latter is due to the strong scattering in the optically thick region. We study the fully implicit scheme for this equation to account for the stiffness. The main challenge in the implicit treatment is the coupling between the spacial and angular coordinates that requires the large size of the to-be-inverted matrix, which is also ill-conditioned and not necessarily symmetric. Our main idea is to utilize the spectral structure of the ill-conditioned matrix to construct a pre-conditioner, which, along with an exquisite split of the spatial and angular dependence, significantly improve the condition number and allows a matrix-free treatment. We also design a fast solver to compute this pre-conditioner explicitly in advance. Our method is shown to be efficient in both diffusive and free streaming limit, and the computational cost is comparable to the state-of-the-art method. Various examples including anisotropic scattering and two-dimensional problems are provided to validate the effectiveness of our method.