In this paper, we first consider the numerical method that Lin and Xu proposed and analyzed in [Finite difference/spectral approximations for the time-fractional diffusion equation, JCP 2007] for the time-fractional diffusion equation. It is a method based on the combination of a finite different scheme in time and spectral method in space. The numerical analysis carried out in that paper showed that the scheme is of $(2-\alpha)$-order convergence in time and spectral accuracy in space for smooth solutions, where $\alpha$ is the time-fractional derivative order. The main purpose of this paper consists in refining the analysis and providing a sharper estimate for both time and space errors. More precisely, we improve the error estimates by giving a more accurate coefficient in the time error term and removing the factor in the space error term, which grows with decreasing time step. Then the theoretical results are validated by a number of numerical tests.