In this paper, we extend a diagonalization-based parallel-in-time (PinT) algorithm to the viscoelastic equation. The central difference method is used for spatial discretization, while for temporal discretization, we use the Crank-Nicolson scheme. Then an all-at-once system collecting all the solutions at each time level is formed and solved using a fixed point iteration preconditioned by an $α$-circulant matrix in parallel. Via a rigorous analysis, we find that the spectral radius of the iteration matrix is uniformly bounded by $α/(1 − α),$ independent of the model parameters (the damping coefficient $\varepsilon$ and the wave velocity $\sqrt{\gamma}$) and the discretization parameters (the time step $\tau$ and the spatial mesh size $h$). Unlike the classical wave equation with Dirichlet boundary condition where the upper bound $α/(1 − α)$ is very sharp, we find that the occurrence of the damping term $−\varepsilon∆y_t,$ as well as the large final time $T,$ leads to even faster convergence of the algorithm, especially when $α$ is not very small. We illustrate our theoretical findings with several numerical examples.