The baroclinic-barotropic mode splitting technique is commonly employed in numerical solutions of the primitive equations for ocean modeling to deal with the multiple time scales of ocean dynamics. In this paper, a second-order implicit-explicit (IMEX) scheme is proposed to advance the baroclinic-barotropic split system. Specifically, the baroclinic mode and the layer thickness of fluid are evolved explicitly via the second-order strong stability preserving Runge-Kutta scheme, while the barotropic mode is advanced implicitly using the linearized Crank-Nicolson scheme. At each time step, the baroclinic velocity is first computed using an intermediate barotropic velocity. The barotropic velocity is then corrected by re-advancing the barotropic mode with an improved barotropic forcing. Finally, the layer thickness is updated by coupling the baroclinic and barotropic velocities together. In addition, numerical inconsistencies on the discretized sea surface height caused by the mode splitting are alleviated via a reconciliation process with carefully calculated flux deficits. Temporal truncation error is also analyzed to validate the second-order accuracy of the scheme. Finally, two benchmark tests from the MPAS-Ocean platform are conducted to numerically demonstrate the performance of the proposed IMEX scheme.