In this paper, we present a novel adaptive sampling strategy for enhancing the performance of physics-informed neural networks (PINNs) in addressing inverse problems with low regularity and high dimensionality. The framework is based on failure-informed PINNs, which was recently developed in [Gao et al., SIAM J. Sci. Comput., 45(4), 2023]. Specifically, we employ a truncated Gaussian mixture model to estimate the failure probability; this model additionally serves as an error indicator in our adaptive strategy. New samples for further computation are also produced using the truncated Gaussian mixture model. To describe the new framework, we consider two important classes of inverse problems: the inverse conductivity problem in electrical impedance tomography and the inverse source problem in a parabolic system. The effectiveness of our method is demonstrated through a series of numerical examples.