This paper presents a fourth-order Cartesian grid based boundary integral method (BIM) for heterogeneous interface problems in two and three dimensional space, where the problem interfaces are irregular and can be explicitly given by parametric curves or implicitly defined by level set functions. The method reformulates the governing equation with interface conditions into boundary integral equations (BIEs) and reinterprets the involved integrals as solutions to some simple interface problems in an extended regular region. Solution of the simple equivalent interface problems for integral evaluation relies on a fourth-order finite difference method with an FFT-based fast elliptic solver. The structure of the coefficient matrix is preserved even with the existence of the interface. In the whole calculation process, analytical expressions of Green’s functions are never determined, formulated or computed. This is the novelty of the proposed kernel-free boundary integral (KFBI) method. Numerical experiments in both two and three dimensions are shown to demonstrate the algorithm efficiency and solution accuracy even for problems with a large diffusion coefficient ratio.