In this paper, a fourth order augmented matched interface and boundary (AMIB) method is proposed for solving a three-dimensional elliptic interface problem which involves a smooth material interface inside a cuboid domain. On the boundary of the cuboid domain, the fourth order AMIB method can handle different types of boundary conditions, including Dirichlet, Neumann, Robin and their mix combinations in fictitious value generation. Moreover, zero-padding solutions are introduced so that the fast Fourier transform (FFT) algorithm is still valid near the boundary. In dealing with the interior interface, a fourth order ray-casting matched interface and boundary (MIB) scheme is proposed, which enforces the jump conditions along the normal direction for calculating fictitious values. Comparing with the existing MIB scheme, the ray-casting scheme naturally bypasses the corner issue and becomes more robust in handling complex geometry. Based on fictitious values generated near interface and boundary, the fourth order central difference can be corrected at various irregular points including corner points, by introducing Cartesian derivative jumps as auxiliary variables. This gives rise to an enlarged linear system, which can be efficiently solved by the Schur complement procedure together with the FFT inversion of the discrete Laplacian. Extensive numerical experiments have been carried to test the proposed ray-casting AMIB method for numerical accuracy, efficiency, and robustness in corner treatment. The numerical results demonstrate that the ray-casting AMIB scheme not only maintains a fourth order of accuracy in treating various interfaces and boundaries for both solutions and solution gradients, but also attains an overall efficiency on the order of O(n3logn) for a n×n×n uniform grid.