In modern optics, particular interest is devoted to the phase singularities that yield complicated and twisted phase structures by photons carrying optical angular momentum. In this paper, the traditional M-line method is applied to a vortex beam (VB) by a symmetric metal cladding waveguide chip, which can host numerous oscillating guided modes via free space coupling. These ultrahigh-order modes (UOMs) result in high angular resolution due to the high finesse of the resonant chip. Experiments show that the reflected pattern of a VB can be divided into a series of inner and outer rings, whilst both of them are highly distorted by the M-lines due to the UOMs’ leakage. Taking the distribution of the energy flux into account, a simple ray-optics-based model is proposed to simulate the reflected pattern by calculating the local incident angle over the cross section of the beam. The theoretical simulations fit well with the experimental results, and the proposed scheme may enable new applications in imaging and sensing of complicated phase structures.