Magnetic Josephson junctions are the preferred candidate devices for designing fast and scalable cryogenic memory elements. This is especially the case for rapid single-flux quantum-based superconducting electronics, where the speed mismatch between logic and memory elements have remained a long-standing challenge. In this Letter, we demonstrate a simple tri-layer Josephson memory device using ferromagnetic insulating (FI) GdN-based S/FI/S vertical mesa-type junctions, with reliable nonvolatile memory operation without the need of a shunt resistor at 4.2 K. The characteristic frequency of our devices is approximately 90 GHz, corresponding to an IcRn product of 177 μV. We demonstrate a thorough study of the parameter spaces required for designing these devices and identify the scope for future improvements that can lead to further miniaturization and higher operating speed of these devices.