Due to the potential decrease of the computation time for problems with fractional order derivatives, and due to the extension of the range of applicable solvers for a given problem, the approximation of fractional derivatives (e.g., using Oustaloup’s method) is an important and frequently discussed topic. A significant problem that can occur (although one that is not discussed very often) when approximations are applied concerns the potential numerical issues that can arise when applying these approximations in a transient analysis, e.g., as shown in the paper, in a time-stepping solver. This paper examines five different methods for approximating the fractional derivative (Oustaloup’s method, a refined Oustaloup method, a modified continued fraction expansion method, Matsuda’s method and Charef’s method). An important preliminary step in the analysis was the establishment of the general form of FDAE (fractional differential-algebraic equations), the general form of the applied approximations and the general form of the DAE (differential-algebraic equations) resulting from the approximation. This allowed for a precise description of the equations according to which the transformation from the FDAE to the approximating DAE takes place. Three problems derived from studies in electrical engineering have been introduced into the analysis (a linear circuit problem, a circuit problem with nonlinear fractional elements, and a problem featuring nonlinearities and pregenerated noise). For these problems, the conditions in which numerical issues appear have been studied in detail. They are the main motivation of this study as these issues are so significant that they often result in zero-solutions or solutions tending to infinity (giving an impression as if the system is unstable). Modifications and alternatives of the transformation into the DAE, that aim at the mitigation of these numerical errors, are mentioned later. The final result is very satisfactory, where the algorithm for the transformation of the FDAE to the approximating DAE practically eliminates the most important barriers (which was the impossibility of using approximations above a certain order due to the mentioned numerical issues). The study presented in this paper is motivated by problems in electrical engineering but due to its generality it is also applicable in other fields where fractional calculus is used.