The Energy Conserving Semi-Implicit Method (ECSIM) introduced by Lapenta (2017) has many advantageous properties compared to the classical semi-implicit and explicit PIC methods. Most importantly, energy conservation eliminates the growth of the finite grid instability. We have implemented ECSIM in a different and more efficient manner than the original approach. More importantly, we have addressed two major shortcomings of the original ECSIM algorithm: there is no mechanism to enforce Gauss's law and there is no mechanism to reduce the numerical oscillations of the electric field. A classical approach to satisfy Gauss's law is to modify the electric field and its divergence using either an elliptic or a parabolic/hyperbolic correction based on the Generalized Lagrange Multiplier method. This correction, however, violates the energy conservation property, and the oscillations related to the finite grid instability reappear in the modified ECSIM scheme. We invented a new alternative approach: the particle positions are modified instead of the electric field in the correction step. Displacing the particles slightly does not change the energy conservation property, while it can satisfy Gauss's law by changing the charge density. We found that the new Gauss's Law satisfying Energy Conserving Semi-Implicit Method (GL-ECSIM) produces superior results compared to the original ECSIM algorithm. In some simulations, however, there are still some numerical oscillations present in the electric field. We attribute this to the simple finite difference discretization of the energy conserving implicit electric field solver. We modified the spatial discretization of the field solver to reduce these oscillations while only slightly violating the energy conservation properties. We demonstrate the improved quality of the GL-ECSIM method with several tests.