I realize that I did't thank you for you advice. So thanks.
Running the calculation for a regular DFT guess function gave me better results but it only reduced the distribution of energies. I will try to run two successive SO-DFT calculation but it's getting a bit complicated for a single point energy.