The issue was that sounds were always faded from 0 volume when they started and depending on the currently used buffer size, the fading took longer or shorter. The solution stores whether the sound has ever been played back and consequently does not fade when starting to play back.