This makes more sense to have no displacement if baking non-normalized displacement map or have mid-level displacement (which is likely be close to no displacement as well) when baking normalized displacement. In any way it's better than using default black color.