Completely skip the work of interpolating domains for single values
for many to and from combinations. Similar to 535f50e5a6,
but slightly more complex because of the possibility of loose elements
on some mesh domains.
From D16054, with added comments.