Underground material should affect fertility of farms

Right now, you can dump veritable tonnes of waste into the ground, and as long as you overlay it with a thin strip of dirt you can farm on it just fine.
This feels wrong.
When you build a farm, there should be a one-time (to keep it from affecting performance) calculation of the voxels under it to determine the farm's fertility regeneration rate. It should check 5 levels deep. Dirt increases fertility regen (compost gives a larger increase), rock and the like is neutral and does nothing, waste applies a penalty.
This means that farms cannot be effective planted on waste, but on the plus side being able to layer compost in this way means that in the late game you could prepare land for highly effective farms that need very little fertilizer to stay productive.
Note that this does not prevent you from farming effectively as long as you use enough fertilizer to compensate, and since dirt is 5 levels deep from the start it will also not affect farms planted on virgin soil.

This game does not use 'voxels'. (Unity Game Engine.)
I do get what you mean though.
Currently I found that placing one layer of Dirt over Coal prevents grass from growing.
(Although that might be an issue caused by using a Mod.)
I also think rocks should have some effect, maybe by changing the water table underneath?