Unity/Health Decrease from Boosting Productivity

Yes, I know...something to make the game MORE challenging?
Right now, as long as you have the Unity, and the necessary inputs, you can leave a building Boosted forever. You're essentially saying that workers can do double shifts with no drawbacks forever.
My suggestion is to have a timer on buildings that require workers and can be Boosted. After a certain amount of time, there is either a decrease in productivity or a decrease in health as workers become exhausted.
Possible modifications:
1) If there are enough unemployed workers to equal the Boosted building, there is no drawback because fresh workers can swap out with tired ones.
2) This could be tied to Difficulty Level so on Easy mode it doesn't work at all but on the hardest setting can really be a factor.