I find they won't work if they haven't slept in a while, why I always build a house with cots first. You can also try having your work crews specialize for particular jobs, helps to level their skills as well.