The trick is to wait before you make new buildings...you get charged upkeep per archer/wizard/craftsman and if you dont have the dough, you lose them.
What I'm doing is waiting for ages before training units, I now have a few archers and a hundered craftsmen. I can just let it run in the background, if they make it to the walls, they aint gonna tear the castle down as fast as the crafties can rebuild it!