I think the US Navy SEALS have a motto which goes something along the lines of "A man is only defeated if he gives up or he dies".
In other words, being truly 'hard', to a great extent, is about mental resilience - the human body can be pushed much further than many people have the mental toughness, the "inner metal" to take it. They will be defeated long before their physical ability is. When you are defeated mentally, the physical defeat will be soon to follow.
If you were to look at any of the world's best special forces units, I'm sure they won't be packed with the biggest, hardest looking guys - but they will be the kind of people who will keep getting back up in a fight, until they win or they die.