It's good practice when attending the flag to remove it without regard for where the shot is going.
There's no penalty involved in the situation you outline - if you decided he had hit it too hard and a would benefit from leaving it in (and it hit the flagstick) you would be penalised.
If he was on the green and hit the flagstick then he would be penalised unless you deliberately left it in to get him a penalty...
