Remember when Joseph was thinking about "quietly divorcing" Mary? He pretty much had two choices: a) accuse her of adultery, or b) divorce her and let people think the baby was his while still leaving Mary free to go be with The Other Guy.
If Joseph accused Mary of adultery, she would be stoned. But not just Mary. If she were killed, then her baby would also die. Joseph's compassion (even prior to the message from God) was not only for this woman, but he also was showing compassion to that baby in sparing His life.