Not being there for someone doesn't mean that you are bad.
It means that you can't be there because you have set your priorities.
Why must there usually be a misconception on this ruling? Then you will get people talking back about your wrongdoings and forgetting what good you have done in just a split second.
Whatever it takes, do it as long as it comes from your heart.