I know this is not an easy thing to do in all cases, but with people I know (wife, family, friends, etc), I set the expectations that I'm not on the phone at all times and it might take me a long while to answer. If they don't like it, well, too bad.
A few can moan and whine from time to time...