Yes, sandbox elements can be really annoying and kill the fun out of the game. Unless of course you like to be blueballed and think waiting or grinding requirements is exciting.
Not a sandbox fan either. By now I sometimes even exclude the tag. It's not even getting to the fapping. It just often kills the pacing of storytelling for me.
I prefer some sandbox options or at least well written illusions of sandbox options. I find it more difficult to get into a VN if you rarely get a chance to decide what to do next and the entire thing is just written like a movie with no choices. Doesn't have to be an extreme sandbox, but having the options, in game, to go to different areas or redo scenes without needing to go to a scene replay mode make it more enjoyable for me, personally.
If the devs idea of "sandbox" is "find the one place to go that advances the story" rather than actual open world, it sucks donkey balls. The donkey has several STDs if the game has time and it has to be at a specific time.
Gotta say that i enjoy it sometimes, even the grinding part. It's not always about the fap lol.
But most of the sandbox games i played are kinda annoying...
Not really. Too much clicking and figuring out where to click. I'm sure you could do it in a way, but it needs to be simple, clear, and obvious. Either way, the real idea behind it is choice. Role-playing to make the player feel like they have some agency in what is going on. You can do that in so many ways and usually sandbox games just do it to pad playtime, not enhance the experience. There's a way to do it well, or better. I'm not saying I know that way, but I'm sure it exists.