To all? Ofc it does to some people, rarely anything deals in absolutes. It's like with penis it does matter to some.
1. Well, actually there is certainly a minimum when it comes to pleasure and reproduction, but these are extreme cases, and yes, I believe many men don't like being on the left side of the curve, because again, it's men who make a big deal out of this.
2. Really? I don't think it is a bold claim. Since humans wore clothes penis size isn't really a factor, and even before that it wasn't as important as other factors. Almost all biology books say women seeked fat men because fat meant access to food and men seeked chubby girls because chubby meant bigger chance for a woman to produce healthy offspring. Today, penis size is not something that women base their opinion on, excluding, you know, sluts, penis is the last thing woman discovers regarding a man. You claiming it matters for majority - last time I read papers about it most women were only concerned about penis being too big, the rest didn't really matter.
3. Ofc penis is associated with masculinity just like breasts are associated with feminity and there are proofs of that since prehistoric times. Also I believe you some women think that size reflect the masculinity. Some people believe earth is flat. I am not sure what is a dominant trait for women judging masculinity - muscles? Being a good father? Making a lot of money? Sexual prowess? I would guess one of those, but not penis size, but again, some women probably do think so.
4. Both things are true but I don't see how do you fit them here in this argument. Fight flight or freeze - yes, what other options do you have? Maslow's hierarchy - yes, but it doesn't mean humans don't actively act against it when needed. Like I said, instincts are coded within us, but they do not steer us anymore.