|Dogs can be fearful of men too - Dreyfuss in the picture is.|
He likes cats, though. Photo by Elisa Black-Taylor.
Cats can be fearful of people in general, men and women. You could argue that it is a "default position", meaning it is normal for a cat (especially a kitten) to be fearful of the human simply because we are so much larger and rather frightening to be honest. Even people are frightened of other people!
Individual cats might be afraid of women for whatever reason - a certain event might have happened with a women at some stage in the cat's development. Men should be particularly careful, quiet and gentle when in the vicinity of a domestic cat. Approaching a cat slowly and quietly while giving warning with a melodious voice can be important too. Handling cats properly is also important.