Most samurai prized their relationships with other men far above those they had with women. Women had very little status in this militarist society, and never functioned as ideals or enobling inspirations the way they did in medieval Europe; Japanese culture has no counterpart to the Virgin Mary and no tradition of putting women on a pedestal. Of course samurai married and had children, but as in ancient Greece, a man's sexual/mentoring relationship with a male youth and the lifelong close bonds that resulted were considered the highest form of love. This was called the "beautiful way", since young males were the ideal of beauty in every respect. The ages of fifteen to seventeen were the most admired, which fits the Japanese aesthetic of fleeting perfection, as in the short-lived cherry blossom. Many of these ideas grew out of Buddhist monasteries; the pursuit of young male acolytes was practically a holy and purifying activity, since sex with women was forbidden and ritually polluting.
More on that subject:
A samurai who showed excessive attachment to a particular woman, or to women in general, was considered weak and unmanly. Grooming himself to appeal to women or acquiring polished teahouse manners was the height of limp-wristed effeminacy and not worthy of a warrior. A macho man was one who shunned women, even if he were married, and dedicated himself mostly to the society of men, which usually included having a male lover. Samurai were not supposed to be promiscuous, so the ideal was loyalty to one man until death. Obviously that wasn't always the case, but anyone who completely avoided any sexual contact with other males would probably have had his masculinity put in serious question!
This isn't an easy concept for modern Americans to grasp, naturally. :D Human sexual expression has existed in almost infinite variations throughout history, and labels and roles such as "masculine" and "feminine" are culturally defined norms rather than innate. (I could go on for pages on that subject, but I ain't got time.) When speaking of men of the Edo period, ideas like "gay", "straight", "bi" and even "ephebophile" mean less than nothing; they didn't think of themselves in divisions like that and trying to force modern assumptions and judgments into a society so different from our own only distorts the truth. Samurai defined male/male sexual attraction as an extension of their warrior role; wealthy townspeople, who often aped samurai morals, visited prostitutes of both sexes without assigning any particular meaning to their sexual behavior other than pleasure.
This has changed in modern Japan, which imbibed European ideas about sexual morality wholesale in the later 19th century. The "beautiful way" of the samurai died out in the Meiji period. But the antique preference for sex partners much younger and of lesser status than oneself lives on in the "loli-con" obsession with schoolgirls.