I can understand how distressing it would be to experience a sudden reduction in libido, since sexuality can be such a revitalizing part of life. However, I think you are being overly hasty in concluding that a nearly vegan diet was responsible for your troubles. My reasons:
(1) Well-planned vegan diets, which lack any dietary cholesterol, are known to be healthful: Vegetarian Diets
It is possible, of course, that your particular physiology prevented you from thriving even on a well-planned vegan diet...although I'm not sure what the reason for that might be.
(2) Humans are generally able to biosynthesize all the cholesterol their bodies need, and dietary sources of cholesterol have much less impact on blood cholesterol levels than people once thought:
(3) It isn't clear that increasing a person's cholesterol levels through diet will cause his or her testosterone levels to increase, with free testosterone being the form that appears to be active.
For example, here is a study on testosterone levels in vegans and omnivores that found that vegan men actually had higher testosterone levels and nearly the same levels of free testosterone as omnivores:
(4) The relationship between testosterone levels and libido isn't simple or clear, since sex drive depends on many factors, not just testosterone levels. Testosterone levels don't perfectly predict libido:
A study in which young men were given testosterone injections to nearly double their testosterone levels found that the heightened testosterone levels had little effect on the men's sexual behavior:
Lucifer, considering your own bad experiences when eating a vegetarian diet that appears to have lacked dairy but should have included eggs (a source of cholesterol that could be used instead of dairy or meat if you really wanted to increase your cholesterol intake), one possible explanation is that your dairy-free vegetarian diet simply was not a balanced one.
For example, perhaps you weren't eating sufficient levels of monounsaturated and polyunsaturated fats such as from nuts. Your rapid weight loss is a red flag that your diet may not have been adequate.
You might also be someone with hypocholesterolemia whose body has very low cholesterol levels (such as less than 50 mg/dL) due to a disease or an inherited problem in your cholesterol biosynthesis pathway:
Hypocholesterolemia. [Curr Vasc Pharmacol. 2011] - PubMed - NCBI
If I were your doctor (and if I *were* a doctor 
), I would have:
(1) referred you to a nutritionist to ensure your diet was providing you with all the nutrients you needed;
(2) drawn your blood to obtain your fasting blood levels of cholesterol (total, HDL, and LDL);
(3) measured your total testosterone and free testosterone levels.
I agree that your health is most important, yet your description of what happened does not convince me that a nearly vegan diet was the cause of the problem, especially since I've not heard of vegan men in general having disappointing sex lives or sex drives.
I'm curious now what your doctor found out about you during your experience with vegetarianism. Do you have records of your cholesterol levels before and during your vegetarian phase, for example? As a vegan, the lowest my Total Cholesterol has been is 110 mg/dL and the highest is 127 mg/dL. I've been biosynthisizing all my own cholesterol for 14 years now...and I also produced all the cholesterol in a 9 lb 10 oz baby!
What makes you think that meat is a necessary component for survival? If it were, how could I be typing this?
How could 30% of the population of India be vegetarian?
Can you think of any situation in which meat-eaters *should* apologize for the harm they're doing to the animal kingdom? Will you eat any animal without any twinge of conscience? A dolphin? A dog?
Finally, about my sympathy for animals and my moral outrage: what determines whether an emotional response is "irrational" or not? Aren't emotions distinct from reasoning and thus can't be determined to be "rational" or "irrational?"
I would say that a person's emotional response might be "unusual" yet not "irrational." Most people don't feel sad like I do when I see someone eating a chicken or part of a pig. I acknowledge that. Yet rather than viewing my response as irrational, I view *their* response as callous and oddly emotionally empty...and all too normal. From my vantage point as a vegan, it is as if part of their emotional center is dead and they seem emotionally blind when it comes to animals.
I recall how I felt when I was emotionally "blind" to food animals. Thinking about their feelings and experiences just wasn't something I did very often. It was as if their experiences didn't exist. In my view, ignoring other beings' existence, and one's impact on these animals, doesn't seem very rational.