Sex or the lack thereof

I know most poly people seem to be shy when talking about sex. Although, I would like to add here, that I am really not writing about sex, but I am writing about the lack thereof.  Poly seems to have the reputation from those who really have no clue about it that it is all about sex.  In reality though, it is about love, family and committment.

Over the years, our husband has gone from being very sexually active and vibrant to being exhausted, stressed, and barely sexually active. We are fortunate if each of us gets to have sex with him at least once a week and sometimes that includes group sex.  I know most of it is due to extreme stress.  The stress has also caused each of adults to gain weight, so I know the weight increase has some to do with it as well. 

Before we started courting Sarah, our sex life was fairly decent.  Leah and I would have sex with him almost three times a week each.  I was actually quite happy with this arrangement.  I warned him though that if the intimacy between each of us actually diminished too much, like only once a week or less, then I would not be willing to ask Sarah to marry us.  He assured me that it would not and that everything would be ok.

We have been seriously courting her for over a year now, but I am still holding back on asking her to marry us.  This is due to the fact that since we have been courting her that sex has diminshed to once a week or even less than that.  I love Sarah, but I am not willing to bring in another wife if our husband cannot fulfill his duty to us now. I have already shared this with Sarah, and she understands.  Of course, it also hurts her, but I felt I needed to be honest with her. 

This is not the first that he has lacked in the sex or intimacy department.  It seems that major stress greatly affects his libido.  A few years back before Sarah, we courted another woman, and when we were courting her, he acted the same.  This is beginning to make me believe that he cannot handle more than two women. He says it has nothing to do with the amount of women, but it has everything to do with stress. 

I am not sure if I should believe him.  I know he really loves each of us, including Sarah.  Leah had begun to question if she is even desirable to him anymore. I told her that it had nothing to do with her, but everything to do with him and his level of stress.  I told him he needed to speak with his doctor about anti-anxiety medication or something, but he does not think they will help.

I feel at a loss though because I am not happy with our current sex life.  Any suggestions?