In some ways, it's quite convenient for your wife's boyfriend to be living so close. It means that she can drop by his place easily whenever they're in the mood for sex -- or he can drop by your house, whichever is easiest at the moment. And you and he have become pretty good friends, now that you see quite a lot of each other. But it also means that everyone in the neighborhood knows about their relationship -- and know that when there are neighborhood events of any kind, she is very likely to attend as part of a "couple" with him instead of with you.
Your wife decided it was time for the two of you to have "the cuckolding talk" -- and however painful it may be for you to hear this, you can't deny that what she has said about your sexual inadequacy is true, and you know there is nothing you can do at this point to change her mind about starting to date other men. You do wish she weren't going out with one of your friends, but that's also not something you can change.