Won't let me join my friends session on xbox live? Help?

When i go to join my friends xbox live session to play borderlands it doesn't work. I click 'Find Xbox Live Party' and i press the X button to join the session in progress. But it says connecting and then just closes and nothing happens. This happened when he sent me an invite to his game aswell. I have a moderate NAT type and he has an open but when he used ot have moderate it still worked for us. Any suggestions on whats up or what i should do?