Is it ok to talk about sexual stuff in mixed company?

It's ok if your comfortable with it, which you weren't. I get the feeling these friends were testing the waters about having a foursome or small orgy and wanted to see your reaction. Make sure you let your feelings be known if this happens again.