There's absolutely no problem if you don't want to play. Just say no, but we can schedule a time. You send the invitation and after enough time has passed to discern who was calling, you enter a chat with someone else. A simple explanation of why you're waiting and building expectations would have sufficed.