I'm afraid if we come across each other, and then I listen to English or speak it badly, as did he, like you suggest, because we may not know each other speaking subjects.
I'm afraid if we met, I listen to English or speak wrong, it is a problem. You don't like yoga practise because we may have to be unable to understand each other.