If your colleague is annoying you, be quiet when he greeted and chatted with him at lunch so that you can tell your annoyance to him; if he does not accept, ignore him