Enterprise Manager Remote Data Collector Connection Status Down

Hi guys, we are having a scenario with the Remote Data Collector Connection Status "Down" to Primary Data Collector. But sometimes its up and then after a few seconds its down and then its up again and so on. We double checked all the requirements especially the ports required, and its all open. We tried to remove then reconnect it again and even re-installed it also but same scenario happens. Is there anyone can shed light on this matter. Thanks.