conversation: fix swarm synchronization bug

This patch fixes a bug that could cause the members in a swarm to
repeatedly fetch from each other, generating large amounts of
unnecessary network traffic in the process.
jami-client-qt#1851

Change-Id: I3e4d2f2b3b9b659a9a51ad6fe00c4f08f4c923a6