swarm: add possibility to remove the conversation without the contact

if there is other conversations with a contact, only the conversation
is removed, not the others

Change-Id: I6a7aa6fc4953c34fb46e68c7682d1891511b0017