divergent unsyncable single-user multi-device private swarm

I've long used a "Reminders" private swarm to send reminders to myself from one device to another.

While jami-client-qt#1634 (moved) was preventing message delivery between them, I used it for testing, and tried to send messages from both ends.

IIUC, this shouldn't have been a problem, but the swarm seems to have gone hard out of sync. Even after hours with both devices connected, having got plenty of message syncing between them, this swarm seems to have got stuck.

I even tried to add a third device to the mix, to see if that would somehow help. It didn't. Other swarms synced beautifully, but this one didn't even show up on the newly-added device.

The git repos on both sides seems to be consistent. The /main tag corresponding to the other device on each device point to the commit corresponding to the last post that they managed to sync.

I also tried to create a new swarm, on one of the devices. It wouldn't appear on the other. I also noticed that these two single-user swarms wouldn't appear as choices in the list of users to add to a swarm, even though other swarms, in which other users are present, appear on that list.

Perhaps there is something special going on about single-user swarms, maybe, that's preventing (optimized away?) synchronization across multiple devices associated with the same single user?

mentioned in issue jami-client-qt#1636

marked this issue as related to jami-client-qt#1006 (closed)

moved from jami-client-qt#1635 (moved)

removed the relation with jami-client-qt#1006 (closed)

marked this issue as related to #1006 (closed)

added platformlinux BacklogMedium swarm bug labels

Logs while sending a message should give what's wrong

Indeed. Here's the oddity I see:

[1718176127.198|35614] [device 967c0526bfcc42347683deffa32c75997bb839cb21ae372856ff3f6d615a3210] Received request [1718176127.199|35614] Found peer device: 967c0526bfcc42347683deffa32c75997bb839cb21ae372856ff3f6d615a3210 account:31fa4751059b176c0d78891564d0520f7e49e72c CA:830b19215d499be2e487e5f617df569debf69c92 [1718176127.200|35614] [device 967c0526bfcc42347683deffa32c75997bb839cb21ae372856ff3f6d615a3210] New connection request [1718176127.205|35614|account_manager.cpp :431 ] Found peer device: 967c0526bfcc42347683deffa32c75997bb839cb21ae372856ff3f6d615a3210 account:31fa4751059b176c0d78891564d0520f7e49e72c CA:830b19215d499be2e487e5f617df569debf69c92 [1718176127.208|35614] %s [1718176127.208|35614|account_manager.cpp :473 ] [Auth] Discarding message from unauthorized peer 31fa4751059b176c0d78891564d0520f7e49e72c. [1718176127.208|35614|jamiaccount.cpp :2002] Discarding ICE request from 0000000000000000000000000000000000000000 [1718176127.208|35614] [device 967c0526bfcc42347683deffa32c75997bb839cb21ae372856ff3f6d615a3210] Refusing connection

On the other device, I'm getting symmetric messages.

The "from 000...000" looks very suspicious. So is the stray "%s", I wonder what it meant to print there.

I suppose I may be able to get out of this without losing anything by adding a third device and allowing it to sync with both of them. But it shouldn't be a "device" associated with an "instance" (for lack of a better name) that already holds one of the active "device"s. That doesn't seem to have worked very well, and it may even have aggravated the current difficulties.

After the syncing is done, I suppose I could roll back the current devices to known-good backups, or replace them with new "devices", and allow them to resync.

mentioned in issue #1006 (closed)

Actually [Auth] Discarding message from unauthorized peer 31fa4751059b176c0d78891564d0520f7e49e72c is the log

3 possibilities (cause some context is missing):

1. Can you check your blocked list (in the account settings), because 31fa4751059b176c0d78891564d0520f7e49e72c can be blocked (it's in the account settings)
2. else it's a revoked device (there is probably more description in the logs at this point).
3. Another possibility is that you decline ICE request from untrusted peers. It's in the advanced account settings (but you can copy show cat ~/.local/share/jami/*/config.yml to confirm this)

1. I can't find any mention of blocked lists anywhere in account settings, but really, this is my own account, how could it have got blocked?
2. neither device is revoked, but there was a third device that I added and later revoked; the problem was already there, but I don't have logs from before to compare
3. here's config.yml: alias: Alexandre Oliva username: 31fa4751059b176c0d78891564d0520f7e49e72c activeCodecs: 139/27/12/4/86076/536956963/268521507/69660/86051/65543/65542 useragent: Jami Daemon displayName: Alexandre Oliva defaultModerators: "" uiCustomization: "{}" dhtPublicInCalls: false Account.peerDiscovery: true proxyServer: dhtproxy.jami.net:[80-100] RingNS.uri: ns.jami.net Account.registeredName: lxo Account.archivePath: export.gz Account.deviceName: aoliva DHT.PublicInCalls: false ringAccountReceipt: ... ringAccountReceiptSignature: ... tls: certificate: ring_device.crt privateKey: ring_device.key

Ugh, after connecting a new device to my account, it still wouldn't sync. It would get information from groups I belong to from my peers, but those in which I'm not only online member display as myself. The old devices won't allow the new device to connect to them, and the new device won't allow hte old devices to connect to it, so syncing cannot make progress.

Now, if I create a brand new account, on the same third machine, and add it to the group, then it can sync with the other devices, and it could fetch the messages in that group that were stuck on both preexisting devices into the new account on the new device, and then it could sync messages and push them onto the preexisting devices, both onto the old account and onto the new account, that I replicated there.

So, that solved the immediate problem of finding a way to sync the messages in that group. Unfortunately, dding the new device, or even the new account, doesn't seem to have fixed the problem that both of my primary devices won't talk to each other any more. I've added some logs to #1006 (closed). Those running 6.6 were from before I created the new device and the alt account, but maybe they will reveal what's going on, if you know what to look for. The 6.9 log there was also from before, but that one didn't do any messaging whatsoever, so I don't expect it would be of any use for this issue.

I've set myself up for some jami debugging (thanks for providing dgbsym packages, that is quite convenient), for #1006 (closed), and the first hit of the "Discarding message from unauthorized peer" already gave me ideas of things to try, and now I have some theories on how the problem may have come about, and how I may have worked around (or fixed?) it.

Deducing that allowPublic was false in the isAllowed call just before the message (src/jamidht/account_manager.cpp:473 in 20240529), I started wondering why that setting was that way, and how it could lead to the denial of connections between devices associated with my account on all of my devices, even new ones.

These are hypotheses based on fuzzy recollection of past events, for which I have neither logs nor notes.

How it started

Not long ago, probably long before I understood the correlation between kernel version and #1006 (closed), but possibly after I started noticing its symptoms, I started going through settings, and I recall noticing, under call settings, "Allow incoming calls from unknown contacts". I don't recall how it was set, but I'm sure I tried toggling it, possibly more than once, and it is now disabled on both of my local devices.

I expect this setting is supposed to be higher-level, blocking audio/video calls only, but I am guessing that there might be some misuse of it that affects the ICE subsystem. It could be the case that I disabled it, noticed that it didn't prevent syncing between my devices (as long as at least one of them had it enabled), and then proceeded to block the other one, without immediately associating it with difficulties in syncing because I don't use my notes-to-self group very often.

How it may have got worse

Then, after I started looking into this and the other issue in earnest, but IIRC still before I filed the tickets and learned how to collect logs, I added the third device mentioned in the description of this issue, alongside my primary device (same jami session). That account managed to sync groups that had other contacts online, but it doesn't seem to have succeeded in syncing this notes-to-self group, and other groups that didn't have other online contacts.

As a result, these groups were displayed in the contact list as myself, and because syncing wouldn't complete, they remained so. Now, I'd never seen myself listed as a contact, and I found that disturbing. It seemed to be something wrong. I recall being very puzzled and wondering whether to delete those stray contacts.

I am not sure I did, but if I did, it may have made things worse, because then I was no longer a contact of myself, and if this removal sets some flag such as the block setting that you suggested, then that might explain why it got set, and why the blocking remained in effect even after I deleted this co-located third device and added another one on a third machine.

(at some point I realized that these appearances of myself as a contact were unsynced groups, and that their settings mentioned the group hash id, but it is conceivable that the removal of the unsynced group would have removed myself as a contact, since AFAICT I didn't get removed me from any group)

How I worked around it

I looked up myself in the account list, found my account, and added it as a contact. Voila! I am no longer getting the unauthorized connections, and message flow is now unimpaired, even without resorting to the separate account I set up for testing purposes.

Recommendations

• review the uses of the 'call from unknown contacts' setting and make sure it does not affect syncing of a single-user group between multiple devices

• keep oneself as a hidden (or visible) contact at all times, even after adding and removing oneself explicitly, perhaps making it unremovable. IIRC having oneself as a contact was not even possible before swarms came about, and I was surprised that this was possible now. I recall missing the possibility of using jami to send messages to myself on my other device. when swarms came about, I could create the notes-to-self group and satisfy that convenience/need

• use a less confusing representation for unsynced groups, displaying the group id rather than self, so that it doesn't appear to be the self contact

Here it is

jamidht/jamiaccount.cpp-1999-                    auto res = accountManager_->onPeerCertificate(cert,
jamidht/jamiaccount.cpp:2000:                                                                  this->config().dhtPublicInCalls,
jamidht/jamiaccount.cpp-2001-                                                                  peer_account_id);

Nailed!

Now, was this really what this flag meant, and if so, should it really be under 'Call settings', with that wording, or should I file a separate issue to make that less likely to be disabled without understanding the implications.