From fe21ff654bed3129e069f6b3b26a5816c137497b Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?S=C3=A9bastien=20Blin?=
<sebastien.blin@savoirfairelinux.com>
Date: Wed, 17 Aug 2022 17:09:51 -0400
Subject: [PATCH] developer/calls: add some docs about multistream
Change-Id: Ibafd8e94979a325002c65543a87f7a96ccaf4c96
---
developer/calls.md | 154 ++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 152 insertions(+), 2 deletions(-)
diff --git a/developer/calls.md b/developer/calls.md
index cc0ab7cf..7af4941c 100644
--- a/developer/calls.md
+++ b/developer/calls.md
@@ -73,8 +73,158 @@ Media sockets are SRTP sockets where the key is negotiated through the TLS Sessi
### Architecture
+**TOOD**
-## Client APIs
+## Multi-stream
-## LRC
+Since daemon's version 13.3.0, multi-stream is fully supported. This
+feature allows users to share multiple videos during a call at the
+same time. In the following parts, we will describe all related
+changes.
+
+### pjsip
+
+The first part is to negotiate enough media streams. In fact, every
+media stream uses 2 UDP sockets. We consider three scenarios:
+
+1. If it's the host of a conference who wants to add media, there is
+ nothing more to negotiate, because we already mix the videos into
+ one stream. So, we add the new media directly to the video-mixer
+ without negotiations.
+
+2. If we're in 1:1, for now, as there is no conference information,
+ multi-stream is not supported.
+
+3. Else, 2 new sockets are negotiated for new media.
+
+To make pjsip able to generate more sockets per ICE session,
+`PJ_ICE_COMP_BITS` was modified to `5` (which corresponds to `2^5`, so
+32 streams).
+
+### Deprecate switchInput, support requestMediaChange
+
+In the daemon, the old API `switchInput` is now **DEPRECATED**; same
+for `switchSecondaryInput`:
+
+```xml
+<method name="switchInput" tp:name-for-bindings="switchInput">
+ <tp:docstring>
+ Switch input for the specified call
+ </tp:docstring>
+ <arg type="s" name="accountId" direction="in"/>
+ <arg type="s" name="callId" direction="in"/>
+ <arg type="s" name="input" direction="in"/>
+ <arg type="b" direction="out" />
+</method>
+
+<method name="switchSecondaryInput" tp:name-for-bindings="switchSecondaryInput">
+ <tp:added version="11.0.0"/>
+ <tp:docstring>
+ Switch secondary input for the specified conference
+ </tp:docstring>
+ <arg type="s" name="accountId" direction="in" />
+ <arg type="s" name="conferenceId" direction="in"/>
+ <arg type="s" name="input" direction="in"/>
+ <arg type="b" direction="out" />
+</method>
+```
+
+`requestMediaChange` replaces this, for both calls and conferences:
+
+```xml
+<method name="requestMediaChange" tp:name-for-bindings="requestMediaChange">
+ <tp:added version="11.0.0"/>
+ <tp:docstring>
+ <p>Request changes in the media of the specified call.</p>
+ </tp:docstring>
+ <arg type="s" name="accountId" direction="in" />
+ <arg type="s" name="callId" direction="in">
+ <tp:docstring>
+ The ID of the call.
+ </tp:docstring>
+ </arg>
+ <annotation name="org.qtproject.QtDBus.QtTypeName.In2" value="VectorMapStringString"/>
+ <arg type="aa{ss}" name="mediaList" direction="in">
+ <tp:docstring>
+ A list of media attributes to apply.
+ </tp:docstring>
+ </arg>
+ <arg type="b" name="requestMediaChangeSucceeded" direction="out"/>
+</method>
+```
+
+### Compability
+
+If a call is done with a peer where the daemon's version is < 13.3.0,
+multi-stream is not enabled and the old behavior is used (1 video
+only).
+
+### Identifications of streams
+
+Because there can be multiple streams now, every media stream is
+identified by its identifier, and the format is "<type>_<idx>"; for
+example: "audio_0", "video_2", etc.
+
+### Rotation
+
+The XML was updated to add the wanted stream:
+
+```
+<?xml version="1.0" encoding="utf-8" ?>
+<media_control>
+ <vc_primitive>
+ <stream_id>{}</stream_id>
+ <to_encoder>
+ <device_orientation>0</device_orientation>
+ </to_encoder>
+ </vc_primitive>
+</media_control>
+```
+
+### Key-frame
+
+The XML was updated to add the wanted stream:
+
+```
+<?xml version="1.0" encoding="utf-8" ?>
+<media_control>
+ <vc_primitive>
+ <stream_id>{}</stream_id>
+ <to_encoder><picture_fast_update/></to_encoder>
+ </vc_primitive>
+</media_control>
+```
+
+### Voice activity
+
+The XML was updated to add the wanted stream:
+
+```
+<?xml version="1.0" encoding="utf-8" ?>
+<media_control>
+ <vc_primitive>
+ <stream_id>{}</stream_id>
+ <to_encoder>
+ <voice_activity>true</voice_activity>
+ </to_encoder>
+ </vc_primitive>
+</media_control>
+```
+
+## Conference
+
+Reflected changes are documented {doc}`here <conference-protocol>`.
+
+## Client
+
+Even if the back-end supports up to 32 media at the same time, except
+for custom clients we currently recommend only giving the ability to
+share one camera and one video at the same time. The camera is
+controlled via the camera button, and the other media via the "Share"
+button.
+
+In client-qt, the interesting part is in `AvAdapter` (methods like
+`isCapturing`, `shareAllScreens`, `stopSharing`). In the library's
+logic, `addMedia` and `removeMedia` in the `callModel` directly use
+the `requestMediaChange` and can be used as a design reference.
--
GitLab