diff --git a/technical/7. Jami plugins.md b/technical/7. Jami plugins.md index 408abd471d569d64a1938ea1fda98571f34b0642..b56827fdc372744e3a5a4e375c9f9a7a85216980 100644 --- a/technical/7. Jami plugins.md +++ b/technical/7. Jami plugins.md @@ -1,49 +1,74 @@ **NOTE: this page describes what is a Jami Plugin and how to install and use them.** # Jami Plugins - As from September of 2020, Jami team has added plugins as a call/chat feature for Linux and Android users. This meaning that now they can personalize their call/chat experience by using one of our available plugins. But that is not all, you can also transform your awesome ideas into a brand new plugin! -To properly setup a plugin you must follow the step bellow. -To `build` a plugin, please refer to build instructions at the end of this page. +To properly setup a plugin you must follow the steps bellow. + +To build a plugin, please refer to [#How to build?]("") instructions. + +If you are interested in knowing how our Plugin system works, please refer to [#How it works?]("") ## How to use it? -### Install / Uninstall -A Jami plugin is a file of the type plugin-name.jpl, and it must be installed to your Jami. -For Android You must go under Setting, enable plugins, and select a plugin file from your phone. -Similarly, for Linux, one must go to Jami Prefences, enable plugins, and select a plugins file from your computer. -Once installed, Jami will add your new plugin to the available plugins list. +### Setup +A Jami plugin is a file of the type `pluginname.jpl`, and it must be installed to your Jami. +Once installed, Jami will add your new plugin to the available plugins list but they will not be available for use yet. Plugins are libraries and must be loaded if you want to expose them. +Besides install/uninstall and load/unload actions, it is possible to modify plugin's preferences. +For example, our Foreground Segmentation plugin allows the user to change the background image displayed. -For Android uninstall, one must click on the plugin and a uninstall option will appear. -For Linux, each plugins in plugins list shows a uninstall button. +#### Android +To setup a plugin for Android you must go under Setting, enable plugins if it disabled, and select a plugin file from your phone. +After installed it is automaticaly loaded. +Optionally, the user can manually perform load/unload using the checkbox button on the plugin list. -### Load / Unload -Once installed, a plugins is not available for use yet. Plugins are libraries and must be loaded if you want to expose their functionalities. + + -For Android and Linux, there is checkbox that indicates plugin's load state. You can load or unload it by clicking on this checkbox. +For Android uninstall, one must click on the plugin and a uninstall option will appear allong with the preferences and a reset preferences option. +In order to a preference modification can take effect the plugin has to be reloaded. -### Change Parameters -Each plugin has configurations, and some of them are editable by the user. For example, our Foreground Segmentation plugin allows the user to change the background image displayed. + + -For client-gnome it is not yet possible to change those preferences, but for Android you can do so. -Preferences are displayed in the same page one must access to uninstall a plugin. +#### Linux +Similarly, for Linux, one must go to Jami Prefences, enable plugins if it disabled, and select a plugins file from your computer. +For Linux, each plugins in plugins list shows a uninstall button and a checkbox button corresponding to the load/unload action. +For client-gnome it is not yet possible to change plugin's preferences. -To take effect, the plugin has to be reloaded. + + ### Use! -Call plugins only take place if you turn them on during a call. -For both Android and Linux you can do so by clicking on the plugins icon on your call screen. +A plugins only take place if you turn them on during a call. +For either Android or Linux you can do so by clicking on the plugins icon on your call screen. + + + + + ## How to build? +If you want to make something with your video call, it is possible that you will do so with OpenCV and/or Tensorflow. +So, before going to the plugin, it is necessary to build plugin's dependencies. ### Dependencies -If you want to make something with your video call, it is possible that you will do so with OpenCV and/or Tensorflow. -So, before going to the plugin, it is necessary to build the plugin dependencies. +Here we give you the steps to build OpenCV and Tensorflow but do not feel limited to these libraries. +Other ones should work as long they and the plugin are correctly built! #### OpenCV 4.1.1 +We kindly added OpenCV 4.1.1 as a contrib in [daemon](https://git.jami.net/savoirfairelinux/ring-daemon/tree/master/contrib). +This way you can easily build OpenCV for Android or Linux. You only have to set `export DAEMON=<path/to/daemon>` and follow the corresponding instructions. + +##### Linux +```bash +cd ${DAEMON}/contrib/native +./../bootstrap --enable-opencv --enable-opencv_contrib +make +``` + ##### Android ```bash cd ${DAEMON} @@ -61,32 +86,31 @@ export STRIP=$TOOLCHAIN/bin/$TARGET-strip export PATH=$PATH:$TOOLCHAIN/bin cd contrib mkdir native-${TARGET} -cd native +cd native-${TARGET} ../bootstrap --build=x86_64-pc-linux-gnu --host=$TARGET$ANDROID_API --enable-opencv --enable-opencv_contrib make ``` -##### Linux -```bash -cd ${DAEMON}/contrib/native -./../bootstrap --enable-opencv --enable-opencv_contrib -make -``` -#### Tensorflow 2.1 -Dependencies: +#### Tensorflow 2.1.0 +A difficulty for a lot of people working with tensorflow is how to properly build it. Here we carefully guide you through the proper build of tensorflow LITE Native and Tensorflow C++ API. + +You will need: * Python 3 * Bazel 0.29.1 +* Tensorflow 2.1.0 repository: -```bash -git clone https://github.com/tensorflow/tensorflow.git -cd tensorflow -git checkout -b v2.1.0 -``` + ```bash + git clone https://github.com/tensorflow/tensorflow.git + cd tensorflow + git checkout -b v2.1.0 + ``` -Assemble instructions are available under [jami-plugins](https://review.jami.net/admin/repos/jami-plugins) README file +Assemble instructions for Tensorflow LITE Native and C++ API are available under [jami-plugins](https://git.jami.net/savoirfairelinux/jami-plugins) README_ASSEMBLE file #### Linux +Tensorflow LITE does not support desktops GPU. If you want to use them, please consider using C++ API + If you want to build Tensorflow C++ API with GPU suport, be sure to have a CUDA capable GPU and that you have followed all installation steps for the Nvidia drivers, CUDA Toolkit, CUDNN, Tensor RT, that their versions matches and that they are correct for the Tensorflow version you want to build. @@ -98,18 +122,26 @@ The following links may be very helpfull: * https://developer.nvidia.com/cuda-toolkit-archive * https://developer.nvidia.com/cudnn +Setup your build options with `./configure`. -`./configure` - -##### LITE -`bazel build //tensorflow/lite:libtensorflowlite.so` -##### C++ API -`bazel build --config=v1 --define framework_shared_object=false //tensorflow:libtensorflow_cc.so` -#### Android - LITE -Depencencie: +* Tensorflow LITE Native + + ```bash + bazel build //tensorflow/lite:libtensorflowlite.so + ``` +* Tensorflow C++ API + + ```bash + bazel build --config=v1 --define framework_shared_object=false //tensorflow:libtensorflow_cc.so + ``` + +#### Android - Tensorflow LITE Native +For mobile applications Tensorflow LITE is the only option you want to consider and to succesfully build it you will also need: * Android NDK 18r +Setup your build options with: + ```bash ./configure >> Do you wish to build TensorFlow with XLA JIT support? [Y/n]: n @@ -117,22 +149,58 @@ Depencencie: >> Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: y >> Please specify the home path of the Android NDK to use. [Default is /home/<username>/Android/Sdk/ndk-bundle]: put the right path to ndk 18r ``` +And build as desired: -```bash -bazel build //tensorflow/lite:libtensorflowlite.so --crosstool_top=//external:android/crosstool --cpu=armeabi-v7a --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --cxxopt="-std=c++11" -``` +* armeabi-v7a + + ```bash + bazel build //tensorflow/lite:libtensorflowlite.so --crosstool_top=//external:android/crosstool --cpu=armeabi-v7a --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --cxxopt="-std=c++11" + ``` +* arm64-v8a + + ```bash + bazel build //tensorflow/lite:libtensorflowlite.so --crosstool_top=//external:android/crosstool --cpu=arm64-v8a --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --cxxopt="-std=c++11" + ``` + +### Plugin +To exemplify a plugin build, we will use the Foreground Segmentation plugin available [here](https://git.jami.net/savoirfairelinux/jami-plugins). + +First you need to clone the repository to your ring-project path: ```bash -bazel build //tensorflow/lite:libtensorflowlite.so --crosstool_top=//external:android/crosstool --cpu=arm64-v8a --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --cxxopt="-std=c++11" +cd ring-project +git clone https://review.jami.net/jami-plugins ``` -### Plugin -##### Android -##### Linux +Inside ForegroundSegmentation folder there is three possible build files: + +* build.sh +* builtfcc.sh +* buildandroid.sh + +Each of these files corresponds to a platform and tensorflow build to be used. + +* Android: for this platform we do `sh buildandroid.sh`; +* Linux: for this platform we can do `sh build.sh` for Tensorflow LITE Native and `sh buildtfcc.sh` for Tensoflorw C++ API; + +The foregroundsegmentation.jpl file will be available under `<jami-plugins/build/>`. + +### How it works? +Jami can be break down to three main components that interact together: Daemon, LRC and clients. +Daemon is the core of Jami, and although it does not interact with users, it is involved in every +command. Therefore it is only natural that Daemon is responsible for everything related to Plugin's management, and among other actions it perfoms install/uninstall, load/unload, edit preferences and control plugins' usage. +Despite Daemon importance, what a plugin effectivelly does to a call video or a chat message is unknown to it the same way Daemon does not know what is effectivelly done by LRC or the clients interfaces. +Plugins then can be seen as a forth interacting component in Jami. + +By it's time Plugins are composed by two main components: the media handler and the process. While the latter actually implements plugin's functionality and waits data to perform, the former is the interface between Daemon and the plugin. +It is, the media handler is attached or detached to a media stream from Daemon and passes it to plugin's process. + +To use a plugin it is necessary that the Plugin Manager knows the available media stream and plugin's media handlers. +Plugin's media handlers are created once a plugin is loaded and they are shared with Daemon's Plugin Manager. +In the event of a new SIP call, the Plugin Manager creates and stores the corresponding media stream subjects. +Once a user puts a plugin in action the Plugin Manager attaches this plugin's media handler and the available call media streams. When deactivated, our Plugin Manager dettaches these two instances. -## TODO -### How it works Related articles: -+ https://jami.net/jami-plugins-foreground-segmentation-artificial-inteligence/ +- [ ] https://jami.net/jami-plugins-foreground-segmentation-artificial-inteligence/ diff --git a/uploads/plugins/Screenshot_20200716-131836_Jami.jpg b/uploads/plugins/Screenshot_20200716-131836_Jami.jpg new file mode 100755 index 0000000000000000000000000000000000000000..55386bb63cf84303194b37a72e1421a433d5c758 Binary files /dev/null and b/uploads/plugins/Screenshot_20200716-131836_Jami.jpg differ diff --git a/uploads/plugins/Screenshot_20200716-131841_Jami.jpg b/uploads/plugins/Screenshot_20200716-131841_Jami.jpg new file mode 100755 index 0000000000000000000000000000000000000000..f11c435637947af686bdd178f85f56658666e4ea Binary files /dev/null and b/uploads/plugins/Screenshot_20200716-131841_Jami.jpg differ diff --git a/uploads/plugins/change_preference_android.jpg b/uploads/plugins/change_preference_android.jpg new file mode 100755 index 0000000000000000000000000000000000000000..1b7a192d24f9a44e112660b3fe48620c7c596eee Binary files /dev/null and b/uploads/plugins/change_preference_android.jpg differ diff --git a/uploads/plugins/install_android.jpg b/uploads/plugins/install_android.jpg new file mode 100755 index 0000000000000000000000000000000000000000..31fd7e382bee0d3a9d0d9dcb105e77e932110ee2 Binary files /dev/null and b/uploads/plugins/install_android.jpg differ diff --git a/uploads/plugins/install_linux.png b/uploads/plugins/install_linux.png new file mode 100644 index 0000000000000000000000000000000000000000..00b008b80d7487f70cfd6c98ca8fd07d3b8a1b8c Binary files /dev/null and b/uploads/plugins/install_linux.png differ diff --git a/uploads/plugins/installed_android.jpg b/uploads/plugins/installed_android.jpg new file mode 100755 index 0000000000000000000000000000000000000000..492cf3f80b488cae5da406ffd3f4e238304b8fd0 Binary files /dev/null and b/uploads/plugins/installed_android.jpg differ diff --git a/uploads/plugins/installed_linux.png b/uploads/plugins/installed_linux.png new file mode 100644 index 0000000000000000000000000000000000000000..9034d99269fa9cf0e8ec77a2d4ab79e4385b084c Binary files /dev/null and b/uploads/plugins/installed_linux.png differ diff --git a/uploads/plugins/plugin_disabled_android.jpg b/uploads/plugins/plugin_disabled_android.jpg new file mode 100755 index 0000000000000000000000000000000000000000..7f1dea5ef72ac3e6a759114dba2bf0d2439cbb6b Binary files /dev/null and b/uploads/plugins/plugin_disabled_android.jpg differ diff --git a/uploads/plugins/uninstall_android.jpg b/uploads/plugins/uninstall_android.jpg new file mode 100755 index 0000000000000000000000000000000000000000..45f40300a287ffb1551ed93fce562e320394b34f Binary files /dev/null and b/uploads/plugins/uninstall_android.jpg differ diff --git a/uploads/plugins/use_android.jpg b/uploads/plugins/use_android.jpg new file mode 100755 index 0000000000000000000000000000000000000000..6893fb0dc0e3a166bdc9181c92dddfe3ef9051e9 Binary files /dev/null and b/uploads/plugins/use_android.jpg differ diff --git a/uploads/plugins/use_linux.png b/uploads/plugins/use_linux.png new file mode 100644 index 0000000000000000000000000000000000000000..66ab92c3bf4ac65ef93b45aacb1f5d713c2252d3 Binary files /dev/null and b/uploads/plugins/use_linux.png differ diff --git a/uploads/plugins/using_android.jpg b/uploads/plugins/using_android.jpg new file mode 100755 index 0000000000000000000000000000000000000000..eea0b1d8496094d4d879eb040e8ae21ab2fc776f Binary files /dev/null and b/uploads/plugins/using_android.jpg differ