OpenGL Video PipeLine
https://review.jami.net/c/jami-client-qt/+/16795
Currently, on client-qt, we are using converting the AvFrame data to QImages and draw the QImages in the Qml layer,
but this will increase the memory usage and CPU usage comparing with using only the AvFrame raw data,
so the plan is to use OpenGL API (or OpenGL Qt API wrapper) to do the raw data drawing by using shaders and textures.
(Qml layer is based on OpenGL 2.0, so we have to use their wrapper to be able to interact with the rendering event oop)
Generally, from what I've found, there are three ways to do that
-
QSGNode* updatePaintNode(QSGNode* old, UpdatePaintNodeData*) -> Patchset: 13
-
QQuickWindow::afterRendering -> Patchset:14
-
QQuickFramebufferObject -> Patchset: 16 and on
The first two methods could support protocols other than OpenGL (possible but not tested)
.
But, you have to manage the viewport itself, since the gl context is the whole window by default instead of individual QQuickItem.
(Not sure if this will be fixed on Qt 6.0 -> https://wiki.qt.io/New_Features_in_Qt_6.0)
The third one does not have the viewport problem, but it will only support OpenGL.
https://doc.qt.io/qt-5/qquickframebufferobject-renderer.html#render
https://doc.qt.io/qt-5/qquickframebufferobject.html#details
For now, we will integrate the QQuickFramebufferObject approach.