Video GPU hardware decoding on Android
Issue generated from Tuleap's migration script. Originally submitted by: Ciro Santilli (cirosantilli)
This is a followup to https://tuleap.ring.cx/plugins/tracker/?aid=293 specifically to implement GPU decoding.
Current status: patches uses AMediaCodec on https://gerrit-ring.savoirfairelinux.com/\#/c/3799/ and https://gerrit-ring.savoirfairelinux.com/\#/c/3780/
Video works most of the time, but fails some, see the commit messages.
We are not sure if AMediaCodec is the best way of doing it, but it looks like a good choice because it is the JNI version of a Java API, thus "well documented": http://developer.android.com/reference/android/media/MediaCodec.html
There is an official example of using AMediaCodec at https://github.com/googlesamples/android-ndk/tree/3cd41e1f5280443665ca98463c7a76e80bf0b96c/native-codec The problem is that example uses MediaExtractor to read a video file instead of decoding streaming bytes directly.
The pure Java example at: https://github.com/googlesamples/android-BasicMediaDecoder/tree/39d62cafb747176b96a385404952f1b8439f7b2e may be useful since the native API is basically identical. But once again that example uses a video file instead of a stream.
https://github.com/fyhertz/libstreaming appears to be doing H.264 streaming with Java MediaCodec, so it might be an useful example to look at. It implements RTSP, concrete examples at: https://github.com/fyhertz/libstreaming-examples
## Other possibilities
Openmax would likely work as well, as there is an example at https://github.com/googlesamples/android-ndk/tree/3cd41e1f5280443665ca98463c7a76e80bf0b96c/native-media Like the MediaCodec example, the problem is that the example decodes a file, not a stream.
Openmax is used inside of MediaCodec, as can be seen by the fact that libopenmax must be linked to as well to use AMediaCodec.
The major problem of Openmax is that it feels like an unmaintained Khronos API from 2011, examples are extremely scarce, and it is even difficult to get a quick GNU Linux implementation running. MediaCodec on the other hand seems to get more and more support with time.
The name comes up often, likely because FFmpeg had support for it, but it was dropped because it is an internal API. See:
So this is likely not a good idea.
The perfect solution would be of course to find an FFmpeg build option that magically works on Android.
We haven't been able to find one however.
There is some info at https://trac.ffmpeg.org/wiki/HWAccelIntro which mentions
-hwaccel, but it does not mention Android.
Most FFmpeg Android hardware threads are about libstagefright, which as mentioned above is not an option.
So we have opted to do a custom quick-and-dirty solution first. But ideally, our solution should be merged back to FFmpeg one day.
## Measuring the improvement
We expect two improvements with this change:
- higher FPS
- lower energy consumption
To test FPS, use the patch: https://gerrit-ring.savoirfairelinux.com/\#/c/3728/ and then compile with:
We haven't investigated energy consumption yet. The Android guide says that constant networking is the major power consumption in Android devices, so maybe it is not significant: http://developer.android.com/training/monitoring-device-state/index.html To be confirmed.
## How to generate a minimal example
In order to really understand what is going on, it might be necessary to generate a minimal FFmpeg encoding to MediaCodec decoding example.
The best way to do that might be to:
make a minimal Android socket-based server app that reads packets and feeds them to AMediaCodec.
Maybe this can be used as a starting point: https://github.com/cirosantilli/android-cheat/tree/92de020d0b708549a444ebd9f881de7b240b3fbc/socket for the
encode camera data from a GNU Linux computer and send it with FFmpeg over the LAN.
It would be even more minimal if we could have the video source inside Android as well, but installing FFmpeg on Android is not so simple (Ring does it of course, but maybe https://github.com/WritingMinds/ffmpeg-android would be simpler).
## If you really want this to work at any cost
Email: https://www.linkedin.com/in/andymcfadden http://stackoverflow.com/users/294248/fadden and ask if he does consulting. He's answered tons of android video questions on SO. If it's doable, I bet he could do it in a week, and the right way.
He replied to me:
I'm not available for consulting at this time.
If you want this to work on Android API 18+, you should use MediaCodec, in either its Java or native form. The output should go directly to the display Surface. The Java API is just a thin wrapper around the native API, so there's no performance reason to use the native MediaCodec API. Using a stable API will make your life much easier than attempting to use libstagefright or OMX directly, as those can change without warning and break everything on future devices.
Breaking a stream into NAL units will likely be faster in native code, but I don't know how much of a difference it would make in practice. Either way you have to identify the start and end, and copy the data into an input buffer, because that's how OMX rolls.
Or maybe https://github.com/fyhertz He's from Montreal as well.
## Misc links
This is a huge semi-organized dump of a great number of links that we have looked into to different degrees. Very few examples everywhere. Just to save some googling time.
- openmax android https://www.khronos.org/openmax/ VLC has the option. By Khronos. Previously called iomx?
- last API update 2011, almost no desktop implementation... old forgotten stuff
- TODO: how is frame-rate controlled there? no usleep. Just do it as fast as possible?
- http://stackoverflow.com/questions/15305241/how-do-i-feed-h-264-nal-units-to-android-mediacodec-for-decoding fadden says: can't feed fixed size buffers, need to parse H.264 and send some variable size unit
- https://developer.android.com/reference/android/media/MediaCodec.html official API?
- With FFmpeg
- https://github.com/taehwandev/MediaCodecExample could not get working
- VLC example https://github.com/videolan/vlc/blob/master/modules/codec/omxil/mediacodec\_ndk.c
- FFmpeg dropped it because this API is not public, says that MediaCodec should be used instead: https://github.com/FFmpeg/FFmpeg/commit/72673ad7eae2d4f685f3c0a895558502bfe07c8e
- part of
git clone https://android.googlesource.com/platform/frameworks/avat path
- there was a serioius vuln in it: https://nakedsecurity.sophos.com/2015/07/28/the-stagefright-hole-in-android-what-you-need-to-know/