encoder: use avoption api

FFmpeg's AVOption API is more flexible and better suited than
AVDictionary for codec and format options.

This also allows to use strings in encoder.json, such as setting profile
to high (instead of 100).

Change-Id: I2039b2d039a87f8b3ed8ecb197dfa7b734c27950