encoder: cleanup and simplify

Cleans up the encoder a bit and removes methods that weren't used or did
nothing. In any case, it should not be the encoder's responsibility to
mute audio/video.

Removes direct accesses to private data. All these options are accessible
through the av_opt_* API with the AV_OPT_SEARCH_CHILDREN flag.

Adding streams to the encoder is now done separately than opening an RTP
output in an effort to streamline the encoder setup (file vs RTP).

Change-Id: I7a868d098fa942697cfbe3246f368fb9fc7bfb0f