encoder: add error checking to videoFrameSize

av_image_get_buffer_size can return a negative error code. Prevents
casting such an error to a large positive integer that may cause a
bad_alloc exception on the subsequent buffer allocation.

Change-Id: Ie8a7987dcf16957a21496e791c34957bb2e35bfc