Had a bit of a limp stab at a/v sync today.
I started with something simple although it ended up a bit too complicated - trying to synchronise multiple decoding and display threads is a little messy. I was trying to hide as much as possible inside MediaReader and the Audio/VideoDecoder classes, but it got ugly.
But as far as the timing, the simple approach nearly worked as far as sync goes, there's just a couple of issues remaining:
- After seeking on the container, the audio stream starts well before the video stream, so one ends up with a visual pause before starting, or even the video decoding never catching up (wrt packet buffering) and some nasty frame jumping.
Hopefully this can be solved as I do in JJMediaReader: after a seek I discard packets and/or frames until the seek position is reached.
- Getting the amount audio data "buffered" is a bit of a pain. Trying to use the reported position is fine until you seek since it throws the position out.
The simple approach was to have a central MediaClock object which tracks the audio renderer position, and then does various timing calculations for the video sync. It manages pause as well as interacts with seek. Maybe it isn't so simple ...
Eventually I should work it out.