Sometimes the problem is in the file itself. An Ogg file missing its initialization headers will trigger download fallbacks.
When a browser or media player encounters an Ogg file, it has to decode the container, find the audio/video streams, and begin playback. Ogg Stream Init Download