Archive outage Jan 1 and 2, 2022

On Jan 1 and until about 10am eastern on Jan 2 2022 archived recordings made in the new year were not available. Sorry about that. This has been fixed now and I can listen to recent recordings. There may be some discontinuity around the time I was working on the repair but it seems back on line now.

What happened?

Pretty weird actually. We use FFmpeg to connect to your webcast streams and write the audio data roughly like this (this shows only the options critical in the incidence, there are lots more):

ffmpeg -i STREAMURL -c copy -f segment -segment_time 300 -segment_atclocktime 1 \
  -strftime 1 /path/to/file/%G%m%dT%H%M%SZ.mp3

So FFmpeg connects to your stream and continuously listens, copying the MPEG audio data into .mp3 files (or .m4a if you are using AAC). It opens a new file every 5 minutes. Each file is named according to the current date and time. For example, KZSC-20211231T143000Z.mp3 is the 5-minute long segment for KZSC starting at 14:30 UTC on Hogmanay 2021. (The Z* suffix denotes Zulu time, i.e. UTC).

When the new year rang in UTC (UK, Portugal, Morocco, Iceland, Liberia, etc.) the first file to write should have been KZSC-20220101T000000Z.mp3 but FFmpeg instead wrote KZSC-20210101T000000Z.mp3, i.e. it didn’t advance the year part of the file name. ◔_◔

Our software that handles the finished files, copying them to the HLS server, and trans-mux them to .mp4 format, doesn’t concern itself with these names so it didn’t notice the problem, it just used the year-old file names written by FFmpeg. But the software that maps archive player requests to HLS playlists does scan and interpret the timestamps in the file names. That software hasn’t been able to find files recorded this (UTC) year.

So I modified the file processing software to rename new January 2021 files to 2022, renamed all the existing incorrectly named files and flushed all the HLS mapper and server caches. This seems to have restored the missing archives. I’ll need to keep an eye on what FFmpeg does going forwards.


* The Z refers to the zero-hour time zone, often spoken as Zulu time, because Zulu is the NATO phonetic alphabet for Z. But this is a good moment to listen again to the fine album by Caspar Brötzmann and Page Hamilton called Zulutime.

a drag for you that this happened on a holiday - no rest for the weary sigh

And no peace for the wicked.

I hope you enjoyed my write-up because I think the FFmpeg bug is hilarious. (ง ͡ʘ ͜ʖ ͡ʘ)ง