When I went to investigate the problem yesterday, I found that the ffmpeg process for KCSM was running but not writing any archive files. I donât know what caused that. Part of my job now is to figure out how to monitor the processes so as to understand these failure modes and either prevent them or automatically recover from them.
A little problem with WOWD/Takoma Radio archives. Weâre noticing that during some of our archived episodes, there are 2-3 minute drop-outs ⊠For example, on my own Tuesday Morning Mix (Nov. 10), this happens at about 8:10 AM. The song Iâm playing â18 With a Bulletâ ends abruptly ⊠my entire mic break is missing ⊠and it jumps into the middle of the next song (âAt Seventeenâ).
Another programmer discovered a similar glitch on her Nov. 10 Blues Come Calling. It happened twice during her 2-hour show. First at around 6:49 pm at the end of the Meade Lux Lewis song it skipped the next song by Archie Edwards and jumped to the middle of the following song by Louisiana Red. And around 7:20 pm - it skips from near the end of the second Blind Willie McTell song (Wee Midnight Hours) to the middle of the second Barbecue Bob song (Waycross Georgia), completely skipping My Mistake Blues.
Yet another programmer notes that the first song from her Nov. 8 Night Ride Home is completely missing. She played a Prince song to open the show at 8:00 PM but the archived episode starts a few minutes late, during her first mic break.
When we previously had a somewhat similar glitch, you thought it might have to do with our sending you an AAC stream rather than an mp3 stream as the other beta-testing stations are doing. Could that be the problem here as well?
If this is not easily fixable, I will ask my IT guys to switch our feed to you, we could send you an mp3 feed as we used to do for Radio Free America.
Sorry to be messenger bearing news of a bug, but âŠ!
Our IT guy Matt Toigo notes that we have our mp3 stream running (the one we used to send to RFA), so all youâd have to do is use this URL â https://live2.takomaradio.org/stream-mp3 â instead of the ACC one you currently use. (Let us know if weâd have to do anything on our end.)
-Steve H.
I think I identified the problem, @Steve_from_WOWD, and added a fix to prevent it happening again. I try to monitor it for the next few days. Let me know if it recurs.
In addition, I attempted to fix the problem in the existing archives. Separate fix for the prior effects of the bug.
I canât get the ark player to play on either of m;y two browsers, vivaldi or chrome. it used to work fine. Java is enabled in both. It flashes with no info and just hangs. I hadnât tried it in a while so I donât know how long this has been an issue.
-Judy
See this?
Hi Judy. Itâs working for me for a couple of random recent dates and times I tried. Can you direct me either to the links or dates and times you tried that filed?
Hey Tom -
So I tried a lot of different files now and I see that some work. It may just be files in the new year? The one I first tried was Democracy now on Friday. Hereâs 5pm on Friday, it wonât open
When I try older files they seem to work - Thanks- Judy
Afaict, the archives for all stations stopped working for all program dates starting at midnight UTC at the start of the new year. I think we identified the problem and got the system working again for the new year.
Itâs very strange.
We use software called FFmpeg to connect to the streams and write audio files (usually MP3 files) with copies of the stream audio. FFmpeg segments the stream audio into chunks of a preset duration, one chunk per file. FFmpeg names each file with its timestamp, for example the file for 21:00 UTC yesterday is named
WZBC-20210101T210000Z.mp3
The bug is kinda Y2Kish. After midnight UTC at the new year rang (in UK, Portugal, etc. where they are on UTC) FFmpeg started names the files with WZBC-2020⊠instead of WZBC-2120âŠ
I tried restarting all the FFmpeg processes but that didnât fix the names of files it was writing. So I wrote a script that renames the files from 2020 to 2021 as they are copied to the AWS S3 where they live for two weeks so Ark players can read them. And I wrote another one to rename about 17 thousand files that were already in S3.
Thatâs all now done and I think weâve recovered everyoneâs missing archives. But now we need a more permanent solution to our problem, i.e. fixing the problem in FFmpeg.
I have been listening to David Lynch for an hour now and I love it. There is no where else like KPoo. My ex husband worked for you in the in the 1980ies I think and you are always button one on my car radio. I have been sending you money to keep you around. 100.00 12/17/20 and 100.00 9/6 /20. Keep up the good weird work.
1 Like
More feedback on the Ark archive player on mobile apps:
-Tried Ark archive player (playing from WRFI, Ithaca NY) on iPhone 7 iOS 14.3 using Safari. Every time the screen went to sleep, the archive player would stop. When you woke up the screen, you could restart the player, and it would work fine. Until the screen went to sleep again, and the archive player stopped.
-Then tried the Ark archive player (same station) on an Android Moto g6 using Android 9 and Chrome. Ark archive player worked fine, even when screen went to sleep. But then after about 5 minutes, the archive player stopped making noise. When I woke up the screen, the archive player still looked like it was playing (i.e. time was passing in the counter and the play button indicated it was playing), but no sound came out. Tried pause, unpause, but no success.
I saw reports of both of these behaviors above in the comments, so just confirming that this is still an issue.
1 Like
Thanks for the report Peter.
It is a sad and vexing matter. I have a concept for a new architecture and implementation of an archive service that I think can be made to work on mobile and can be legal too. But itâs quite a lot of work and all Iâve managed to do so far is run some experiments to validate the ideas. If anyone wants to contribute to the development, Iâm happy to open the project to them.
1 Like
Hey Tom -
I was listening to some ZBC archives today and noticed the new ark player and followed the link to this forum. Iâm a programmer and happy to contribute how ever I can. See my forum profile for github link.
@newtnik Hi Jon, Thanks for the offer. Can we try to set up a call so I can explain where things are and we can discuss what we could do together? My email is tom@spinitron.com
My iPhone 6s listening to kmuz archive on safari 5:50pm pacific time March 9.
The player stops playing on dark screen or if I leave spinitron pages to any other page or app.
This is not useful if it wonât play on dark screen!! Any tips on that? My other players work fine.
This is what I wanted to say tooâŠ
Thereâs a tune called Beyond the Tagus River that was played on Vetaâs program Bluegrass and Beyond where the art cover is not showned, Andre Dal is misspelled, itâs not possible to listen to the tune when you clik on it and doesnât go to apple or spotify or any other when you click on the box. Why?
Hi Marc, thanks for joining the forum.
Itâs a known problem. And we understand how big a drawback it is.
Let me give a rundown on where we are, how we got here and what might happen next.
Radio Free America (RFA) was providing a free two week archive that lots of Spinitron stations were using. We thought it was a pretty good service for $0 and easy for stations using Spinitron to try so we often mentioned it to stations. But it is in the nature of that kind of business that if they cease operations they will do so without warning and that worried us. We would feel bad if stations we had pointed their way were left hanging overnight.
When that happened we quickly moved to fill the gap. I figured we could quickly set up a system that connects to stationâs webcast streams, automatically writes the digital audio data in the streams to audio files, stores those in the cloud and to provide a web-based archive player that uses those files to play back from an arbitrary point in the last two weeks.
And I was right, we got that up and running quickly. But we soon found that on mobile the web player would continue running only if the mobile deviceâs screen stays on and the browser tab itâs in stays in the foreground. This is, as you pointed out, a severe limitation and we started to work on fixing it as though it were a bug. But it eventually became clear that fixing this bug would require redesigning the whole system using different web-audio technology. That was going to take considerable time, way more than we had planned and more than stations who had lost RFA should have to wait. So we decided to offer what we had commercially at a low price with a clear notice about the limitation on mobile.
We were unable to quickly fix the mobile issue is because of a malign combination of legal and technical constraints. As a service provider we have to make it unfeasible for a user to download a copy of the stationâs audio without resorting to obvious hacking and violation of the terms of service. So we segmented the audio into 5 minute chunks, each stored in one file. The audio player downloads these files on the fly and plays them back to back. To download, say, an hour of programming, a user would have to hack the system to download twelve such chunks and then stitch them together to have an hour of seamless audio.
And this is where the problem on mobile arises. The archive web player uses JavaScript running in a web browser to download the chunks and start playing them at the right time. When a browser tab goes into the background, or the user switches to a different app, or the screen turns off, the browser stops running the playerâs JavaScript. Thereâs nothing we can do to prevent that.
I can see two ways out of this. One is to use long audio files, so long, maybe an hour or more, that the user doesnât mind reopening the browser tab to restart the archive playback when it stops. Weâre not prepared to do this for legal reasons. A station could reasonably operate such a service for themselves since I believe their liability would be acceptable. But itâs different for a commercial service provider like Spinitron. Litigation for us would likely be the end for Spinitron and our livelihood.
The other is to use a streaming video protocol such as DASH or HLS. Iâve done enough experiments with HLS to be fairly confident that it can be made to work. I can specify how such a system can work but I havenât had time to build it.
1 Like
Hi Andre, thanks for asking your question on the forum.
I suspect the problem is what you said: Andre Dal is misspelled. That sort of data entry error can prevent Spinitron and the external services we connect to (Apple Music, Amazon, âŠ) from identifying the recording. That prevents us from finding the cover art, Apple from providing the 30 second preview, etc.
We have put more effort into features to avoid/correct this kind of data-entry error than into any of Spinitronâs other features. Auto-completion, suggestions, Did-you-mean? search among other other tools and nudges. But if, after all that, Spinitron still cannot connect a spin to a recording in a trusted reference database, we do not force a correction on the DJ.
I suggest you ask the DJ that played the song to make the correction. Iâm happy to help or even do it myself but I need the DJâs request.
So youâre a bluegrass banjo player in Portugal, eh? How cool is that! Iâm enjoying this recording and video for the second time now. Itâs lovely!
