AudioSocket and CPU spikes

[daninmadison]

We use AMI to call the AudioSocket application. We discovered a CPU core spikes to 100% while the call is in AudioSockets.
I found this article discussing the issue https://github.com/NormHarrison/audiosocket_server.

He wrote "When AudioSocket is used like a channel driver, for example Dial(AudioSocket/127.0.0.1:3278/), CPU usage remains perfectly normal, but... depending on the other channel its going to be bridged with (for example, a softphone connected via SIP), the audio sent to your Audiosocket server instance will no longer be in 16-bit, 8KHz, mono LE PCM format.
Instead... It will be encoded and sent as whatever audio codec was agreed upon between the two channels. So in my experience, when a SIP softphone that uses the u-law (G.711) codec makes a call to a place in the Dialplan that eventually invokes Audiosocket, the audio you will be sent will also be in encoded as u-law, which can be both a positive and negative."

First, is there a solution in development for the CPU core spike? Obviously, this would not scale very well.
Second, is Norm correct about AudioSocket encoding using the Dial? We have some WebRTC (opec), some ulaw, and potentially some additional codecs for various channels. If the AudioSocket is encoded according to the channel's codec, wouldn't the AudioSocket server need an indication of the encoding it is receiving.

[asternic]

The audiosocket channel driver included in Asterisk18 can be configured to use slinear (code for channel driver in Asterisk is a bit different from the one in this repository). You can force slinear in this way:

 same => n,Dial(AudioSocket/127.0.0.1:3278/${uuid}/c(slin))

[NormHarrison]

I know this is quite old, but I thought it would be important to bring up again in this issue since its still open. While the codec solution above for theDial() application does remedy this issue, I was able to find the problem causing high CPU usage within the dialplan application variant of AudioSocket.

Looking at the code for app_audiosocket.c, specifically in the function audiosocket_run, there is a while loop that begins with a call to ast_waitfor_nandfds. As that function's documentation says, it waits for activity on both an Asterisk channel and standalone file descriptors, returning whichever has activity first. The documentation also incorrectly mentions the *ms argument as being an "out" parameter. In reality, the flow of data is both ways. The current behavior passes in 0, and then whatever the function places in that variable is then sent back into it the next time around. What this seems to result in is the while loop running as fast as possible, while still technically working (just working way too much).

The *ms parameter is eventually passed to the underlying OS's poll() implementation, which commonly treats 0 as "don't wait at all" and -1 as "wait forever". I modified the module to pass in -1 each time and this does indeed fix the issue.

I was considering submitting a patch to Asterisk for it, but thought it would be good to bring it to @Ulexus's attention first.

[Ulexus]

Oh, wow. Nice work!

Yeah, this definitely should definitely be submitted to Asterisk; that's where development is occurring for the Asterisk side of this project. What I have here is merely for archival reference.

I really appreciate the notice and research, though. Thanks!

[marcelwgn]

Awesome that you found a solution for this @NormHarrison ! Did you find the time to contribute a patch to the Asterisk project? If not, do you still want to or would you like someone else to contribute that patch?

[NormHarrison]

I was planning to yes, though I didn't act immediately since there were a few other aspects I wanted to fix/improve (the dialplan application variant never exiting cleanly, a few more log messages etc.). Asterisk's patch submission process was a bit more involved than I expected. Specifically, I was waiting on the contributor license agreement to be accepted, but it seems to still be pending. I noticed that a previous AudioSocket-related post in their issue tracker (related to the clean app exit) seemed to dwindle since no patch was submitted directly to Gerrit (seems to require the contributor license be signed), so I wanted to wait and do it the "right" way.

I was about to open a Jira issue anyways until I checked my email and saw this, I would certainly be open to and appreciate someone else submitting it to them for me. I will look over the changes once more to ensure they are as clean and ready-for-review as possible. Thank you.

[marcelwgn]

Totally understand you on that, the contribution process is indeed quite involved. If you want to contribute the fix, I can also help you with that, otherwise I would go ahead and contribute a fix for this in the near future.

Slight side note, is there already a Jira issue regarding the CPU usage? I tried finding it but haven't found anything that mentions the CPU usage. If not would you like to create it or is there any way I can mention you/give credit to you? After all you found the fix for this issue.

[NormHarrison]

There was not a preexisting issue regarding the CPU usage problem, I didn't think anyone else had noticed until stumbling coming across the report made here. I wasn't quite sure what the best way to go about creating the issue was, mostly in terms of keeping different problems separate. The various changes I made were done all at once to the source tree downloaded from asterisk.org (missing the git information). Would it be best to piece out each one separately or somehow bundle them all together? This is really the first time I've ever engaged in such a process so I apologize for my lack of experience.

[marcelwgn]

Ah ok, guess not many noticed this issue before or bothered reporting it upstream. If you are able to separate the changes out to their purposes, that would probably make reviewing them a lot easier for other. Otherwise, when you have a question there is a mailing list you can ask questions on or maybe even create an issue if you feel that's appropriate 🙂

[NormHarrison]

Apologies for the delay, I have created an official issue for them in their Jira instance here https://issues.asterisk.org/jira/browse/ASTERISK-30482 where the future discussion will occur.

I also asked what the best way to potentially contribute the other fixes would be as well, hopefully the inclusion process for all of them can be relatively quick and smooth.

[arthurblake]

@NormHarrison is by any chance your fix (and other improvements as well) available as a github fork for those that might want to incorporate it early?

[NormHarrison]

@NormHarrison is by any chance your fix (and other improvements as well) available as a github fork for those that might want to incorporate it early?

I didn't think of that for some reason for some reason actually... it is now available here (applied to 20.2 branch):
https://github.com/NormHarrison/asterisk/tree/20_2_audiosocket_fix

[mtryfoss]

I've pulled the official code from for this from the asterisk repo to a 16.x build. It's working fine, but after applying the code above I'm getting these randomly (using the example playback server in this repo):
"failed to get call ID: wrong message type 16"

It can be success for a number of tries, and then suddenly a bunch of these. I'm using the channel variant.
Here's a pcap of what's happening, if that's useful.

audiosocket.pcap.zip

I've not studied the protocol and/or code yet, but can it be that some audio is being transferred before the call id info?

[arthurblake]

/c(slin)

As far as I can tell, the option to force slin mentioned by @asternic above is completely missing in the Asterisk docs. In fact the docs imply that slin is the default behavior. That definitely threw me off and I thought I was dealing with a bug when I was receiving non slin audio (which as mentioned before, there is no way for the protocol to identify non-slin audio -- but that's a side issue.)

@NormHarrison if you do submit an official patch maybe you can get the doc updated at the same time.
I will try to do the same at some point if nobody else does so.

And it also leaves me wondering what else may be undocumented?

[NormHarrison]

/c(slin)

As far as I can tell, the option to force slin mentioned by @asternic above is completely missing in the Asterisk docs. In fact the docs imply that slin is the default behavior. That definitely threw me off and I thought I was dealing with a bug when I was receiving non slin audio (which as mentioned before, there is no way for the protocol to identify non-slin audio -- but that's a side issue.)

That does not surprise me too much, as the CPU usage issue itself was caused by incorrect documentation for the internal API, which of course also needs correcting too. And yes, I was also wanting to add a lot more detail to the AudioSocket protocol page on the wiki, mentioning some of the somewhat odd things we've discovered from using it and how to work around them.

@NormHarrison if you do submit an official patch maybe you can get the doc updated at the same time. I will try to do the same at some point if nobody else does so.

There hasn't been any activity within the Jirra issue I opened, and the contributor license agreement I signed which is needed to submit patches and wiki changes is still pending. However... Upon visiting Asterisk's Jira instance I see the following displayed at the top:

JIRA will be going read-only at the end of April, 2023. We will be starting fresh on Github at https://github.com/asterisk/asterisk. No issues or patches will be copied to Github. If you file an issue on JIRA at this time you will need to recreate it on Github after this date. The same applies if you have a patch. There may be a period of time when JIRA is read-only and Github is not yet available. We urge you to check back regularly on Github for when issues can be created and contributions made.

Pull requests and issues are now appearing on their Github repository (but don't have responses yet), which is no longer listed as a mirror. So it seems the timing of all this was rather bad, but hopefully once they complete the transition, the process will be easier and quicker for everyone.