T41S: severe audio interruptions

[Anatoli]

Hi,

We purchased a single T41S for evaluation of the new T4S line, configured it with FreeSWITCH and experiencing strange intermittent audio interruptions.

We made 2 recordings: r1 (link) is when the phone experiences extremely server audio interruptions playing music on hold (in the background there is an iPhone with Linphone reproducing the same channel, connected to the same PBX so you hear that neither the audio streaming from the PBX, nor the network connectivity is the problem).

r2 (link) is a second call just after finishing the first one (r1), nothing changed, iPhone continues on the channel. As you can appreciate, there are almost no interruptions (just 2: one at 00:17, the other at 00:53).

The Yealink phone is connected by cable directly to the machine running FreeSWITCH BPX, no firewalls in between, same /24 network, the latency between them is less than 1ms with very low jitter.

We tried changing jitter buffer, codecs (G722 was used during the recordings, but Opus behaves the same). Tried a lot of other settings, tried to check the log files at level 6, network dumps. Nothing special appears that would clearly identify the cause.


Another, probably related, issue is that Yealink constantly lags audio some milliseconds (like 15-30) behind other devices. On the recordings you probably will hear that the audio in the background (from Linphone on iPhone which is connected via WiFi with some 25ms of network latency) goes some 15-30ms ahead of the main sound (from Yealink). Same happens during calls. So it appears that Yealink itself adds some 30-60s of latency.

What could be wrong?

Firmware: 66.81.0.110
Hardware Version: 68.0.0.0.0.0.0

Current config is attached.

[Kevin_Yealink]

(08-14-2017 03:43 PM)Anatoli Wrote:  Hi,

We purchased a single T41S for evaluation of the new T4S line, configured it with FreeSWITCH and experiencing strange intermittent audio interruptions.

We made 2 recordings: r1 (link) is when the phone experiences extremely server audio interruptions playing music on hold (in the background there is an iPhone with Linphone reproducing the same channel, connected to the same PBX so you hear that neither the audio streaming from the PBX, nor the network connectivity is the problem).

r2 (link) is a second call just after finishing the first one (r1), nothing changed, iPhone continues on the channel. As you can appreciate, there are almost no interruptions (just 2: one at 00:17, the other at 00:53).

The Yealink phone is connected by cable directly to the machine running FreeSWITCH BPX, no firewalls in between, same /24 network, the latency between them is less than 1ms with very low jitter.

We tried changing jitter buffer, codecs (G722 was used during the recordings, but Opus behaves the same). Tried a lot of other settings, tried to check the log files at level 6, network dumps. Nothing special appears that would clearly identify the cause.


Another, probably related, issue is that Yealink constantly lags audio some milliseconds (like 15-30) behind other devices. On the recordings you probably will hear that the audio in the background (from Linphone on iPhone which is connected via WiFi with some 25ms of network latency) goes some 15-30ms ahead of the main sound (from Yealink). Same happens during calls. So it appears that Yealink itself adds some 30-60s of latency.

What could be wrong?

Firmware: 66.81.0.110
Hardware Version: 68.0.0.0.0.0.0

Current config is attached.

Hi

Thanks for your information. I will did my best to help you fix this issue soon.

I need to check with you for more detailed information:
1. Audio issue only happened on MOH?
If A call B, will Audio issue happened?
2. Do you have other yealink phone or other brand? Can they work well?
3. To check this issue, i need PCAP file and syslog file. So please help refer this FAQ: http://support.yealink.com/faq/faqInfo?id=311
4. May i know how many phones will you need If yealink phone pass testing.

BR
Kevin

[Anatoli]

Hi Kevin,

Thanks for following up the issue.

Answering your questions:

1. The audio issues happen in any type of communications: A>BPX, A>B, A<B, Opus/G722 (A/B/AB), MOH/speech/music. We use music to perceive better the interruptions as it's a continuous reproduction and playing MOH directly from the BPX excludes any quality degradations caused by the other endpoint, but human speech with another phone has the same problems.

After some extensive tests simulating different environment conditions (and rebooting the device multiple times) we were not able to consistently reproduce the more severe interruptions. It looks like there were 2 issues: some strange interruptions that could last for 3-4 seconds or more and repeat very frequently (and sporadically, like in the first post where a second call had no such issues) and small interruptions due to environment issues (network delays, packets reordering, etc.).

We came to a conclusion that the small interruptions issue is due to deviations of the packets order/timing from the absolute perfect condition. It looks like our Yealink phone is extremely sensitive to any deviation in order or latency, it doesn't compensate for this imperfections with adaptive jitter buffer (was configured with "adaptive,20,200,20"). At the same time, Linphone was able to "eat" most of the environment issues without audible defects or with light sound distortions.

At the same time, for unknown reason, Yealink phone was always behind Linphone in audio reproduction, sometimes with a delay of up to 300-500ms, most of the time around 80-100ms (subjective judgment). One could guess that this could be an extremely large jitter buffer, but the small interruptions were happening with this delay too.

When enabled Settings > Voice monitoring > Display Report options on phone, there were strange statistics. JitterBufferMax, Packets lost and RoundTripDelay were always at 0, but Jitter had strange numbers: it could stay at 0 when there really was jitter, after one reboot it stayed constantly at 950-960 without dropping to normal values though there was no such jitter. At other times it was varying from 0 to 400, but not consistently, i.e. when there was jitter, the value sometimes was staying without change, at other times it was following the real jitter pattern. At the same time Linphone was changing its jitter buffer from 0 to 90ms, staying most of the time at around 40ms (via WiFi).

We got an impression that there was no jitter buffer enabled at all or it was discarding buffered data too quickly as the phone was not able to compensate for most deviations. I'm not sure if this is a standard behavior or we got a defective unit, or there are some options we're not aware of that deal with this sort of issues.


2. We don't have another Yealink or any other (non-Cisco) hardphone at hand, but were using multiple smartphones (iOS/Android) and PCs (Win/Mac/Linux) with Linphone, WiFi and cable connections. Everything works well in any combination, except the Yealink. Even tried disconnecting Yealink's cable and connecting it to a PC with Linphone.

3. Syslog and PCAP files attached: yealink.pcap is a pcap generated by the phone for about a minute of a call to PBX with MOH, fs.pcap is a pcap generated at the Linux machine running FreeSWITCH BPX almost at the same time as yealink.pcap (was started some seconds before and was turned off some seconds after). We generated varying CPU load at the PBX machine during this recording to cause some jitter and got a lot of small interruptions on Yealink. During this recording the same channel was reproducing on Linphone without interruptions (there were 2-3 very light sound distortions for about half a second).

4. Depending on the demand from our clients, we expect that we could purchase some 20-30 phones this year and more in the future. We're currently using Cisco 7900 + chan_sccp_b + Asterisk with G722 and would like to setup new deployments with Yealink T4S + FreeSWITCH using Opus (and, according to our clients' decision, possibly migrate existing deployments).

We really like the new T4S line for the features, ease of configuration (compared to the Cisco phones) and its look&feel, but we can't put it in production with this performance.

Please let me know if you need additional details.

Regards,
Anatoli

[Kevin_Yealink]

(08-18-2017 05:49 PM)Anatoli Wrote:  Hi Kevin,

Thanks for following up the issue.

Answering your questions:

1. The audio issues happen in any type of communications: A>BPX, A>B, A<B, Opus/G722 (A/B/AB), MOH/speech/music. We use music to perceive better the interruptions as it's a continuous reproduction and playing MOH directly from the BPX excludes any quality degradations caused by the other endpoint, but human speech with another phone has the same problems.

After some extensive tests simulating different environment conditions (and rebooting the device multiple times) we were not able to consistently reproduce the more severe interruptions. It looks like there were 2 issues: some strange interruptions that could last for 3-4 seconds or more and repeat very frequently (and sporadically, like in the first post where a second call had no such issues) and small interruptions due to environment issues (network delays, packets reordering, etc.).

We came to a conclusion that the small interruptions issue is due to deviations of the packets order/timing from the absolute perfect condition. It looks like our Yealink phone is extremely sensitive to any deviation in order or latency, it doesn't compensate for this imperfections with adaptive jitter buffer (was configured with "adaptive,20,200,20"). At the same time, Linphone was able to "eat" most of the environment issues without audible defects or with light sound distortions.

At the same time, for unknown reason, Yealink phone was always behind Linphone in audio reproduction, sometimes with a delay of up to 300-500ms, most of the time around 80-100ms (subjective judgment). One could guess that this could be an extremely large jitter buffer, but the small interruptions were happening with this delay too.

When enabled Settings > Voice monitoring > Display Report options on phone, there were strange statistics. JitterBufferMax, Packets lost and RoundTripDelay were always at 0, but Jitter had strange numbers: it could stay at 0 when there really was jitter, after one reboot it stayed constantly at 950-960 without dropping to normal values though there was no such jitter. At other times it was varying from 0 to 400, but not consistently, i.e. when there was jitter, the value sometimes was staying without change, at other times it was following the real jitter pattern. At the same time Linphone was changing its jitter buffer from 0 to 90ms, staying most of the time at around 40ms (via WiFi).

We got an impression that there was no jitter buffer enabled at all or it was discarding buffered data too quickly as the phone was not able to compensate for most deviations. I'm not sure if this is a standard behavior or we got a defective unit, or there are some options we're not aware of that deal with this sort of issues.


2. We don't have another Yealink or any other (non-Cisco) hardphone at hand, but were using multiple smartphones (iOS/Android) and PCs (Win/Mac/Linux) with Linphone, WiFi and cable connections. Everything works well in any combination, except the Yealink. Even tried disconnecting Yealink's cable and connecting it to a PC with Linphone.

3. Syslog and PCAP files attached: yealink.pcap is a pcap generated by the phone for about a minute of a call to PBX with MOH, fs.pcap is a pcap generated at the Linux machine running FreeSWITCH BPX almost at the same time as yealink.pcap (was started some seconds before and was turned off some seconds after). We generated varying CPU load at the PBX machine during this recording to cause some jitter and got a lot of small interruptions on Yealink. During this recording the same channel was reproducing on Linphone without interruptions (there were 2-3 very light sound distortions for about half a second).

4. Depending on the demand from our clients, we expect that we could purchase some 20-30 phones this year and more in the future. We're currently using Cisco 7900 + chan_sccp_b + Asterisk with G722 and would like to setup new deployments with Yealink T4S + FreeSWITCH using Opus (and, according to our clients' decision, possibly migrate existing deployments).

We really like the new T4S line for the features, ease of configuration (compared to the Cisco phones) and its look&feel, but we can't put it in production with this performance.

Please let me know if you need additional details.

Regards,
Anatoli

Hi Anatoli

Sorry for late reply. I miss this forum as i don't receive reply nitification. Really sorry about this.

For this issue, i already submit it to R&D, and i will give you update.
I will also send you email by my private email. Maybe we can follow this case via email system.

Best Regards,
Kevin