Grading VoIP Call Quality

We've been doing a lot of testing of our Voice-over-IP systems lately. We experiences some pretty horrendous voice quality in a series of high-profile conference calls last week, and are looking to ferret out the root causes and fix the problem.

One of the questions we've had to ask ourselves is, "How do you measure and record the audio quality of a call?" There are some pretty sophisticated methodologies and grading scales out there for assessing call quality, but everything I can find seems to be overkill for what we're trying to do -- test various configurations, record the results, and be able to look at the results and choose the configurations that work best.

Since I couldn't find a system that was simple enough to implement very quickly, we started assigning letter grades to the calls we were listening to. After a few calls, some patterns emerged and we were able to clearly state why we thought each call deserved a particular grade. Here's what we came up with.

Grade Call Characteristics User's Response
A No problems, Clear natural sound. I didn't even notice the audio quality.
B Some minor delays or tinny unnatural sound. It sounded good enough.
C Delays, tinny unnatural sound, minor dropouts, or a combination of minor problems. The sound was annoying.
D Barely intelligible, very unnatural sound, "underwater," periodic dropouts. We had to work to have a conversation.
F UnintelligibleWe were not able to communicate.

VOIP Testing

To grade calls, we rigged a handset over the speaker on a laptop and played a sample from an audio book. We chose a recording where the narrator uses as steady cadence and low dynamic range, the version of Malcom Gladwell's Tipping Point. This worked much better than having someone talk into the phone. (You really can't assess quality very well if all the other person is saying is "Can you hear me now?")

And finally, here's a table showing call quality degrading as we add callers. We're routing each call out and back over a T1 line using the uncompressed G.711 codec. We graded audio quality on both the conference call as well as a second call between two endpoints over the same T1.

Callers to Conference G.711 Channels Routed across T1 Conference Quality Other Call Quality
6 14 A A
716A A
8 18 B (some delays) A
9 20 D (periodic dropouts) F (unintelligible)

I was pleased that the results of our test are aligned with these figures posted on the wiki. The 1.54 Mbps capacity of our T1 divided by the Nominal Ethernet Bandwidth of a G.711 SIP channel (87.2Kbps) gives a maximum of 18 simultaneous calls, which is exactly how many we were able to place before things broke down.

Geek side note: A T1 carrying regular voice calls on for the phone company, or PRI, uses the G.711 audio format, but without the overhead for IP headers -- a PRI uses time division to split the line into channels -- phone companies are able to fit 24 channels on a single line (64Kbps x 24 = 1.54Mbps.) Usually, 23 are used for voice and the 24th is used for data like Caller ID information.

In the next week or so we'll be testing other voice codecs. We're leaning toward G.729. Although it has a license cost of $10 per simultaneous channel, it promises good call quality and 40+ channels on a T1.

blog comments powered by Disqus

Posted in Tech Notes
on April 14, 2009