Reduce VoIP incidents and costs

Customer sector

Car Manufacturer

Technological challenge

The Customer Voice team, one of Europe's leading car manufacturers, manages a complex and heterogeneous telephony system on which nearly 100,000 users depend. This environment is based on 80 Alcatel Lucent nodes, a Cisco node managing 4,000 phones in Spain and a Lync collaboration environment with 3,000 voice users. Also, the evolution of this architecture is federated around SIP operators and the implementation of SBC and EBC Oracle.

In order to find a supervision solution that would allow them to reduce both the number of incidents and VoIP costs, they called on an integrator who offered them ServicePilot.

Monitoring solution

ServicePilot

Results post-monitoring

In a single day, we managed to deploy our VoIP monitoring solution for this international car manufacturer, covering its entire VoIP perimeter. By choosing ServicePilot, its IT teams have been able to become more proactive in detecting incidents; thus improving the experience of their many users, telephony security and thus reducing the costs of IP telephony incidents.

Implementation and Monitoring Scope

The technical scope is defined by the following multi-vendor components:

• Alcatel-Lucent OXE communication server in a spatial redundancy configuration. The requirement is to monitor all elements of the platform, including trunks, gateway, couplers, call servers and passive call servers. An additional requirement, equally important, is to collect and analyze call detail records (CDRs) in order to monitor call activity and call quality.
• Acme Packet Session Border Controllers (SBC): devices Net-Net Central in a redundant configuration, for the live system, and as standalone, for testing purposes. The requirement is to monitor the system as well as the SIP trunks that act as gateways between the Alcatel-Lucent platform, the Microsoft Lync Unified Communication platform and the carrier trunks.
• NET VX devices: our requirement for these SIP gateways is to monitor the status of the SIP trunks between the carrier and the Alcatel-Lucent solution.

Progress of the monitoring mission

First, Alcatel-Lucent and Acme Packet, are both supported right out-of-the box by our VoIP End-User Monitoring Solution.

Alcatel Lucent VoIP Monitoring:

All that is required is to import the 'Alcatel VoIP Portal' and the 'Alcatel OXE Monitoring' built-in packages, configuring the IP addresses, the SNMP community, the user credentials for the interactive session and the collection method for the retrieval of CDRs. I personally prefer the so called 'ticket on the fly over Ethernet' method, as it allows gathering call data more rapidly than via FTP, where the buffering time is up 1 hour. For this proof-of-concept time is of the essence!

Acme Packet VoIP Monitoring:

All that is required is to import the 'Acme Packet VoIP Portal' package and the 'Acme Packet Monitoring' packages, configuring the IP addresses, the SNMP community and the collection method for the retrieval of CDRs. I take this opportunity to praise the quality of the Acme Packet MIBs, which are very comprehensive and allow the retrieval of plenty of information about the operating status as well as the monitoring of the trunks – it is such a pleasure to use them! The customer demands a minute-by-minute monitoring in order to assess the load of the carrier trunks and of the trunks internal to the OXE. First, I show them the addition and setup of the packages, then, through the automatic discovery of the other network elements, I explain that it is possible to set thresholds for performance alarms, for example for the purpose of detecting heavy usage conditions for the carrier trunks. I also take the opportunity to point out that these measured values will allow us to take advantage of our solution’s extensive reporting capabilities, which are as important as its monitoring functionality. This is exactly what grabbed the customer’s attention, precisely because they had not thought about bringing up the subject of reporting for the purpose of this proof-of-concept. Why not, after all?

NET VX devices:

That's the part that took a little time to implement, because we didn't have a package built into the product available at the time. Fortunately, we had already studied this technology internally. Here is the "recipe" applied in this case for the creation of a new supervision package:

• We gather the MIBs and check which ones respond to a MIB browser
• We identify the parts relevant to this project: System (CPU and memory), physical status of the network interfaces (MIB-2) and usage status of the trunks (MIB NET VX)
• Depending on the results from the previous action, we create specific classes and their associated indicators within our product
• We configure thresholds for availability and performance-related alarms
• We set up tables and charts for the reporting
• Test phases and iterations for KPI adjustments
• And hey presto, the package is done!
• After the completion of the steps above, all the devices are added to the configuration and the monitoring is started. I test the configuration one last time and I then move on to the reporting

Automatic Morning Checks Reporting

Time is running out so I need to focus on the essential points: provide measurements on the usage of the carrier SIP and IP trunks, as well as on the call quality, by means of tables, top-bottom rankings and capacity planning. I rely exclusively on the web interface to prepare the report: for each paragraph I select the desired functionality (tables, top-bottom rankings or capacity planning) and the equipment class to apply it to. Then, for the automatic creation of the reports, I select the frequency (e.g. weekly, monthly or quarterly) and the distribution list for the emails, if required.

Looking back on the deployment

• The whole deployment is completed before lunch time!
• The afternoon is then devoted to the presentation and customization of the web interface (user-side interface), e.g. how to access and present all the available data.
• We take advantage of this time to re-assess one by one the points stated as part of the mission statement, and we then go separate ways with the plan to reconvene in a month.
• It was a very productive day!
• The deployment of our product progressed very well, because the data gathering and the analysis of the customer requirements had been thoroughly carried out in advance. What I personally appreciate the most, when implementing a proof-of-concept, is to be able to show ServicePilot’s ease of installation and configuration, by focusing on the technical perimeter chosen by the customer. This is precisely what helps directing the customer’s attention on the product itself, and specifically on the user-friendliness of the web interface. After all, it is in the customer’s hands that the product will ultimately prove its worth.

User zoom and the benefits of supervision

After only a few months of using the ServicePilot supervision software, here are the results that the customer's voice teams were able to achieve:

1) Reduce the number of telecom incidents

Accelerated resolution of existing incidents

Before they started using ServicePilot, and because of the complexity of their telephony environment, the Voice team had a hard time finding the cause of VoIP quality problems. Indeed, they could not correlate the availability and performance of different brand elements. ServicePilot is capable of collecting data from Alcatel-Lucent, Cisco, Skype Enterprise, Oracle and other technologies, giving them access to key indicators such as :

• Network equipment utilization rate (CPU, memory, etc.)
• SIP and SBC availability and performance
• Call quality (MOS, jitter, latency and more) and call activity (incoming, outgoing, internal calls) over time

This brings them the following benefits:

__-Although the members of the Voice of the Customer team are each specialists in a particular technology, the simplicity of ServicePilot's dashboards allows them to carry out a complete and rapid diagnosis, even on equipment of a technology of which they are not experts. __
- They can now monitor their entire telephony perimeter from a single software package, and track call degradation to their cause by seamlessly switching from one technology to another.

Anticipate more than 20% of future incidents

However, resolving existing incidents is not enough to ensure optimal service quality. Before using ServicePilot, the Voice team did not have a capacity planning tool to predict the future performance of its infrastructure elements. With ServicePilot, they can anticipate saturations up to 3 months in advance, thanks to projections calculated on the basis of historical data for each resource. The customer in question estimated that he reduced the total number of incidents by almost 20% thanks to this feature.

2) Improve detection of security incidents

Although SIP is used by many large enterprises, and today considered one of the most widespread VoIP signalling protocols, it has security weaknesses that leave its users at the mercy of hackers. Before using ServicePilot, our customer had already been the victim of several attacks that had resulted in significant financial losses (abnormally high telephone bills and indirect costs related to the interruption of VoIP service).

The ServicePilot solution, in addition to providing the company with global visibility of its infrastructure and greater responsiveness to resolve incidents, has enabled it to block and prevent this type of attack.

Collection of SIP traffic from SBCs

First and foremost, it is important to know that ServicePilot collects indicators from all of the customer's infrastructure equipment (Alcatel-Lucent, Cisco, Lync, Oracle, Sonus). In particular, our supervision solution collects, on SBCs and PBXs, CDR information concerning the volume of calls made.

Setting zones and alerts

In addition, the customer was able to configure the tool so that its data is displayed by geographic area, and create alerts with customized thresholds, to detect abnormal changes in call volume.

Detect attacks with custom alerts and reports

It was after deploying ServicePilot that the Customer Voice team received an alert indicating that one of its SBCs was experiencing a sudden increase in outgoing international call volume (see the "London" area below). Indeed, hundreds of calls were being made from an unusual IP address and passing through one of the group's Oracle SBCs.

Blocking the attack and preventing future fraud

It is thanks to this precise diagnosis that the Voice team was able to modify the configuration of the SBC in question in order to block the IP from which the attack came.

3) Reduce VoIP costs and telephone incidents

Fewer VoIP support tickets mean lower VoIP costs

Above all, the ability to prevent and diagnose incidents more quickly has reduced the company's Mean Time to Recovery and the workload on its Voice and Help Desk teams. As a result, the customer saw an automatic decrease in the total cost of resolving incidents, as well as the costs associated with degradations.

SIP trunk costs halved

The capacity planning functionality described above has also resulted in significant savings for the Voice team. Indeed, the analysis of the future use of its trunks has led to a halving of the number and cost of subscriptions to its SIP channels.

Prevention of financial loss

In the past, the Voice team's telephone system has been the victim of fraudulent attacks. Unfortunately, without the right tools, it was difficult to detect, block and prevent attacks. Shortly after deploying ServicePilot, the customer received an alert indicating that one of his SBCs was experiencing a sudden increase in outgoing international call volume (see "London" area 2 screenshots above).

Indeed, hundreds of calls were being made from an unusual IP address and passing through one of the group's Oracle SBCs. ServicePilot transmitted this information thanks to its configuration based on the collection of SIP traffic on the group's SBCs, on alerts with customized thresholds and on the display of call volume by geographic zone.

It is thanks to this precise diagnosis that the Voice team was able to modify the configuration of the SBC in question in order to block the IP from which the attack came.

Increase team productivity to reduce costs

ServicePilot allowed the customer to create a personalized dashboard called "Morning Check" that gives them a clear daily view of incidents requiring their attention. In addition to facilitating decision making, these dashboards can be generated as PDF reports sent to local technical teams. According to the Director of Telecommunications of the client:

"We are quickly familiar with the ServicePilot solution, and are now able to create dashboards and reports that respond to each new problem encountered, in addition to supervising new equipment. Moreover, despite the complexity of our environment and the large number of users of our system, ServicePilot requires only one hour of maintenance per week."

Conclusion and results of the ServicePilot solution

Using ServicePilot, the customer, a major European car manufacturer, benefited from the following:

• Acceleration of incident diagnostics
• Proactive resolution of 20% of incidents
• Automatic reduction of costs caused by incidents and related to their resolution
• SIP trunk costs cut in half
• VoIP attack detection and associated financial loss prevention
• Increase productivity of Voice and Help Desk teams
• By reacting quickly with ServicePilot, the customer was able to:
  • Limit the financial impact of this attack and protect your system from future attacks from the hacker
  • Avoid service interruptions affecting the service provided to system users