Services : Case Studies

Liberty Conducts Telecommunications Performance Measures and Service Quality Audits for Several State Commissions

Liberty completed a review of Verizon New Jersey’s monthly performance reports and associated incentive plan payment reports for the New Jersey Board of Public Utilities. The Verizon New Jersey audit addressed roughly 50 metrics (which encompassed approximately 750 separate reported results) in the areas of pre-ordering, ordering, provisioning, maintenance and repair, network performance, operator services and databases, billing, and general.

Liberty conducted a comprehensive review of Verizon’s reporting integrity to assess whether its data generation, collection, analysis, retention and reporting were sound, accurate, complete, and complied with relevant plans, guidelines and Board orders. Some of the specific questions that Liberty’s audit sought to answer included:

Whether Verizon’s interpretation of language in the performance measure guidelines, including allowable exclusions, is reasonable, and whether it is accurately reflected in its metrics calculations
Whether complete and accurate documentation exists for performance measure calculations, and whether it faithfully reflects the performance standards and Board orders
Whether adequate and up-to-date supporting documentation exists for source data storage, backup, retrieval, and extraction, and whether the performance measurement process starts with complete and accurate data
Whether change control procedures are reasonable and fully implemented
Whether “parity by design” measures are in fact “parity by design”
Whether Verizon accurately reports its incentive plan results, makes correct determinations of payments and severity levels, and uses correct calculation and statistical techniques.

Liberty also conducted audits of Verizon’s Performance Assurance Plan (PAP) on behalf of the regulatory commissions of Virginia, Maryland, and the District of Columbia. The commissions designed the PAP, which consists of both performance metrics and methods to determine billing credits, to help ensure that Verizon provides quality wholesale services to competitive carriers after it gained entry into the long distance market under Section 271 of the Telecommunications Act of 1996.

Performance measures or metrics and incentive or performance assurance plans are essential tools for regulators concerned with monitoring the competitive local exchange marketplace. Performance measures in such areas as ordering, provisioning, maintenance and repair, and billing provide a method for regulators to correlate an incumbent local exchange carrier’s (ILEC) performance between its wholesale and retail services. Performance measure results allow regulators to monitor whether there is a level playing field between the ILEC and competitive local exchange carriers (CLECs). Performance assurance plans allow regulators to provide remedies to CLECs or incentives to ILECs to encourage ILECs to satisfy their commitments regarding the provision of service to CLECs.

Virginia and other states have recently adopted a revised set of performance measurement guidelines, based on those adopted by the New York Commission. Rather than proscribing a given set of activities for this audit, the commissions have asked Liberty to instead design the audit to capitalize on the knowledge that Liberty has already gained in its audit of Verizon for the New Jersey Board.

During 2001 and 2002, Liberty conducted a Performance Measures Audit for the Regional Oversight Committee (ROC), which consists of commissioners, commission staff, and stakeholders with responsibility for or substantial interests in the local exchange operations of US WEST (now Qwest) across the 14-state region in which this local exchange carrier provides such service. Thirteen of those 14 state commissions participated in this audit. The Arizona Commission conducted a separate audit of performance measures; however it later asked Liberty to include its state in subsequent Qwest data reconciliation work.

The Qwest audit specifically addressed a series of more than 50 indicators (which, with sub-measures, impose several hundred quantified performance standards) that measure Qwest’s performance for CLECs. These measures, among the country’s most detailed and quantitative, govern nearly every measurable aspect of Qwest’s performance. Measures included in the audit covered the areas of electronic gateway availability, pre-ordering, ordering and provisioning, maintenance and repair, billing, database updates (including E911), directory assistance, network performance, and collocation.

Liberty’ audit included the three following principal audit task areas:

Examining the systems, processes, and procedures by which Qwest measures and records performance results
Conducting detailed tracking of the data Qwest uses to measure performance (to verify accuracy, completeness, and use in actual measurements of performance results)
Replicating Qwest’s calculations of performance results to verify their accuracy and completeness.

As part of its continuing work after the audit, Liberty conducted extensive data reconciliation work using data provided by CLECs operating in the Qwest region. Liberty geared this data reconciliation effort to determining whether there were reasonable assurances that the performance as measured and reported by Qwest was equivalent to the performance that Qwest actually delivered.

During 2001, Liberty conducted an investigation of Ameritech-Ohio service delivery quality for the Public Utilities Commission of Ohio. This investigation addressed aspects of two related, yet distinct, elements of Ameritech-Ohio’s service quality: (a) how well Ameritech managed the performance of the work activities that it takes to deliver that service, and (b) how well Ameritech measured the quality of its service. Liberty examined how Ameritech-Ohio managed Ohio Minimum Telephone Service Standard (MTSS) performance and compliance, how it managed MTSS record keeping and provision of credits to customers for poor performance, and how it complied with selected marketing and disclosure requirements in communications with its customers.

As part of this engagement, Liberty provided its assessment of Ameritech-Ohio’s installation and repair operations and processes. Liberty also reviewed the policies, procedures, and work practices of Ameritech-Ohio’s call center management to better understand how Ameritech handled Ohio customer calls and how it had trained, coached, and prepared its workforce to deliver Ohio Commission disclosures. Liberty conducted an extensive investigation into the methods used by the company to calculated its MTSS performance measures and service quality benchmarks (required under a merger stipulation), as well as the credits provided to customers for missed installation and repair appointments or lapsed service.