Robust Assessment of Changes in Cellular Networks

Presenter: Ajay Mahimkar
Authors:  Ajay Mahimkar (AT&T Labs - Research), Zihui Ge (AT&T Labs - Research), Jennifer Yates (AT&T Labs - Research), Chris Hristov (AT&T Mobility Services), Vincent Cordaro (AT&T Mobility Services), Shane Smith (AT&T Mobility Services), Jing Xu (AT&T Mobility Services), Mark Stockert (AT&T Mobility Services)

Changes in the sense of software upgrades, configuration, hardware deployment... The question is how those affect user perception of service quality - accessibility, retainability, throughput, minutes of usage, etc. But no lab can fully replicate scale, complexity and diversity of operational networks! To measure these details also need to take into account external factors - seasonal changes (leaves on trees!), weather (worst when coincide with configuration changes), traffic pattern changes, other network events such as outages...

Idea: Litmus - compare performance between study and control group:

• study group - network elements where change is implemented
• control group - network elements without the change

Going to discuss the methodology of selection of the groups. Spatial regression in study and sampled control group to learn the coefficients and compare the differences before a change and after it. Using domain knowledge to select control group: select control group subject to same external factors and sharing similar properties with study group. Geo distance, topological structure of the cell net, etc.

Evaluation: Litmus outperforms study-group only analysis because of robustness to external factors. Also outperforms Difference in Differences analysis. Some operational experiences: self optimizing network doing automated load balancing, neighbor discovery, etc. - how did it perform during hurricane Sandy. Both study and control group were impacted due to Sandy (everything went down!), but study group did better than control - faster recovery, so the feature was rolled out network-wide.

Q: There is a way to do A/B testing in offline manner if you log enough data without actually running experiments - do you think it might be applicable?
A: Definitely would be, but the question is how to select the control group - e.g. completely random selection might not work

Q: How do you identify external factors?
A: It is very hard, but with this analysis you don't need to know what external factor is there, just automatically discount their impact. But external factor identification is not plausible without additional information.