Home


Project Overview

Our Publications


Personnel


Reference Shelf


Links


Data


Private


News

Distributed Network Anomaly Detection



Project Title: Distributed Network Anomaly Detection and Visualization

PI Involved: Alfred O. Hero III,
Professor, Electrical Engineering & Computer Science, University of Michigan

Student: Neal Patwari
Ph.D. 2005, Electrical Engineering & Computer Science, University of Michigan

Project Description

Data Dimension Reduction

Previously, we had tested existing manifold learning algorithms in an attempt to reduce high-dimensional data to a lower dimension while preserving inter-data distances between neighboring devices [1]. This past year, joint work with Jose Costa has resulted in a new manifold learning method called the distributed weighted multi-dimensional scaling (dwMDS) [2,3]. This new method has been demonstrated to provide an energy-efficient, distributed estimator of sensor localization which nearly achieves the Cram\'er-Rao bound in a variety of situations. A key attribute of the method is its distributed implementation - it allows nodes in a network to collaboratively estimate a global solution with minimal communication requirements. We have also demonstrated an adaptive neighbor selection method that helps to keep the estimator nearly unbiased, as shown in Figure 1.
Hierarchical Module of 'censoring sensors' change detector
Fig.1 : Hierarchical Module of 'censoring sensors' change detector


Traffic Anomaly Visualization

Manifold learning in general is a data visualization tool. We have shown how the dwMDS and other manifold learning algorithms can be used to 'locate' anomalies in Internet traffic data [4]. We have developed and made publicly available online applets and command-line utilities which can be used in combination with NetFlow data to visualize very high-dimensional traffic data in two-dimensions. We have also tested the tools on data from the Abilene backbone network, and shown that they can show in two dimensions dramatic changes when traffic anomalies such as port and network scans, alpha flows, and worm propagation activity are measured on the network (more information is available online at http://www-personal.engin.umich.edu/~npatwari/mnd05/). Such visualization tools will complement detection tools by providing more information to operators and allowing them to quickly visually investigate network traffic before taking corrective action.
Wednesday, 12-Jan-2005 at 20:15 UTD
Fig.2 : Wednesday, 12-Jan-2005 at 20:15 UTD: Legend: The 4-week mean location (o) is connected to the current estimate (O) by a dashed red line (- - - -). The shading of the circle is proportional to the residual value e_i: dark indicates high residual and white indicates low residual. Event Description: There is a large anomaly of 71,000 flows at the STTL, LOSA, and SNVA routers. These flows are single, 29-byte UDP packet flows from source IP address 163.30.88.0/21 (possibly tyc.edu.tw) to destination IP address 134.71.24.0/21 (csupomona.edu, California Poly in Pomona). The packets are from source port 40150 to random destination ports. Since the traffic was observed on LOSA, SNVA, and STTL but no other router, these routers are placed far away to the West, while the rest of the routers, due to the constraint on total distances, are placed very close together.


Wednesday, 12-Jan-2005 at 20:15 UTD
Fig. 3: Total traffic, and port 80 traffic on the Abilene Backbone on 05-Jan-2005, displayed using the Java-based space-time traffic visualization applet. The router map is calculated for 08:25 UTD, during scheduled maintenance of the CHIN-IPLS link, during which traffic drops at CHIN and IPLS and increases dramatically at the HSTN and ATLA routers.

References

1. N. Patwari, A. O. Hero, "Manifold Learning Algorithms for Localization in Wireless Sensor Networks", in Proceedings of the 2004 IEEE Int. Conf. on Acoustics, Speech, and Signal Processing (ICASSP), Montreal, Quebec, Wed. May 19, 2004.
2. J. A. Costa, N. Patwari, A. O. Hero (2006), "Distributed Weighted Multidimensional Scaling for Node Localization in Sensor Networks",   in ACM Journal on Networking, (revised, Jan 2005).
3. J. A. Costa, N. Patwari, A. O. Hero, "Achieving High-Accuracy Distributed Localization in Sensor Networks",     in Proceedings of the 2005 IEEE Int. Conf. on Acoustics, Speech, and Signal Processing (ICASSP), Philadelphia, PA, March, 2005 (A Student Paper Contest Finalist).
4. N. Patwari, A. O. Hero, and A. Pacholski (2005), "Manifold Learning Visualization of Network Traffic Data",     2005 SIGCOMM Workshop on Mining Network Data, Philadelphia, PA, August 26, 2005.
5. N. Patwari, A. O. Hero(2005), "Adaptive Neighborhoods for Manifold Learning-based Sensor Localization",     2005 IEEE Signal Processing and Wireless Communications Conf., New York City, Jun 7, 2005.
6. N. Patwari (Hero advisor), "Location Estimation in Sensor Networks",   PhD Thesis, Dept. EECS, Sept 2005