Home > Topics > Passive Monitoring > Load management in network monitoring > Traffic aggregation

Traffic aggregation

Several network monitoring systems use data reduction techniques, such as packet filtering, traffic sampling, flow aggregation or a combination of them to handle overload situations. The most representative example is arguably Cisco’s NetFlow. NetFlow is considered the state-of-the-art technology for network monitoring. It is a widely deployed, general-purpose solution supported in most of today’s routers. It extracts pre-defined per-flow information (depending on the version of NetFlow) and periodically reports to a central collection server. In order to handle the large volumes of data exported and to reduce the load on the router, Sampled NetFlow resorts to packet sampling. The sampling rate must be defined at configuration time, and to handle unexpected traffic scenarios network operators tend to set it to a low “safe” value (e.g., 1/100 or 1/1000 packets). NetFlow input filters also permits to configure different sampling rates to different groups of flows defined by the network administrator.

Adaptive NetFlow [Estan et al., 2004] allows routers to dynamically tune the sampling rate to the memory consumption in order to maximize the accuracy given a specific incoming traffic mix. Flow Slices [Kompella and Estan, 2005] uses a combination of packet sampling, sample-and-hold and a variant of threshold sampling to independently control the CPU, memory and reporting bandwidth usage of routers. While the sampling parameters used to control the memory and bandwidth usage are dynamically adapted to runtime conditions, the sampling rate used to control the CPU usage is statically set at configuration time to a conservative value as in Sampled NetFlow.

ProgME [Yuan et al., 2007] uses aggregation instead of sampling to control the memory consumption. ProgME is a programmable flow aggregation engine based on the novel concept of flowset. A flowset is an arbitrary set of flows that is defined using a composition language based on set algebra. The main advantage of ProgME is that the memory consumption depends only on the number of flowsets the user is interested in and not on the observed traffic mix, which can significantly reduce the memory consumption for certain types of applications.

[Keys et al., 2005] extend the approach used in NetFlow by extracting and exporting a set of 12 traffic summaries that allow the system to answer a fixed number of common questions asked by network operators. The summaries focus on the detection of heavy hitters. The system deals with extreme traffic conditions and anomalous traffic patterns by gracefully degrading the accuracy of the summaries using adaptive sample-and-hold and memory-efficient counting algorithms.

References

  • [conf-sigcomm-EstanKMV04] bibtex
    C. Estan, K. Keys, D. Moore, and G. Varghese, "Building a better NetFlow.," in SIGCOMM, 2006-02-15 2004, pp. 245-256.
    @inproceedings{conf/sigcomm/EstanKMV04, added-at = {2006-02-15T00:00:00.000+0100},
      author = {Estan, Cristian and Keys, Ken and Moore, David and Varghese, George},
      biburl = {http://www.bibsonomy.org/bibtex/214d0cee3ce772a683b9afb272cbda43f/dblp},
      booktitle = {SIGCOMM},
      crossref = {conf/sigcomm/2004},
      date = {2006-02-15},
      description = {dblp},
      editor = {Yavatkar, Raj and Zegura, Ellen W. and Rexford, Jennifer},
      ee = {http://doi.acm.org/10.1145/1015467.1015495},
      interhash = {f17dc73d490161f4c32110158c585f6b},
      intrahash = {14d0cee3ce772a683b9afb272cbda43f},
      isbn = {1-58113-862-8},
      keywords = {dblp},
      pages = {245-256},
      publisher = {ACM},
      timestamp = {2006-02-15T00:00:00.000+0100},
      title = {Building a better NetFlow.},
      url = {http://dblp.uni-trier.de/db/conf/sigcomm/sigcomm2004.html#EstanKMV04},
      year = 2004 }
  • [conf-sigmetrics-KeysME05] bibtex
    K. Keys, D. Moore, and C. Estan, "A robust system for accurate real-time summaries of internet traffic.," in SIGMETRICS, 2006-02-15 2005, pp. 85-96.
    @inproceedings{conf/sigmetrics/KeysME05, added-at = {2006-02-15T00:00:00.000+0100},
      author = {Keys, Ken and Moore, David and Estan, Cristian},
      biburl = {http://www.bibsonomy.org/bibtex/21b57ac829b98cc6ae0896f715f9a7183/dblp},
      booktitle = {SIGMETRICS},
      crossref = {conf/sigmetrics/2005},
      date = {2006-02-15},
      description = {dblp},
      editor = {Eager, Derek L. and Williamson, Carey L. and Borst, Sem C. and Lui, John C. S.},
      ee = {http://doi.acm.org/10.1145/1064212.1064223},
      interhash = {7f161d4180033c2aa43ff66b63d07df8},
      intrahash = {1b57ac829b98cc6ae0896f715f9a7183},
      isbn = {1-59593-022-1},
      keywords = {dblp},
      pages = {85-96},
      publisher = {ACM},
      timestamp = {2006-02-15T00:00:00.000+0100},
      title = {A robust system for accurate real-time summaries of internet traffic.},
      url = {http://dblp.uni-trier.de/db/conf/sigmetrics/sigmetrics2005.html#KeysME05},
      year = 2005 }
  • [conf-imc-KompellaE05] bibtex
    R. R. Kompella and C. Estan, "The Power of Slicing in Internet Flow Measurement.," in Internet Measurment Conference, 2007-05-08 2005, pp. 105-118.
    @inproceedings{conf/imc/KompellaE05, added-at = {2007-05-08T00:00:00.000+0200},
      author = {Kompella, Ramana Rao and Estan, Cristian},
      biburl = {http://www.bibsonomy.org/bibtex/22becacc46d1df41fb54417f8c0caf676/dblp},
      booktitle = {Internet Measurment Conference},
      crossref = {conf/imc/2005},
      date = {2007-05-08},
      description = {dblp},
      ee = {http://www.usenix.org/events/imc05/tech/kompella.html},
      interhash = {f12960b3c794dc75e9d7776669e5b9a6},
      intrahash = {2becacc46d1df41fb54417f8c0caf676},
      keywords = {dblp},
      pages = {105-118},
      publisher = {USENIX Association},
      timestamp = {2007-05-08T00:00:00.000+0200},
      title = {The Power of Slicing in Internet Flow Measurement.},
      url = {http://dblp.uni-trier.de/db/conf/imc/imc2005.html#KompellaE05},
      year = 2005 }
  • [conf-sigcomm-YuanCM07] bibtex
    L. Yuan, C. Chuah, and P. Mohapatra, "ProgME: towards programmable network measurement.," in SIGCOMM, 2007-10-23 2007, pp. 97-108.
    @inproceedings{conf/sigcomm/YuanCM07, added-at = {2007-10-23T00:00:00.000+0200},
      author = {Yuan, Lihua and Chuah, Chen-Nee and Mohapatra, Prasant},
      biburl = {http://www.bibsonomy.org/bibtex/2c8cda65ec76be3fb567266c4b4ad224a/dblp},
      booktitle = {SIGCOMM},
      crossref = {conf/sigcomm/2007},
      date = {2007-10-23},
      description = {dblp},
      editor = {Murai, Jun and Cho, Kenjiro},
      ee = {http://doi.acm.org/10.1145/1282380.1282392},
      interhash = {34561c34c498ca0f36257f49667e5f65},
      intrahash = {c8cda65ec76be3fb567266c4b4ad224a},
      isbn = {978-1-59593-713-1},
      keywords = {dblp},
      pages = {97-108},
      publisher = {ACM},
      timestamp = {2007-10-23T00:00:00.000+0200},
      title = {ProgME: towards programmable network measurement.},
      url = {http://dblp.uni-trier.de/db/conf/sigcomm/sigcomm2007.html#YuanCM07},
      year = 2007 }