Typical event rate · category
Distributed system log rate: 100 events/s per device
A distributed system source typically produces 100 events per second per device, bursting to 1,000: per node at INFO; log level is the lever. The rate is an informed estimate, a starting point for a sizing, not a measurement; the event sizes below are measured.
Measured sources in this category
Average bytes per event, measured, and one device's volume at the typical 100 events per second.
| Source | Bytes per event | A day | A year |
|---|---|---|---|
| Open Stack | 297 | 2.6 GB | 937 GB |
| Apache Hadoop | 192 | 1.7 GB | 605 GB |
| Apache Zookeeper | 139 | 1.2 GB | 438 GB |
| Apache Spark | 97 | 838 MB | 306 GB |
At the median size, a day from one device, 1.4 GB, is about 2 times a CD-ROM.
Size a fleet
Raw bytes, before compression, indexing overhead and replication.
Measure a sample of your own
The figure above is an average; yours will differ, and for distributed system logs what moves it most is this: per node at INFO; log level is the lever. The profiler reads a sample from distributed system logs in your browser, line by line, one event per line, and reports the bytes per line to set against it. Nothing is uploaded.
Logcaliper, free on iOS, has these sizes and rates built in and sizes a whole list of sources at retention: storage, cluster capacity, nodes.