Typical event rate · category

Distributed system log rate: 100 events/s per device

A distributed system source typically produces 100 events per second per device, bursting to 1,000: per node at INFO; log level is the lever. The rate is an informed estimate, a starting point for a sizing, not a measurement; the event sizes below are measured.

100
events per second, typical
1,000
at the burst
166 B
median event size, 4 sources
1.4 GB
per device per day

Measured sources in this category

Average bytes per event, measured, and one device's volume at the typical 100 events per second.

SourceBytes per eventA dayA year
Open Stack2972.6 GB937 GB
Apache Hadoop1921.7 GB605 GB
Apache Zookeeper1391.2 GB438 GB
Apache Spark97838 MB306 GB

At the median size, a day from one device, 1.4 GB, is about 2 times a CD-ROM.

Size a fleet

1,000
events per second
14.3 GB
per day
1.3 TB
at retention

Raw bytes, before compression, indexing overhead and replication.

Measure a sample of your own

The figure above is an average; yours will differ, and for distributed system logs what moves it most is this: per node at INFO; log level is the lever. The profiler reads a sample from distributed system logs in your browser, line by line, one event per line, and reports the bytes per line to set against it. Nothing is uploaded.

Logcaliper, free on iOS, has these sizes and rates built in and sizes a whole list of sources at retention: storage, cluster capacity, nodes.