Measured event size · Distributed system
Apache Hadoop log size: 192 bytes per event
The average Apache Hadoop event is 192 bytes, measured from samples of its logs. A distributed system source typically produces 100 events per second per device, bursting to 1,000: per node at INFO; log level is the lever. The rate is an informed estimate; the size is measured.
What that comes to
One device, raw bytes, at the typical rate for a distributed system source.
| Period | At 100 events/s | At the 1,000 events/s burst |
|---|---|---|
| An hour | 69.1 MB | 691 MB |
| A day | 1.7 GB | 16.6 GB |
| 90 days | 149 GB | 1.5 TB |
| A year | 605 GB | 6.1 TB |
A day of Apache Hadoop logs from one device at the category's typical rate, 1.7 GB, is about 2 times a CD-ROM.
Size your own
Raw bytes, before compression, indexing overhead and replication.
Measure a sample of your own
The figure above is an average; yours will differ, and for Apache Hadoop what moves it most is this: per node at INFO; log level is the lever. The profiler reads a sample from Apache Hadoop in your browser, line by line, one event per line, and reports the bytes per line to set against it. Nothing is uploaded.
Other distributed system sources
- Open Stack297 B
- Apache Zookeeper139 B
- Apache Spark97 B
The distributed system category: its typical rate and every source in it →
Logcaliper, free on iOS, has these sizes and rates built in and sizes a whole list of sources at retention: storage, cluster capacity, nodes.