Measured event size · Distributed system
Apache Spark log size: 97 bytes per event
The average Apache Spark event is 97 bytes, measured from samples of its logs. A distributed system source typically produces 100 events per second per device, bursting to 1,000: per node at INFO; log level is the lever. The rate is an informed estimate; the size is measured.
What that comes to
One device, raw bytes, at the typical rate for a distributed system source.
| Period | At 100 events/s | At the 1,000 events/s burst |
|---|---|---|
| An hour | 34.9 MB | 349 MB |
| A day | 838 MB | 8.4 GB |
| 90 days | 75.4 GB | 754 GB |
| A year | 306 GB | 3.1 TB |
A day of Apache Spark logs from one device at the category's typical rate, 838 MB, is about the size of a CD-ROM.
Size your own
Raw bytes, before compression, indexing overhead and replication.
Measure a sample of your own
The figure above is an average; yours will differ, and for Apache Spark what moves it most is this: per node at INFO; log level is the lever. The profiler reads a sample from Apache Spark in your browser, line by line, one event per line, and reports the bytes per line to set against it. Nothing is uploaded.
Other distributed system sources
- Open Stack297 B
- Apache Hadoop192 B
- Apache Zookeeper139 B
The distributed system category: its typical rate and every source in it →
Logcaliper, free on iOS, has these sizes and rates built in and sizes a whole list of sources at retention: storage, cluster capacity, nodes.