Typical event rate · category
Application log rate: 100 events/s per device
An application source typically produces 100 events per second per device, bursting to 1,000: web access logs high, service audit trails low. The rate is an informed estimate, a starting point for a sizing, not a measurement; the event sizes below are measured.
Measured sources in this category
Average bytes per event, measured, and one device's volume at the typical 100 events per second.
| Source | Bytes per event | A day | A year |
|---|---|---|---|
| Apache Flink | 2,000 | 17.3 GB | 6.3 TB |
| Apache Kafka | 2,000 | 17.3 GB | 6.3 TB |
| Office 365 | 1,958 | 16.9 GB | 6.2 TB |
| Google Audit | 400 | 3.5 GB | 1.3 TB |
| Apache HTTP Server | 300 | 2.6 GB | 946 GB |
| Twitter message | 140 | 1.2 GB | 442 GB |
| Open SSH | 111 | 959 MB | 350 GB |
At the median size, a day from one device, 3.5 GB, is about the size of an hour of Netflix in HD.
Size a fleet
Raw bytes, before compression, indexing overhead and replication.
Measure a sample of your own
The figure above is an average; yours will differ, and for application logs what moves it most is this: web access logs high, service audit trails low. The profiler reads a sample from application logs in your browser, line by line, one event per line, and reports the bytes per line to set against it. Nothing is uploaded.
Logcaliper, free on iOS, has these sizes and rates built in and sizes a whole list of sources at retention: storage, cluster capacity, nodes.