3 / 36 chapters
    DocumentationDeployment

    03

    Sizing & Bandwidth

    How Much Data Will Flow In?

    This chapter answers a practical question: how strong an internet connection and how big a disk do you need for the number of agents and network devices you plan to monitor? You do not need to understand the technical details — the first table gives you three numbers that are enough for a rough estimate.

    All numbers below were measured on a running FlowPRO server with real traffic (agents with all datasets enabled, syslog from firewalls, NetFlow from a live network) — they are not vendor estimates.

    Quick estimate (rules of thumb)

    Data sourceLoad on the connectionNew data on disk
    1 agent on a server or PC~50 MB per day~90 MB per day
    1 ordinary router/firewall sending syslog~2–20 MB per day~3–30 MB per day
    NetFlow from a network device~0.5 % of the traffic it monitors~4 GB per day per 1 Gbit/s of traffic

    Example: 100 agents + 20 routers + monitoring of a 1 Gbit/s line means roughly 6 Mbit/s of incoming data and about 20 GB of new data on disk every day. In terms of bandwidth this is very little — a standard 100 Mbit connection is plenty for 100 agents. The disk is always the tighter constraint, not the connection.

    Agents (servers and workstations)

    An agent sends two kinds of data: metrics (CPU, RAM, disks, network, processes — every 10 seconds) and logs (logins, system messages, audit records). Data is compressed before sending, so on the connection it takes roughly one eighth of its original size.

    AgentsBasic profileFull profileConnection (full profile)Disk / dayDisk / 30 days
    100,1 GB0,5 GB0,05 Mbit/s0,9 GB27 GB
    250,25 GB1,3 GB0,12 Mbit/s2,3 GB68 GB
    500,5 GB2,5 GB0,23 Mbit/s4,5 GB135 GB
    1001,0 GB5,0 GB0,46 Mbit/s9 GB270 GB
    2502,5 GB12,5 GB1,2 Mbit/s22 GB675 GB
    5005,0 GB25 GB2,3 Mbit/s45 GB1,35 TB
    1 00010 GB50 GB4,6 Mbit/s90 GB2,7 TB
    2 00020 GB100 GB9,3 Mbit/s180 GB5,4 TB
    Basic profile means per-process monitoring (system.process) and disk operations (system.diskio) are turned off. These two datasets alone account for about 70 % of all agent data. You can turn them off in Settings → Data collection without reinstalling any agent — it is by far the most effective way to reduce the data volume if you do not need that level of detail.

    Two cases where an agent sends much more

    • •Hosts with Docker or with many disks and network interfaces — the agent measures every disk and every interface separately. Measured up to 90 MB/day instead of 50 MB/day, almost double.
    • •Mail and web servers — Postfix logs on a busy mail server alone produce about 1 GB of data per day, which is more than 20 ordinary agents combined.

    Syslog from network devices

    One syslog message takes about 227 bytes on the connection and about 376 bytes on disk (measured values).

    Messages per secondConnectionReceived data / dayDisk / day
    100,02 Mbit/s0,2 GB0,33 GB
    500,09 Mbit/s1,0 GB1,6 GB
    1000,18 Mbit/s2,0 GB3,3 GB
    5000,9 Mbit/s9,8 GB16 GB
    1 0001,8 Mbit/s20 GB33 GB
    2 5004,5 Mbit/s49 GB81 GB
    5 0009,1 Mbit/s98 GB163 GB
    10 00018 Mbit/s196 GB325 GB

    How many messages does a device actually send?

    It depends only on what the device is configured to log — not on its size and not on how many devices you have.

    Logging configuration on the deviceMessages/s
    Firewall drops, DHCP, logins only (typical)0,2–2
    Logging of all connections20–80
    Per-packet logging (e.g. Kerio ips_pktlog)~760
    With syslog, do not size by the number of devices — size by their configuration. On the measured server, a single firewall with per-packet logging enabled generated 65 million messages a day (25 GB of disk every day), the same as 500 ordinary routers. If your disk is filling up unexpectedly, look for such a device first: Log Explorer → group by source IP.

    NetFlow / IPFIX (network flow analysis)

    NetFlow does not send the traffic itself, only records about connections — so its volume is a small fraction of the monitored traffic. Measured: 0.5 % of the monitored line. For sizing we recommend planning with 1 % as a reserve.

    Monitored trafficNetFlow (measured)Plan forFlows/sDisk / day
    100 Mbit/s0,5 Mbit/s1 Mbit/s~2200,4 GB
    500 Mbit/s2,5 Mbit/s5 Mbit/s~1 1001,9 GB
    1 Gbit/s5 Mbit/s10 Mbit/s~2 2003,8 GB
    5 Gbit/s25 Mbit/s50 Mbit/s~11 00019 GB
    10 Gbit/s50 Mbit/s100 Mbit/s~22 00038 GB
    If this is too much for your connection, enable sampling on the exporting device. A 1:100 setting reduces both the data volume and the level of statistical detail a hundredfold.

    Practical scenarios

    DeploymentData sourcesIncoming connectionDisk / dayDisk / 90 days
    Small business25 agents, 5 routers, 200 Mbit/s NetFlow~1,5 Mbit/s~4 GB~360 GB
    Medium business100 agents, 20 routers, 1 Gbit/s NetFlow~6 Mbit/s~20 GB~1,8 TB
    ISP / large network300 agents, 50 routers, 10 Gbit/s NetFlow~55 Mbit/s~90 GB~8 TB

    What to keep in mind

    • •Reserve for peaks — the tables show averages. When an agent loses the connection to the server it stores data locally and sends it all at once when the connection is restored. Plan for 3–5 times the average: for 1 000 agents that means 20–25 Mbit/s rather than 5 Mbit/s.
    • •Agent installation and updates — each agent downloads about 0.5 GB of installation package from the internet (artifacts.elastic.co), not from the FlowPRO server. It does not load the server, but during a mass rollout it does load the connection of that location.
    • •The disk is always a problem before the connection is. Even with 1 000 agents the data flow is a few Mbit/s, but the disk grows every single day. Before you add more disk, check your retention settings (Settings → Data retention) and turn off datasets you do not use.
    • •Communication between agent and server (Fleet enrollment, status check-ins) is already included in the numbers above and is negligible.