Concepts

Flow telemetry

Flow telemetry summarizes network conversations. Instead of storing every packet payload, flow records describe who communicated, when they communicated, how much data moved which protocol or port was used and other such data.

Spilled supports common flow sources such as NetFlow, IPFIX and sFlow.

Collector

The collector handles ingestion. It listens for IPFIX, NetFlow, and sFlow over UDP, enriches and batches the records it receives, and writes them to object storage.

After each write, the collector sends a manifest to the control plane describing what it just wrote, so the control plane can plan queries without reading the data itself.

Query engine

The query engine executes queries against the flow records in object storage. It uses metadata from the control plane to read only the files relevant to a query before returning the results. The query engine reads data in place. Your flow records never leave your cloud, only query results do.

Fusion

Fusion combines the collector and query engine into a single container, making it easy to deploy and get started with Spilled. It is currently the only way to deploy Spilled. It is suitable for smaller workloads. We are working on supporting separate collector and query engine deployments.

Bring Your Own Cloud (BYOC)

Spilled uses a Bring Your Own Cloud (BYOC) model. The data plane, consisting of the flow collector and query engine, runs in your cloud environment. Durable storage is provided by your cloud object storage service, such as AWS S3 or Google Cloud Storage. Due to this split, the flow collector and query engine are completely stateless and can therefore be scaled with ease. The Spilled control plane, responsible for managing the data plane, runs in our cloud. It routes queries to the query engine, performs compaction and other management-related tasks.

With this deployment model, raw flow data never leaves your cloud or object storage bucket. Spilled never sees or accesses this data.

Object storage

Object storage, such as AWS S3 or Cloudflare R2, is the durable store for Spilled data. It is the single source of truth for your flow data.

The collector writes telemetry under the /spilled prefix in the configured bucket, and the query engine reads from that same location when executing queries. Spilled never holds credentials for the bucket; every read and write happens from inside your cloud.

VPC

A virtual private cloud (VPC) is an isolated, private network inside a public cloud provider such as AWS, GCP or Azure that you own and control.

Spilled's data plane runs inside your VPC so your flow data stays within that boundary. The data plane connects outbound to Spilled; nothing needs to connect inbound to your network.

Organizations

Spilled organizes your account and data into a hierarchy:

Organization
└── Team
    ├── Query clusters  (query-engine instances allocated for querying)
    └── Datasets        (collections of flow records)
        └── Manifests → Granules (files in object storage)

An organization is the top of that hierarchy and the unit of billing and plan management. Plan tier, ingest limits, spend controls and credit balance all apply at the organization level. You will typically have one organization per company.

Teams

A team lives inside an organization and is where people and resources come together. Team members get access to the query clusters and datasets the team owns. Teams are the natural boundary for separating groups that should not see each other's data. Spend controls and ingest limits can also apply at this level.

Query clusters

A query cluster is a logical group of query-engine instances used for query execution. Typically one cluster will be used per team but multiple clusters can be created for different use cases or workloads.

Datasets

A dataset is a named collection of flow records within a team. It is roughly analogous to a table: queries run against a dataset, a time range and a set of predicates. By default, all flows go to the default dataset. More can be created, alongside dataset rules (coming soon) in order to segment flows into different datasets based on their source or other criteria.

Manifests

A manifest is the metadata descriptor for a raw data file on object storage. Each manifest records where the file lives, the time range it covers, and per-column statistics that let the query engine skip files it does not need to read.

Manifests live in the control plane; the data they describe live in your object storage. Together they bridge the two planes: the control plane plans a query from manifests, and the query engine fetches only the matching files.