Spilled
Get Started

Using Spilled

Architecture

Spilled separates the systems that handle flow data from the systems that coordinate them. The data plane runs in your cloud and reads from and writes to your object storage. The control plane runs in Spilled's cloud and works only with metadata and query results.

The diagram below shows the Spilled architecture and how the components interact with each other, as well as where they run.

The diagram shows the collector and query engine as separate logical components. This will be a deployment option in the future to meet the scalability and performance needs of larger customers. However, today we package both components into a single deployable runtime, aptly named Fusion.

Spilled architecture Flow data stays in your cloud. Only metadata and control messages cross the boundary into the Spilled control plane. Your network routers · switches Your cloud Collector ingests flow telemetry NetFlow / IPFIX / sFlow write Object storage your bucket — flow data at rest read Query engine reads data in place Spilled cloud Control plane metadata · coordination Web console explore · query · manage sends metadata polls for queries sends control messages Users

We can trace the flow of data through Spilled's architecture:

  1. Ingest. Routers and switches export NetFlow, IPFIX or sFlow telemetry to the collector.
  2. Persist. The collector enriches, batches and then writes the records to object storage in your cloud.
  3. Catalog. The collector sends a manifest describing each write to the control plane. The manifest contains metadata, not the flow records themselves.
  4. Plan. When a user submits a query, the control plane uses manifests to identify the relevant files and sends that work to the query engine.
  5. Execute. The query engine reads the selected files directly from your bucket and returns only the query result.

Designed around separation

Spilled is able to meet its design principles because it separates:

  1. Storage from compute
  2. Data from metadata
  3. The data plane from the control plane

Separating storage and compute

Spilled stores all durable flow data in object storage rather than on the collector or query engine. Compute nodes do not own a particular subset of the data, so they can be added or removed without moving or rebalancing flow records.

This separation lets compute respond to ingest and query load while storage grows at its own rate.

Separating data and metadata

Flow records are the data in Spilled. They remain in your object storage. Manifests are the metadata: lightweight descriptions of which files exist, the time ranges they cover, and statistics that help the query planner decide which files to read.

The collector sends manifests to the control plane after writing files. The control plane can then catalog the data and plan queries without reading or storing the flow records themselves.

Separating the data plane from the control plane

Your cloud. The data plane runs inside infrastructure you control. It receives telemetry from your network, stores flow records in your bucket, and executes queries against those records. Bucket credentials remain with the data plane and are never sent to Spilled.

Spilled cloud. The control plane maintains the metadata catalog, coordinates query and compaction work, and serves the web console.

The trust boundary. Only metadata, control messages, and query results cross the boundary. Raw flow records remain in your object storage throughout ingestion and query execution. Data-plane components establish outbound connections to the control plane; Spilled does not require inbound access to your network.