Engine sizing isn’t an exact science. It involves testing, monitoring, and trial and error. However, there are some best practices and rules of thumb that can guide your decision-making process.
It is important to note that there isn’t a “right” and “wrong” engine. Some engine topologies will work better for certain queries than others. Choosing the best engine means finding the most cost-effective solution that provides the best balance of query performance across all workloads.
In addition to the information here, there is a section in the Firebolt documentation about Choosing an engine.
This article describes ways that engine utilization can be monitored.
What are the variables?
There are four factors that go into planning a Firebolt engine deployment.
-
Instance Family
This is the series of instances that defines the hardware specs.
-
Node Type
This is the specific instance type within the instance family.
-
Scale
This is the number of nodes in the cluster.
Whether to use a single-engine or separate ingestion and analytics
Choosing your instance family
There are a number of different engine specifications that fall into 4 different families:
-
Memory Optimized (M series)
Choose this family if queries involve large aggregated queries or joins. This is typically the default option and serves a wide variety of workloads.
-
Compute optimized (C series)
This series is a good choice if you don’t have many joins, and you’re doing a large amount of filtering. Firebolt will use every CPU core available for data scanning and calculations. CPU is also very important if you have high concurrency workloads.
-
Balanced (B series)
If you do both a lot of aggregation, and a lot of filtering, you can use the B series to balance both.
-
Storage optimized (S series)
Having data in the cache is very important to the performance of Firebolt. If your use case requires having a lot of data in the cache ready to be queried, and that doesn’t fit in the above nodes, using the S series can eliminate the performance degradation from having to fetch data from F3.
Choosing your node type
Bigger is better
The node type is the biggest factor in determining Firebolt performance and cost. Firebolt will use all available CPUs of a cluster even for a single query. Generally, the more cores you add, the faster queries will run. Going with too small of an engine will lead to poor performance and OOM errors.
Choosing your engine scale
A small number of large nodes will generally provide better performance than a large number of small nodes
Core for core, scaling up before you scale out will provide better query performance, as it requires less merging of results.
Scaling out helps with concurrency
Having a higher number of nodes allows queries to load balance among all nodes more efficiently.
When to use a separate ingestion engine from an analytics engine
In general, it is easier and safer to use a single engine for both ingestion and querying. This provides the fastest availability to data and eliminates query inconsistency while data is ingesting.
There are some cases, however, where it is preferred to separate the ingestion and analytics engine.
-
With large ingestion tasks, that run infrequently.
By separating the ingestion engine, you can run a very large cluster to ingest data quickly, and then shut the engine down to reduce cost.
When multiple engines are needed for concurrency.
-
When ingestion volume cannot be predicted.
A large ingestion task can negatively impact query performance. Running separate engines isolates the workloads.