Creating an Execution
Theodolite Executions look similar to the following example.
apiVersion: theodolite.com/v1
kind: execution
metadata:
name: theodolite-example-execution
spec:
benchmark: "uc1-kstreams"
load:
loadType: "NumSensors"
loadValues: [25000, 50000]
resources:
resourceType: "Instances"
resourceValues: [1, 2]
slos:
- sloType: "lag trend"
prometheusUrl: "http://prometheus-operated:9090"
offset: 0
properties:
threshold: 2000
externalSloUrl: "http://localhost:80/evaluate-slope"
warmup: 60 # in seconds
execution:
strategy: "LinearSearch"
duration: 300 # in seconds
repetitions: 1
loadGenerationDelay: 30 # in seconds
restrictions:
- "LowerBound"
configOverrides:
- patcher:
type: "SchedulerNamePatcher"
resource: "uc1-kstreams-deployment.yaml"
value: "random-scheduler"
Naming and Labeling
Similar to Kubernetes Jobs, Theodolite Executions are not automatically deleted after they are finished. Therefore, it is recommended to choose a meaningful name for your Execution. Additionally, you can add labels to your Executions as with any other Kubernetes resource.
Reference to a Benchmark
An Execution always refers to a Benchmark. For the Execution to run, the Benchmark must be registered with Kubernetes and it must be in state Ready. If this is not the case, the Execution will remain in state Pending.
As a Benchmark may define multiple supported load and resource types, an Execution has to pick exactly one of each by its name. Additionally, it defines the set of load values and resource values the benchmark should be executed with. Both these values are represented as integers, which are interpreted in a Benchmark-specific way to configure the SUT and load generator.
Definition of SLOs
SLOs provide a way to quantify whether a certain load intensity can be handled by a certain amount of provisioned resources. In Theodolite, SLO are evaluated by requesting monitoring data from Prometheus and analyzing it in a benchmark-specific way. An Execution must at least define one SLO to be checked.
A good choice to get started is defining an SLO of type generic
:
- sloType: "generic"
prometheusUrl: "http://prometheus-operated:9090"
offset: 0
properties:
externalSloUrl: "http://localhost:8082"
promQLQuery: "sum by(job) (kafka_streams_stream_task_metrics_dropped_records_total>=0)"
warmup: 60 # in seconds
queryAggregation: max
repetitionAggregation: median
operator: lte
threshold: 1000
All you have to do is to define a PromQL query describing which metrics should be requested (promQLQuery
) and how the resulting time series should be evaluated. With queryAggregation
you specify how the resulting time series is aggregated to a single value and repetitionAggregation
describes how the results of multiple repetitions are aggregated. Possible values are
mean
, median
, mode
, sum
, count
, max
, min
, std
, var
, skew
, kurt
as well as percentiles such as p99
or p99.9
. The result of aggregation all repetitions is checked against threshold
. This check is performed using an operator
, which describes that the result must be “less than” (lt
), “less than equal” (lte
), “greater than” (gt
) or “greater than equal” (gte
) to the threshold.
In case you need to evaluate monitoring data in a more flexible fashion, you can also change the value of externalSloUrl
to your custom SLO checker. Have a look at the source code of the generic SLO checker to get started.
Experimental Setup
According to Theodolite’s measurement method, isolated SLO experiments are performed for different combinations of load intensity and resource amounts. The experimental setup can be configured by:
- A search strategy (
strategy
), which determines which load and resource combinations should be tested. Supported values areFullSearch
,LinearSearch
andBinarySearch
. Additionally, arestrictions
can be set toLowerBound
. - The
duration
per SLO experiment in seconds. - The number of repetitions (
repetitions
) for each SLO experiment. - A
loadGenerationDelay
, specifying the time in seconds before the load generation starts.
Configuration Overrides
In cases where only small modifications of a system under test should be benchmarked, it is not necessary to create a new benchmark. Instead, also Executions allow to do small reconfigurations, such as switching on or off a specific Pod scheduler.
This is done by defining configOverrides
in the Execution. Each override consists of a patcher, defining which Kubernetes resource should be patched in which way, and a value the patcher is applied with.