Theodolite Executions look similar to the following example.
apiVersion: theodolite.com/v1 kind: execution metadata: name: theodolite-example-execution spec: benchmark: "uc1-kstreams" load: loadType: "NumSensors" loadValues: [25000, 50000] resources: resourceType: "Instances" resourceValues: [1, 2] slos: - sloType: "lag trend" prometheusUrl: "http://prometheus-operated:9090" offset: 0 properties: threshold: 2000 externalSloUrl: "http://localhost:80/evaluate-slope" warmup: 60 # in seconds execution: strategy: "LinearSearch" duration: 300 # in seconds repetitions: 1 loadGenerationDelay: 30 # in seconds restrictions: - "LowerBound" configOverrides: - patcher: type: "SchedulerNamePatcher" resource: "uc1-kstreams-deployment.yaml" value: "random-scheduler"
Similar to Kubernetes Jobs, Theodolite Executions are not automatically deleted after they are finished. Therefore, it is recommended to choose a meaningful name for your Execution. Additionally, you can add labels to your Executions as with any other Kubernetes resource.
An Execution always refers to a Benchmark. For the Execution to run, the Benchmark must be registered with Kubernetes and it must be in state Ready. If this is not the case, the Execution will remain in state Pending.
As a Benchmark may define multiple supported load and resource types, an Execution has to pick exactly one of each by its name. Additionally, it defines the set of load values and resource values the benchmark should be executed with. Both these values are represented as integers, which are interpreted in a Benchmark-specific way to configure the SUT and load generator.
SLOs provide a way to quantify whether a certain load intensity can be handled by a certain amount of provisioned resources. In Theodolite, SLO are evaluated by requesting monitoring data from Prometheus and analyzing it in a benchmark-specific way. An Execution must at least define one SLO to be checked.
A good choice to get started is defining an SLO of type
- sloType: "generic" prometheusUrl: "http://prometheus-operated:9090" offset: 0 properties: externalSloUrl: "http://localhost:8082" promQLQuery: "sum by(job) (kafka_streams_stream_task_metrics_dropped_records_total>=0)" warmup: 60 # in seconds queryAggregation: max repetitionAggregation: median operator: lte threshold: 1000
All you have to do is to define a PromQL query describing which metrics should be requested (
promQLQuery) and how the resulting time series should be evaluated. With
queryAggregation you specify how the resulting time series is aggregated to a single value and
repetitionAggregation describes how the results of multiple repetitions are aggregated. Possible values are
kurt as well as percentiles such as
p99.9. The result of aggregation all repetitions is checked against
threshold. This check is performed using an
operator, which describes that the result must be “less than” (
lt), “less than equal” (
lte), “greater than” (
gt) or “greater than equal” (
gte) to the threshold.
In case you need to evaluate monitoring data in a more flexible fashion, you can also change the value of
externalSloUrl to your custom SLO checker. Have a look at the source code of the generic SLO checker to get started.
According to Theodolite’s measurement method, isolated SLO experiments are performed for different combinations of load intensity and resource amounts. The experimental setup can be configured by:
- A search strategy (
strategy), which determines which load and resource combinations should be tested. Supported values are
BinarySearch. Additionally, a
restrictionscan be set to
durationper SLO experiment in seconds.
- The number of repetitions (
repetitions) for each SLO experiment.
loadGenerationDelay, specifying the time in seconds before the load generation starts.
In cases where only small modifications of a system under test should be benchmarked, it is not necessary to create a new benchmark. Instead, also Executions allow to do small reconfigurations, such as switching on or off a specific Pod scheduler.
This is done by defining
configOverrides in the Execution. Each override consists of a patcher, defining which Kubernetes resource should be patched in which way, and a value the patcher is applied with.