Please note that to simply run a benchmark, it is not required to define one. Theodolite comes with a set of benchmarks, which are ready to be executed. See the fundamental concepts page to learn more about our distinction between benchmarks and executions.
A typical benchmark looks like this:
apiVersion: theodolite.com/v1 kind: benchmark metadata: name: example-benchmark spec: sut: resources: - configMap: name: "example-configmap" files: - "uc1-kstreams-deployment.yaml" loadGenerator: resources: - configMap: name: "example-configmap" files: - uc1-load-generator-service.yaml - uc1-load-generator-deployment.yaml loadTypes: - typeName: "NumSensors" patchers: - type: "EnvVarPatcher" resource: "uc1-load-generator-deployment.yaml" properties: variableName: "NUM_SENSORS" container: "workload-generator" - type: "NumSensorsLoadGeneratorReplicaPatcher" resource: "uc1-load-generator-deployment.yaml" properties: loadGenMaxRecords: "150000" kafkaConfig: bootstrapServer: "theodolite-kafka-kafka-bootstrap:9092" topics: - name: "input" numPartitions: 40 replicationFactor: 1 - name: "theodolite-.*" removeOnly: True
In Theodolite, the system under test (SUT), the load generator as well as additional infrastructure (e.g., a middleware) are described by Kubernetes resources files. All resources defined for the SUT and the load generator are started and stopped for each SLO experiment, with SUT resources being started before the load generator. Infrastructure resources live over the entire duration of a benchmark run. They avoid time-consuming recreation of software components like middlewares, but should be used with caution to not let previous SLO experiments influence latte ones.
The recommended way to link Kubernetes resources files from a Benchmark is by bundling them in one or multiple ConfigMaps and refer to that ConfigMap from
To create a ConfigMap from all the Kubernetes resources in a directory run:
kubectl create configmap <configmap-name> --from-file=<path-to-resource-dir>
Add an item such as the following one to the
resources list of the
configMap: name: example-configmap files: - example-deployment.yaml - example-service.yaml
Alternatively, resources can also be read from the filesystem, Theodolite has access to. This usually requires that the Benchmark resources are available in a volume, which is mounted into the Theodolite container.
filesystem: path: example/path/to/files files: - example-deployment.yaml - example-service.yaml
Sometimes it is not sufficient to just define resources that are created and deleted when running a benchmark. Instead, it might be necessary to define certain actions that will be executed before running or after stopping the benchmark.
Theodolite allows to execute commands on running pods. This is similar to
kubectl exec or Kubernetes’ container lifecycle handlers. Theodolite actions can run before (
beforeActions) or after
infrastructure resources are deployed.
For example, the following actions will create a file in a pod with label
app: logger before the SUT is started and delete if after the SUT is stopped:
sut: resources: # ... beforeActions: - selector: pod: matchLabels: app: logger exec: command: ["touch", "file-used-by-logger.txt"] timeoutSeconds: 90 afterActions: - selector: pod: matchLabels: app: logger exec: command: [ "rm", "file-used-by-logger.txt" ] timeoutSeconds: 90
Theodolite checks if all referenced pods are available for the specified actions. That means these pods must either be defined in
infrastructure or already deployed in the cluster. If not all referenced pods are available, the benchmark will not be set as
Ready. Consequently, an action cannot be executed on a pod that is defined as an SUT or load generator resource.
Note: Actions should be used sparingly. While it is possible to define entire benchmarks imperatively as actions, it is considered better practice to define as much as possible using declarative, native Kubernetes resource files.
Benchmarks need to specify at least one supported load and resource type for which scalability can be benchmarked.
Load and resource types are described by a name (used for reference from an Execution) and a list of patchers. Patchers can be seen as functions, which take a value as input and modify a Kubernetes resource in a patcher-specific way. Examples of patchers are the ReplicaPatcher, which modifies the replica specification of a deployment, or the EnvVarPatcher, which modifies an environment variable. See the patcher API reference for an overview of available patchers.
Theodolite allows to automatically create and remove Kafka topics for each SLO experiment by setting a
bootstrapServer needs to point your Kafka cluster and
topics configures the list of Kafka topics to be created/removed.
For each topic, you configure its name, the number of partitions and the replication factor.
removeOnly: True property, you can also instruct Theodolite to only remove topics and not create them.
This is useful when benchmarking SUTs, which create topics on their own (e.g., Kafka Streams and Samza applications).
For those topics, also wildcards are allowed in the topic name and, of course, no partition count or replication factor must be provided.