Creating a Benchmark

Please note that to simply run a benchmark, it is not required to define one. Theodolite comes with a set of benchmarks, which are ready to be executed. See the fundamental concepts page to learn more about our distinction between benchmarks and executions.

A typical benchmark looks like this:

apiVersion: theodolite.com/v1
kind: benchmark
metadata:
  name: example-benchmark
spec:
  sut:
    resources:
      - configMap:
         name: "example-configmap"
         files:
           - "uc1-kstreams-deployment.yaml"
  loadGenerator:
    resources:
      - configMap:
         name: "example-configmap"
         files:
            - uc1-load-generator-service.yaml
            - uc1-load-generator-deployment.yaml
  loadTypes:
    - typeName: "NumSensors"
      patchers:
        - type: "EnvVarPatcher"
          resource: "uc1-load-generator-deployment.yaml"
          properties:
            variableName: "NUM_SENSORS"
            container: "workload-generator"
        - type: "NumSensorsLoadGeneratorReplicaPatcher"
          resource: "uc1-load-generator-deployment.yaml"
          properties:
            loadGenMaxRecords: "150000"
  kafkaConfig:
    bootstrapServer: "theodolite-kafka-kafka-bootstrap:9092"
    topics:
      - name: "input"
        numPartitions: 40
        replicationFactor: 1
      - name: "theodolite-.*"
        removeOnly: True

System under Test (SUT), Load Generator and Infrastructure

In Theodolite, the system under test (SUT), the load generator as well as additional infrastructure (e.g., a middleware) are described by Kubernetes resources files. All resources defined for the SUT and the load generator are started and stopped for each SLO experiment, with SUT resources being started before the load generator. Infrastructure resources live over the entire duration of a benchmark run. They avoid time-consuming recreation of software components like middlewares, but should be used with caution to not let previous SLO experiments influence latte ones.

Resources

ConfigMap

The recommended way to link Kubernetes resources files from a Benchmark is by bundling them in one or multiple ConfigMaps and refer to that ConfigMap from sut.resources, loadGenerator.resources or infrastructure.resources. To create a ConfigMap from all the Kubernetes resources in a directory run:

kubectl create configmap <configmap-name> --from-file=<path-to-resource-dir>

Add an item such as the following one to the resources list of the sut, loadGenerator or infrastructure fields.

configMap:
  name: example-configmap
  files:
  - example-deployment.yaml
  - example-service.yaml

Filesystem

Alternatively, resources can also be read from the filesystem, Theodolite has access to. This usually requires that the Benchmark resources are available in a volume, which is mounted into the Theodolite container.

filesystem:
  path: example/path/to/files
  files:
  - example-deployment.yaml
  - example-service.yaml

Actions

Sometimes it is not sufficient to just define resources that are created and deleted when running a benchmark. Instead, it might be necessary to define certain actions that will be executed before running or after stopping the benchmark.

Theodolite allows to execute commands on running pods. This is similar to kubectl exec or Kubernetes’ container lifecycle handlers. Theodolite actions can run before (beforeActions) or after afterActions all sut, loadGenerator or infrastructure resources are deployed. For example, the following actions will create a file in a pod with label app: logger before the SUT is started and delete if after the SUT is stopped:

  sut:
    resources: # ...
    beforeActions:
      - selector:
          pod:
            matchLabels:
              app: logger
        exec:
          command: ["touch", "file-used-by-logger.txt"]
          timeoutSeconds: 90
    afterActions:
      - selector:
          pod:
            matchLabels:
              app: logger
        exec:
          command: [ "rm", "file-used-by-logger.txt" ]
          timeoutSeconds: 90

Theodolite checks if all referenced pods are available for the specified actions. That means these pods must either be defined in infrastructure or already deployed in the cluster. If not all referenced pods are available, the benchmark will not be set as Ready. Consequently, an action cannot be executed on a pod that is defined as an SUT or load generator resource.

Note: Actions should be used sparingly. While it is possible to define entire benchmarks imperatively as actions, it is considered better practice to define as much as possible using declarative, native Kubernetes resource files.

Load and Resource Types

Benchmarks need to specify at least one supported load and resource type for which scalability can be benchmarked.

Load and resource types are described by a name (used for reference from an Execution) and a list of patchers. Patchers can be seen as functions, which take a value as input and modify a Kubernetes resource in a patcher-specific way. Examples of patchers are the ReplicaPatcher, which modifies the replica specification of a deployment, or the EnvVarPatcher, which modifies an environment variable. See the patcher API reference for an overview of available patchers.

If a benchmark is executed by an Execution, these patchers are used to configure SUT and load generator according to the load and resource values set in the Execution.

Kafka Configuration

Theodolite allows to automatically create and remove Kafka topics for each SLO experiment by setting a kafkaConfig. bootstrapServer needs to point your Kafka cluster and topics configures the list of Kafka topics to be created/removed. For each topic, you configure its name, the number of partitions and the replication factor.

With the removeOnly: True property, you can also instruct Theodolite to only remove topics and not create them. This is useful when benchmarking SUTs, which create topics on their own (e.g., Kafka Streams and Samza applications). For those topics, also wildcards are allowed in the topic name and, of course, no partition count or replication factor must be provided.