Skip to content

Custom Workloads and Datasets

Mako supports benchmarking and tuning with custom workloads.

In order to use them, you will have to create or reuse a relevant dataset with pairs of (input, output) sentences, and then extend it with a set of extra parameters specific to the workload you are interested in. This is done primarily by using a YAML file with all the necessary parameters, but you might also need to convert the actual data to a supported JSON format.

Defining your dataset

Note

This step is optional and can be skipped if you are fine using one of the provided datasets.

First, you need to decide what dataset will be used to provide specific data used by the workload. Note that the dataset does not have to be used in its entirety, but it will limit the workload in what source data is available to it.

In particular, the dataset should provide a set of user prompts and expected replies from the server. The user prompt is always used as-is, while the expected reply is only used to calculate the amount of output tokens returned by the server, and can be overwritten by some of the parameters mentioned later.

The records from the dataset are also subject to filtering, based on the the parameters of a specific workload, hence many workloads can be defined from the same source dataset. See a section about workloads in the technical specification for details.

The dataset file

In order to define the dataset, you have to create (or otherwise obtain) a .json file with the dataset's contents. Multiple files can be used, in which case they should hold disjoint subsets of the records.

Each file should follow the format below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
[
  // (1)!
  {
    "conversations": [ // (2)!
      {
        "value": "User prompt #1" // (3)!
      },
      {
        "value": "Expected reply #1", // (4)!
      }
      // (5)!
    ]
  },
  // (6)!
]
  1. The top-level structure should be a list of dictionaries, each dictionary representing a single record.
  2. Each record should include "conversations" in its content, which should hold the user prompt and the reply. Other keys are permitted, but they will be ignored. Technically, it should be another list of dictionaries.
  3. The first element inside the "conversations" list should be a dictionary specifying the user (input) prompt. The prompt should be provided under the "value" key. Other keys may be present, but they will be ignored. In particular, the code will NOT check if the there is a "role" element and if it is in fact set to "user", or any similar checks.
  4. The second element of the list should be the expected reply from the server. It follows the same convention as the user's prompt.
  5. More elements can be present, but they will be ignored.
  6. The entry above constitutes one possible request to the server. It should be repeated as many times as needed, in oder to define the overall set of possible requests.

The format above defines a list of records that will be added to the dataset. If multiple files are used, their respective lists will simply be concatenated.

Added in version 1.4.0

You can also use mako datagen to generate synthetic datasets following the format above.

The dataset description

In order to use your dataset in any of the workloads, you will have to first describe its content such that MakoOptimize can correctly load it.

This is done by providing the first part of the YAML file, which should define its content under a dictionary key named "datasets".

Describing a dataset requires you to define all of its attributes, unless they provide a default value and you are fine with it. Below is an example with some comments:

1
2
3
4
5
6
datasets: # (1)!
  my_dataset: # (2)!
    files: # (3)!
      - filename: my_data.json
        url: https://example.com/my_data.json # (4)!
        checksum: <sha256> # (5)!
  1. Different datasets should be specified under top-level datasets entry. Note that all datasets used to define workloads in the file have to also be defined in the same file. I.e., cross-referencing entries from different files is not supported.
  2. Each dataset should be a named entry in the datasets dict, with the key being the dataset's name. The provided name should later be referenced when defining a workload. The values defining a dataset are described in the technical notes. Note that omitted values will use their defaults, and that the name should be skipped in favour of the entry's key.
  3. files should be a list of the .json files constituting the dataset. See the section above about what the content of each file should look like.
  4. The url is optional. If provided, it will be used to download the file if it is missing from the local storage. See the technical notes for details.
  5. Like url, checksum is optional and will be used to verify the content of a file - both when checking for its presence on the local machine and after downloading. If not provided, the file is always assumed to be correct, if it exists.

Defining the workload

After you have decided on the dataset to use as the source of data for your workload - either by defining your own, or by using one of the provided ones - you will then need to describe the workload itself.

You can think about the workload as the receipt of how exactly the data from the dataset will be used to benchmark the server. In particular: 1) what subset of the records will be used, and 2) how fast the requests will be made; are two main questions for which a workload definition should provide answers. This is done by providing a set of parameters, which are described in details in a relevant technical section.

Practically speaking, for each workload you want to define, you will have to add a relevant entry in a custom YAML file under a top-level workloads: key. The entry should consists of the key, being the workload name, and a dictionary of other properties defined as key: value pairs underneath it. For the dataset property, you should use the name of one of the datasets defined in the same file. A particular property can be omitted, if it provides a default value and you do not wish to overwrite it.

Following on the custom dataset example from above, the file can be extended by adding a workload definition that uses the custom dataset like that:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
datasets:
  my_dataset:
    files:
      - filename: my_data.json
        url: https://example.com/my_data.json
        checksum: <sha256>

workloads: # (1)!
  my_workload: # (2)!
    dataset: my_dataset # (3)!
    num_requests: 100
    max_prompt: 512
    max_seq: 2048
  1. Similar to datasets, all workloads should be grouped under the top-level workloads entry.
  2. Each workload should be defined with a key: value pair, where they key is the workload's name, and value is a dictionary describing the workload's properties. Possible values for workloads are listed in the technical notes. Properties with default values can be skipped.
  3. dataset should be a name of one of the datasets defined in the same file.

The example above will use the first 100 records from the my_dataset dataset that have user prompts with length of at most 512 tokens, and the total sequence length (input + output) is at most 2048. The requests will all be send as fast as possible, per the default value of request_rate.

Then pass it to any mako command using --workloads-spec, while specifying the name of the workload to use with --workload:

mako benchmark meta-llama/Llama-3.1-8B-Instruct --workloads-spec path/to/workloads.yaml --workload my_workload

Note

The default workload name is default - you can make it your workload's name if you do not want to have to specify --workload each time.