Skip to content

mako tune

The tune command performs automated tuning for a specified LLM model.
Runs automatic tuning for a given model, testing different configurations (e.g., GPU batch sizes, attention backends, cache settings) to find the one that provides the best performance under a specific workload and objective.

This command is especially useful for preparing a model to be later served and benchmarked under optimal conditions.

Command Syntax

mako tune [OPTIONS...] MODEL_NAME

Required arguments

  • MODEL_NAME (str, positional) - Name of the model to tune.

Options

  • --engine-type (str)- Engine to use (vllm or sglang). If not provided, Mako will auto-detect.

  • --objective (str) - Name of the metrics(s) to optimize when tuning (e.g., input_throughput). Default: output_throughput. See technical notes.

    Changed in version 1.3.0

    --objective can now be specified multiple times, to start multi-objective optimization (but note it might not be supported in all cases). Also see --constraint below.

  • --constraint (str) - Add a constraining expression to the optimization problem. The expression should be in the form of {metric} <> {threshold}, where metric is one of the possible objectives, <> is one of the following comparators: <, >, <=, or >=, and threshold is a Python floating-point literal. Can be specified multiple times, in which case all provided constraint have to be met for a point to be considered valid.

    Added in version 1.3.0

  • --results, -r (str) - Path to a directory or file containing tuning results.

  • --session, -s (str) - Directory for saving a full tuning session (config trials, logs, etc.). Technical notes.
  • --workload, -w (str) - Workload name to use for evaluation. Technical notes.
  • --workloads-spec (str) - Path to a YAML file that defines custom workload specifications. See: custom workloads.
  • --search-space (str) - A YAML file specifying the search space for tuning. See: custom search space.
  • --extra-args (str) - YAML file containing additional fixed arguments for the engine. See: fixed parameter.
  • --fresh (flag) - Start the search from scratch, even if the same setting has already been run. This does not remove any old data, simply forces a new session directory.
  • --reset (flag) - Remove any existing results for the same setting before starting a search.
  • --configs-dir, -c (str) - Adds a location to look for config files (only relevant if config files are not explicitly specified with other arguments). Can be specified multiple times, each time a location is appended. Locations are searched in-order.
  • --data-dir (str) - Adds a location to look for data files. Can be specified multiple times, the location will be first searched in order and then used in-order to try to save the file into.
  • --tensor-parallel-size (int) - Enforces particular tensor parallel size when running servers, if neither this nor --data-parallel-size is provided, it will be detected automatically. See technical notes on model splitting.
  • --data-parallel-size (int) - Enforces particular data parallel size when running servers. If not provided, data parallelism will not be used.
  • --skip-hf-download (flag) - Don’t pre-download the model from HuggingFace Hub (useful for local-only models).
  • --max-evals (int) - Maximum number of evaluations that the search is allowed to perform before stopping. This does not include evaluation requests that hit the cache. Zero or None means unlimited (default). Note: this is clamped to the number of elements in the search space. See technical notes on stopping criteria.
  • --max-history (int) - Maximum number of entries in the search history allowed (including reused results). Zero or None means unlimited (default).
  • --timeout (int) - Timeout in seconds for the entire tuning process to finish. When the timeout expires, any pending evaluations will still be finished but no new ones will be queued. Zero, None or negative values mean no timeout (default). Note that the timeout is only one of the stopping criteria, others include max-evals, max-history and running out of points in a search space.

Notes

  • Tuning automatically launches short-lived servers for each configuration trial, benchmarks them, and shuts them down.
  • Once tuning is complete, the best configuration can be reused in mako serve or mako benchmark, either by pulling from Bench Hub (if tuning results were being uploaded) or from locally-stored results (see the --results flag).

Examples

Quick start: just specify the model name

mako tune meta-llama/Llama-3.1-8B-Instruct

Tune a model with a specified workload

mako tune meta-llama/Llama-3.1-8B-Instruct --workload rand_10k

Specify a custom workload spec file

mako tune meta-llama/Llama-3.1-8B-Instruct --workloads-spec ./custom_workload.yaml

Tune for a specific metric

mako tune meta-llama/Llama-3.1-8B-Instruct --objective input_throughput

Save full tuning session

mako tune meta-llama/Llama-3.1-8B-Instruct --session ./sessions/llama_run