Skip to content

mako serve

The serve command launches an LLM inference server using either the vLLM or sglang engine, and a pre-computed optimized configuration. You specify a model name (from Hugging Face), and Mako will handle engine setup, configuration loading, and server launch. This command is typically used before benchmarking or tuning. Most of the optional arguments can be additionally used to differentiate between configurations optimized for different use cases.

Command Syntax

mako serve [OPTIONS...] MODEL_NAME

Required arguments

  • MODEL_NAME (str, positional) - Name of the model to serve.

Requesting arbitrary configuration

  • --engine-config-id (str) - Load a specific configuration from Bench Hub using its ID. If not provided, Mako tries to find the best-known configuration for the model automatically, using the arguments listed below.

(or) Letting Mako choose the right configuration

  • --results, -r (str) - Path to a local directory or file containing additional tuning results (e.g., from mako tune) to what is available in Bench Hub.

  • --objective (str) - Name of the metrics(s) for which the configuration should have been optimized (e.g., input_throughput). To view all available options, run: mako info --objective. Default: output_throughput.

    Changed in version 1.3.0

    --objective can now be specified multiple times, to support multi-objective optimization (but note it might not be supported in all cases). Also see --constraint below.

  • --constraint (str) - Add a constraining expression to the optimization problem. The expression should be in the form of {metric} <> {threshold}, where metric is one of the possible objectives, <> is one of the following comparators: <, >, <=, or >=, and threshold is a Python floating-point literal. Can be specified multiple times, in which case all provided constraint have to be met for a configuration to be considered valid.

    Added in version 1.3.0

  • --workload, -w (str) - Name of the workload for which the configuration should have been optimized. For details, see technical notes on workloads.

Common arguments

  • --engine-type ([sglang|vllm]) - which engine to use for serving (vLLM or sglang). If unspecified, the engine will be auto-detected from the environment. This argument will also influence the automatic configuration lookup.

  • --workloads-spec (str) - Path to a YAML file that defines custom workload specifications. See: custom workloads.

  • --configs-dir, -c (str) - Additional directory to search for config files. Can be specified multiple times to add multiple locations.
  • --verbose, -v (int) - Verbosity level as an integer (e.g., -v 1, -v 2). The higher the number, the more verbose is the output.
  • --num-gpus (int) - How many GPUs to target. If unspecified or set to a negative number, Mako will asume all visible GPUs can be used. Note: how many GPUs are actually used will depend on the configuration loaded, this is only an upper limit.
  • --dump-config (str) - Save the final config used to launch the server to a file (for reproducibility or debugging).

Notes

  • All unrecognized arguments are passed to the underlying engine.
  • Any of such arguments are subject to being overwritten by the loaded configuration. You can use --dump-config to inspect the effective configuration and arguments for the engine.

Examples

Launch with default settings

mako serve meta-llama/Llama-3.1-8B-Instruct --engine-type vllm

Launch with a custom objective

mako serve meta-llama/Llama-3.1-8B-Instruct --objective input_throughput