Skip to content

mako benchmark

Benchmarks a running or locally launched LLM server using a selected workload. You can use this command to evaluate performance with various configurations or measure results of external servers.

Command Syntax

mako benchmark [OPTIONS...] MODEL_NAME

Required arguments

  • MODEL_NAME (str, positional) - Name of the model to benchmark.

If benchmarking an external server

  • --address (str) - Benchmark an already running external server at the specified address. If set, results will not be uploaded to Bench Hub or otherwise recorded since the configuration of the server is unknown. The address should be in the form of: [proto://]host[:port], where proto:// is either http:// or https:// and defaults to http://, host is either an IP address or a name of the target host, and :port is optional port number (defaults to the default for a relevant protocol).

    Changed in version 1.3.0

    --address now accepts the optional proto:// prefix, and allows omitting :port. Before http:// was always used and :port was required.

  • --api-key (str) - API key to use when benchmarking an external server (relevant only if --address is provided).

Benchmarking locally (auto-launching server)

  • --engine-type (str) - Which engine to use (vllm or sglang). If not set, Mako will try to auto-detect it.
  • --baseline (flag) - Start the server using vllm or sglang directly, otherwise they will be launched via mako serve and subject to using optimized settings.
  • --skip-hf-download (flag) - Don’t pre-download the model from HuggingFace Hub (useful for local-only models).

Common arguments

  • --workload, -w (str) - Name of the workload to use for benchmarking. See technical notes on workloads. This is also passed to mako serve, if relevant.
  • --workloads-spec (str) - Path to a YAML file that defines custom workloads. See: custom workloads.
  • --dont-upload (flag) - Skip uploading benchmark results to Bench Hub.
  • --verbose, -v (int) - Increase verbosity level (e.g., -v 1, -v 2). Controls debug output during the launch.

Notes

  • If you do not specify --address, Mako will automatically launch a local server, benchmark it, and shut it down afterward.'
  • If --address is provided, no local server will be launched and benchmark results will not be uploaded to Bench Hub.
  • Without --tensor-parallel-size, your benchmark will run fine, but results might not be uploaded to Bench Hub. This argument is required for proper identification and validation on the platform.

Examples

Benchmark a local server with optimized settings

mako benchmark meta-llama/Llama-3.1-8B-Instruct

Benchmark a server with default (non-optimized) config

mako benchmark meta-llama/Llama-3.1-8B-Instruct --baseline

Benchmark a remote server at given address

mako benchmark meta-llama/Llama-3.1-8B-Instruct --address localhost:8000