# Introduction

`srtctl` is a command-line tool for running distributed LLM inference benchmarks on SLURM clusters. It replaces complex shell scripts and 50+ CLI flags with clean, declarative YAML configuration files.

## Table of Contents

* [Why srtctl?](#why-srtctl)
* [How It Works](#how-it-works)
* [Commands](#commands)
* [Next Steps](#next-steps)

## Why srtctl?

Running large language models across multiple GPUs and nodes requires orchestrating many moving parts: SLURM job scripts, container mounts, SGLang configuration, worker coordination, and benchmark execution. Traditionally, this meant maintaining brittle bash scripts with hardcoded parameters.

`srtctl` solves this by providing:

* **Declarative configuration** - Define your entire job in a single YAML file
* **Validation** - Catch configuration errors before submitting to SLURM
* **Reproducibility** - Every job saves its full configuration for later reference
* **Parameter sweeps** - Run grid searches across configurations with a single command
* **Profiling support** - Built-in torch/nsys profiling modes

## How It Works

When you run `srtctl apply -f config.yaml`, the tool:

1. Validates your configuration against the schema
2. Resolves any aliases from your cluster config (`srtslurm.yaml`)
3. Generates a SLURM batch script and SGLang configuration files
4. Submits to SLURM

Once allocated, workers launch inside containers, discover each other through ETCD and NATS, and begin serving. If you've configured a benchmark, it runs automatically against the serving endpoint and saves results to the log directory.

## Commands

| Command                                            | Description                             |
| -------------------------------------------------- | --------------------------------------- |
| `srtctl apply -f <config>`                         | Submit job(s) to SLURM                  |
| `srtctl apply -f <config> --setup-script <script>` | Submit with custom setup script         |
| `srtctl apply -f <config> --tags tag1,tag2`        | Submit with tags for filtering          |
| `srtctl dry-run -f <config>`                       | Validate and preview without submitting |
| `srtctl validate -f <config>`                      | Alias for dry-run                       |

## Next Steps

* [Installation](/srtctl-docs/getting-started/installation.md) - Set up `srtctl` and submit your first job
* [Monitoring](/srtctl-docs/operations/monitoring.md) - Understanding job logs and debugging
* [Parameter Sweeps](/srtctl-docs/configuration/sweeps.md) - Run grid searches across configurations
* [Config Overrides](https://github.com/ishandhanani/srt-slurm/blob/main/docs/overrides.md) - Multi-variant jobs from a single file
* [Profiling](/srtctl-docs/benchmarking/profiling.md) - Performance analysis with torch/nsys
* [Analyzing Results](/srtctl-docs/operations/analyzing.md) - Dashboard and visualization
* [SGLang Router](/srtctl-docs/architecture-and-frontends/sglang-router.md) - Alternative to Dynamo for PD disaggregation


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://srtctl.gitbook.io/srtctl-docs/getting-started/readme.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.