32a241c28f
Some checks failed
nightly / Test against latest dependencies (py3.10) (push) Has been cancelled
nightly / Test against latest dependencies (py3.13) (push) Has been cancelled
tests / tests (3.13, locked) (push) Has been cancelled
tests / tests (3.13, unlocked) (push) Has been cancelled
tests / pyright (3.10) (push) Has been cancelled
tests / lint-format (push) Has been cancelled
tests / tests (3.10, locked) (push) Has been cancelled
tests / tests (3.11, locked) (push) Has been cancelled
tests / tests (3.12, locked) (push) Has been cancelled
tests / pyright (3.11) (push) Has been cancelled
tests / pyright (3.12) (push) Has been cancelled
tests / pyright (3.13) (push) Has been cancelled
tests / ty-check (3.10) (push) Has been cancelled
tests / ty-check (3.11) (push) Has been cancelled
tests / ty-check (3.12) (push) Has been cancelled
tests / ty-check (3.13) (push) Has been cancelled
tests / stubs (push) Has been cancelled
tests / smoke-test (push) Has been cancelled
Docker / check_paths (push) Has been cancelled
docs / build (push) Has been cancelled
Docker / build (push) Has been cancelled
Upstream: https://github.com/michaelgillett/mjlab Upstream-Commit: c19f713c415a699a79d71cd96aa13c3104a05047 Upstream-Branch: main
77 lines
1.9 KiB
Markdown
77 lines
1.9 KiB
Markdown
# mjlab Nightly Benchmarks
|
|
|
|
This directory contains scripts for automated nightly benchmarking of mjlab.
|
|
|
|
## Overview
|
|
|
|
The nightly benchmark system:
|
|
1. Trains a tracking policy on the latest commit
|
|
2. Evaluates the policy across 1024 trials
|
|
3. Measures simulation throughput
|
|
4. Generates an HTML report with historical trends
|
|
5. Publishes results to GitHub Pages
|
|
|
|
## Usage
|
|
|
|
### Run the full nightly benchmark
|
|
|
|
```bash
|
|
./scripts/benchmarks/nightly_train.sh
|
|
```
|
|
|
|
### Skip training (regenerate report only)
|
|
|
|
```bash
|
|
SKIP_TRAINING=1 ./scripts/benchmarks/nightly_train.sh
|
|
```
|
|
|
|
### Skip training and throughput
|
|
|
|
```bash
|
|
SKIP_TRAINING=1 SKIP_THROUGHPUT=1 ./scripts/benchmarks/nightly_train.sh
|
|
```
|
|
|
|
### Regenerate report directly (no git operations)
|
|
|
|
```bash
|
|
uv run python scripts/benchmarks/generate_report.py \
|
|
--entity gcbc_researchers \
|
|
--tag nightly \
|
|
--output-dir benchmark_results
|
|
```
|
|
|
|
### Measure throughput only
|
|
|
|
```bash
|
|
uv run python scripts/benchmarks/measure_throughput.py \
|
|
--num-envs 4096 \
|
|
--output-dir benchmark_results
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Environment variables for `nightly_train.sh`:
|
|
|
|
- `CUDA_DEVICE` - GPU device to use (default: 0)
|
|
- `WANDB_TAGS` - Comma-separated tags for the run (default: nightly)
|
|
- `SKIP_TRAINING` - Set to "1" to skip training
|
|
- `SKIP_THROUGHPUT` - Set to "1" to skip throughput benchmarking
|
|
|
|
## Automated Setup
|
|
|
|
See [systemd/README.md](systemd/README.md) for instructions on setting up automated nightly runs using systemd timers.
|
|
|
|
## Report Options
|
|
|
|
The `generate_report.py` script supports:
|
|
|
|
- `--eval-limit N` - Maximum number of NEW runs to evaluate per invocation (default: 10)
|
|
- Set to 0 for no limit
|
|
- Historical cached results are always preserved
|
|
- `--tag TAG` - Filter runs by WandB tag (default: "nightly")
|
|
- `--num-envs N` - Number of parallel environments for evaluation (default: 1024)
|
|
|
|
## Viewing Results
|
|
|
|
Reports are published to: https://mujocolab.github.io/mjlab/nightly/
|