32a241c28f
Some checks failed
nightly / Test against latest dependencies (py3.10) (push) Has been cancelled
nightly / Test against latest dependencies (py3.13) (push) Has been cancelled
tests / tests (3.13, locked) (push) Has been cancelled
tests / tests (3.13, unlocked) (push) Has been cancelled
tests / pyright (3.10) (push) Has been cancelled
tests / lint-format (push) Has been cancelled
tests / tests (3.10, locked) (push) Has been cancelled
tests / tests (3.11, locked) (push) Has been cancelled
tests / tests (3.12, locked) (push) Has been cancelled
tests / pyright (3.11) (push) Has been cancelled
tests / pyright (3.12) (push) Has been cancelled
tests / pyright (3.13) (push) Has been cancelled
tests / ty-check (3.10) (push) Has been cancelled
tests / ty-check (3.11) (push) Has been cancelled
tests / ty-check (3.12) (push) Has been cancelled
tests / ty-check (3.13) (push) Has been cancelled
tests / stubs (push) Has been cancelled
tests / smoke-test (push) Has been cancelled
Docker / check_paths (push) Has been cancelled
docs / build (push) Has been cancelled
Docker / build (push) Has been cancelled
Upstream: https://github.com/michaelgillett/mjlab Upstream-Commit: c19f713c415a699a79d71cd96aa13c3104a05047 Upstream-Branch: main
1.9 KiB
1.9 KiB
mjlab Nightly Benchmarks
This directory contains scripts for automated nightly benchmarking of mjlab.
Overview
The nightly benchmark system:
- Trains a tracking policy on the latest commit
- Evaluates the policy across 1024 trials
- Measures simulation throughput
- Generates an HTML report with historical trends
- Publishes results to GitHub Pages
Usage
Run the full nightly benchmark
./scripts/benchmarks/nightly_train.sh
Skip training (regenerate report only)
SKIP_TRAINING=1 ./scripts/benchmarks/nightly_train.sh
Skip training and throughput
SKIP_TRAINING=1 SKIP_THROUGHPUT=1 ./scripts/benchmarks/nightly_train.sh
Regenerate report directly (no git operations)
uv run python scripts/benchmarks/generate_report.py \
--entity gcbc_researchers \
--tag nightly \
--output-dir benchmark_results
Measure throughput only
uv run python scripts/benchmarks/measure_throughput.py \
--num-envs 4096 \
--output-dir benchmark_results
Configuration
Environment variables for nightly_train.sh:
CUDA_DEVICE- GPU device to use (default: 0)WANDB_TAGS- Comma-separated tags for the run (default: nightly)SKIP_TRAINING- Set to "1" to skip trainingSKIP_THROUGHPUT- Set to "1" to skip throughput benchmarking
Automated Setup
See systemd/README.md for instructions on setting up automated nightly runs using systemd timers.
Report Options
The generate_report.py script supports:
--eval-limit N- Maximum number of NEW runs to evaluate per invocation (default: 10)- Set to 0 for no limit
- Historical cached results are always preserved
--tag TAG- Filter runs by WandB tag (default: "nightly")--num-envs N- Number of parallel environments for evaluation (default: 1024)
Viewing Results
Reports are published to: https://mujocolab.github.io/mjlab/nightly/