Reference · v1.1.0

Declare how your task judges candidates

Define what scientific progress means before running experiments. The optional evaluation: section in task.yaml describes your metrics, requirements, and trade-offs. Every research agent reads the same rules.

Source of truth: the v1.1.0 evaluation guide (opens in a new tab)

Keep three decisions separate: Health checks whether a result is valid, constraints check whether it meets your requirements, and selection ranks the eligible results.

Evaluation context

Explain the science to every agent

Declare what each quantity means, its units, and which direction is better. Agents use the same rules when proposing, implementing, and interpreting models.

Weighted formula

Balance numerical objectives

Combine normalized metrics or measurements using positive weights and explicit bounds. For example, balance signal recovery with model size. The framework computes the score.

Hard constraints

Set requirements that cannot be traded away

Require accuracy of at least 90% before comparing speed. A candidate that violates a requirement cannot become the best result; a missing or non-finite required value also fails.

Candidate measurements

Measure the model itself

Measure trainable or total parameters, model bytes, and inference latency. Latency uses a declared batch shape, warm-up, and repeated forward passes on the training device; it excludes data loading.

Rubric components

Describe qualities that need judgment

Give agents explicit criteria for interpretability, scientific evidence, or practical usefulness. These guide their work; they do not automatically produce numerical scores for trained models.

Rubric balancing

Explain your preferred trade-offs

Tell agents, for example, to favor a simpler model when accuracy is similar. Numerical selection still uses your declared primary metric or weighted formula; the rubric adds no separate judging calls.

These capabilities are available in SIDERIUS v1.1.0. See the evaluation guide (opens in a new tab) for declarations and examples.

Formulas, constraints, and candidate measurements use native task-owned scoring. Measurement failures make a candidate unscoreable. Rubric guidance is separate from automated model judging and the paper’s offline behavior review.

Start with the scientific question

For example: require balanced accuracy of at least 90%, then prefer a faster model. Health checks reject invalid outputs; constraints enforce the accuracy requirement; selection ranks the remaining candidates.

If a single metric is enough, leave evaluation: out. Candidates are ranked by the main metric in its declared direction. Otherwise, ask your coding agent to describe the quantities and implement the evaluation contract:

My scientific goal is [goal]. Define a primary metric, useful secondary metrics, validity checks, and any non-negotiable requirements. Use the SIDERIUS v1.1.0 evaluation guide to implement them. Explain what ranks candidates automatically and what is guidance for the agents. Check the task declaration before running experiments.

Give each metric meaning

This example assumes both metrics are declared by the task. The ranges describe the quantities; they do not enforce constraints or normalize a formula.

evaluation:
  components:
    - kind: metric
      id: balanced_accuracy             # your name for this component
      metric: balanced_accuracy         # a declared metric id
      meaning: Mean per-class recall on validation.csv; each species counts equally.
      units: fraction (0 to 1), mean over classes
      range: {lower: 0.0, upper: 1.0}   # descriptive only
    - kind: metric
      id: worst_class_error
      metric: worst_class_error         # declared under secondary_metrics:
      meaning: Error rate on the hardest species.
      units: fraction (0 to 1) of the hardest class's rows
      range: {lower: 0.0, upper: 1.0}
  selection:
    kind: primary                       # still rank by the main metric

For weighted formulas, constraints, measurement declarations, and rubric examples, follow the versioned evaluation guide (opens in a new tab). Formula ranking uses normalized utility, not a raw sum of incompatible units.

Check before running

Ask your agent to save this script outside the SIDERIUS checkout as check_evaluation.py, then run it from the checkout with the task manifest path. It checks composition and runtime support without training or calling an LLM. Success ends with Launch check: admitted; this checks the declaration, not the scientific quality of a result.

import json
import sys

from agent.evaluation_context import build_evaluation_prompt_context
from agent.prompt_templates.evaluation import render_evaluation_spec
from workflows.evaluation_composition import require_evaluation_runtime
from workflows.task_composition import TaskCompositionError, compose_run_task_bindings

try:
    task = compose_run_task_bindings(sys.argv[1])
except TaskCompositionError as error:
    sys.exit(f"Composition refused: {error}")
if task.evaluation is None:
    print("No evaluation section: candidates are ranked by the main metric.")
    sys.exit(0)
context = build_evaluation_prompt_context(task.evaluation, task.metric.spec, task.judge_task)
print(render_evaluation_spec(context))
print(json.dumps(task.evaluation.inspect(), indent=2))
try:
    require_evaluation_runtime(task.evaluation)
except TaskCompositionError as error:
    sys.exit(f"Launch check: {error}")
print("Launch check: admitted")

.venv/bin/python /absolute/project/check_evaluation.py /absolute/project/tasks/iris/task.yaml