Optimize a Strategy
A backtest gives you one result from one setup on one window of history. Optimization asks whether that number is a real property of your logic, or whether you tuned your way into a lucky corner of the past that will not repeat live.
Optimization is no longer a separate button in the toolbar. It is the middle stage of the Backtesting Suite: replay first, then compare parameter sets, then validate. Everything below happens inside that one pane, attached to one backtesting report.
Backtests, parameter comparisons, and validation checks are all simulations on historical market data, never live orders. Check for overfitting now, before real funds are on the line.
The three stages
The suite runs one evidence path, and each stage names the action that comes next.
| Stage | Question it answers |
|---|---|
| Historical replay | How did this exact strategy behave over the chosen window |
| Compare parameters | Which values score best on your objective |
| Historical validation | Does the winner hold up on data it never saw |
Each stage feeds the next. A parameter comparison needs a completed replay behind it. The validation checks need a comparison to test, because two of the three grade the selection process, not just the values it picked.
Compare parameters
Run a backtest first. Then, from Advanced tools in the suite, click Compare parameters. The Tune parameters panel opens and configures the comparison attached to that report.
Choose what to vary
The comparison works on the numeric knobs already in your strategy: indicator periods, constants, comparison thresholds, order sizes, take-profit and stop-loss levels. It detects those locations for you, and highlights the node on the canvas as you hover a parameter, so you always know which knob you are about to move.
You then decide which ones to vary and over what values, either as a range (min, max, step) or an explicit list. Locations that look like the same knob, an EMA period used in two places at the same value, merge into a single parameter so they move together. Split them if you want them tuned independently, or link two that should stay in step.
Every combination is a full historical backtest, so the panel shows a live count of the grid against a hard cap and blocks a run that exceeds it. Three parameters with ten values each is a thousand backtests. Vary the knobs that matter.
If your strategy has no numeric fields to vary, the panel says so: add an indicator period, a constant, or a condition threshold before comparing parameters.
Choose an objective
An objective is the single metric the comparison ranks by, plus a direction, either Maximize or Minimize. Pick Maximize for a metric where more is better (return, Sharpe) and Minimize where less is (drawdown).
| Objective | What it rewards |
|---|---|
| Total return % | The largest total percentage gain over the window |
| Sharpe | The best return per unit of total volatility |
| Sortino | The best return per unit of downside volatility only |
| Max drawdown % | The shallowest worst peak-to-trough decline (minimize this) |
| Profit factor | The highest gross profit divided by gross loss |
| Win rate | The highest share of profitable trades |
| Trade count | The most trades taken |
| Closed trade count | The most positions actually closed |
Profit factor above 1 means the winners outweigh the losers in dollars. It is a favorite because it resists the trap of a high win rate hiding a few catastrophic losses.
The overfitting-risk check works on return-path objectives only: Total return %, Sharpe, Sortino, and Max drawdown %. Comparing on any other objective leaves that check with limited evidence, because it cannot compute a probability of backtest overfitting from it.
The frozen plan
The panel shows the plan the comparison will run under: the fitting history it will train on, and the planned number of combinations against the cap. Those dates come from the evaluation parameters already frozen into the report, so the comparison and the replay are graded on the same terms.
Click Compare parameters to run it.
Apply a candidate
The result attaches to the review rather than landing as a separate node. Where several candidates come back, pick one from the Candidate parameter set list, then click Preview changes to see exactly which values move and where, written as current → next against the node each one belongs to. Anything that cannot be mapped back onto the current canvas is flagged instead of silently applied.
Apply and review writes those values into your strategy’s nodes and starts a fresh baseline review on the updated canvas. That boundary matters: applying a candidate changes numbers on your canvas. It does not deploy anything, place an order, or take a position. You still review the updated strategy and deploy it yourself, the same as any other change.
Historical validation
Three checks sit under Historical validation in the suite. Each one carries the question it answers, its method, the exact test conditions it ran under, and its result. Run them, retry a failed one with its frozen configuration, or cancel one in flight.
The panel is explicit about what these are: historical validation, not a forecast and not a live-trading approval.
Out-of-sample test
Fixed parameters, later windows.
Holds the applied parameter values constant and tests them in later windows they were never fitted on. The question: do the exact applied values keep positive risk-adjusted performance out of sample? It needs at least three complete out-of-sample windows with positive mean and minimum Sharpe, and no material caveats, to count as supportive. Fewer windows, missing Sharpe values, or a mismatch with the saved parameters or dates produces limited evidence rather than a verdict.
Walk-forward analysis
Re-optimize, then test the next window.
Repeats the parameter search in each in-sample window, then scores its winner on the window that follows. This grades the selection process rather than one lucky set of values. A setup that only shines in-sample and collapses out-of-sample is exactly the trap walk-forward is built to catch. Same bar: positive mean and minimum out-of-sample Sharpe across at least three windows.
Backtest overfitting risk
CSCV and PBO.
Combinatorially symmetric cross-validation reshuffles your data into many train-and-test splits and measures how often the candidate that looked best in training ranked below the out-of-sample median. The headline number is PBO, the probability of backtest overfitting.
| PBO | Reading |
|---|---|
| Below 25% | Low overfitting risk |
| 25% up to 50% | Elevated risk |
| 50% or higher | High risk |
Low risk also requires at least four usable splits, no skipped splits, and no material warnings. Without those, the check reports limited evidence rather than a clean pass.
Use the three in order: the out-of-sample test on your chosen values, walk-forward on the process that chose them, then the overfitting-risk check to put a number on it. A setup that survives all three has earned a small live allocation rather than a conviction.
Reading the outcome
The suite’s headline status carries the verdict of the checks, not the size of the return. OOS supported means replay, sensitivity, and out-of-sample evidence are all available. Fragile means a check produced adverse evidence or the evidence has material caveats. A check whose evidence is too thin to conclude says so, rather than being counted as a pass.
Alongside the checks, the suite reports the in-sample history the comparison fitted on, and how many later test windows were reserved for grading. Read those two lines before you read any result. They tell you what the number was actually earned against.
Treat a clean walk-forward and a low PBO as permission to test further with a small live allocation, never as a promise. The market regime you fitted on is one of many.
Run it with Signac
Signac can drive most of this from a sentence. Describe the goal in plain words: “compare the fast and slow EMA periods for the best Sharpe,” or “check whether this is overfit.”
- Signac can open the parameter-comparison form prefilled with a plan and a short rationale, so you review the parameters and objective before anything runs.
- It can run a comparison, apply the exact candidate you select, and run a named validation check.
- It cannot invent an objective or a candidate on your behalf. Those have to be grounded in what you asked for.
- Cancelling a run is yours. Use Cancel backtest in the toolbar or the cancel action on a running check.
Signac reads the same review you do and will not claim performance until the evidence returns.
Limits
Optimization inherits every limit of the backtests it runs. Fills are simulated at historical prices, so it cannot reproduce live order-book depth, latency, or market impact. See the backtesting limitations for the full list, and weight the results the same way.
The deeper trap is statistical. The more combinations you try, the more likely one of them looks good by chance alone, which is why walk-forward and CSCV exist. A backtest gives you a number. Validation tells you how much to trust it.