QQ 201, assessment

Problem set

Some of these have a number for an answer. Others ask for a judgement, and the solution gives ours.

1. Problem 1

   You test 50 parameter combinations of a strategy that has no real edge, each at the 5% significance level. What is the chance at least one looks significant?

   Show the solution

   1 - 0.95 to the power 50 = 1 - 0.077 = about 92%. Finding one that "works" is close to certain.

2. Problem 2

   You want the chance of any false positive across those 50 tests to stay at 5%. Using the simplest correction, what significance level should each test use?

   Show the solution

   Divide by the number of tests: 0.05 / 50 = 0.001, or 0.1% per test. This is the Bonferroni correction. It is deliberately conservative.

3. Problem 3

   A strategy averages 12 dollars per trade before costs and makes 400 trades a year. A round trip costs 7.50 dollars. What is the net profit for the year? What if the real cost is double?

   Show the solution

   Net per trade is 12 - 7.50 = 4.50, so 4.50 x 400 = 1,800 dollars, against 4,800 before costs. At 15 dollars a round trip, net per trade is -3, so the year is minus 1,200 dollars. The strategy's sign depends on the cost assumption.

4. Problem 4

   A rule reads: "If today's close is above the highest close of the last 20 days, including today, buy at today's close." What is wrong with it, and how would you fix it?

   Show the solution

   The decision uses today's close, which is only known once the bar has finished, and then asks to be filled at that same price. The fill is not available. Fix: decide on the close and fill at the next bar's open. Then check the rule survives a one-bar delay.

5. Problem 5

   A walk-forward test shows 30,000 dollars a year in the tuning windows and 12,000 dollars a year in the test windows. What is the walk-forward efficiency, and how would you read it?

   Show the solution

   12,000 / 30,000 = 0.40. Less than half of the tuned performance carried over, so much of the tuning was fitting noise. The terminal's walk-forward check asks for an efficiency of at least 0.5 and a positive out-of-sample profit, so this would not pass.

6. Problem 6

   You sealed the last two years of a ten-year history as a holdout. You have now opened it three times, changing the strategy after each look. A colleague says the third result is your out-of-sample performance. Are they right?

   Show the solution

   No. Each look that led to a change made the holdout part of the search. The third result was chosen with knowledge of the first two, so it is in sample in everything but name. The honest options are to report it as such, or to wait for new data.