Design Effectiveness and Operating Effectiveness

Design effectiveness and operating effectiveness. Two questions that sound almost the same and are not, and testing the second before you have established the first produces findings that do not hold up.

Start with a situation. Every quarterly access review in the period was completed on time and signed by the right manager. The evidence is immaculate, and you are about to conclude that access review is operating effectively.

Then you look at what the reviewers were actually sent. A list of user names. No entitlements, no roles, no permissions. They were asked to confirm that these people still work here, which is not the same question as whether their access is still appropriate.

So separate the two questions cleanly. Design effectiveness asks: if this control were performed exactly as intended, would it address the risk? Operating effectiveness asks: was it actually performed that way, consistently, throughout the period? One is about whether the control can work. The other is about whether it did.

And that access review is designed ineffectively. It cannot identify inappropriate access, because the reviewer never sees the access. Its perfect operation is irrelevant. Reporting it as an operating issue would misdirect management entirely, they would tighten the schedule when they need to redesign the report.

Which is why design comes first, and it is not merely tidy sequencing. Testing the operation of a control that cannot work is wasted effort, and worse, it risks a clean conclusion on a control that never addressed the risk. You assess design by walking the control through once: the risk, the objective, who performs it, what they see, what they compare against, and what happens when they find an exception. That last part is where design deficiencies most often hide.

So hold the three combinations in your head. Design effective and operation effective, you can rely on the control. Design effective but operation deficient, the control could work but did not, so there is no reliance for the affected period. And design deficient, in which case you do not test operation at all, because a control that is designed ineffectively cannot be operating effectively. That combination does not exist, and any option offering it is wrong.

When a control is deficient, you consider whether another control addresses the same risk. But a genuine compensating control has to qualify, and candidates wave things through that do not.

First, it must address the same risk, not a related one. A monthly review of totals does not compensate for an access control failure, because it is aimed at a different thing.

Second, it must operate at sufficient precision to catch what the failed control would have caught. This is the test candidates skip most. A high-level management review would not detect a single inappropriate transaction, so it does not compensate for a control that was meant to.

And third, it must be independent, so the same weakness does not disable both controls at once. Same risk, sufficient precision, independent. Miss any one of the three and it is not a compensating control.

So carry away the rule that sits underneath all of it. A control cannot be operating effectively if it is designed ineffectively. Establish that the control can work before you ever test whether it did.