mirror of
https://github.com/open-policy-agent/opa.git
synced 2026-08-14 12:22:51 -06:00
Torin Sandall
00a71ef465
This commit adds a new kind of indexing to the compiler and topdown to help avoid recomputing comprehensions. This helps with queries that perform "group by" operations. This optimization allows policies to perform group-by/aggregation in O(n) instead of O(n^2). The optimization works by computing a set of index keys for the comprehension at compile-time and then computing the collection once at evaluation-time and indexing the result based on the keys. The index keys are variables in the outer query that limit the values produced by the comprehension. In the simple group-by case these are the object values themselves. During evaluation, topdown checks if indexing is possible and builds the index by computing the comprehension without creating a closure over the outer query. This computes ALL values in the collection defined by the comprehension. The results are keyed by the assignments to the variables indicated in the comprehension index. This way the comprehension does not have to be recomputed for each set of assignments in the outer query. The index is exposed on both the compiler and the query compiler so that ad-hoc queries can benefit from the indexing as well. This is important for things like the playground where users may select a rule body and run it. If that exhibited n^2 behaviour it would be quite confusing. In order to be indexed, the comprehension must meet a few conditions. Importantly, the indexing should not worsen overall performance. To ensure this, comprehensions containing refs or walk() calls that include output vars that close over the outer query are not indexed. This means that if the caller were pushing down assignments to those vars, OPA will not compute the entire collection. In the future we can improve the index to cover more kinds of comprehensions. One improvement that would be particularly nice would be to allow the comprehension index to close over specific local variables in the parent scope. This would let us build the index in more cases--however, the analysis would need to be careful to take into account the count of closure variables. Variables with multiple assignments would be poor candidates. Benchmark results (before, O(n^2) runtime): BenchmarkComprehensionIndexing/10-16 13831 85821 ns/op BenchmarkComprehensionIndexing/100-16 208 5662625 ns/op BenchmarkComprehensionIndexing/1000-16 2 549295038 ns/op Benchmark results (after, O(n) runtime): BenchmarkComprehensionIndexing/10-16 35809 33369 ns/op BenchmarkComprehensionIndexing/100-16 3756 274546 ns/op BenchmarkComprehensionIndexing/1000-16 438 2725152 ns/op Fixes #2276 Signed-off-by: Torin Sandall <torinsandall@gmail.com>