Torin Sandall 00a71ef465 ast, topdown: Index comprehensions to avoid unnecessary work
This commit adds a new kind of indexing to the compiler and topdown to
help avoid recomputing comprehensions. This helps with queries that
perform "group by" operations.

This optimization allows policies to perform group-by/aggregation in
O(n) instead of O(n^2). The optimization works by computing a set of
index keys for the comprehension at compile-time and then computing
the collection once at evaluation-time and indexing the result based
on the keys.

The index keys are variables in the outer query that limit the values
produced by the comprehension. In the simple group-by case these are
the object values themselves. During evaluation, topdown checks if
indexing is possible and builds the index by computing the
comprehension without creating a closure over the outer query. This
computes ALL values in the collection defined by the
comprehension. The results are keyed by the assignments to the
variables indicated in the comprehension index. This way the
comprehension does not have to be recomputed for each set of
assignments in the outer query.

The index is exposed on both the compiler and the query compiler so
that ad-hoc queries can benefit from the indexing as well. This is
important for things like the playground where users may select a rule
body and run it. If that exhibited n^2 behaviour it would be quite
confusing.

In order to be indexed, the comprehension must meet a few
conditions. Importantly, the indexing should not worsen overall
performance. To ensure this, comprehensions containing refs or walk()
calls that include output vars that close over the outer query are not
indexed. This means that if the caller were pushing down assignments
to those vars, OPA will not compute the entire collection.

In the future we can improve the index to cover more kinds of
comprehensions. One improvement that would be particularly nice would
be to allow the comprehension index to close over specific local
variables in the parent scope. This would let us build the index in
more cases--however, the analysis would need to be careful to take
into account the count of closure variables. Variables with multiple
assignments would be poor candidates.

Benchmark results (before, O(n^2) runtime):

BenchmarkComprehensionIndexing/10-16 	   13831	     85821 ns/op
BenchmarkComprehensionIndexing/100-16         	     208	   5662625 ns/op
BenchmarkComprehensionIndexing/1000-16        	       2	 549295038 ns/op

Benchmark results (after, O(n) runtime):

BenchmarkComprehensionIndexing/10-16 	   35809	     33369 ns/op
BenchmarkComprehensionIndexing/100-16         	    3756	    274546 ns/op
BenchmarkComprehensionIndexing/1000-16        	     438	   2725152 ns/op

Fixes #2276

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2020-05-01 07:59:40 -04:00
2019-08-14 17:21:17 -04:00
2020-04-09 10:46:13 -07:00
2016-10-17 11:03:48 -07:00
2020-04-09 10:46:13 -07:00
2020-02-24 14:03:42 -08:00
2020-03-30 11:25:21 -07:00
2020-04-27 15:54:33 -07:00
2020-04-16 16:10:53 -07:00
2020-04-30 14:53:20 -04:00
2020-04-09 10:46:13 -07:00
2020-04-09 10:46:13 -07:00
2015-12-28 14:08:25 -08:00
2020-03-26 04:57:12 -07:00
2019-09-18 13:36:15 -04:00
2020-03-26 04:57:12 -07:00

logo Open Policy Agent

Slack Status Build Status Go Report Card CII Best Practices Netlify Status

The Open Policy Agent (OPA) is an open source, general-purpose policy engine that enables unified, context-aware policy enforcement across the entire stack.

OPA is hosted by the Cloud Native Computing Foundation (CNCF) as an incubating-level project. If you are an organization that wants to help shape the evolution of technologies that are container-packaged, dynamically-scheduled and microservices-oriented, consider joining the CNCF. For details read the CNCF announcement.

Want to learn more about OPA?

Want to get OPA?

Want to integrate OPA?

  • See GoDoc to integrate OPA with services written in Go.
  • See REST API to integrate OPA with services written in other languages.

Want to contribute to OPA?

How does OPA work?

OPA gives you a high-level declarative language to author and enforce policies across your stack.

With OPA, you define rules that govern how your system should behave. These rules exist to answer questions like:

  • Can user X call operation Y on resource Z?
  • What clusters should workload W be deployed to?
  • What tags must be set on resource R before it's created?

You integrate services with OPA so that these kinds of policy decisions do not have to be hardcoded in your service. Services integrate with OPA by executing queries when policy decisions are needed.

When you query OPA for a policy decision, OPA evaluates the rules and data (which you give it) to produce an answer. The policy decision is sent back as the result of the query.

For example, in a simple API authorization use case:

  • You write rules that allow (or deny) access to your service APIs.
  • Your service queries OPA when it receives API requests.
  • OPA returns allow (or deny) decisions to your service.
  • Your service enforces the decisions by accepting or rejecting requests accordingly.

The examples below show different kinds of policies you can define with OPA as well as different kinds of queries your system can execute against OPA. The example queries are executed inside OPA's REPL which was built to make it easy to develop and test policies.

For concrete examples of how to integrate OPA with systems like Kubernetes, Terraform, Docker, SSH, and more, see openpolicyagent.org.

Example: API Authorization

This example shows how you can enforce access controls over salary information served by a simple HTTP API. In this example, users are allowed to access their own salary as well as the salary of anyone who reports to them.

The management chain is represented in JSON and stored in a file (data.json):

{
    "management_chain": {
        "bob": [
            "ken",
            "janet"
        ],
        "alice": [
            "janet"
        ]
    }
}

Start OPA and load the data.json file:

opa run data.json

Inside the REPL you can define rules and execute queries. Paste the following rules into the REPL.

default allow = false

allow {
    input.method = "GET"
    input.path = ["salary", id]
    input.user_id = id
}

allow {
    input.method = "GET"
    input.path = ["salary", id]
    managers = data.management_chain[id]
    input.user_id = managers[_]
}

Example Queries

Is someone allowed to access their own salary?

> input := {"method": "GET", "path": ["salary", "bob"], "user_id": "bob"}
> allow
true

Display the management chain for Bob:

> data.management_chain["bob"]
[
    "ken",
    "janet"
]

Is Alice allowed to access Bob's salary?

> input := {"method": "GET", "path": ["salary", "bob"], "user_id": "alice"}
> allow
false

Is Janet allowed to access Bob's salary?

> input := {"method": "GET", "path": ["salary", "bob"], "user_id": "janet"}
> allow
true

Example: App Placement

This example shows how you can enforce where apps are deployed inside a simple orchestrator. In this example, apps must be deployed onto clusters that satisfy PCI and jurisdiction requirements.

app_placement[cluster_id] {
    cluster = data.clusters[cluster_id]
    satisfies_jurisdiction(input.app, cluster)
    satisfies_pci(input.app, cluster)
}

satisfies_jurisdiction(app, cluster) {
    not app.tags["requires-eu"]
}

satisfies_jurisdiction(app, cluster) {
    app.tags["requires-eu"]
    startswith(cluster.region, "eu-")
}

satisfies_pci(app, cluster) {
    not app.tags["requires-pci-level"]
}

satisfies_pci(app, cluster) {
    level = to_number(app.tags["requires-pci-level"])
    level >= cluster.tags["pci-level"]
}

Example Queries

Where will this app be deployed?

> input := {"app": {"tags": {"requires-pci-level": "3", "requires-eu": "true"}}}
> app_placement
[
    "prod-eu"
]

Display clusters in EU region:

> startswith(data.clusters[cluster_id].region, "eu-")
+------------+
| cluster_id |
+------------+
| "prod-eu"  |
| "test-eu"  |
+------------+

Display all clusters:

> data.clusters[cluster_id]
+------------+------------------------------------------------+
| cluster_id |           data.clusters[cluster_id]            |
+------------+------------------------------------------------+
| "prod-eu"  | {"region":"eu-central","tags":{"pci-level":2}} |
| "prod-us"  | {"region":"us-east"}                           |
| "test-eu"  | {"region":"eu-west","tags":{"pci-level":4}}    |
| "test-us"  | {"region":"us-west"}                           |
+------------+------------------------------------------------+

Example: SSH Auditing

This example shows how you can audit who has SSH access to hosts within different clusters. We will assume that SSH access is granted via group access in LDAP.

import data.ldap
import data.clusters

ssh_access[[cluster_name, host_id, user_id]] {
    host_id = clusters[cluster_name].hosts[_]
    group_id = ldap.users[user_id].groups[_]
    group_id = clusters[cluster_name].groups[_]
}

prod_users = {user_id | ssh_access[["prod", _, user_id]]}

Example Queries

Who can access production hosts?

> prod_users
[
  "alice",
  "bob"
]

Display all LDAP users:

> data.ldap.users[user_id]
+-------------------------------+---------+
|   data.ldap.users[user_id]    | user_id |
+-------------------------------+---------+
| {"groups":["dev","platform"]} | "alice" |
| {"groups":["dev","ops"]}      | "bob"   |
| {"groups":["dev"]}            | "janet" |
+-------------------------------+---------+

Display all cluster/group pairs:

> data.clusters[cluster_id].groups[_] = group_id
+------------+------------+
| cluster_id |  group_id  |
+------------+------------+
| "test"     | "dev"      |
| "test"     | "ops"      |
| "prod"     | "ops"      |
| "prod"     | "platform" |
+------------+------------+

Does Janet have access to the test cluster?

> ssh_access[["test", _, "janet"]]
true

What are the addresses of the hosts in the test cluster that Janet can access?

> ssh_access[["test", host_id, "janet"]]; addr = data.hosts[host_id].addr
+------------+------------+
|    addr    |  host_id   |
+------------+------------+
| "10.0.0.1" | "host-abc" |
| "10.0.0.2" | "host-cde" |
| "10.0.0.3" | "host-efg" |
+------------+------------+

Further Reading

Presentations

  • Open Policy Agent Introduction @ CloudNativeCon EU 2018: video, slides
  • Rego Deep Dive @ CloudNativeCon EU 2018: video, slides
  • How Netflix Is Solving Authorization Across Their Cloud @ CloudNativeCon US 2017: video, slides.
  • Policy-based Resource Placement in Kubernetes Federation @ LinuxCon Beijing 2017: slides, screencast.
  • Enforcing Bespoke Policies In Kubernetes @ KubeCon US 2017: video, slides.
  • Istio's Mixer: Policy Enforcement with Custom Adapters @ CloudNativeCon US 2017: video, slides.

Security

Security Audit

A third party security audit was performed by Cure53, you can see the full report here

Reporting Security Vulnerabilities

Please report vulnerabilities by email to open-policy-agent-security. We will send a confirmation message to acknowledge that we have received the report and then we will send additional messages to follow up once the issue has been investigated.

Languages
Go 85.1%
C 7.5%
C++ 5.6%
Open Policy Agent 0.8%
Open-Policy-Agent 0.4%
Other 0.5%