mirror of
https://github.com/open-policy-agent/opa.git
synced 2026-08-13 03:42:35 -06:00
website: Import blog from medium (#8898)
This PR imports the blogs from https://medium.com/open-policy-agent into the docusaurus site. This is being done as the blog has stopped working due to the custom domain hosting feature being removed from our tier. The pages have been imported from manually downloaded webarchive files from each post to ensure we captured the assets and rendered code example too. --------- Signed-off-by: Charlie Egan <charlie_egan@apple.com>
This commit is contained in:
Charlie Egan
@@ -25,3 +25,6 @@ BasedOnStyles =
|
||||
|
||||
[docs/style-guide.md]
|
||||
BasedOnStyles =
|
||||
|
||||
[blog/**.md]
|
||||
OPA.WeOur = NO
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
---
|
||||
title: "What is Policy? Part One: Enforcement"
|
||||
authors: ["tsandall"]
|
||||
date: 2017-02-28
|
||||
slug: what-is-policy-part-one-enforcement-bad8ea8eb35c
|
||||
---
|
||||
|
||||
_Welcome to the Open Policy Agent project. If you're interested in topics like policy, enforcement, remediation, and compliance we'd love to hear from you! Join us on [Slack](http://slack-inviter-1327627577.us-west-2.elb.amazonaws.com) or check out the project on [GitHub](https://github.com/open-policy-agent/opa)._
|
||||
|
||||
This is the first in a two-part series about policy where we introduce definitions, concepts, and challenges in policy enforcement. In future series we'll examine the state of policy in the cloud-native ecosystem.
|
||||
|
||||
The word **policy** means different things to different people in different contexts. In the context of software systems, policies are the rules that govern how the system behaves.
|
||||
|
||||
Computers and humans use policy to answer questions such as:
|
||||
|
||||
- Is this user allowed to change the config of that service?
|
||||
- Is this VM allowed to accept TCP connections from that VM?
|
||||
- Which host should this container be deployed on?
|
||||
- Which workloads are running in the wrong geographic region?
|
||||
|
||||
|
||||
|
||||
We define policy to avoid repeating mistakes and ensure important requirements are met around cost, technology, security, legislation, internal conventions, and so on.
|
||||
|
||||
Understanding where requirements are met and where they aren't is by itself quite valuable (more on this in a later post), but eventually we need to **enforce** policy to ensure that systems behave the way policy says they should.
|
||||
|
||||
We enforce policy differently depending on the organization, the technology the policy applies to, and of course, the policy in question. In some cases, we rely on in-person communication whereas in other cases we embed special-purpose components into our systems that enforce policy automatically.
|
||||
|
||||
|
||||
|
||||
At one end of the spectrum, policies are **tribal knowledge**. They are not recorded anywhere, so if we to want to modify the system, or even understand how it's expected to behave, we ask someone.
|
||||
|
||||
Eventually, we tire of answering (or asking) the same question so we record the answer **on the wiki** or **in the docs**. These docs always fall out of date eventually.
|
||||
|
||||
When our companies grow rapidly, introduce automation, or are faced with frequent policy violations, we usually start **hard-coding** policy into our software. If policy could be defined once and forgotten, we would stop there. In reality, policy evolves. With hard-coded solutions we have to read the code to understand (let alone modify) policy. As a result policy becomes less accessible and more expensive to maintain.
|
||||
|
||||
Once the pain of hard-coding becomes evident, we make our policies **configurable** in software (e.g., including config parameters or even going as far as defining custom [DSLs](https://en.wikipedia.org/wiki/Domain-specific_language).) However, our [abstractions often leak](https://en.wikipedia.org/wiki/Leaky_abstraction) or cannot be adapted for future requirements. As a result, we spend more time and money on development, re-education, and upgrades. It turns out that predicting future requirements around cost, technology, internal conventions, and so on, is HARD.
|
||||
|
||||
The eventual conclusion of this progression is the **policy engine**: a tool for codifying and enforcing policies that is flexible enough to encompass a wide range of future requirements around cost, technology, internal conventions, and the like. It balances the desire to have programmatic enforcement (like hard-coding and configuration) with the need to update policy frequently and inexpensively (like tribal knowledge and the wiki).
|
||||
|
||||
In our next post we'll look at policy engines, declarative languages, and the decoupling of policy decisions from enforcement. Thanks for reading!
|
||||
@@ -0,0 +1,46 @@
|
||||
---
|
||||
title: "What is Policy? Part Two: Policy Engines"
|
||||
authors: ["tsandall"]
|
||||
date: 2017-03-20
|
||||
slug: what-is-policy-part-two-policy-engines-7ee1d972386b
|
||||
---
|
||||
|
||||
_This is the second post in a two-part series about policy enforcement. If you're unfamiliar with the idea of policy enforcement check out [Part One: Enforcement](/blog/what-is-policy-part-one-enforcement-bad8ea8eb35c)._
|
||||
|
||||
Last time we examined alternative approaches to policy enforcement. We saw that tribal knowledge and documentation on wikis provide few guarantees about enforcement but also that automated solutions often make policy difficult to understand and expensive to maintain. In this post, we'll examine how policy engines can help balance the desire for automated enforcement and the need for ease-of-use.
|
||||
|
||||
Policy engines have been around for years. If you've worked on networking or authorization, you've used one when setting up [ACLs](https://en.wikipedia.org/wiki/Access_control_list) or writing [RBAC](https://en.wikipedia.org/wiki/Role-based_access_control) policies. You may also have heard of more general-purpose policy technologies, such as [XACML](https://en.wikipedia.org/wiki/XACML), or protocols like [RADIUS](https://tools.ietf.org/html/rfc2865).
|
||||
|
||||
At a high level, policy engines take policy and data as input and produce answers to policy questions as output. We can design policy engines as libraries, [sidecars](https://www.usenix.org/system/files/conference/hotcloud16/hotcloud16_burns.pdf), or full-blown services. In a future post, we'll examine the trade-offs with each approach.
|
||||
|
||||
When policy engines are integrated into our systems, we refer to those systems as **policy-enabled**. The goal of policy-enabling systems is to **decouple policy decisions from policy enforcement**. This decoupling results in policy implementations that are easier to understand, flexible enough to handle future requirements, and less expensive to maintain.
|
||||
|
||||
For example, a policy-enabled API-gateway asks its policy engine whether a client request should be allowed. The policy engine makes a decision, and the API-gateway rejects the request or forwards it along. Without the policy engine, the logic that decides whether to accept or reject is hard-coded/configured into the API-gateway.
|
||||
|
||||
|
||||
|
||||
Decoupling allows us to define policy in a language different from the one used to implement the service that enforces policy. We can choose a higher-level language for expressing policy that makes policy easier to write, update, understand, analyze, and optimize. For example, it's much simpler (for most people) to read and write **"permit tcp host 1.2.3.4 port http"** than it is read or write the equivalent C (or Java or Python or …) code.
|
||||
|
||||
Policy engines usually support **declarative languages** for defining policy. A declarative language lets us tell the system _what_ we want it to do as opposed to imperative code where we tell the system _how_ to do what we want. Declarative languages balance peoples' need for expressing policy with the policy engine's need to understand the policy definition.
|
||||
|
||||
Declarative languages are also nice because they can provide:
|
||||
|
||||
- **Guarantees that code will return an answer (i.e., it will not run forever.)
|
||||
- **Concise and readable syntax designed to express constraints.
|
||||
- **Consistent and repeatable results given the same code and data.
|
||||
- **Dry-run features to see what would happen if code or data changes somehow.
|
||||
- **Hot-reloading when we want to change the deployed policy.
|
||||
- **Performance optimizations without requiring us to change our code.
|
||||
- **Debugging support to answer questions like "why was decision X made?"
|
||||
|
||||
Beyond the strengths of using declarative languages for policy, decoupling also enables:
|
||||
|
||||
- **Visibility into policy violations that have occurred in the system.
|
||||
- **Automatic remediation when the policy or relevant state changes in the system.
|
||||
- **Sharing across different components in the system (which may be written in different languages).
|
||||
|
||||
All of the benefits described above become available when we decouple policy decisions from policy enforcement. Furthermore, when we use high-level declarative languages to express policy, we simplify the task of reading, writing, and managing the rules that govern our systems.
|
||||
|
||||
That said, it's a lot of work to build policy engines with everything described above. We have to create well-defined languages, implement parsers, compilers, and query evaluation. We also need solid APIs and powerful tools to ingest data, execute queries, debug errors, profile performance, and so on.
|
||||
|
||||
In upcoming series we'll dive into existing policy efforts in the cloud native ecosystem and talk more about [the Open Policy Agent project](http://www.openpolicyagent.org). Thanks for reading!
|
||||
@@ -0,0 +1,177 @@
|
||||
---
|
||||
title: "Authorizing HTTP APIs, SSH, and Puppet with OPA"
|
||||
authors: ["tsandall"]
|
||||
date: 2017-05-12
|
||||
slug: authorizing-http-apis-ssh-and-puppet-with-opa-dc5341602ed5
|
||||
---
|
||||
|
||||
_This is a short post that shows how you can use the Open Policy Agent (OPA) project to enforce authorization policies across HTTP APIs, SSH, and Puppet. If you're interested in policy, authorization, compliance, or other related topics, check out [openpolicyagent.org](http://www.openpolicyagent.org/) or come chat with us on [Slack](http://slack.openpolicyagent.org)._
|
||||
|
||||
One goal of OPA is to solve authorization (who can do what) across the stack. To achieve this goal, OPA provides a **simple HTTP API to integrate at enforcement points and a** high-level declarative language to codify authorization policies. The policy language (Rego) is domain-agnostic and let's you define rich, fine-grained access controls over arbitrary JSON data.
|
||||
|
||||
When you write authorization policy in Rego, you're writing assertions over the state of the world represented as JSON. The state available to the authorization policy is provided either as **input to the authorization query or** pushed from an external data source and stored inside OPA. Because external state can be pushed into OPA, policies can leverage all kinds of context when making their authorization decisions.
|
||||
|
||||
Recently we built a handful of authorization integrations that use OPA at different points in the stack. As part of this effort we're reaching out to other projects that are looking to solve authorization in their domain. We've already built several integrations and examples spanning multiple layers:
|
||||
|
||||
- [Micro-service API authorization with Linkerd](https://github.com/open-policy-agent/contrib/tree/master/linkerd_authz)
|
||||
- [SSH and sudo authorization with a custom PAM module](https://github.com/open-policy-agent/contrib/tree/master/pam_authz)
|
||||
- [Provisioning authorization with Puppet](https://github.com/open-policy-agent/contrib/tree/master/puppet_example)
|
||||
|
||||
Let's look at some examples.
|
||||
|
||||
## HTTP API Authorization
|
||||
|
||||
This simple example shows how to limit read access to an employee's salary in a web app.
|
||||
|
||||
```rego
|
||||
package httpapi.authz
|
||||
|
||||
default allow = false
|
||||
|
||||
# Allow users to get their own salaries.
|
||||
allow {
|
||||
input.method = "GET"
|
||||
input.path = ["finance", "salary", user]
|
||||
user = input.user
|
||||
}
|
||||
|
||||
# Allow managers to get their subordinates' salaries.
|
||||
allow {
|
||||
input.method = "GET"
|
||||
input.path = ["finance", "salary", user]
|
||||
manager_of[user] = input.user
|
||||
}
|
||||
```
|
||||
|
||||
In this case, the policy allows exactly two people to access an employee's salary:
|
||||
|
||||
- The employee themselves (first rule)
|
||||
- The manager of the employee (second rule)
|
||||
|
||||
This is a simplistic example but it helps show how OPA lets you leverage arbitrary data to make policy decisions. In this case, the second rule contains a reference to data ("manager_of") that maps an employee to their manager. For example:
|
||||
|
||||
```
|
||||
{"alice": "bob", "charlie": "betty"}
|
||||
```
|
||||
|
||||
It's worth pointing out that "manager_of" mapping could be defined statically or _dynamically_ in the policy itself. Defining "manager_of" statically would look very familiar:
|
||||
|
||||
```
|
||||
manager_of = {"alice": "bob", "charlie": "betty"}
|
||||
```
|
||||
|
||||
Alternatively, we could define "manager_of" dynamically based on some other data source (e.g., WorkDay, LDAP, etc.). For example:
|
||||
|
||||
```rego
|
||||
package httpapi.authz
|
||||
|
||||
manager_of[employee] = manager {
|
||||
data.employees[employee].team = team_id
|
||||
data.teams[team_id].lead = manager
|
||||
}
|
||||
```
|
||||
|
||||
## SSH Authorization (using Linux-PAM)
|
||||
|
||||
This example shows how to restrict SSH access to users who have contributed to services running on individual hosts. Again, this policy shows how we can leverage external data to make policy decisions.
|
||||
|
||||
```rego
|
||||
package ssh.authz
|
||||
|
||||
default allow = false
|
||||
|
||||
# Allow access to any user that has the "admin" role.
|
||||
allow {
|
||||
data.roles["admin"][_] = input.user
|
||||
}
|
||||
|
||||
# Allow access to any user who contributed to the code running on the host.
|
||||
allow {
|
||||
data.hosts[input.host_identity.host_id].contributors[_] = input.user
|
||||
}
|
||||
```
|
||||
|
||||
In this case, the "roles" and "hosts" refer to external data loaded into OPA:
|
||||
|
||||
```json
|
||||
{
|
||||
"hosts": {
|
||||
"frontend": {
|
||||
"contributors": [
|
||||
"frontend-dev"
|
||||
]
|
||||
},
|
||||
"backend": {
|
||||
"contributors": [
|
||||
"backend-dev"
|
||||
]
|
||||
}
|
||||
},
|
||||
"roles": {
|
||||
"admin": [
|
||||
"ops"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
With PAM you can also control who can run sudo commands. Since our PAM module offloads authorization decisions to OPA, we can extend our authorization policy to cover sudo access without changing any code in the enforcement point. This example shows how you can restrict sudo access to users with an _admin_ role:
|
||||
|
||||
```rego
|
||||
package sudo.authz
|
||||
|
||||
default allow = false
|
||||
|
||||
# Allow sudo access to any user that has the "admin" role.
|
||||
allow {
|
||||
data.roles["admin"][_] = input.user
|
||||
}
|
||||
```
|
||||
|
||||
## Puppet Authorization
|
||||
|
||||
Finally, let's look at how OPA can be used to enforce authorization decisions over more complex data structures such as Puppet catalogs.
|
||||
|
||||
In this case, we assume:
|
||||
|
||||
- **The infrastructure team is responsible for config stored inside /etc/infra
|
||||
- **The app team is responsible for config stored inside /etc/app
|
||||
|
||||
```rego
|
||||
package puppet.authz
|
||||
|
||||
default allow = false
|
||||
|
||||
allow { not deny }
|
||||
|
||||
deny {
|
||||
resource = catalog.resources[resource_index]
|
||||
resource.type = "File"
|
||||
startswith(resource.title, "/etc/infra")
|
||||
resource_author[resource_index] = email
|
||||
not infra_team[email]
|
||||
}
|
||||
|
||||
deny {
|
||||
resource = catalog.resources[resource_index]
|
||||
resource.type = "File"
|
||||
startswith(resource.title, "/etc/infra")
|
||||
resource_author[resource_index] = email
|
||||
not infra_team[email]
|
||||
}
|
||||
```
|
||||
|
||||
This policy combines data from Puppet and Git (blame) to determine if an infrastructure team member has modified files belonging to the app team (or vice-versa).
|
||||
|
||||
## Wrap Up
|
||||
|
||||
If you're interested in trying out these examples, check out the OPA documentation:
|
||||
|
||||
- [HTTP API Authorization (Python)](http://www.openpolicyagent.org/tutorials/http-api-authorization/)
|
||||
- [SSH and sudo Authorization](http://www.openpolicyagent.org/tutorials/ssh-sudo-authorization/)
|
||||
|
||||
We also have examples showing Puppet and Linkerd-based micro-service authorization that can be found in the [open-policy-agent/contrib](http://github.com/open-policy-agent/contrib) repository.
|
||||
|
||||
In upcoming posts we'll dive into more detail on authorization use cases such as conflict resolution, consistency guarantees, performance, visibility, and so on.
|
||||
|
||||
Thanks for reading!
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
title: "Optimizing OPA: Rule Indexing"
|
||||
authors: ["tsandall"]
|
||||
date: 2017-06-06
|
||||
slug: optimizing-opa-rule-indexing-59f03f17caf3
|
||||
---
|
||||
|
||||
One of the hallmarks of declarative languages is that people shouldn't worry about performance — the software should. People write down the logic of a policy decision (e.g. should this API be allowed), and the software figures out how to evaluate that logic efficiently.
|
||||
|
||||
For example, it's common to use OPA to write whitelist (or blacklist) policies by defining multiple rules that all say to 'allow' (or 'deny') some operation. The example below defines a simple whitelist authorization policy for HTTP API operations (see the [HTTP API Authorization tutorial](https://www.openpolicyagent.org/docs/latest/http-api-authorization/) for more details). Notice how there are multiple statements that 'allow' the operation, any of which could apply to a given input.
|
||||
|
||||
```rego
|
||||
package acmecorp.api
|
||||
|
||||
import data.acmecorp.roles
|
||||
|
||||
default allow = false
|
||||
|
||||
allow {
|
||||
input.method = "GET"
|
||||
input.path = ["accounts", user]
|
||||
input.user = user
|
||||
}
|
||||
|
||||
allow {
|
||||
input.method = "GET"
|
||||
input.path = ["accounts", "report"]
|
||||
roles[input.user][_] = "admin"
|
||||
}
|
||||
|
||||
allow {
|
||||
input.method = "POST"
|
||||
input.path = ["accounts"]
|
||||
roles[input.user][_] = "admin"
|
||||
}
|
||||
```
|
||||
|
||||
To ask if user "felix" is authorized to GET the /accounts API, you would query OPA as shown below.
|
||||
|
||||
```
|
||||
POST /v1/data/acmecorp/api/allow HTTP/1.1
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"input": {
|
||||
"method": "POST",
|
||||
"user": "felix",
|
||||
"path": ["accounts"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
If "felix" has the "admin" role, OPA responds with true, since the 3rd "allow" rule matches the path ["accounts"] and requires the user to be in the "admin" role. Otherwise, none of the "allow" rules match the input above, and OPA responds with false (because of the "default" rule).
|
||||
|
||||
In this blog post we describe an optimization that improves how efficiently OPA computes the answer to such a query. The simplest way to compute an answer is to evaluate each of the "allow" rules one-by-one. This is exactly what OPA did before v0.4.9. The obvious problem with this approach is that query latency increases as the total size of the rule set grows, even if the additional rules are irrelevant to most queries.
|
||||
|
||||
In many cases, rule sets share similar expressions that differ slightly but in ways that make some of them mutually exclusive. This means that if one rule would generate a value, another could not. In the example above, the 3rd rule only applies to the path ["accounts"], whereas the other two rules only apply to longer paths, e.g. ["accounts", "report"].
|
||||
|
||||
As of v0.4.9, OPA indexes rule sets to quickly determine which subset of rules must be evaluated given data stored in OPA or passed as input to the query. This allows OPA to evaluate only the rules required to generate the correct result. Rules that would not generate a value for the query input are ignored.
|
||||
|
||||
Conceptually, the index is a function that takes the query input as an argument and returns the rules OPA must evaluate:
|
||||
|
||||
```go
|
||||
func (index Index) GetRules(input Value) []Rule {
|
||||
// perform index lookup
|
||||
}
|
||||
```
|
||||
|
||||
To make the index lookup efficient, OPA builds a [Trie data structure](https://en.wikipedia.org/wiki/Trie) from equality expressions contained in rule sets. For example, OPA would build the following structure for the rule set above:
|
||||
|
||||
|
||||
|
||||
When OPA looks up the rules to evaluate the query above, it traverses the Trie and collects rules that are required for the input. For simple equality expressions involving scalars this is straightforward however it becomes more complex when taking into account composites (arrays, objects), vars, and undefined values.
|
||||
|
||||
|
||||
|
||||
As we would expect, this approach provides an asymptotic performance improvement on queries that match the form supported by the index:
|
||||
|
||||
|
||||
|
||||
Currently, OPA can build this rule index for expressions of the form:
|
||||
|
||||
```
|
||||
<ref> = <scalar | array | var> # or vice-versa
|
||||
```
|
||||
|
||||
Here are a few examples:
|
||||
|
||||
```
|
||||
input.foo = "bar"
|
||||
["foo", "bar", x] = input.baz
|
||||
data.foo.bar = x
|
||||
```
|
||||
|
||||
There's an additional constraint on the `<ref>` term: it must:
|
||||
|
||||
- be ground (e.g., `data.example.foo[x]` is excluded)
|
||||
- be non-nested (e.g., `data.example.foo[data.example.bar]` is excluded)
|
||||
- refer to a base document (e.g., if `data.example.p` refers to a rule, it is excluded)
|
||||
|
||||
Nonetheless, in practice, there are many cases covered by the currently supported form and we're pleased to have this optimization in place. In the future, we'll extend the index to support other types of terms (e.g., objects, non-ground refs, etc.) as needed.
|
||||
@@ -0,0 +1,192 @@
|
||||
---
|
||||
title: "Orderly versus Disorderly Policies"
|
||||
authors: ["timhinrichs"]
|
||||
date: 2017-09-07
|
||||
slug: orderly-versus-disorderly-policies-717475c23d2f
|
||||
---
|
||||
|
||||
|
||||
|
||||
One of the age-old questions with policy languages is: does the order of policy statements matter?
|
||||
|
||||
- For Orderly policies where statement-order matters, the policy engine makes snap judgments. It uses the first statement that applies to make a decision. Examples are Firewalls and IPTables.
|
||||
- For Disorderly policies where statement-order doesn't matter, the policy engine is more contemplative. It finds all the statements that apply and combines them to make a decision. Examples are query languages like SQL and Datalog.
|
||||
|
||||
Are Orderly or Disorderly policies better? In this blog post we lay out the tradeoffs and conclude there's no clear winner (which is why the [Open Policy Agent](http://openpolicyagent.org) supports **both**). You want a policy language that lets you mix and match Orderly and Disorderly policies so you can leverage the strengths of each along the following dimensions.
|
||||
|
||||
- **Conflict resolution**: Implicit (Orderly) vs. Explicit (Disorderly)
|
||||
- **Overrides and Defaults**: Easy (Orderly) vs. Difficult (Disorderly)
|
||||
- **Composability**: Difficult (Orderly) vs. Easy (Disorderly)
|
||||
- **Performance**: At most 1 (Orderly) vs. More than 1 (Disorderly)
|
||||
- **Understandability:** Misleading (Orderly) vs. Requires Tools (Disorderly)
|
||||
|
||||
## Terminology
|
||||
|
||||
Throughout this blog post we'll use the following terms.
|
||||
|
||||
- A **policy** is a collection of statements.
|
||||
- A **statement** makes a decision, such as whether an HTTP API call should be allowed or denied.
|
||||
- A **policy language** defines what statements are possible. It defines what a statement, policy, or collection of policies means and what decisions it makes.
|
||||
- A **policy engine** is a piece of software that ingests policies and evaluates them to make decisions.
|
||||
|
||||
As a running example, we'll use HTTP API authorization. The input to the policy has two components: a user making an API request and the HTTP path for that request. For example, the user might be `alice` and the HTTP path might be `/finance/alice/salary`. The policy must decide whether to allow or deny the input.
|
||||
|
||||
## Conflict Resolution
|
||||
|
||||
A conflict happens when a policy says something contradictory, like a request is both allowed and denied. A policy language that can't express conflicts is usually too impoverished in practice, so a policy language must have a way of eliminating conflicts if it's to make an actual decision.
|
||||
|
||||
For example, the following policy has a conflict: any URL of the form `/finance/<user>/salary` is both allowed and denied.
|
||||
|
||||
```
|
||||
Deny /finance
|
||||
Allow /finance/<user>/salary
|
||||
```
|
||||
|
||||
An Orderly policy resolves conflicts automatically because there's only ever 1 statement that applies: the first one. You order your policy statements so that conflicts get resolved the way you want.
|
||||
|
||||
A Disorderly policy on the other hand requires you to choose an explicit conflict resolution strategy for when multiple statements make contradicting decisions. For example, you might dictate that Allow overrides Deny (or vice versa).
|
||||
|
||||
The snippets below show the different options for the example above.
|
||||
|
||||
Option 1: Orderly
|
||||
|
||||
```
|
||||
Deny /finance
|
||||
Allow /finance/<user>
|
||||
Final decision: Deny
|
||||
```
|
||||
|
||||
Option 2: Orderly (statements in a different order)
|
||||
|
||||
```
|
||||
Allow /finance/<user>
|
||||
Deny /finance
|
||||
Final decision: Allow
|
||||
```
|
||||
|
||||
Option 3: Disorderly with Deny overrides Allow
|
||||
|
||||
```
|
||||
Deny /finance
|
||||
Allow /finance/<user>
|
||||
Final decision: Deny
|
||||
```
|
||||
|
||||
Option 4: Disorderly with Allow overrides Deny
|
||||
|
||||
```
|
||||
Deny /finance
|
||||
Allow /finance/<user>
|
||||
Final decision: Allow
|
||||
```
|
||||
|
||||
On the surface, Orderly policies seem to handle conflicts better than Disorderly policies, but in reality you're thinking about conflicts in both cases. For Orderly policies, you think about conflicts when you figure out what order to put your statements in, and for Disorderly policies you think about conflicts when choosing a conflict-resolution strategy. Here there is no clear winner.
|
||||
|
||||
## Overrides and Defaults
|
||||
|
||||
One of the best applications of an Orderly policy is to handle default conditions and overrides. Imagine you have an existing policy and temporarily need to block all requests to a particular HTTP API because of a recent security vulnerability or bug.
|
||||
|
||||
Using an Orderly policy you can add the appropriate policy statements to the top of the policy and rest assured that those statements will be the ones that get applied. In contrast, for a Disorderly policy you figure out what all statements are that conflict with your new statement and modify them so they no longer apply.
|
||||
|
||||
In our running example, imagine you have 4 different policy statements and now you want to block all API calls to `/finance`. With an Orderly policy, you simply add a single statement `Deny /finance` to the top of the policy. You don't even need to know what the other policy statements say.
|
||||
|
||||
**Orderly**
|
||||
|
||||
```
|
||||
Deny /finance
|
||||
Allow /finance/<user> by <user>
|
||||
Allow /finance/<user> for HR group members
|
||||
Allow /finance/<user> for <user>'s manager
|
||||
Deny /finance/<user> for subordinates of <user>
|
||||
```
|
||||
|
||||
In a Disorderly policy where Allow overrides Deny, we would need to identify and then temporarily remove or disable all of the Allow statements shown above.
|
||||
|
||||
Orderly policies clearly excel at the override use-case.
|
||||
|
||||
## Composability and Collaboration
|
||||
|
||||
Based on the last section, it might seem that Orderly policies are better because they make defaults and overrides so easy. But as we're about to see Disorderly policies are vastly better for the case of composition (the ability to combine different policies).
|
||||
|
||||
Composing (combining) different policies happens whenever multiple people are collaboratively defining a policy. Imagine you write policy A and someone else writes policy B (both about HTTP APIs). For Disorderly policies combining A and B is easy: union the statements of A and B together. Any conflicts between A and B are handled by conflict resolution.
|
||||
|
||||
In contrast, for Orderly policies, there's no clear way to combine A and B into a single policy. Put policy A before B, and all conflicts are resolved in your favor. Put policy B before A, and all of policy B's decisions win out. Orderly policies don't have conflicts or conflict resolution, so you need additional tooling to even identify conflicts between A and B.
|
||||
|
||||
For example, if you were to combine Orderly policies A and B below, which order would the authors of A and B agree on? (A then B does NOT produce the Orderly policy given in the last section.) If the policies are Disorderly, combining them is easy.
|
||||
|
||||
**Policy A (Ordered)**
|
||||
|
||||
```
|
||||
Allow /finance/<user> by <user>
|
||||
Allow /finance/<user> for <user>'s manager
|
||||
```
|
||||
|
||||
**Policy B (Ordered)**
|
||||
|
||||
```
|
||||
Allow /finance/<user> for HR group members
|
||||
Deny /finance/<user> for subordinates of <user>
|
||||
```
|
||||
|
||||
You might wonder how often composition happens. Anytime you have hierarchically organized resources (like files in a directory structure or HTTP API resources), you'll typically end up composing policies. It's natural to attach different policies to different points in the hierarchy and then make decisions based on the policies attached at a parent and child in the hierarchy. Sometimes you want the policy at the lower level to take precedence over the policy at the higher level, and sometimes the reverse is true.
|
||||
|
||||
Composability is the single greatest strength of Disorderly policies. People (and machines) can contribute what they know as policy statements, and the language lets you understand where the disagreements are, and what the final decision is.
|
||||
|
||||
## Performance
|
||||
|
||||
For Orderly policies, evaluation only executes until it finds one applicable statement. If that statement is number 10 out of 100,000 then the performance improvement is substantial. While it is unknown how many policy statements will need to be evaluated, at most 1 will ever be fully evaluated.
|
||||
|
||||
For Disorderly policies, it's important to implement indexing algorithms. Indexing algorithms analyze a policy and organize its statements so that by inspecting a request, it can quickly zero in on the (hopefully) small number of statements that apply.
|
||||
|
||||
In our running example, an indexer might identify that just the following two statements need to be evaluated. If it's smart, it can even conclude that the answer must be Allow without evaluating anything because all of the statements Allow the result.
|
||||
|
||||
```
|
||||
Allow /finance/<user> by <user>
|
||||
Allow /finance/<user> for HR group members
|
||||
```
|
||||
|
||||
Of course, you can use indexing for ordered evaluation, but it requires deeper analysis because statement k only applies if none of statements 1…k-1 apply, meaning that the conditions for statement k include some of the conditions from previous statements.
|
||||
|
||||
In terms of performance, it's hard to say whether Orderly or Disorderly policies are better. Indexing algorithms can be complex and lead to variability, but can also improve performance asymptotically (infinitely).
|
||||
|
||||
## Familiarity and Understandability
|
||||
|
||||
Technical considerations are important, but the best technical solution in the world won't be used if people don't understand it. How familiar and understandable a policy language is has a great impact on its adoption.
|
||||
|
||||
Orderly policies require the reader/writer to think top-to-bottom. That's the same way we train people to read, and it's the same way we train programmers to write most code in imperative, object-oriented, and functional languages. Orderly policies are familiar to a wide range of people.
|
||||
|
||||
Just because they're familiar doesn't mean Orderly policies are easy to understand (especially at scale). To know what statement number k means, you need to know which of the 1…k-1 statements conflict with statement k since they all take precedence over statement k. So to understand statement number 1,000 you need to understand those 999 statements that come before it. The worst part is that when people authored the policy, they may have known that there were only 5 statements that mattered, but they were forced to write them so that it looks like all 999 statements matter.
|
||||
|
||||
For example, understanding which requests get denied by the last statement in the Orderly policy below means understanding which requests get allowed by the first 3 statements. Understanding what happens to a request for a user's financial data by a member of HR only requires looking at the first 2 statements — it's safe to ignore all the rest.
|
||||
|
||||
**Orderly**
|
||||
|
||||
```
|
||||
Allow /finance/<user> by <user>
|
||||
Allow /finance/<user> for HR group members
|
||||
Allow /finance/<user> for <user>'s manager
|
||||
Deny /finance/<user> for subordinates of <user>
|
||||
```
|
||||
|
||||
Disorderly statements are no less familiar to the general populace than ordered. Every time you read two different newspaper articles, you get different sides of the story and need to figure out what really happened. For programmers, the order in which you define functions usually makes no difference; any time you create a set, use a boolean OR, launch multiple threads, or work on a distributed system you're working with unorderedness.
|
||||
|
||||
A single statement in a Disorderly policy stands on its own. Once you understand that statement, you need to understand which other statements conflict with it and how those conflicts are resolved. Tooling helps immensely here.
|
||||
|
||||
For example, in our running example you may want to understand the impact of the statement `Allow /finance/<user> for HR group members`. The statement by itself is simple enough, but you need to find all the statements that might conflict with it (e.g. the last statement conflicts if one of the HR group members is a subordinate), and how that conflict would be resolved.
|
||||
|
||||
**Disorderly**
|
||||
|
||||
```
|
||||
Allow /finance/<user> by <user>
|
||||
Allow /finance/<user> for HR group members
|
||||
Allow /finance/<user> for <user>'s manager
|
||||
Deny /finance/<user> for subordinates of <user>
|
||||
```
|
||||
|
||||
In the end, understanding any single statement in any policy (Orderly or Disorderly) requires understanding all the other statements that conflict with it. Orderly policies tell you that the conflicts all come before the statement you're looking at, but force you to order statements that have no inherent ordering, resulting in the appearance of conflicts when none exist. Disorderly policies give you no indication about which policy statements might conflict and rely on tooling to help you find conflicting statements.
|
||||
|
||||
## Summary
|
||||
|
||||
When all is said and done, there are times when you want statement order to matter (overrides), and there are times when you don't want statement order to matter (composition and hierarchies). If you have a tightly-constrained use case, you might be able to choose one or the other. But if you're interested in a general-purpose policy language that works for many different use cases, across many different domains, and is used by a wide population, there's no clear winner. You'll want your policy language to support both Orderly and Disorderly policies.
|
||||
|
||||
Given that this is a blog post for the [Open Policy Agent](http://openpolicyagent.org), it's probably not surprising that OPA supports both options and lets you mix and match as appropriate. See the [docs](http://www.openpolicyagent.org/docs/) and [FAQ](http://www.openpolicyagent.org/docs/faq.html) for more details, or reach out on the [slack channel](http://a5ec585d42ace11e7aaad0260e83477e-1095713357.us-west-2.elb.amazonaws.com).
|
||||
@@ -0,0 +1,170 @@
|
||||
---
|
||||
title: "OPA's Full Stack Policy Language"
|
||||
authors: ["timhinrichs"]
|
||||
date: 2017-12-14
|
||||
slug: opas-full-stack-policy-language-caeaadb1e077
|
||||
---
|
||||
|
||||
|
||||
|
||||
The [Open Policy Agent](http://www.openpolicyagent.org/) (OPA) has been used to policy-enable software across several different domains across several layers of the stack: container management (Kubernetes), servers (Linux), public cloud infrastructure (Terraform), and microservice APIs (Istio, Linkerd, CloudFoundry). In this post we describe how OPA's policy language Rego manages to work for all these different domains and layers of the stack without requiring any changes or extensions to the language.
|
||||
|
||||
## Rego Overview
|
||||
|
||||
Rego's sole purpose is to make policy decisions that other products/services need to take action. For example…
|
||||
|
||||
- Is this API request allowed or denied?
|
||||
- What's the hostname of the backup server for this application?
|
||||
- What's the risk score for this proposed infrastructure change?
|
||||
- What list of clusters should this container be deployed to for high-availability?
|
||||
- What's the routing information that should be used for this microservice?
|
||||
|
||||
Rego lets you write policy to answer all of those questions (and many more). It lets you write policy about any layer of the stack and any domain (e.g. APIs, servers, infrastructure, clusters, networking), without requiring you to change or extend the language. There are two key insights to Rego:
|
||||
|
||||
> Every domain can be encoded as JSON/YAML data.
|
||||
>
|
||||
> Policy is logic applied to data.
|
||||
|
||||
When you're writing policy, you should only be thinking about the domain you care about, how to encode that domain as data, and the logic you need to make a policy decision given that data. In this post, we show you how to do exactly that with Rego. Along the way you'll learn that:
|
||||
|
||||
- Rego data can be any JSON/YAML data
|
||||
- Rego is a query language. You write logic to search and combine JSON/YAML data from different sources.
|
||||
|
||||
## Rego Data is JSON/YAML
|
||||
|
||||
As mentioned above, Rego is a general-purpose policy language, meaning that it works for any layer of the stack and any domain. The key idea is that while you as an author are thinking about servers, containers, or APIs, Rego just sees JSON/YAML data. So you can write policy about any domain as long as the information you need to make a decision can be stuffed into JSON/YAML.
|
||||
|
||||
For example, you can think of authorizing an HTTP API call as making a true/false decision about the YAML data shown below.
|
||||
|
||||
```yaml
|
||||
user: alice
|
||||
method: GET
|
||||
path: /finance/salary/bob
|
||||
headers:
|
||||
- JWT: …
|
||||
```
|
||||
|
||||
And you can think of authorizing someone to SSH into a server as a true/false decision about:
|
||||
|
||||
```yaml
|
||||
user: alice
|
||||
server_id: s12345
|
||||
role: webapp
|
||||
environment: prod
|
||||
```
|
||||
|
||||
Besides the data describing the input to the decision, Rego lets you incorporate background information from many different sources of JSON/YAML to make a decision. You could, for example, tell Rego who each person's manager is:
|
||||
|
||||
```yaml
|
||||
manager:
|
||||
charlie: bob
|
||||
dave: bob
|
||||
bob: alice
|
||||
```
|
||||
|
||||
Then you could authorize managers to execute API calls that return their subordinates' salaries, or that a manager can SSH into their subordinates' desktop servers.
|
||||
|
||||
The key point here is that Rego does not understand what the data (or even the schema for the data) means in the real world. Because of that, you can use Rego to make policy decisions about APIs, servers, containers, risk-management, or any other domain you can imagine. It's the policy author that knows what the data means in the real world and writes logic to make a policy decision.
|
||||
|
||||
## Rego is a Query Language
|
||||
|
||||
To make a policy decision in Rego, you write logical tests on the data that comes in as input (such as the API or SSH data from the last section).
|
||||
|
||||
For example, if you want to allow a user to run the API call that reports her own salary, you write a policy where the input is a JSON/YAML document representing the API call (shown below tweaked slightly to represent the URL as an array instead of a string) and call it `input`.
|
||||
|
||||
```yaml
|
||||
input:
|
||||
user: alice
|
||||
method: GET
|
||||
path: ["finance", "salary", "bob"] # /finance/salary/bob
|
||||
headers:
|
||||
- JWT: …
|
||||
```
|
||||
|
||||
Then you write boolean logic that decides whether or not the API call represented by that JSON/YAML data is authorized. For example, you could write conditions that authorize `bob` to see his own salary.
|
||||
|
||||
```rego
|
||||
input.method = "GET"
|
||||
input.path = ["finance", "salary", "bob"]
|
||||
input.user = "bob"
|
||||
```
|
||||
|
||||
Implicitly all of the conditions above are ANDed together. In Rego, logic like this doesn't stand on its own — it needs a name. Below we've given the logic the name `allow`.
|
||||
|
||||
```rego
|
||||
allow {
|
||||
input.method = "GET"
|
||||
input.path = ["finance", "salary", "bob"]
|
||||
input.user = "bob"
|
||||
}
|
||||
```
|
||||
|
||||
Of course, you don't want to write one `allow` statement for every employee in the company. You want to use a variable so that the statement applies to everyone. In Rego, a variable is basically any symbol that's not a string or number, so to allow all employees to see their own salary, you replace `"bob"` with a variable like `employee`.
|
||||
|
||||
```rego
|
||||
allow {
|
||||
input.method = "GET"
|
||||
input.path = ["finance", "salary", employee]
|
||||
input.user = employee
|
||||
}
|
||||
```
|
||||
|
||||
This statement says that `allow` is true if there is some value for the `employee` variable that makes all of the conditions true. Rego treats a set of conditions as a query and finds variable assignments that makes them all true. So while Rego has a syntax closer to programming languages than to SQL, it's really a query language underneath.
|
||||
|
||||
Now that you've let everyone see their own salary, you might decide to allow managers to see their subordinates' salaries. But to do that the Rego policy needs to know who manages whom, information that isn't included in the API call. In OPA, you make managerial data available by inserting it into the `data` namespace. (OPA has exactly two toplevel namespaces for JSON/YAML data: `data` and `input`.)
|
||||
|
||||
```yaml
|
||||
data:
|
||||
manager:
|
||||
charlie: bob
|
||||
dave: bob
|
||||
bob: alice
|
||||
...
|
||||
```
|
||||
|
||||
Now you can write a second query also named `allow` as shown below. When you have multiple queries with the same name, they are ORed together.
|
||||
|
||||
```rego
|
||||
allow {
|
||||
input.method = "GET"
|
||||
input.path = ["finance", "salary", employee]
|
||||
input.user = data.manager[employee]
|
||||
}
|
||||
```
|
||||
|
||||
This query looks up whether the user requesting an employee's salary is the manager of that employee using the `manager` dictionary. In addition to looking up key/value pairs in a dictionary, Rego lets you search through JSON data to find values and even cross-reference (or join) multiple JSON data sources during a search.
|
||||
|
||||
Additionally, Rego lets you make policy decisions that are more sophisticated than the allow/deny decisions shown above. You can make decisions that are numbers (e.g. rate-limits), strings (e.g. hostnames), arrays (e.g. servers), or dictionaries (microservice route-mappings). For more examples, see the [Open Policy Agent tutorials](http://www.openpolicyagent.org/docs/get-started.html).
|
||||
|
||||
The key takeaway is that Rego logic lets you write queries about multiple JSON/YAML data-sources to make a policy decision, and a decision can be a boolean, number, string, array, or even a dictionary.
|
||||
|
||||
## Rego is for Policy, not Programming
|
||||
|
||||
The goal of Rego is to help you tell software systems how to behave in the world by writing logic about (collections of) JSON/YAML data. Programming languages (e.g. C, Java, Go, Python) are the usual solution to this problem, but Rego was purpose-built to let you focus on just the data that represents the world and the logic that makes policy decisions about that data. Below we contrast policy and programming by showing what you SHOULD be thinking about when writing policy (logic and data) and what you SHOULD NOT be thinking about when writing policy (programming).
|
||||
|
||||
When writing policy about HTTP APIs…
|
||||
|
||||
- you SHOULD be thinking about whether the `method` is a `GET` or `POST`
|
||||
- you SHOULD be thinking about whether the `employee` is requesting her own salary or someone else's.
|
||||
- you SHOULD be thinking about whether the `manager` data has an entry stating that the `employee` requesting a salary is the manager of the person's salary in the request.
|
||||
|
||||
But…
|
||||
|
||||
- You SHOULD NOT be thinking about opening a network socket to retrieve the `manager` data.
|
||||
- You SHOULD NOT be thinking about the object classes or class inheritance used to store the managerial data.
|
||||
- You SHOULD NOT be thinking about which methods to use to access the fields of a class.
|
||||
- You SHOULD NOT be deciding between different looping constructs (`for`, `do-until`, `while`, `iterators`, `recursion`) to search through the data or worrying about if those loops will terminate.
|
||||
- You should NOT be thinking about a multitude of data structures and their subtle tradeoffs, like splay trees versus red-black trees.
|
||||
|
||||
Rego has some of the same primitives as programming languages for sharing common logic and coping with a large number of policies (modules and functions), but you're always thinking about exactly two things: logic and data. You're not thinking about sockets, object classes, method calls, non-terminating loops, or binary trees. You're thinking about logic and data.
|
||||
|
||||
## Wrap Up
|
||||
|
||||
Hopefully that helps shed some light on Rego and how OPA works for any domain and any layer of the stack. Here are the key takeaways:
|
||||
|
||||
- Rego lets you write policy about any domain and any layer of the stack: APIs, servers, risk-management, containers, networking.
|
||||
- Rego makes you think about policy (logic and data), not programming (sockets, object classes, method calls, non-terminating loops, binary trees).
|
||||
- Rego operates over JSON data. You can supply JSON data as input to every decision and as background information for making decisions.
|
||||
- Rego logic is all queries. A query finds values for variables that make boolean conditions true.
|
||||
|
||||
For more information, check out the [Open Policy Agent](http://www.openpolicyagent.org/) project.
|
||||
@@ -0,0 +1,302 @@
|
||||
---
|
||||
title: "Partial Evaluation"
|
||||
authors: ["tsandall"]
|
||||
date: 2018-02-05
|
||||
slug: partial-evaluation-162750eaf422
|
||||
---
|
||||
|
||||
We'd like to introduce a new OPA feature called _partial evaluation_ which has several interesting applications. In this post we'll explain how partial evaluation works, and how it leverages rule indexing to optimize policy evaluation for low-latency use cases (e.g. microservice API authorization).
|
||||
|
||||
## What Is Partial Evaluation?
|
||||
|
||||
With partial evaluation, callers specify that certain inputs or pieces of data are _unknown_. OPA evaluates as much of the policy as possible without touching parts that depend on unknown values. The result of partial evaluation is _a new policy_ that can be evaluated more efficiently than the original.
|
||||
|
||||
|
||||
|
||||
By partially evaluating policies, OPA can perform computation at compile time instead of runtime.
|
||||
|
||||
For example, in API authorization use cases, expensive operations that are independent of incoming API requests are evaluated once (during partial evaluation) and their results can be cached for subsequent evaluation runs. This helps remove costly operations from the critical path of API request processing.
|
||||
|
||||
The big difference between partial evaluation and normal evaluation is the result. Normal evaluation produces a decision that can be enforced (e.g. allow or deny), but partial evaluation produces new policies that can be evaluated later when the unknowns become known.
|
||||
|
||||
|
||||
|
||||
Let's look at an example.
|
||||
|
||||
## Partial Evaluation and Rule Indexing
|
||||
|
||||
Imagine we have a simple authorization policy that decides whether to allow requests against appliances.
|
||||
|
||||
```rego
|
||||
package smart_home
|
||||
|
||||
default allow = false
|
||||
|
||||
allow = true {
|
||||
op = allowed_operations[_]
|
||||
input.method = op.method
|
||||
input.resource = op.resource
|
||||
}
|
||||
|
||||
allowed_operations = [
|
||||
{"method": "PUT", "resource": "air-conditioner"},
|
||||
{"method": "GET", "resource": "security-camera"},
|
||||
{"method": "POST", "resource": "garage-door"},
|
||||
]
|
||||
```
|
||||
|
||||
To use this policy, callers query OPA by asking for the value of **smart_home.allow** which will be **true** or **false** depending on whether the request should be allowed or denied. To answer the policy query, OPA _searches_ the allowed_operations data to find a match.
|
||||
|
||||
The nice thing about this policy is that we only have to write the search logic (**allow**) once, up front. After that, we can expand the data set (**allowed_operations**) to cover more and more appliances.
|
||||
|
||||
> In real scenarios, **allowed_operations** would be loaded into OPA as _data_ instead of hardcoded in the policy. It's hardcoded into the policy to make the example simpler.
|
||||
|
||||
The downside is that OPA needs to potentially search over the entire data set each time an API request is received. As we automate more of our home, the authorization decision will take longer to compute.
|
||||
|
||||
Performance is an important requirement for OPA: we need to minimize the latency introduced into the request path as much as possible. OPA has addressed use cases like this in the past by [precomputing a data structure that quickly looks up applicable rules instead of searching through them](/blog/optimizing-opa-rule-indexing-59f03f17caf3). However to accomplish this the rules have to be written in a manner that OPA's indexer can understand. For example, we would have to take the policy above rewrite it as follows:
|
||||
|
||||
```rego
|
||||
package smart_home
|
||||
|
||||
default allow = false
|
||||
|
||||
allow {
|
||||
input.method = "PUT"
|
||||
input.resource = "air-conditioner"
|
||||
}
|
||||
|
||||
allow {
|
||||
input.method = "GET"
|
||||
input.resource = "security-camera"
|
||||
}
|
||||
|
||||
allow {
|
||||
input.method = "POST"
|
||||
input.resource = "garage-door"
|
||||
}
|
||||
```
|
||||
|
||||
While this works, no one wants to write out these kinds of rules by hand. Moreover, the policy becomes painful to read because we end up with thousands of duplicated rules. What we want is a simple, concise policy that OPA evaluates efficiently.
|
||||
|
||||
This is where partial evaluation comes in. Looking at the **allow** rule from earlier, we can see that there are two pieces of potentially unknown information: **input.method** and **input.resource**.
|
||||
|
||||
```rego
|
||||
allow {
|
||||
op = allowed_operations[_]
|
||||
input.method = op.method
|
||||
input.resource = op.resource
|
||||
}
|
||||
```
|
||||
|
||||
With partial evaluation, we can evaluate everything in the policy that does not depend on these two input values. The result of partial evaluation, used internally by OPA, looks almost identical to the unrolled version we made above by hand.
|
||||
|
||||
Of course, not all policies are this simple. In some cases, expressions that depend on unknown values may produce outputs that are required in the rest of the policy:
|
||||
|
||||
```rego
|
||||
allow {
|
||||
risk_score = (input.num_deletes * 10) + input.num_adds
|
||||
risk_score < risk_limit
|
||||
}
|
||||
|
||||
risk_limit = 100
|
||||
```
|
||||
|
||||
In this case, OPA will determine that **risk_score** cannot be computed if **input** is unknown and so the second expression will not be processed during partial evaluation. Instead it gets saved as part of the partial evaluation result.
|
||||
|
||||
This was a quick overview of how partial evaluation works. Let's look at how you can use partial evaluation today!
|
||||
|
||||
## Using Partial Evaluation
|
||||
|
||||
You enable partial evaluation by specifying the **partial** query parameter when you ask OPA for policy decisions.
|
||||
|
||||
OPA performs partial evaluation lazily: the first time you ask for a policy decision with partial evaluation, OPA will compute the partially evaluated policy and then cache it for later. If the data or policy in OPA changes such that partial evaluation needs to be re-run, OPA invalidates the cache entry.
|
||||
|
||||
Let's look at some examples.
|
||||
|
||||
Suppose we have the policy from above, but instead of 3 resources, we have hundreds. We can construct a query to see if we're allowed to GET the data from one of security cameras:
|
||||
|
||||
```json
|
||||
{
|
||||
"input": {
|
||||
"method": "GET",
|
||||
"resource": "security-camera-493"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This object is saved into a file called req.json that we can use to execute the query below with cURL:
|
||||
|
||||
```bash
|
||||
curl localhost:8181/v1/data/smart_home/allow?metrics -d @req.json
|
||||
```
|
||||
|
||||
In this case, the request is allowed and normal evaluation takes around **5ms** on an Intel Core i7 @ 2.9Ghz:
|
||||
|
||||
```json
|
||||
{
|
||||
"metrics": {
|
||||
"timer_rego_query_compile_ns": 71412,
|
||||
"timer_rego_query_eval_ns": 5185152,
|
||||
"timer_rego_query_parse_ns": 194950
|
||||
},
|
||||
"result": true
|
||||
}
|
||||
```
|
||||
|
||||
We'll change the caller to specify the **partial** query parameter.
|
||||
|
||||
```bash
|
||||
curl localhost:8181/v1/data/smart_home/allow?metrics&partial \
|
||||
-d @req.json
|
||||
```
|
||||
|
||||
Now when we run the call, we get back the same results, but we can see the query latency (**timer_rego_query_eval_ns**) is much lower, around **50µs**.
|
||||
|
||||
```json
|
||||
{
|
||||
"metrics": {
|
||||
"timer_rego_partial_eval_ns": 10613556,
|
||||
"timer_rego_query_compile_ns": 137427,
|
||||
"timer_rego_query_eval_ns": 43921,
|
||||
"timer_rego_query_parse_ns": 200871
|
||||
},
|
||||
"result": true
|
||||
}
|
||||
```
|
||||
|
||||
Since the above call was the first with the **partial** parameter set, the response included the time it took to partially evaluate the policy, around 10 milliseconds (**timer_rego_partial_eval_ns**). If we query OPA again with **partial** set the step is skipped because the result was cached:
|
||||
|
||||
```bash
|
||||
curl localhost:8181/v1/data/smart_home/allow?metrics&partial \
|
||||
-d @req.json
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"metrics": {
|
||||
"timer_rego_query_compile_ns": 78901,
|
||||
"timer_rego_query_eval_ns": 50405,
|
||||
"timer_rego_query_parse_ns": 22203
|
||||
},
|
||||
"result": true
|
||||
}
|
||||
```
|
||||
|
||||
You can also use partial evaluation if you embed OPA as a library. In that case:
|
||||
|
||||
- You must invoke partial evaluation yourself.
|
||||
- You should invoke partial evaluation offline, outside the request path.
|
||||
- You should cache the partial evaluation result.
|
||||
- You can use the partial evaluation result when input values are known (e.g., when an API request is received.)
|
||||
|
||||
For more information on how to embed OPA as a library and leverage the partial evaluation feature, see [this GoDoc example](https://godoc.org/github.com/open-policy-agent/opa/rego#example-Rego-PartialEval).
|
||||
|
||||
## Benchmark: Role-based Access Control (RBAC)
|
||||
|
||||
An immediate application for partial evaluation is [RBAC](https://en.wikipedia.org/wiki/Role-based_access_control) policy enforcement. RBAC provides a simple, coarse-grained way of granting permissions by groupings. Determining whether to allow requests under RBAC involves identifying whether the caller has been associated with a role that grants permission to the perform the operation.
|
||||
|
||||
In projects like [Kubernetes](https://kubernetes.io/docs/admin/authorization/rbac/) and [Istio](https://istio.io/docs/concepts/security/rbac.html), RBAC configuration is specified using _roles_ and _role bindings_. Roles grant permission to perform operations and role bindings associate subjects (e.g., users or service accounts) to roles. Below is an example of some role and role binding data:
|
||||
|
||||
```json
|
||||
{
|
||||
"roles": [
|
||||
{
|
||||
"operation": "read",
|
||||
"resource": "widgets",
|
||||
"name": "widget-reader"
|
||||
},
|
||||
{
|
||||
"operation": "write",
|
||||
"resource": "widgets",
|
||||
"name": "widget-writer"
|
||||
}
|
||||
],
|
||||
"bindings": [
|
||||
{
|
||||
"user": "inspector-alice",
|
||||
"role": "widget-reader"
|
||||
},
|
||||
{
|
||||
"user": "maker-bob",
|
||||
"role": "widget-writer"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
In OPA, we can define an RBAC policy that evaluates the configuration above as follows:
|
||||
|
||||
```rego
|
||||
package example_rbac
|
||||
|
||||
default allow = false
|
||||
|
||||
allow {
|
||||
user_has_role[role_name]
|
||||
role_has_permission[role_name]
|
||||
}
|
||||
|
||||
user_has_role[role_name] {
|
||||
role_binding = data.bindings[_]
|
||||
role_binding.role = role_name
|
||||
role_binding.user = input.subject.user
|
||||
}
|
||||
|
||||
role_has_permission[role_name] {
|
||||
role = data.roles[_]
|
||||
role.name = role_name
|
||||
role.operation = input.action.operation
|
||||
role.resource = input.action.resource
|
||||
}
|
||||
```
|
||||
|
||||
First, the policy searches for "bindings" that match the subject. Second, the policy checks to see if the role associated with the binding grants permission to perform the operation on the resource.
|
||||
|
||||
In this example, applying partial evaluation would yield a set of rules where the roles and role bindings have been inlined and the implicit loops that perform the search have been unrolled. For example:
|
||||
|
||||
```rego
|
||||
# … more rules
|
||||
allow {
|
||||
"admin" = input.subject.user
|
||||
"read" = input.action.operation
|
||||
"resource-lktj" = input.action.resource
|
||||
}
|
||||
allow {
|
||||
"admin" = input.subject.user
|
||||
"write" = input.action.operation
|
||||
"resource-rsiq" = input.action.resource
|
||||
}
|
||||
# … more rules
|
||||
```
|
||||
|
||||
The table below summarizes the performance when using OPA to evaluate this RBAC policy on increasingly large RBAC configuration data sets.
|
||||
|
||||
```
|
||||
# Roles | # Bindings | Normal Eval (ms) | With Partial Eval (ms)
|
||||
--------+------------+------------------+-----------------------
|
||||
250 | 250 | 5.4982303 | 0.0468107
|
||||
500 | 500 | 11.8662272 | 0.0591411
|
||||
1,000 | 1,000 | 21.6441107 | 0.0542551
|
||||
2,000 | 2,000 | 45.4870310 | 0.0623962
|
||||
```
|
||||
|
||||
This benchmark demonstrates the power of partial evaluation. Not only is the policy quick to evaluate but the evaluation latency remains relatively stable as the data set grows ([because of rule indexing](/blog/optimizing-opa-rule-indexing-59f03f17caf3)).
|
||||
|
||||
But it's only fair to warn that nothing comes for free. As the data set grows the time taken to run partial evaluation (and the memory required to store the result) also grow:
|
||||
|
||||
```
|
||||
# Roles | # Bindings | Partial Eval + Compile (seconds) | Heap (MB)
|
||||
--------+------------+----------------------------------+----------
|
||||
250 | 250 | 0.503920151 | 11.714843
|
||||
500 | 500 | 1.998275287 | 13.496093
|
||||
1,000 | 1,000 | 8.085092923 | 18.449218
|
||||
2,000 | 2,000 | 31.972901290 | 29.097656
|
||||
```
|
||||
|
||||
It's important to reiterate that the partial evaluation step is done offline, outside of the request path: this means the partial evaluation latency only affects the time taken to propagate policy updates. From our point of view, given other system behaviour (such as caching and failures), this is an acceptable price to pay.
|
||||
|
||||
## Future Work and Wrap Up
|
||||
|
||||
With v0.6, we've laid the groundwork for partial evaluation. Going forward we're planning to work on extending coverage over the kinds of expressions that can be partially evaluated, and looking at strategies like performing partial evaluation in multiple passes.
|
||||
|
||||
Hopefully this post was useful (or at least interesting)! Partial evaluation is a big step for OPA as it allows policies to be expressed concisely while still benefiting from optimizations like rule indexing. We're looking forward to applying partial evaluation to other use cases in the near future.
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
title: "v0.7 Release"
|
||||
authors: ["tsandall"]
|
||||
date: 2018-03-03
|
||||
slug: v0-7-release-2ef1c158613b
|
||||
---
|
||||
|
||||
We're happy to announce the v0.7 release of OPA. This release includes language improvements as well as initial support for monitoring via Prometheus. For the full list of interesting changes see [GitHub Releases](https://github.com/open-policy-agent/opa/releases). The rest of this post introduces the major features introduced in v0.7.
|
||||
|
||||
## Nested Expressions
|
||||
|
||||
Before v0.7, OPA only supported one level of embedding on expressions. For example:
|
||||
|
||||
- You could write `x = y * 2; z = x / 3`
|
||||
- But you could not write `z = (y * 2) / 3`
|
||||
|
||||
For simple policies that only rely on boolean expressions and perform a few built-in calls, the limitation was not an issue. However, in more sophisticated policies that perform a series of arithmetic or string manipulation, policy authors had to introduce intermediate variables, which meant coming up with good variable names (which we all know is hard.)
|
||||
|
||||
With support for nested expressions, policies can be expressed more concisely and without the need for extraneous variables. For example, the policy below computes a risk score for Terraform changes. Before v0.7, the policy required extra statements to sum the products:
|
||||
|
||||
```rego
|
||||
# compute risk score before v0.7
|
||||
score = s {
|
||||
all = [x | w = weights[type]
|
||||
del = w.delete * num_deletes[type]
|
||||
cre = w.create * num_creates[type]
|
||||
upd = w.update * num_updates[type]
|
||||
x1 = del + cre
|
||||
x2 = x1 + upd]
|
||||
sum(all, s)
|
||||
}
|
||||
```
|
||||
|
||||
In v0.7 with support for nested expressions we can write this simply as:
|
||||
|
||||
```rego
|
||||
score = sum([x | w = weights[type]
|
||||
x = (w.delete * num_deletes[type] +
|
||||
w.create * num_creates[type] +
|
||||
w.update * num_updates[type])])
|
||||
```
|
||||
|
||||
Similarly, chains of string manipulation operations are much more concise now. Before v0.7:
|
||||
|
||||
```rego
|
||||
decoded_str = base64url.decode(str)
|
||||
trimmed_str = trim(decoded_str, "/")
|
||||
parts = split(trimmed_str, "/")
|
||||
```
|
||||
|
||||
After v0.7:
|
||||
|
||||
```rego
|
||||
parts = split(trim(base64url.decode(str), "/"))
|
||||
```
|
||||
|
||||
## Assignment and Comparison Operators
|
||||
|
||||
In OPA, the `=` operator performs assignment and comparison _at the same time_. If you come from an imperative or functional programming background, this can be a bit confusing at first. To help improve the user experience for first time OPA users, we have introduced new operators for assignment (`:=`) and comparison (`==`) that behave the way you would expect coming from a traditional programming language.
|
||||
|
||||
The `:=` operator allows users to declare local variables within the current scope by assigning a value. Unlike the `=` operator, variables appearing on the left hand side of the expression will _shadow_ symbols in the global scope. Furthermore, redeclaration of a variable in the same scope is treated as an error.
|
||||
|
||||
The `==` operator allows users to compare two values. Unlike the `=` operator, variables appearing on either side of the operator must be set elsewhere in the query (otherwise an error is reported.)
|
||||
|
||||
These changes are backwards compatible. Existing policies are unaffected and you can continue to develop policies without using the new operators if you like.
|
||||
|
||||
## Built-in Functions
|
||||
|
||||
OPA supports 50+ built-in functions. The documentation for built-in functions can be found in the [Language Reference](http://www.openpolicyagent.org/docs/language-reference.html). Internally, OPA makes it relatively easy to [add new built-in functions.](https://godoc.org/github.com/open-policy-agent/opa/topdown#example-RegisterFunctionalBuiltin1)
|
||||
|
||||
This release adds several new built-in functions:
|
||||
|
||||
- `regex.glob_intersect` checks if two regex patterns intersect.
|
||||
- `http.send` executes a HTTP request and returns the response.
|
||||
- `time.date` returns `[YYYY, MM, DD]` from the time since epoch provided as input.
|
||||
- `time.clock` returns `[HH, mm, SS]` for the day of the time since epoch provided as input.
|
||||
- `intersection` returns the n-way intersection of sets provided as input.
|
||||
- `union` returns the n-way union of sets provided as input.
|
||||
|
||||
## Prometheus /metrics Endpoint
|
||||
|
||||
As we continue to harden and optimize OPA, we have begun to focus more on monitoring and diagnostic features. This release adds support for a /metrics API endpoint that exposes high level performance metrics so that users can monitor OPA using the Prometheus project. See the new [Monitoring & Diagnostics](http://www.openpolicyagent.org/docs/monitoring-diagnostics.html) documentation for more details.
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
title: "Managing OPA"
|
||||
authors: ["tsandall"]
|
||||
date: 2018-04-19
|
||||
slug: managing-opa-203a3bc86c5d
|
||||
---
|
||||
|
||||
OPA is described as "a general-purpose policy engine that let's you offload decisions from your service," requiring access to policy and data for decision-making.
|
||||
|
||||
Before version 0.8, OPA "only exposed low-level HTTP APIs that let you push policy and data into the engine." Version 0.8 adds new management capabilities for distributing policies/data and monitoring agent health.
|
||||
|
||||
|
||||
|
||||
## Bundle API
|
||||
|
||||
OPA can now be configured to download "bundles" — described as "gzipped tarballs containing Rego and JSON files" — from remote HTTP endpoints on a periodic basis.
|
||||
|
||||
```
|
||||
GET /bundles/example/authz HTTP/1.1
|
||||
```
|
||||
|
||||
Services should respond with a gzipped tarball. If an ETag header is included, it gets reused in subsequent `If-None-Match` requests, allowing servers to reply with an HTTP 301 Not Modified instead of resending the same bundle.
|
||||
|
||||
More info: [http://www.openpolicyagent.org/docs/bundles.html](http://www.openpolicyagent.org/docs/bundles.html)
|
||||
|
||||
## Status API
|
||||
|
||||
Since OPA already supports Prometheus-based metrics reporting ([http://www.openpolicyagent.org/docs/monitoring-diagnostics.html#prometheus](http://www.openpolicyagent.org/docs/monitoring-diagnostics.html#prometheus)), the new Status API answers two additional questions: which policy/data version is active, and what errors occurred during load.
|
||||
|
||||
OPA can be configured to periodically POST status updates to a remote endpoint:
|
||||
|
||||
```json
|
||||
POST /status HTTP/1.1
|
||||
Content-Type: application/json
|
||||
{
|
||||
"labels": {
|
||||
"app": "my-example-app",
|
||||
"id": "1780d507-aea2-45cc-ae50-fa153c8e4a5a"
|
||||
},
|
||||
"bundle": {
|
||||
"name": "http/example/authz",
|
||||
"active_revision": "660daf152602bd95ea2d2139d215678b0ed[...]",
|
||||
"last_successful_download": "2018-04-17T12:34:40.258Z",
|
||||
"last_successful_activation": "2018-04-17T12:34:42.196Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The payload also includes labels that "uniquely identify the agent."
|
||||
|
||||
More info: [http://www.openpolicyagent.org/docs/status.html](http://www.openpolicyagent.org/docs/status.html)
|
||||
|
||||
## Decision Log API
|
||||
|
||||
For auditing and debugging, OPA can buffer and periodically upload batches of policy decisions to a remote endpoint. Each event includes:
|
||||
|
||||
- The name of the policy decision requested by your service.
|
||||
- The input provided in the query by your service.
|
||||
- The result returned by OPA to your service.
|
||||
|
||||
Events also include metadata such as bundle revision and agent labels.
|
||||
|
||||
More info: [http://www.openpolicyagent.org/docs/decision_logs.html](http://www.openpolicyagent.org/docs/decision_logs.html)
|
||||
|
||||
## Wrap Up
|
||||
|
||||
The author frames these features as simplifying large-scale OPA management while enabling more advanced tooling. Feedback is welcomed via Slack: [http://slack.openpolicyagent.org](http://slack.openpolicyagent.org)
|
||||
@@ -0,0 +1,61 @@
|
||||
---
|
||||
title: "v0.9 Release"
|
||||
authors: ["tsandall"]
|
||||
date: 2018-07-31
|
||||
slug: v0-9-release-4eb605bd0989
|
||||
---
|
||||
|
||||
This post provides a quick overview of the work that has gone into OPA v0.9. As usual, the release is published on [GitHub Releases](https://github.com/open-policy-agent/opa/releases) and [Docker Hub](https://hub.docker.com/r/openpolicyagent/opa/tags/).
|
||||
|
||||
## Securing the Data Lake: Ceph and Minio Integrations
|
||||
|
||||
During this release cycle we worked with the upstream [Ceph](https://ceph.com/) and [Minio](https://github.com/minio/minio) communities to introduce fine-grained access control into data lakes using OPA. Both Ceph and Minio support the standard S3 object storage APIs and are popular choices for deploying object storage services.
|
||||
|
||||
The integrations into Ceph and Minio allow administrators to express fine-grained attribute-based access control (ABAC) policies over requests to the object storage layer. Compared to Bucket Policies and Object ACLs, these integrations give administrators greater control over sensitive data stored in these services. Using OPA you can enforce policies over object and file access based on context such as:
|
||||
|
||||
- Time of day
|
||||
- Multi-Factor Authentication (MFA) attributes
|
||||
- Geographic region of the connecting client
|
||||
|
||||
We expect to see more adoption of OPA to control access to sensitive data as more and more organizations build out data lakes using object storage services like Ceph and Minio.
|
||||
|
||||
For more information on the integrations see the PRs:
|
||||
|
||||
- [Ceph integration](https://github.com/ceph/ceph/pull/22624) and [documentation preview](http://docs.ceph.com/ceph-prs/22624/radosgw/opa/)
|
||||
- [Minio integration](https://github.com/minio/minio/pull/6168)
|
||||
|
||||
## Profiling Policy Evaluation
|
||||
|
||||
OPA now includes a command-line tool that helps you understand the performance of your policies.
|
||||
|
||||
Given a policy query, the tool reports per-expression metrics including:
|
||||
|
||||
- Time spent evaluating each expression (non-recursive)
|
||||
- Number of times each expression is called from the outside
|
||||
- Number of times each expression is re-evaluated due to backtracking
|
||||
|
||||
The need for profiler support became obvious after reviewing and optimizing several large policies. These policies relied on sophisticated search logic split across hundreds of lines of policy statements in multiple files. In many cases it was possible to improve performance significantly by simply tweaking how context was structured.
|
||||
|
||||
We hope that by providing a profiler, policy authors will be able to quickly identify the root cause of performance issues.
|
||||
|
||||
You can invoke the profiler on the command line via the `opa eval` subcommand. For example:
|
||||
|
||||
```bash
|
||||
opa eval data.rbac.allow --data rbac.rego --format pretty --profile
|
||||
```
|
||||
|
||||
By default, when profiling is enabled, OPA will output the top 10 most expensive expressions in the policy.
|
||||
|
||||
|
||||
|
||||
Example profiler output.
|
||||
|
||||
For more information on the profiler tool, see the new [documentation page describing how to test your policies.](https://www.openpolicyagent.org/docs/how-do-i-test-policies.html)
|
||||
|
||||
## Compile API
|
||||
|
||||
Finally, as of v0.9, OPA includes new APIs that allow callers to invoke [Partial Evaluation](/blog/partial-evaluation-162750eaf422) via [HTTP](https://www.openpolicyagent.org/docs/rest-api.html#compile-api) or in Go with the [Rego package](https://godoc.org/github.com/open-policy-agent/opa/rego#example-Rego-Partial).
|
||||
|
||||
In the past, decisions returned by OPA were always definitive, e.g., callers would query for an "allow" or "deny" decision and the answer would always be "true" or "false". With partial evaluation exposed, OPA can return condition answers, e.g., "allow" is "true" if certain conditions are satisfied.
|
||||
|
||||
Later this week we will publish a blog post that describes how you can enforce data filtering policies in your storage layer with OPA by leveraging the new Compile API.
|
||||
@@ -0,0 +1,313 @@
|
||||
---
|
||||
title: "Write Policy in OPA. Enforce Policy in SQL."
|
||||
authors: ["tsandall"]
|
||||
date: 2018-08-17
|
||||
slug: write-policy-in-opa-enforce-policy-in-sql-d9d24db93bf4
|
||||
---
|
||||
|
||||
This post explains how to use OPA and SQL to protect access to sensitive data in your services without impacting consistency, performance, or scalability. We show how to translate OPA policies into SQL and enforce them within the database.
|
||||
|
||||
Throughout this post we will refer to a hypothetical service (`petprofilesv1`) used by a chain of veterinary clinics. The service exposes an HTTP API that serves profiles for pets at the clinics. The service implements the HTTP API by querying a SQL database and returning the results.
|
||||
|
||||
|
||||
|
||||
Services like `petprofilesv1` often implement authorization policies that depend on attributes of the objects being accessed. They often also include authorization policies that must be applied to filter data in API results.
|
||||
|
||||
In this post, we dive into how services can integrate with OPA to enforce these kinds of authorization policies.
|
||||
|
||||
## Replicating Context Is Hard
|
||||
|
||||
Imagine we want to enforce a simple role-based authorization policy in our hypothetical service that says:
|
||||
|
||||
> "Only veterinarians are allowed to read pet profiles."
|
||||
|
||||
|
||||
|
||||
To implement this policy we could create a simple `allow` rule in OPA:
|
||||
|
||||
```rego
|
||||
default allow = false
|
||||
|
||||
allow = true {
|
||||
input.method = "GET"
|
||||
input.path = ["pets", name] # name unused for now
|
||||
input.subject.roles[_] = "veterinarian"
|
||||
}
|
||||
```
|
||||
|
||||
When the service queries OPA, it provides the identity of the caller (`input.subject`), the operation being performed (`input.method`), and the resource being operated on (`input.path`). The response from OPA indicates whether the request should be allowed (`true`) or denied (`false`).
|
||||
|
||||
**This model works well when all of the context required for the decision is carried in the request.** But what happens if the incoming request does not include the necessary context? For example, suppose the policy should say:
|
||||
|
||||
> "Only the treating veterinarian is allowed to read a pet's profile."
|
||||
|
||||
In this case, a mapping from pet to treating veterinarian is required but not included in incoming HTTP GET requests.
|
||||
|
||||
One way of solving this would be to have the service fetch relevant context and provide it as input to the policy query:
|
||||
|
||||
```rego
|
||||
allow = true {
|
||||
input.method = "GET"
|
||||
input.path = ["pets", name] # name unused for now
|
||||
input.subject.user = input.pet.veterinarian
|
||||
}
|
||||
```
|
||||
|
||||
This approach works well for small inputs and keeps the enforcement model relatively simple.
|
||||
|
||||
The downside is that it requires the service to know the exact context required by the policy. Each API endpoint exposed by the service (or set of services) could need custom logic to fetch the context to provide as input to the policy query. **Tight coupling between the service and the policy is difficult to maintain over time.** Moreover, as the size of the input grows, it may become prohibitive to fetch and supply context on every query.
|
||||
|
||||
Another approach is to have the pet-veterinarian mapping replicated into OPA and cached in-memory. This approach requires an additional component to act as a _data source (DS)_ that gathers the extra context required by policies and loads it into OPA.
|
||||
|
||||
|
||||
|
||||
If we used this approach, we could rewrite the OPA rule as follows:
|
||||
|
||||
```rego
|
||||
allow = true {
|
||||
input.method = "GET"
|
||||
input.path = ["pets", name]
|
||||
data.pets[name] = input.subject.user
|
||||
}
|
||||
```
|
||||
|
||||
This avoids directly coupling the service with the policy. However, the obvious challenge is that OPA must maintain a cache of all pet-veterinarian mappings to authorize requests correctly. Depending on the size of the mapping and the consistency requirements of the service, this may not be feasible: OPA may not have the up-to-date mappings while evaluating the policy.
|
||||
|
||||
We could also use built-in functions in OPA to execute HTTP requests or query external databases during policy evaluation however this approach makes policies harder to test in isolation and the extra network hop negatively impacts latency and availability. Similarly, this approach may also violate the service's consistency requirements.
|
||||
|
||||
## Lists Require Filtering
|
||||
|
||||
In addition to challenges with replication, we also have to deal with APIs that return lists of resources.
|
||||
|
||||
When designing service APIs, it's common to expose a _list_ operation to return all the resources in a collection (e.g., the `petprofilesv1` service needs to expose `GET /pets`). Since list operations frequently return the same information (for each resource) as reads for individual resources, it's important that we apply the authorization policy to filter elements in the result.
|
||||
|
||||
While we could execute a policy query for each resource (or structure the policy to accept a list of resources to authorize access to), this approach complicates pagination models and does not scale well compared to filtering in the data store.
|
||||
|
||||
## Partial Evaluation
|
||||
|
||||
In the first section of this post we explained why it may be difficult to replicate context into OPA in a reliable, maintainable, and performant manner.
|
||||
|
||||
As of the latest release of OPA (v0.9), services can leverage the [Partial Evaluation](/blog/partial-evaluation-162750eaf422) feature to avoid replicating context into OPA. When services use Partial Evaluation, they specify what portions of the data or input documents are _unknown_. When OPA evaluates the policy, any statements that depend on unknown values are not evaluated — they are saved and returned to the caller.
|
||||
|
||||
By partially evaluating authorization policies, OPA can treat context that's unavailable during evaluation as unknown. For example, statements that depend on the pet-veterinarian mapping would be saved and returned to the caller.
|
||||
|
||||
|
||||
|
||||
The table below shows the output when the service queries OPA for a request from `alice` trying to access `fluffy`'s profile.
|
||||
|
||||
```
|
||||
+----------+-------------------------------------------------------+
|
||||
| Policy | allow { |
|
||||
| | input.method = "GET" |
|
||||
| | input.path = ["pets", name] |
|
||||
| | data.pets[name].veterinarian = input.subject.user |
|
||||
| | } |
|
||||
+----------+-------------------------------------------------------+
|
||||
| Input | { |
|
||||
| | "method": "GET", |
|
||||
| | "path": ["pets", "fluffy"], |
|
||||
| | "subject": {"user": "alice"} |
|
||||
| | } |
|
||||
+----------+-------------------------------------------------------+
|
||||
| Unknowns | [data.pets] |
|
||||
+----------+-------------------------------------------------------+
|
||||
| Output | data.pets["fluffy"].veterinarian = "alice" |
|
||||
+----------+-------------------------------------------------------+
|
||||
```
|
||||
|
||||
When OPA partially evaluates policies, the output is a simplified version of the policy. The result of partial evaluation can be interpreted as a sequence of Rego expressions ANDed and ORed together:
|
||||
|
||||
```
|
||||
( expr-1 AND expr-2 AND … ) OR ( expr-N AND expr-N+1 AND … ) OR …
|
||||
```
|
||||
|
||||
If we extend the policy to allow pet owners to access their pet's profiles and require that veterinarians be signed in from a device at the pet's clinic, the output would include two queries (which can be ORed):
|
||||
|
||||
```
|
||||
+----------+-------------------------------------------------------+
|
||||
| Policy | allow { |
|
||||
| | input.method = "GET" |
|
||||
| | input.path = ["pets", name] |
|
||||
| | data.pets[name].owner = input.subject.user |
|
||||
| | } |
|
||||
| | |
|
||||
| | allow { |
|
||||
| | input.method = "GET" |
|
||||
| | input.path = ["pets", name] |
|
||||
| | data.pets[name].veterinarian = input.subject.user |
|
||||
| | data.pets[name].clinic = input.subject.location |
|
||||
| | } |
|
||||
+----------+-------------------------------------------------------+
|
||||
| Input | { |
|
||||
| | "method": "GET", |
|
||||
| | "path": ["pets", "fluffy"], |
|
||||
| | "subject": { |
|
||||
| | "user": "alice", |
|
||||
| | "location": "SOMA" |
|
||||
| | } |
|
||||
| | } |
|
||||
+----------+-------------------------------------------------------+
|
||||
| Unknowns | [data.pets] |
|
||||
+----------+-------------------------------------------------------+
|
||||
| Output 1 | data.pets["fluffy"].owner = "alice" |
|
||||
+----------+-------------------------------------------------------+
|
||||
| Output 2 | data.pets["fluffy"].veterinarian = "alice"; |
|
||||
| | data.pets["fluffy"].clinic = "SOMA" |
|
||||
+----------+-------------------------------------------------------+
|
||||
```
|
||||
|
||||
In some cases, OPA can still determine that a request should be allowed or denied unconditionally. In these cases, OPA returns a single empty query or no queries at all (respectively.) For example, if the input above was missing the subject field, neither allow rule would match (even partially) and the result would be empty.
|
||||
|
||||
By returning the simplified remainder of the policy to the service, we avoid evaluating the entire policy in one place — which means we do not have to replicate all of the context into OPA. The tradeoff is that the response from OPA is not simply an allow (`true`) or deny (`false`) value anymore — it's a set of Rego queries that must be evaluated by something (eventually).
|
||||
|
||||
This section explained how we can avoid replicating context into OPA using Partial Evaluation. However, this only solves part of the problem. If the service were to evaluate the result of Partial Evaluation itself, the service would still have to fetch the additional context from the data store (which would likely result in a solution with the same issues as before.)
|
||||
|
||||
## Enforcing OPA policies with SQL
|
||||
|
||||
To overcome the issues we outlined above, the remainder of the policy needs to be evaluated as close to the data as possible — inside the database.
|
||||
|
||||
[Since OPA policies are essentially just queries](/blog/opas-full-stack-policy-language-caeaadb1e077), the translation from Rego into another query language, like SQL, is relatively easy. As long as the policies expressed in Rego do not perform joins, we can translate sets of Rego queries into SQL expressions that get appended onto WHERE clauses. For example, the last policy from above says that:
|
||||
|
||||
> "Pet owners can access their own pet's profiles."
|
||||
> "Veterinarians can access pet profiles from devices at the clinic."
|
||||
|
||||
We can express this policy in OPA as follows:
|
||||
|
||||
```rego
|
||||
package petclinic.authz
|
||||
|
||||
default allow = false
|
||||
|
||||
allow {
|
||||
input.method = "GET"
|
||||
input.path = ["pets", name]
|
||||
allowed[pet]
|
||||
pet.name = name
|
||||
}
|
||||
|
||||
allowed[pet] {
|
||||
pet = data.pets[_]
|
||||
pet.owner = input.subject.user
|
||||
}
|
||||
|
||||
allowed[pet] {
|
||||
pet = data.pets[_]
|
||||
pet.veterinarian = input.subject.user
|
||||
pet.clinic = input.subject.location
|
||||
}
|
||||
```
|
||||
|
||||
In this example we have factored the authorization decision into helper rules named `allowed`. The `allowed` rules generate a _set_ of pets the user is allowed to see.
|
||||
|
||||
To integrate with OPA, the service invokes the Compile API and marks the `data.pets` path as unknown. For example, when `fluffy`'s veterinarian `alice` requests the profile from a device at the clinic, the query to OPA looks like this:
|
||||
|
||||
```json
|
||||
{
|
||||
// The policy query to run.
|
||||
"query": "data.petclinic.authz.allow = true",
|
||||
// The input document to use.
|
||||
"input": {
|
||||
"method": "GET",
|
||||
"path": ["pets", "fluffy"],
|
||||
"subject": {
|
||||
"user": "alice",
|
||||
"location": "SOMA"
|
||||
}
|
||||
},
|
||||
// The values to treat as unknown during evaluation.
|
||||
"unknowns": ["data.pets"]
|
||||
}
|
||||
```
|
||||
|
||||
The response from OPA contains two queries:
|
||||
|
||||
```rego
|
||||
# Query 1
|
||||
pet = data.pets[_];
|
||||
pet.owner = "alice";
|
||||
pet.name = "fluffy"
|
||||
|
||||
# Query 2
|
||||
pet = data.pets[_];
|
||||
pet.veterinarian = "alice";
|
||||
pet.clinic = "SOMA";
|
||||
pet.name = "fluffy"
|
||||
```
|
||||
|
||||
The service can consume these responses by translating them into SQL WHERE clauses. To keep things simple we can interpret references like `data.pets[_].owner` as follows:
|
||||
|
||||
- The prefix `data.pets` refers to the `pets` table in SQL.
|
||||
- The variable `_` refers to a row in the `pets` table.
|
||||
- The suffix `owner` refers to a column in the `pets` table.
|
||||
|
||||
With this interpretation we would produce the following SQL expression:
|
||||
|
||||
```sql
|
||||
(pets.owner = "alice" AND pets.name = "fluffy") OR
|
||||
(pets.veterinarian = "alice" AND
|
||||
pets.clinic = "SOMA" AND
|
||||
pets.name = "fluffy")
|
||||
```
|
||||
|
||||
To show how you can implement a library that converts a fragment of Rego into SQL WHERE clauses, we have prepared an example that includes a library. You can find the example on [GitHub in the OPA contrib repository](https://github.com/open-policy-agent/contrib/tree/master/data_filter_example).
|
||||
|
||||
## Data Filtering with OPA
|
||||
|
||||
In addition to the API to get individual pet profiles, our service also exposes an API to list pet profiles (e.g,. GET /pets). In many APIs, list operations return the full extent of the resources in the collection. Because of this, it's important to apply the same data authorization policy there.
|
||||
|
||||
Even if the list operations only returned a subset of fields (e.g., the resource ID), it's a common requirement to not show clients the IDs of resources they are not allowed to access, so the filtering policy should still be applied.
|
||||
|
||||
We can extend the policy from above to cover list operations by adding the following rule:
|
||||
|
||||
```rego
|
||||
allow {
|
||||
input.method = "GET"
|
||||
input.path = ["pets"]
|
||||
allowed[pet]
|
||||
}
|
||||
```
|
||||
|
||||
This rule is similar to the ones from earlier. The only differences are that it matches on the path `["pets"]` (instead of `["pets", name]`) and there is no condition on `pet.name`.
|
||||
|
||||
The result sent back to the service from OPA will be similar to the cases above. For example, if `alice` tries to list the pet profiles from the same device at the `SOMA` clinic the response from OPA will be:
|
||||
|
||||
```rego
|
||||
# Output #1
|
||||
pet = data.pets[_]
|
||||
pet.owner = "alice"
|
||||
|
||||
# Output #2
|
||||
pet = data.pets[_]
|
||||
pet.veterinarian = "alice";
|
||||
pet.clinic = "SOMA"
|
||||
```
|
||||
|
||||
These statements will get translated into a SQL WHERE clause that return all of the pet profiles that `alice` is allowed to see at the `SOMA` clinic as well as any profiles of her own pets at any clinic:
|
||||
|
||||
```sql
|
||||
(pets.owner = "alice") OR
|
||||
(pets.veterinarian = "alice" AND pets.clinic = "SOMA")
|
||||
```
|
||||
|
||||
This example highlights how you can reuse the same data authorization policy across both the _get_ and _list_ APIs.
|
||||
|
||||
## Try It Out
|
||||
|
||||
You can try out this new capability with the latest release of OPA. If you integrate with OPA in Go, you can use the [rego.Rego#Partial](https://godoc.org/github.com/open-policy-agent/opa/rego#example-Rego-Partial) function to invoke Partial Evaluation and obtain a set of conditions to apply to incoming requests. Similarly, if you integrate with OPA via HTTP, you can use the new [Compile API](https://www.openpolicyagent.org/docs/rest-api.html#compile-api) to obtain identical conditions (represented in JSON) that you can process in any language.
|
||||
|
||||
We have also published a small [example](https://github.com/open-policy-agent/contrib/tree/master/data_filter_example) that shows how to integrate a simple Python service backed by SQLite. The example includes a Python module that translates Rego queries into SQL predicates. The example includes additional support for policies that perform joins as well as relational operators like `!=`, `<=`, `>=`, etc.
|
||||
|
||||
## Wrap Up
|
||||
|
||||
This post explored how you can leverage OPA to enforce context-aware authorization policies that go way beyond simple protocol-level approaches. By using Partial Evaluation to obtain conditions that you evaluate in your service or via database queries, you can implement data protection and filtering policies in your service that are correct, maintainable, consistent, performant, and scalable.
|
||||
|
||||
We anticipate these OPA features will be used to solve a number of interesting problems in the authorization space, such as multi-tenancy support and protection of PII and other sensitive data.
|
||||
|
||||
Next we plan to tackle:
|
||||
|
||||
- Integrations with other kinds of databases, e.g., Elasticsearch.
|
||||
- Tooling to identify when policies exceed the supported fragments of Rego.
|
||||
- Indirection layers between the policy and underlying DB schema.
|
||||
- Other use cases that build on partial evaluation like _column masking_.
|
||||
|
||||
If you have questions or comments, or you are interested in contributing, please reach out via [Slack](https://slack.openpolicyagent.org) or create tickets on [GitHub](https://github.com/open-policy-agent/opa).
|
||||
@@ -0,0 +1,119 @@
|
||||
---
|
||||
title: "v0.10 Release"
|
||||
authors: ["tsandall"]
|
||||
date: 2018-10-25
|
||||
slug: v0-10-release-277da41b9ed1
|
||||
---
|
||||
|
||||
|
||||
|
||||
We're excited to announce the v0.10.0 release of OPA. This release contains more than 60 commits from 8 authors across 5 organizations. For a detailed list of changes see the [GitHub releases](https://github.com/open-policy-agent/opa/releases) page.
|
||||
|
||||
## WebAssembly
|
||||
|
||||
This release adds experimental support for compiling OPA policies into WebAssembly (Wasm) binaries that can be executed in any Wasm runtime (e.g., V8).
|
||||
|
||||
The compiler is designed to be lightweight and embedded inside other programs. The package does not depend on any third-party compiler toolchains like LLVM or Wasm-specific toolchains like Emscripten. The compiled policies are fairly small in size (10KB for toy examples) and have no system call dependencies — making them easy to instantiate inside your app, serverless function, etc.
|
||||
|
||||
We are excited about the potential that Wasm brings to the policy enforcement space because it provides a portable, secure, and efficient runtime for answering policy queries. You can find out more about Wasm at [https://webassembly.org/](https://webassembly.org/).
|
||||
|
||||
If you are feeling adventurous, you can try out the Wasm compiler today with the new `opa build` command. We only support a limited subset of the language today but [we plan to extend coverage over the next few months](https://github.com/open-policy-agent/opa/issues/1024). If you run into any problems, please file an issue on GitHub.
|
||||
|
||||
## Improved Test Support with Data Mocking
|
||||
|
||||
This release adds support for replacing (or mocking) values under the `data` document using the `with` keyword. You can use the `with` keyword to replace both external JSON loaded into OPA as well as JSON generated by rules.
|
||||
|
||||
Prior to v0.10, OPA only allowed you to replace values under the `input` document. This made it hard to test contextual policies using [OPA's test framework](https://www.openpolicyagent.org/docs/how-do-i-test-policies.html). For example, the following policy depends on "roles" data being loaded into OPA:
|
||||
|
||||
```rego
|
||||
package authz
|
||||
|
||||
import data.roles
|
||||
|
||||
default allow = false
|
||||
|
||||
allow {
|
||||
input.method == "GET"
|
||||
input.subject.role == roles[_]
|
||||
}
|
||||
```
|
||||
|
||||
In v0.10.0, you can write a test rule that mocks the value of `data.roles` using the `with` keyword:
|
||||
|
||||
```rego
|
||||
# Define some dummy inputs.
|
||||
dev_input = {"method": "GET", "subject": {"role": "dev"}}
|
||||
hr_input = {"method": "GET", "subject": {"role": "hr"}}
|
||||
|
||||
# Define some dummy role data.
|
||||
fake_roles = ["hr"]
|
||||
|
||||
# Test the allow rule.
|
||||
test_allow {
|
||||
allow with input as hr_input with data.roles as fake_roles
|
||||
not allow with input as dev_input with data.roles as fake_roles
|
||||
}
|
||||
```
|
||||
|
||||
## Partial Evaluation: Negation Optimization
|
||||
|
||||
Earlier this year we added a feature called Partial Evaluation that helps pre-compute portions of your policy. Today, a large fragment of the language is covered by Partial Evaluation but some constructs are not fully supported.
|
||||
|
||||
Prior to v0.10, OPA would generate _support rules_ for negated expressions (which can be difficult to post-process.) This was required because Rego queries only consist of a series of expressions AND-ed together. If you need to express an OR condition, you need multiple queries. When you negate an expression (e.g., **not deny**), the in-lined result may be a series of expressions OR-ed together. For example, given the following policy that says (in English) "no one is allowed to buy more bitcoin or eat pizza":
|
||||
|
||||
```rego
|
||||
allow {
|
||||
not deny
|
||||
}
|
||||
|
||||
deny {
|
||||
input.action = "buy"
|
||||
input.resource = "bitcoin"
|
||||
}
|
||||
|
||||
deny {
|
||||
input.action = "eat"
|
||||
input.resource = "pizza"
|
||||
}
|
||||
```
|
||||
|
||||
We can partially evaluate the `deny` rule to yield the following simplified queries:
|
||||
|
||||
```
|
||||
+---------+----------------------------+
|
||||
| Query 1 | input.action = "buy" |
|
||||
| | input.resource = "bitcoin" |
|
||||
+---------+----------------------------+
|
||||
| Query 2 | input.action = "eat" |
|
||||
| | input.resource = "pizza" |
|
||||
+---------+----------------------------+
|
||||
```
|
||||
|
||||
However, it's a bit trickier to partially evaluate the `allow` rule. OPA does not allow you to negate multiple expressions at once (you have to factor those expressions into a separate rule). In some cases though, it's reasonable to in-line the result of partially evaluating a negated expression by computing the cross-product. In this case the answer is:
|
||||
|
||||
```
|
||||
+---------+--------------------------------+
|
||||
| Query 1 | not input.action = "buy" |
|
||||
| | not input.action = "eat" |
|
||||
+---------+--------------------------------+
|
||||
| Query 2 | not input.action = "buy" |
|
||||
| | not input.resource = "pizza" |
|
||||
+---------+--------------------------------+
|
||||
| Query 3 | not input.resource = "bitcoin" |
|
||||
| | not input.action = "eat" |
|
||||
+---------+--------------------------------+
|
||||
| Query 4 | not input.resource = "bitcoin" |
|
||||
| | not input.resource = "pizza" |
|
||||
+---------+--------------------------------+
|
||||
```
|
||||
|
||||
By in-lining negated expressions like this, we avoid the need for support rules (which are more difficult to optimize and [translate into other languages like SQL and Elasticsearch](/blog/write-policy-in-opa-enforce-policy-in-sql-d9d24db93bf4).) Of course, the size of the cross-product can get quite big, so we put a cap on what OPA will in-line.
|
||||
|
||||
## More Great Contributions
|
||||
|
||||
This release also included a many other contributions from members of the community. Here are some highlights:
|
||||
|
||||
- [JWT decode & verify built-in function.](https://github.com/open-policy-agent/opa/issues/884) This helps implement best practices around checking the aud, exp, and nbf claims automatically.
|
||||
- [Client certificate support for service authentication.](https://github.com/open-policy-agent/opa/issues/684) This allows services to authenticate OPA bundle download, status report, and decision log upload requests using client-side certificates (which may be preferred to bearer tokens.)
|
||||
- [Trace output in test failures.](https://github.com/open-policy-agent/opa/issues/856) This helps policy authors debug test failures faster by pinpointing the source of the issue.
|
||||
- …and many more!
|
||||
+181
@@ -0,0 +1,181 @@
|
||||
---
|
||||
title: "Securing the Kubernetes API with Open Policy Agent"
|
||||
authors: ["timhinrichs"]
|
||||
date: 2019-02-08
|
||||
slug: securing-the-kubernetes-api-with-open-policy-agent-ce93af0552c3
|
||||
---
|
||||
|
||||
|
||||
|
||||
Kubernetes is being rolled out for production — it's mission critical. But it presents unique challenges around the age-old problem of who-can-do-what. Those challenges are exactly the ones the Open Policy Agent was designed to solve.
|
||||
|
||||
## TL;DR
|
||||
|
||||
This post highlights several key ideas:
|
||||
|
||||
- Controlling who-can-do-what on Kubernetes has unique challenges because to make an access control decision you need to inspect an arbitrary chunk of YAML, e.g. the images in all containers in all pods must come from a trusted repository.
|
||||
- The Open Policy Agent was designed around the premise that sometimes you need to write and enforce access control decisions over arbitrary JSON/YAML, so it's a perfect match for Kubernetes's challenges.
|
||||
- OPA supports a class of access control decisions called "context-aware" that enable you to make decisions based on the Kubernetes resources already in the cluster, e.g. no conflicting ingresses.
|
||||
|
||||
## KubeCon Seattle 2018 Debrief
|
||||
|
||||
After each KubeCon we try to document the answer to the question we heard most often while talking to folks at the Open Policy Agent (OPA) booth. For those who don't know, OPA is a general-purpose policy engine for the cloud-native stack and has been applied to solve policy and authorization problems in several different domains, e.g. microservice authorization, data protection, ssh/sudo control, terraform risk-analysis, and most popular at KubeCon this year: Kubernetes admission control. The most common question we heard was
|
||||
|
||||
> Why is OPA so well-suited for securing the Kubernetes API through admission control?
|
||||
|
||||
People heard about this use case in several different talks throughout the week, which is why we think so many people were asking about it at the booth. Here are links to those talks:
|
||||
|
||||
- [Securing Kubernetes With Admission Controllers from Dave Strebel (Microsoft](https://sched.co/GrZQ))
|
||||
- [User Studies from Zach Abrahamson (Capital One) and Todd Ekenstam (Intuit)](https://sched.co/Grbn)
|
||||
- [Liz Rice's keynote](https://youtu.be/McDzaTnUVWs?t=418)
|
||||
|
||||
In this post, when we talk about securing the Kubernetes, we're talking about the Kubernetes API itself — the container management system. We're talking about helping you, the Kubernetes cluster admin, put guardrails in place so that the developers running applications on top of Kubernetes don't need be constantly referring to wikis or PDFs that detail what policies the organization has decided on around Kubernetes. OPA lets you codify those wikis and PDF policies into policy-as-code and enforce them directly on the cluster. For example:
|
||||
|
||||
- every container image must come from a trusted, corporate repository
|
||||
- every application exposed to the internet must use an approved domain name
|
||||
- every resource must include a `costcenter` label
|
||||
- business critical storage volumes must use the `retain` storage policy
|
||||
|
||||
One thing people sometimes mean when they say "Kubernetes" is the applications running on top of the Kubernetes container management system. That's another use case for OPA, but not the one covered in this post. Here are a few references if you're interested in using OPA to provide API security for cloud-native applications themselves (whether or not they run on Kubernetes).
|
||||
|
||||
- [How Netflix is Solving Authorization (with OPA), KubeCon Austin 2017](https://www.youtube.com/watch?v=R6tUNpRpdnY)
|
||||
- [Implementing Authorization (for your app), KubeCon Shanghai 2018](https://sched.co/FuKQ)
|
||||
- [OPA tutorial for HTTP-API authorization](https://www.openpolicyagent.org/docs/http-api-authorization.html)
|
||||
- OPA integrations for [Java Spring](https://github.com/open-policy-agent/contrib/tree/master/spring_authz), [Istio/Envoy](https://github.com/open-policy-agent/opa-istio-plugin), [Linkerd](https://github.com/open-policy-agent/contrib/tree/master/linkerd_authz)
|
||||
|
||||
## The Kubernetes YAML-centric API
|
||||
|
||||
The key reason OPA is such a good choice for securing Kubernetes is that the Kubernetes API is pretty unique, and that presents challenges for authorization and API security. Within the community people refer to it as the Kubernetes Resource Model ([Brian Grant's doc](https://docs.google.com/document/d/1RmHXdLhNbyOWPW_AtnnowaRfGejw-qlKQIuLKQWlwzs/edit), [Tim Hockin tweet](https://twitter.com/thockin/status/1092091311572701184)).
|
||||
|
||||
Each Kubernetes API call requires you to specify the desired-state for one of Kubernetes's many objects: pods, services, ingresses, deployments, etc. For example, here is you define the desired state for an nginx workload.
|
||||
|
||||
```yaml
|
||||
# nginx-pod.yaml
|
||||
kind: Pod
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: nginx
|
||||
labels:
|
||||
app: nginx
|
||||
spec:
|
||||
containers:
|
||||
- image: nginx
|
||||
name: nginx
|
||||
```
|
||||
|
||||
To create this workload, you use `kubectl` and hand it the YAML file (`-f`) above.
|
||||
|
||||
```bash
|
||||
kubectl create -f nginx-pod.yaml
|
||||
```
|
||||
|
||||
Updates happen similarly. Say you want to change the version of `nginx`, mount an external volume, or provide additional configuration. You update the `nginx-pod.yaml` file to whatever the desired state should be and use `kubectl` again, this time using `apply` instead of `create`.
|
||||
|
||||
```bash
|
||||
kubectl apply -f nginx-pod.yaml
|
||||
```
|
||||
|
||||
## The Challenge of Securing the Kubernetes API
|
||||
|
||||
Imagine now that you want to require all images to come from a trusted repository (say, `hooli.com`). Anytime someone runs, say,`kubectl create`, the access control system needs to make a decision based on the user, the action `create` and the YAML that describes the pod, e.g.
|
||||
|
||||
```yaml
|
||||
kind: Pod
|
||||
metadata:
|
||||
labels:
|
||||
app: nginx
|
||||
name: nginx-1493591563-bvl8q
|
||||
namespace: production
|
||||
spec:
|
||||
containers:
|
||||
- image: nginx
|
||||
name: nginx
|
||||
securityContext:
|
||||
privileged: true
|
||||
- image: hooli.com/frontend
|
||||
name: frontend
|
||||
securityContext:
|
||||
privileged: true
|
||||
dnsPolicy: ClusterFirst
|
||||
nodeName: minikube
|
||||
restartPolicy: Always
|
||||
```
|
||||
|
||||
To make the right decision, the access control system needs to extract the list of image names (e.g. `nginx` and `hooli.com/frontend`) and do string manipulation to extract the name of the repository (e.g. the default repo and `hooli.com`). To complicate matters, Kubernetes supports Custom Resource Definitions, which means we can't just build an access control system that knows the layout of these YAML files. We need the access control system to be expressive enough for all of the following:
|
||||
|
||||
- Descending through the hierarchical structure of a YAML file.
|
||||
- Iterating over elements in an array.
|
||||
- Manipulating strings, IPs, numbers, etc.
|
||||
|
||||
## Securing the Kubernetes API with Open Policy Agent
|
||||
|
||||
This is where the Open Policy Agent shines. OPA was designed to express access control policies (as well as other kinds of policies) over arbitrary JSON/YAML, along with a complete toolkit for testing, dry-running, auditing, profiling, and integrating those policies into third party projects. The list of requirements from the last section are first-class citizens in OPA's policy language: dot-notation, iteration, and built-in functions. That means that encoding the policy that says, "all images must come from the repository `hooli.com`" is just a few lines in OPA.
|
||||
|
||||
```rego
|
||||
# deny any pod with an image not from the repository hooli.com
|
||||
deny {
|
||||
image_name := input.spec.containers[_].image
|
||||
not startswith(image_name, "hooli.com")
|
||||
}
|
||||
```
|
||||
|
||||
The logic shown above denies the API call if there is ANY container in the pod whose image fails to start with `hooli.com`. If you want to understand how the code works, the following notes should help:
|
||||
|
||||
- `input` is an OPA keyword that stores the JSON/YAML document representing the Kubernetes YAML shown earlier.
|
||||
- The dot-notation (e.g. `input.spec.containers`) does the obvious thing — descending through the YAML hierarchy.
|
||||
- The underscore (`_`) iterates over all the containers. If the body of the `deny` rule is true for ANY of the containers in the array, the pod is rejected. Note: iteration is not limited to only `_` — see the [docs](https://www.openpolicyagent.org/docs/how-do-i-write-policies.html#variable-keys) for details.
|
||||
- `startswith` is one of [50+ builtins for string, numeric, IP, etc. manipulation](https://www.openpolicyagent.org/docs/language-reference.html).
|
||||
|
||||
## OPA's Context-aware Kubernetes Policies
|
||||
|
||||
That image-repository example is actually one of the simpler access control policies you might need to write for Kubernetes because you can make the decision using just the one YAML file describing the pod. But sometimes you need to know what other resources exist in the cluster to make an allow/deny decision. For example, it's possible to accidentally create two applications serving internet traffic using Kubernetes `ingresses` where one application steals traffic from the other. The policy that prevents that needs to compare a new ingress that's being created/updated with all of the existing ingresses. That leads to another requirement for a Kubernetes access control system that OPA supports:
|
||||
|
||||
- Conditioning decisions based on external information about the world.
|
||||
|
||||
To see the OPA policy that prohibits conflicting ingresses, here is an example `ingress` YAML.
|
||||
|
||||
```yaml
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: test-ingress
|
||||
annotations:
|
||||
nginx.ingress.kubernetes.io/rewrite-target: /
|
||||
spec:
|
||||
rules:
|
||||
- http:
|
||||
paths:
|
||||
- path: /testpath
|
||||
backend:
|
||||
serviceName: test
|
||||
servicePort: 80
|
||||
```
|
||||
|
||||
Below is the (essence of the) OPA policy that stops an ingress from being created/updated if there is an _existing_ ingress that it would conflict with.
|
||||
|
||||
```rego
|
||||
deny {
|
||||
input.kind == "Ingress"
|
||||
host := input.request.object.spec.rules[_].host
|
||||
host == data.kubernetes.ingresses[_][_].spec.rules[_].host
|
||||
}
|
||||
```
|
||||
|
||||
The only new part of this policy is the reference to `data.kubernetes`. Below are some notes explaining that code.
|
||||
|
||||
- `data` is a keyword in OPA (similar to `input`) that stores all of the information about the external world.
|
||||
- The reference `data.kubernetes.ingresses` is a dictionary mapping each namespace into the array of ingresses in that namespace.
|
||||
- `data.kubernetes.ingresses[_][_]` iterates over all ingresses over all namespaces.
|
||||
- The last line checks if the `host` for the new/updated ingress is the same as the host on any of the rules over any of the ingresses in any namespace. (3 "any"s means you need 3 underscores.)
|
||||
|
||||
How you load information about the external world into OPA varies depending on the use case and the kind of data. Typically the information loaded into OPA is eventually-consistent, meaning it's a copy of the data and could be out of date — whether that matters depends entirely on the use case and can be mitigated by using OPA's offline auditing capabilities. To load Kubernetes data, OPA has a sidecar that watches the API server to replicate Kubernetes resources into OPA.
|
||||
|
||||
In this post, we've given 2 simple and common examples of policies using the core Kubernetes objects (pods and ingresses), but there's nothing special about those resources. If you're using Custom Resource Definitions, you can still go ahead and write whatever policies you need, e.g. [knative](https://github.com/knative) or [istio](https://github.com/istio). Any resource managed by Kubernetes is something you can write policy over with OPA — as far as OPA is concerned they're all just YAML/JSON.
|
||||
|
||||
## Summary
|
||||
|
||||
In this post, we dug into the API security challenges faced by Kubernetes and how OPA addresses those challenges.
|
||||
|
||||
- Kubernetes's API is YAML-centric, meaning that the arguments to API calls are (at least conceptually) arbitrary chunks of YAML.
|
||||
- A YAML-centric API is challenging for access control because it requires analyzing that YAML to make a decision. For example, the policy "ensure all images come from a trusted repository" requires navigating the YAML to find the list of all containers, iterating over that list, extracting the image name, and string-parsing that image name to extract the repository.
|
||||
- The Open Policy Agent's declarative policy language was designed to express policy over arbitrary JSON/YAML, so it includes implicit iteration, dot-notation, and 50+ builtins.
|
||||
- OPA also supports _context-aware_ policies that let you analyze both the resource that a user is trying to create/update and all of the other Kubernetes resources that already exist. It's all just JSON/YAML to OPA. For example, the policy "prohibit ingresses with conflicting hostnames" requires comparing any new ingress that is being created to all the existing ingresses.
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
title: "The Rego Playground"
|
||||
authors: ["ashumania"]
|
||||
date: 2019-03-08
|
||||
slug: the-rego-playground-977566855cec
|
||||
---
|
||||
|
||||
OPA gives you a high-level declarative language for authoring policies as code, called Rego. Today we're excited to launch [The Rego Playground](https://play.openpolicyagent.org), a new interactive tool for writing, testing, and sharing Rego policies.
|
||||
|
||||
The initial version of The Rego Playground has support for:
|
||||
|
||||
- **Syntax highlighting** that makes policies easier to read and write.
|
||||
- **Interactive evaluation** of expressions, queries, rules, and packages.
|
||||
- **Input dialog** to specify complex JSON inputs for your policy.
|
||||
- **Sharing** so that you can link to policies online.
|
||||
|
||||
We hope the playground makes it easy for people to try out new policies and learn the language. We plan to keep improving the playground and eventually embed it into the OPA documentation. Check out the video below for a demo of the playground and its features.
|
||||
|
||||
[Watch: Rego Playground Demo](https://www.youtube.com/watch?v=nzcnf83NvxE)
|
||||
@@ -0,0 +1,286 @@
|
||||
---
|
||||
title: "Envoy External Authorization with OPA"
|
||||
authors: ["ashumania"]
|
||||
date: 2019-05-07
|
||||
slug: envoy-external-authorization-with-opa-578213ed567c
|
||||
---
|
||||
|
||||
|
||||
|
||||
Microservices improve productivity of individual development teams by breaking down applications into smaller, standalone parts. However, microservices alone do not solve age-old distributed systems problems like service discovery, authentication, and authorization. In fact, these problems are often more acute due to the heterogenous and ephemeral nature of microservice environments.
|
||||
|
||||
As more organizations adopt microservice architectures, the need for decoupled authentication and authorization has become apparent. This post dives into how you can leverage [Envoy](https://github.com/envoyproxy/envoy), [SPIFFE/SPIRE](https://github.com/spiffe), and the [Open Policy Agent (OPA)](https://github.com/open-policy-agent/opa) to enforce important security policies in microservice environments.
|
||||
|
||||
## Background
|
||||
|
||||
[Envoy](https://www.envoyproxy.io/docs/envoy/latest/intro/what_is_envoy) is a L7 proxy and communication bus designed for large modern service oriented architectures. Envoy (v1.7.0+) supports an [External Authorization filter](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ext_authz_filter.html) which calls an authorization service to check if the incoming request is authorized or not. This feature makes it possible to delegate authorization decisions to an external service and also makes the request context available to the service. The request context contains information such as the source of a network activity, destination of a network activity, the network request (eg. http request). All this information can be used by the external service to make an informed decision about the fate of the incoming request received by Envoy.
|
||||
|
||||
[SPIFFE](https://spiffe.io/spiffe/), the Secure Production Identity Framework for Everyone, is a set of open-source standards for securely identifying software systems in dynamic and heterogeneous environments. Systems that adopt SPIFFE can easily and reliably mutually authenticate wherever they are running. [SPIRE](https://spiffe.io/spire/) (the SPIFFE Runtime Environment) is a toolchain for establishing trust between workloads across a wide variety of platforms.
|
||||
|
||||
The [Open Policy Agent (OPA)](https://www.openpolicyagent.org/docs/v0.10.7/get-started/) is an open source, general-purpose policy engine that enables unified, context-aware policy enforcement across the entire stack. OPA's high-level declarative language [Rego](https://www.openpolicyagent.org/docs/v0.10.7/how-do-i-write-policies/) allows authoring of fine-grained security policies and is purpose built for reasoning about information represented in structured documents.
|
||||
|
||||
## OPA as an External Authorization Service
|
||||
|
||||
We will walkthrough an example of using Envoy's [External authorization filter](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/ext_authz_filter) with OPA as an authorization service.
|
||||
|
||||
|
||||
|
||||
The example consists of three services (web, backend and db) colocated with a running service Envoy. Each service uses the external authorization filter to call its respective OPA instance for checking if an incoming request is allowed or not.
|
||||
|
||||
The web service receives all inbound requests from api-server-1 and api-server-2 which are deployed in different subnets. The request is forwarded to the backend service which then calls the db service.
|
||||
|
||||
Secure communication between the web, backend and db service is established by configuring the Envoy proxies in each container to establish a mTLS connection with each other. Envoy retrieves client and server TLS certificates and trusted CA roots for mTLS communication from a SPIRE Agent which implements an [Envoy SDS](https://www.envoyproxy.io/docs/envoy/v1.10.0/configuration/secret#). The agent in-turn fetches this information from the SPIRE Server and makes it available to an identified workload. In the following example, SPIRE provides each workload an identity, in the form of a **SPIFFE ID** embedded in the TLS certificate, to facilitate mTLS communication. The SPIFFE ID of each workload can then be used by OPA to build the authorization policy. More information on [SPIRE](https://spiffe.io/spire/overview/) can be found **here**.
|
||||
|
||||
- Envoy is listening for ingress on port 8001 in each container.
|
||||
- api-server-1 and api-server-2 are flask apps running on port 5000 and 5001 respectively and forward requests to the web service.
|
||||
- api-server-1 has a static IP in the 172.28.0.0/16 subnet while api-server-2 has one in the 192.28.0.0/16 subnet.
|
||||
- OPA is extended with a GRPC server that implements the [Envoy External authorization API](https://www.envoyproxy.io/docs/envoy/v1.10.0/intro/arch_overview/ext_authz_filter.html).
|
||||
- data.envoy.authz.allow is the default OPA policy that decides whether a request is allowed or not.
|
||||
- Both the GRPC server port and default OPA policy that is queried are configurable.
|
||||
|
||||
## Running the Example
|
||||
|
||||
**Step 1: Install Docker**
|
||||
|
||||
Ensure that you have recent versions of docker and docker-compose installed.
|
||||
|
||||
**Step 2: Clone the repo and Start containers**
|
||||
|
||||
Clone the OPA-Envoy-SPIRE repo with `git clone git@github.com:ashutosh-narkar/opa-envoy-spire-ext-authz.git`
|
||||
|
||||
```bash
|
||||
cd opa-envoy-spire-ext-authz
|
||||
docker-compose up --build -d
|
||||
docker-compose ps
|
||||
```
|
||||
|
||||
The following containers should be running:
|
||||
|
||||
```
|
||||
Name Command State Ports
|
||||
----------------------------------------------------------------------------------------------------------------------
|
||||
opa-envoy-spiffe-ext-authz_api-server-1_1 flask run --host=0.0.0.0 Up 0.0.0.0:5000->5000/tcp
|
||||
opa-envoy-spiffe-ext-authz_api-server-2_1 flask run --host=0.0.0.0 Up 0.0.0.0:5001->5000/tcp, 5001/tcp
|
||||
opa-envoy-spiffe-ext-authz_backend_1 /bin/sh -c /usr/local/bin/ ... Up 10000/tcp
|
||||
opa-envoy-spiffe-ext-authz_db_1 /bin/sh -c /usr/local/bin/ ... Up 10000/tcp
|
||||
opa-envoy-spiffe-ext-authz_opa_be_1 ./opa_istio_linux_amd64 -- ... Up 0.0.0.0:9192->9192/tcp
|
||||
opa-envoy-spiffe-ext-authz_opa_db_1 ./opa_istio_linux_amd64 -- ... Up 0.0.0.0:9193->9193/tcp
|
||||
opa-envoy-spiffe-ext-authz_opa_web_1 ./opa_istio_linux_amd64 -- ... Up 0.0.0.0:9191->9191/tcp
|
||||
opa-envoy-spiffe-ext-authz_spire-server_1 /usr/bin/dumb-init /opt/sp ... Up
|
||||
opa-envoy-spiffe-ext-authz_web_1 /bin/sh -c /usr/local/bin/ ... Up 10000/tcp, 0.0.0.0:8001->8001/tcp
|
||||
```
|
||||
|
||||
**Step 3: Start SPIRE Infrastructure**
|
||||
|
||||
Start the SPIRE Agents and register the web, backend and db servers with the SPIRE Server. More information on the registration process can be found in the [SPIRE workload registration guide](https://spiffe.io/spire/overview/#workload-registration).
|
||||
|
||||
```bash
|
||||
./configure-spire.sh
|
||||
```
|
||||
|
||||
**Step 4: Exercise Ingress Policy**
|
||||
|
||||
The Ingress Policy states that the web service can ONLY be accessed from the subnet 172.28.0.0/16.
|
||||
|
||||
Check that api-server-1 can access the web service.
|
||||
|
||||
```bash
|
||||
$ curl -i localhost:5000/hello
|
||||
HTTP/1.0 200 OK
|
||||
Content-Type: text/html; charset=utf-8
|
||||
Content-Length: 29
|
||||
Server: Werkzeug/0.15.2 Python/2.7.15
|
||||
Date: Thu, 02 May 2019 21:21:48 GMT
|
||||
|
||||
Hello from the web service !
|
||||
```
|
||||
|
||||
Check that api-server-2 cannot access the web service.
|
||||
|
||||
```bash
|
||||
$ curl -i localhost:5001/hello
|
||||
HTTP/1.0 403 FORBIDDEN
|
||||
Content-Type: text/html; charset=utf-8
|
||||
Content-Length: 40
|
||||
Server: Werkzeug/0.15.2 Python/2.7.15
|
||||
Date: Thu, 02 May 2019 21:22:12 GMT
|
||||
|
||||
Access to the Web service is forbidden.
|
||||
```
|
||||
|
||||
**Step 5: Exercise Service-To-Service Policy**
|
||||
|
||||
The Service-To-Service Policy policy states that a request can flow from the web to backend to db service.
|
||||
|
||||
Check that this flow is honored.
|
||||
|
||||
```bash
|
||||
$ curl -i localhost:5000/the/good/path
|
||||
HTTP/1.0 200 OK
|
||||
Content-Type: text/html; charset=utf-8
|
||||
Content-Length: 35
|
||||
Server: Werkzeug/0.15.2 Python/2.7.15
|
||||
Date: Thu, 02 May 2019 21:22:50 GMT
|
||||
|
||||
Allowed path: WEB -> BACKEND -> DB
|
||||
```
|
||||
|
||||
Check that the web service is NOT allowed to directly call the db service.
|
||||
|
||||
```bash
|
||||
$ curl -i localhost:5000/the/bad/path
|
||||
HTTP/1.0 403 FORBIDDEN
|
||||
Content-Type: text/html; charset=utf-8
|
||||
Content-Length: 26
|
||||
Server: Werkzeug/0.15.2 Python/2.7.15
|
||||
Date: Thu, 02 May 2019 21:23:22 GMT
|
||||
|
||||
Forbidden path: WEB -> DB
|
||||
```
|
||||
|
||||
## Example OPA Policy
|
||||
|
||||
Each service calls its respective OPA instance for a decision and loads its desired policies into OPA. To see the OPA policies loaded by a service checkout the docker directory in the repo.
|
||||
|
||||
**Example Policy — 1**
|
||||
|
||||
The following OPA policy used in the example above is loaded into the OPA called by the web service.
|
||||
|
||||
> web service can ONLY be accessed from the subnet 172.28.0.0/16
|
||||
|
||||
```rego
|
||||
import input.attributes.request.http as http_request
|
||||
import input.attributes.source.address as source_address
|
||||
|
||||
default allow = false
|
||||
|
||||
allowed_paths = {"/hello", "/the/good/path", "/the/bad/path"}
|
||||
|
||||
# allow access to the Web service from the subnet 172.28.0.0/16 for the allowed paths
|
||||
allow {
|
||||
allowed_paths[http_request.path]
|
||||
http_request.method == "GET"
|
||||
net.cidr_contains("172.28.0.0/16", source_address.Address.SocketAddress.address)
|
||||
}
|
||||
```
|
||||
|
||||
Try the OPA-Envoy Ingress policy in the [Rego Playground](https://play.openpolicyagent.org/p/aHjJBMqpjC)!
|
||||
|
||||
**Example Policy — 2**
|
||||
|
||||
Another policy used in the example states that:
|
||||
|
||||
> a request can flow from the web to backend to db service
|
||||
|
||||
Below is a policy snippet that is loaded into the OPA called by the db service. This policy allows requests to the db service from ONLY the backend service.
|
||||
|
||||
```rego
|
||||
package envoy.authz
|
||||
|
||||
import input.attributes.request.http as http_request
|
||||
import input.attributes.source.address as source_address
|
||||
|
||||
default allow = false
|
||||
|
||||
# allow Backend service to access DB service
|
||||
allow {
|
||||
http_request.path == "/good/db"
|
||||
http_request.method == "GET"
|
||||
svc_spiffe_id == "spiffe://domain.test/backend-server"
|
||||
}
|
||||
|
||||
svc_spiffe_id = client_id {
|
||||
[_, _, uri_type_san] := split(http_request.headers["x-forwarded-client-cert"], ";")
|
||||
[_, client_id] := split(uri_type_san, "=")
|
||||
}
|
||||
```
|
||||
|
||||
Try the OPA-Envoy Service-Service policy in the [Rego Playground](https://play.openpolicyagent.org/p/6iSVIkK8zW)!
|
||||
|
||||
X-Forwarded-Client-Cert header is injected by the Envoy proxy of the originating service and validated by the Envoy proxy of the destination service. Envoy is configured to forward the URI field in the client certificate. To identify the service making the request, this policy uses the URI field of the X-Forwarded-Client-Cert header which in this case is the SPIFFE ID of the backend server.
|
||||
|
||||
> x-forwarded-client-cert (XFCC) is a proxy header which indicates certificate information of part or all of the clients or proxies that a request has flowed through, on its way from the client to the server. More information about the header and it's supported keys can be found in the [Envoy HTTP headers documentation](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-forwarded-client-cert).
|
||||
|
||||
## Example Envoy configuration
|
||||
|
||||
Here's an example configuration for an Envoy proxy that listens for HTTP client connections on port 80 and then calls OPA's gRPC server that implements the [Envoy External Authorization API](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/ext_authz_filter).
|
||||
|
||||
```yaml
|
||||
static_resources:
|
||||
listeners:
|
||||
- address:
|
||||
socket_address:
|
||||
address: 0.0.0.0
|
||||
port_value: 80
|
||||
use_original_dst: true
|
||||
filter_chains:
|
||||
- filters:
|
||||
- name: envoy.http_connection_manager
|
||||
typed_config:
|
||||
"@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
|
||||
codec_type: auto
|
||||
stat_prefix: ingress_http
|
||||
access_log:
|
||||
- name: envoy.file_access_log
|
||||
config:
|
||||
path: "/dev/stdout"
|
||||
route_config:
|
||||
name: local_route
|
||||
virtual_hosts:
|
||||
- name: backend
|
||||
domains:
|
||||
- "*"
|
||||
routes:
|
||||
- match:
|
||||
prefix: "/hello"
|
||||
route:
|
||||
cluster: web-service
|
||||
- match:
|
||||
prefix: "/the/good/path"
|
||||
route:
|
||||
cluster: web-service
|
||||
- match:
|
||||
prefix: "/the/bad/path"
|
||||
route:
|
||||
cluster: web-service
|
||||
http_filters:
|
||||
- name: envoy.ext_authz
|
||||
config:
|
||||
failure_mode_allow: false
|
||||
grpc_service:
|
||||
google_grpc:
|
||||
target_uri: opa:9191
|
||||
stat_prefix: ext_authz
|
||||
timeout: 0.5s
|
||||
- name: envoy.router
|
||||
config: {}
|
||||
clusters:
|
||||
- name: web-service
|
||||
connect_timeout: 0.25s
|
||||
type: strict_dns
|
||||
lb_policy: round_robin
|
||||
http2_protocol_options: {}
|
||||
load_assignment:
|
||||
cluster_name: web-service
|
||||
endpoints:
|
||||
- lb_endpoints:
|
||||
- endpoint:
|
||||
address:
|
||||
socket_address:
|
||||
address: web-service
|
||||
port_value: 80
|
||||
admin:
|
||||
access_log_path: "/dev/null"
|
||||
address:
|
||||
socket_address:
|
||||
address: 0.0.0.0
|
||||
port_value: 8001
|
||||
```
|
||||
|
||||
## And that's it
|
||||
|
||||
And that's how you use OPA as an External authorization service to enforce ingress and service-to-service security policies using Envoy's External authorization filter. OPA leverages the authentication framework provided by SPIFFE/SPIRE and by configuring Envoy to forward client certificate details, OPA is able to make authorization decisions based on the SPIFFE ID included in the URI SAN of the client X.509 certificate.
|
||||
|
||||
## Source Code
|
||||
|
||||
The code for the example can be found at [https://github.com/open-policy-agent/opa-envoy-spire-ext-authz](https://github.com/open-policy-agent/opa-envoy-spire-ext-authz).
|
||||
+152
@@ -0,0 +1,152 @@
|
||||
---
|
||||
title: "Policy-driven continuous integration with Open Policy Agent"
|
||||
authors: ["lperkins"]
|
||||
date: 2019-06-11
|
||||
slug: policy-driven-continuous-integration-with-open-policy-agent-b98a8748e536
|
||||
---
|
||||
|
||||
_Take sound security policy to the source_
|
||||
|
||||
One of the things that I love most about [Open Policy Agent](https://openpolicyagent.org) (**OPA**) is that it was built to be interoperable with other systems. Anything that produces JSON — and nowadays most things do — can provide OPA with inputs for rendering policy judgments. Due to this interoperability, you can use OPA with container-based development tools like [Docker](https://www.openpolicyagent.org/docs/latest/docker-authorization/), infrastructure provisioning tools like [Terraform](https://www.openpolicyagent.org/docs/latest/terraform/), container orchestration platforms like [Kubernetes](https://www.openpolicyagent.org/docs/latest/kubernetes-admission-control/), and that's just scratching the surface.
|
||||
|
||||
## OPA and continuous integration
|
||||
|
||||
Because OPA can integrate with just about anything, virtually every single part of a modern software "stack" can be policy driven, _including_ continuous integration. With OPA you can create policies that govern which artifacts are allowed to be built _in the first place_, providing a powerful lever for keeping potentially malicious jobs and services from _ever_ running on your systems (make sure those are also governed by OPA policies!).
|
||||
|
||||
Incidents like the recent `event-stream` debacle (see [details](https://blog.npmjs.org/post/180565383195/details-about-the-event-stream-incident)) — amongst many others — demonstrate just how necessary these safeguards are.
|
||||
|
||||
In fact, you're probably _already_ applying policies at the CI level but doing so in an ad hoc way via a loose assemblage of scripts. OPA provides an excellent platform for making those implicit policies explicit and declarative. I'll provide a straightforward example for what policy-driven CI may look like in the next section.
|
||||
|
||||
> "You're probably already _applying policies at the CI level but doing so in an ad hoc way. OPA enables you to make your implicit policies explicit and declarative._"
|
||||
|
||||
## Dependency blacklisting in action
|
||||
|
||||
As an example, let's say that I'm a developer working on a Node.js web server in a large organization. That organization enforces CI policies using a policy written in [Rego](https://www.openpolicyagent.org/docs/latest/how-do-i-write-policies), OPA's policy language. The CI provider is [GitHub Actions](https://developer.github.com/actions/), though the example could easily be ported to other CI providers. The code for this example is in the [lucperkins/opa-ci-example](https://github.com/lucperkins/opa-ci-example/) repository on GitHub.
|
||||
|
||||
_The Rego policy governing package.json `dependencies`_
|
||||
|
||||
```rego
|
||||
package ci
|
||||
|
||||
# The package.json is presumed faulty
|
||||
default allow = false
|
||||
|
||||
# Packages that aren't allowed
|
||||
blacklist = {
|
||||
"event-stream",
|
||||
"left-pad"
|
||||
}
|
||||
|
||||
# Records dependencies that are on the blacklist
|
||||
violations[pkg] {
|
||||
input.dependencies[pkg]
|
||||
blacklist[pkg]
|
||||
}
|
||||
|
||||
# Returns true only if there are no violations
|
||||
allow {
|
||||
count(violations) == 0
|
||||
}
|
||||
```
|
||||
|
||||
This policy takes each project's `package.json` file as an input and applies the policy to that (notice the `input.dependencies`). Two things to note about the policy:
|
||||
|
||||
- `default allow = false` means that my `package.json` dependencies are _presumed_ faulty and must pass muster before the next CI stage (the installation stage) is reached. This is generally a good practice for Rego policies.
|
||||
- The `violations[pkg]` block creates a list of blacklist-violating packages that is returned in the evaluation output in case of violations, making it easier for developers to know _why_ the evaluation is failing.
|
||||
- The evaluation succeeds (i.e. the script returns an exit code of 0) only if there are no violations (`count(violations) == 0`); otherwise, it fails.
|
||||
|
||||
So that covers our dependencies policy. Now let's dive into the GitHub Action workflow definition.
|
||||
|
||||
_The GitHub Actions workflow for this application_
|
||||
|
||||
```hcl
|
||||
workflow "OPA evaluation" {
|
||||
on = "push"
|
||||
resolves = ["install"]
|
||||
}
|
||||
|
||||
# Determines whether the policy has been violated
|
||||
action "evaluate" {
|
||||
uses = "docker://openpolicyagent/opa:0.11.0"
|
||||
args = [
|
||||
"eval",
|
||||
"--fail-defined", "data.ci.violations[pkg]",
|
||||
"--input", "package.json",
|
||||
"--data", "ci.rego",
|
||||
"--format", "pretty"
|
||||
]
|
||||
}
|
||||
|
||||
# Installs the dependencies in package.json
|
||||
# iff the evaluate action succeeds
|
||||
action "install" {
|
||||
uses = "nuxt/actions-yarn@master"
|
||||
args = "install"
|
||||
needs = "evaluate"
|
||||
}
|
||||
```
|
||||
|
||||
There are two Actions in this workflow: `evaluate` and `install` (in a more fleshed-out scenario there may be other stages, like `build-container` or `deploy-to-k8s`). The `evaluate` action runs the following script, using the `openpolicyagent/opa:0.11.0` Docker image:
|
||||
|
||||
```bash
|
||||
opa eval \
|
||||
--fail-defined 'data.ci.violations[pkg]' \
|
||||
--input package.json \
|
||||
--data ci.rego \
|
||||
--format pretty
|
||||
```
|
||||
|
||||
The OPA evaluation fails any time the `violations[pkg]` directive is satisfied, i.e. any time a dependency is both in the `dependencies` block in `package.json` and on the package blacklist. The `--format pretty` flag dictates that the failure output includes a visually appealing table like this:
|
||||
|
||||
```
|
||||
+----------------+-------------------------+
|
||||
| pkg | data.ci.violations[pkg] |
|
||||
+----------------+-------------------------+
|
||||
| "dependency-1" | "dependency-1" |
|
||||
| "dependency-2" | "dependency-2" |
|
||||
+----------------+-------------------------+
|
||||
```
|
||||
|
||||
If I'm a developer working on this project, I get highly readable, actionable feedback about policy violations directly in the CI output. So how is my build currently faring?
|
||||
|
||||
## Results
|
||||
|
||||
Well… not so good. On my current [`dev`](https://github.com/lucperkins/opa-ci-example/tree/dev) branch, my `package.json` file looks like this.
|
||||
|
||||
_The doomed-to-fail package.json_
|
||||
|
||||
```json
|
||||
{
|
||||
"private": true,
|
||||
"dependencies": {
|
||||
"event-stream": "^4.0.1",
|
||||
"express": "^4.17.1",
|
||||
"left-pad": "^1.3.0"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
I've included two dependencies here — `event-stream` and `left-pad` — that very obviously violate the blacklist. You can see the [resulting CI failure run](https://github.com/lucperkins/opa-ci-example/runs/142847261). Let's fix this!
|
||||
|
||||
[This pull request](https://github.com/lucperkins/opa-ci-example/pull/2) gets the job done. It removes the offending dependencies from the `package.json`. As you can see from the [results](https://github.com/lucperkins/opa-ci-example/runs/142848310) of the `evaluate` action, the `opa eval …` command returns `undefined` instead of a table listing violations. And because the `evaluate` action has passed, the [`install` action](https://github.com/lucperkins/opa-ci-example/runs/142848384) has been successfully invoked.
|
||||
|
||||
> You can see the failing policy and input in action in the [Open Policy Agent Playground](https://play.openpolicyagent.org/p/kipUorP7ui). Correct the inputs on your own to fix the build!
|
||||
|
||||
## Related efforts: testing Kubernetes configs
|
||||
|
||||
For another fully fleshed-out example of using OPA as part of a build pipeline, I highly recommend [Unit Testing Your Kubernetes Configurations Using Open Policy Agent](https://www.youtube.com/watch?v=AfTuzonH93U&app=desktop) from [Gareth Rushgrove](https://twitter.com/garethr/) (slides on [Speaker Deck](https://speakerdeck.com/garethr/unit-testing-your-kubernetes-configuration-with-open-policy-agent)), presented at KubeCon/CloudNativeCon EU 2019 in Barcelona.
|
||||
|
||||
Though Gareth's project has different aims from mine, it provides a very nice illustration of using OPA to prevent certain classes of problems from ever arising in production environments by vetting Kubernetes configurations.
|
||||
|
||||
If you have other examples — blog posts, code snippets, anything — please feel free to add a comment to share your work!
|
||||
|
||||
## Implications
|
||||
|
||||
What I've presented here is just a small taste of what's possible. You could use Open Policy Agent to build a much more robust system of CI checks. To give a few examples, you could write policies for:
|
||||
|
||||
- **Linters and formatters**, specifying allowable thresholds for deviance from desired norms
|
||||
- **Code coverage checkers**, with requirements specified for each language and domain within your organization
|
||||
- **Configuration files** for systems like [Kubernetes](https://kubernetes.io), [Prometheus](https://prometheus.io), [Envoy](https://envoyproxy.io), and many others
|
||||
- **Utilize existing integrations** with other tools, such as Terraform and [Docker](https://www.openpolicyagent.org/docs/latest/docker-authorization/), [Terraform](https://www.openpolicyagent.org/docs/latest/terraform/), [Puppet](https://github.com/open-policy-agent/contrib/tree/master/puppet_example), and other CI-related tools.
|
||||
|
||||
Making your production systems policy driven is of the utmost importance, and that has to include sanitizing the inputs to those systems whenever possible. OPA quite simply provides the most robust and flexible platform in the open source world for doing so.
|
||||
@@ -0,0 +1,47 @@
|
||||
---
|
||||
title: "v0.14 Release"
|
||||
authors: ["tsandall"]
|
||||
date: 2019-09-13
|
||||
slug: v0-14-release-55d4ffd70876
|
||||
---
|
||||
|
||||
We're excited to announce the v0.14.0 release of OPA. This release includes 147 commits from 17 authors across 9 organizations! For a detailed list of changes see the [GitHub releases](https://github.com/open-policy-agent/opa/releases) page. In this latest release we focused on the getting started experience for new users.
|
||||
|
||||
## Community Updates
|
||||
|
||||
- The [KubeCon San Diego 2019](https://events.linuxfoundation.org/events/kubecon-cloudnativecon-north-america-2019/#) schedule was posted last week. There will be a number of excellent talks featuring OPA throughout the week.
|
||||
- The [CFP](https://www.papercall.io/opa-summit) for the OPA Summit @ KubeCon San Diego 2019 was posted last month and we received a number of excellent submissions. The event will held on Monday, November 18th from 12–5PM with [games, drinks, and socializing to follow at Nason's Beer Hall](https://www.eventbrite.com/e/pre-kubecon-happy-hour-with-unusual-ventures-and-portfolio-companies-tickets-70986561717). The CFP is open until September 15th at 14:00 UTC.
|
||||
- GSoC 2019 finished a couple weeks ago. GSoC student Urvil Patel completed a project that integrates OPA with IPTables. Check it out in the [open-policy-agent/contrib](https://github.com/open-policy-agent/contrib/tree/master/opa-iptables) repository.
|
||||
- The [OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper) project is developing steadily. [This blog post in August on kubernetes.io](https://kubernetes.io/blog/2019/08/06/opa-gatekeeper-policy-and-governance-for-kubernetes/) talks about the motivations and evolution of the project. The [3.0.4-beta.1 release](https://github.com/open-policy-agent/gatekeeper/releases/tag/v3.0.4-beta.1) includes a dry-run feature, PSP constraints, and more.
|
||||
- The [conftest](https://github.com/instrumenta/conftest) project (which builds on OPA) continues to gain traction. The project recently added support for Dockerfile validation as well as integrations for [Tekton Pipelines](https://github.com/tektoncd/catalog/pull/60), [Helm](https://github.com/instrumenta/helm-conftest), and [GitHub Actions](https://github.com/instrumenta/conftest-action).
|
||||
|
||||
## Docs, Docs, Docs
|
||||
|
||||
Earlier this year we launched [The Rego Playground](/blog/the-rego-playground-977566855cec). The playground provides a way to evaluate and test policies from the browser. Based on positive feedback on the playground we decided to take it further. In the latest version of the docs, policy examples are interactive!
|
||||
|
||||
|
||||
|
||||
In addition to "live docs", we have also re-organized and improved the core content. The docs had grown organically since the project launched and it was time for a rethink. The new structure and content aims to get users started with Rego as quickly as possible. We have also begun carving out dedicated sections for popular integrations like Kubernetes admission control. Finally, we added a search integration powered by Algolia.
|
||||
|
||||
## Performance Optimizations
|
||||
|
||||
This release includes a number of optimizations to the AST and other packages. The optimizations focused on heap allocations during evaluation. With the new optimizations we see about ~25% faster evaluation across-the-board for end-to-end benchmarks in OPA:
|
||||
|
||||
```
|
||||
Test Case Old New Delta
|
||||
-----------------------------------------------------------
|
||||
AuthzForbidAuthn-8 32.3µs±1% 30.5µs±1% −5.53%
|
||||
AuthzForbidPath-8 109µs±1% 85µs±1% −22.14%
|
||||
AuthzForbidMethod-8 115µs±2% 89µs±1% −22.44%
|
||||
AuthzAllow10Paths-8 112µs±2% 87µs±0% −22.95%
|
||||
AuthzAllow100Paths-8 725µs±3% 529µs±3% −27.09%
|
||||
AuthzAllow1000Paths-8 6.27ms±1% 4.55ms±1% −27.40%
|
||||
```
|
||||
|
||||
OPA benchmarks are run automatically on a regular basis and compared against the last stable release using the [benchstat](https://godoc.org/golang.org/x/perf/cmd/benchstat) tool. The results are posted on the [benchmark results page](https://opa-benchmark-results.s3.amazonaws.com/index.html).
|
||||
|
||||
## Improved File Loading in VS Code
|
||||
|
||||
The VS Code extension has been updated to use the new `-b` or `--bundle` flag on `opa eval` to avoid loading _all_ JSON and YAML files inside the workspace. While the old file loading approach was acceptable for Rego-specific workspaces it fell over in larger or mixed workspaces (which most people have!)
|
||||
|
||||
With the new `-b` flag and the latest version of the [Open Policy Agent extension for VS Code](https://marketplace.visualstudio.com/items?itemName=tsandall.opa), file loading is much better.
|
||||
@@ -0,0 +1,81 @@
|
||||
---
|
||||
title: "OPA v0.15.1: Rego on WebAssembly"
|
||||
authors: ["tsandall"]
|
||||
date: 2019-11-19
|
||||
slug: opa-v0-15-1-rego-on-webassembly-81c226c51be4
|
||||
---
|
||||
|
||||
|
||||
|
||||
We're excited to announce that with OPA v0.15.1 you can compile any Rego policy into WebAssembly. This release is the culmination of many incremental improvements to the WebAssembly compiler in OPA that we demonstrated with a proof-of-concept at KubeCon 2018 in Seattle.
|
||||
|
||||
## What is WebAssembly?
|
||||
|
||||
From [WebAssembly.org](https://webassembly.org):
|
||||
|
||||
> "WebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable target for compilation of high-level languages like C/C++/Rust, enabling deployment on the web for client and server applications."
|
||||
|
||||
## What does Wasm have to do with policy enforcement?
|
||||
|
||||
One of the principles behind OPA is that policy decision-making should be decoupled from policy enforcement. For people building software products, decoupling lets them avoid undifferentiated heavy lifting and get to market faster because they do not have to worry about implementing a policy or authorization engine from scratch. For operators running large-scale software systems, decoupling addresses fragmentation from siloed policy implementations enabling unified control and visibility across the stack.
|
||||
|
||||
Cloud native projects have embraced the idea as they matured. Today, projects like Kubernetes and Envoy have strong support for externalized admission control and authorization checks. However, it's not enough to just offer a plugin model for policy decisions. You need a component that can respond to policy queries with answers. This is where OPA comes in. OPA provides a lightweight general-purpose policy engine that can be embedded throughout your infrastructure. When your software needs to make decisions it can query OPA and OPA will reply with the answer based on the rules and data that you distribute to it.
|
||||
|
||||
While OPA is designed to be as lightweight as possible (e.g., by keeping all rules and data in-memory, not introducing any external runtime dependencies, etc.) the fact that the policy evaluation engine is written in Go means that library embedding can be difficult for software not written in Go. Moreover, requiring OPA be installed as a daemon inside (or near) the enforcement point comes with it's own set of challenges and may not always be feasible (e.g., if you are not the owner of the platform.)
|
||||
|
||||
This is where Wasm is relevant because it provides a safe, portable, efficient standards-based execution environment for arbitrary code and logic. In the past year Wasm-based extension mechanisms gained popularity: CDN companies like [Cloudflare](https://blog.cloudflare.com/webassembly-on-cloudflare-workers/) and [Fastly](https://www.fastly.com/press/press-releases/fastly-expands-serverless-capabilities-launch-compute-edge) allow you to execute software at the edge using Wasm, service proxies like [Envoy have integrated Wasm runtimes](https://github.com/envoyproxy/envoy/issues/4272), and even databases like [Postgres can be extended with Wasm built-in functions](https://github.com/wasmerio/postgres-ext-wasm).
|
||||
|
||||
|
||||
|
||||
With Wasm-based extension mechanisms becoming the norm and programming language support for Wasm runtimes maturing, Wasm will naturally become a standard mechanism for offloading policy evaluation from your software. At the same time, Wasm is only a low-level instruction format. The only data types at your disposal are 32/64-bit integer and floating-point numbers. This makes it impractical to use Wasm directly for any kind of policy specification. Enter OPA.
|
||||
|
||||
## How does OPA work with Wasm?
|
||||
|
||||
OPA includes a compiler that accepts Rego policies as input and generates an executable Wasm program as output. This Wasm program can be loaded into any standard Wasm runtime and executed when policy decisions are needed. With the v0.15.1 release we have reached an important milestone in the compiler: with the exception of built-in functions, OPA can now compile any Rego policy into Wasm.
|
||||
|
||||
In the latest OPA release there are two ways of compiling Rego policies into Wasm:
|
||||
|
||||
- On the command-line using the `opa build` CLI tool.
|
||||
- In Go by integrating with the github.com/open-policy-agent/opa/rego package.
|
||||
|
||||
|
||||
|
||||
The compilation process takes a Rego query and zero or more Rego files and generates an execution plan that is compiled into a Wasm module binary. The module binary can then be loaded into any Wasm runtime and executed with different input and data values to obtain decisions. The resulting Wasm module binary sizes are not too significant. The baseline size for a policy with a single statement is ~30KB on disk. A policy containing 300,000 statements consumes ~20MB on disk.
|
||||
|
||||
The initial benchmarks comparing policy evaluation time in Wasm versus the existing Go interpreter implemented in OPA are promising. For example, a relatively simple policy that searches over a large number of data items to decide whether an operation should be allowed (or not) evaluates ~20x faster with the Wasm compiled version. This is to be expected because the overhead of the interpreter in OPA has been completely removed.
|
||||
|
||||
```rego
|
||||
package example
|
||||
|
||||
default allow = false
|
||||
|
||||
allow {
|
||||
rule := data.rules[_]
|
||||
rule.action == input.action
|
||||
rule.resource == input.resource
|
||||
rule.identity == input.identity
|
||||
}
|
||||
```
|
||||
|
||||
Benchmark results:
|
||||
|
||||
```
|
||||
# of data.rules | Existing Go interpreter | Wasm compiled policy
|
||||
----------------+-------------------------+---------------------
|
||||
100 | 0.354ms | 0.0173ms
|
||||
1,000 | 3.1ms | 0.145ms
|
||||
10,000 | 30ms | 1.5ms
|
||||
100,000 | 316ms | 15ms
|
||||
```
|
||||
|
||||
If you are interested in more details about Wasm support in OPA see [this page in the documentation](https://www.openpolicyagent.org/docs/latest/wasm/) and check out [this example on GitHub](https://github.com/open-policy-agent/npm-opa-wasm/).
|
||||
|
||||
## What's Next?
|
||||
|
||||
We have reached an important milestone for OPA with Wasm and we are excited about Wasm's potential as it relates to policy enforcement. However, there is still a lot left to do.
|
||||
|
||||
The main gap (at the moment) is lack of support for the 50+ built-in functions from OPA proper. While you can implement these built-in functions yourself in the host language (e.g., NodeJS) and import them into the Wasm module, we want to make this smoother. Another area that needs work are the management APIs. The OPA daemon and Go library expose APIs that let you control policy distribution, decision logging, and more. In a Wasm-enabled enforcement point these functions do not exist yet.
|
||||
|
||||
In the coming months we are going to build out new integrations that leverage Wasm, create SDKs for languages/runtimes other than Go and NodeJS, and use this experience to harden and optimize the new implementation.
|
||||
|
||||
If you would like to get involved feel free to file issues on [GitHub](https://github.com/open-policy-agent/opa) or [join us on Slack](https://slack.openpolicyagent.org/).
|
||||
@@ -0,0 +1,66 @@
|
||||
---
|
||||
title: "KubeCon US 2019 Recap"
|
||||
authors: ["tsandall"]
|
||||
date: 2019-12-11
|
||||
slug: kubecon-us-2019-recap-3e60c70d633a
|
||||
---
|
||||
|
||||
|
||||
|
||||
A few weeks ago San Diego hosted the largest KubeCon ever with nearly 12,000 attendees. Let's take a look at some OPA highlights!
|
||||
|
||||
## OPA Summit 2019
|
||||
|
||||
|
||||
|
||||
Day-0 (the day before the Kubecon main event) marked a big milestone for the project: we held our first-ever OPA summit!
|
||||
|
||||
_[Tim Hinrichs](https://medium.com/u/d64d0bb99d2f?source=post_page---user_mention--3e60c70d633a---------------------------------------) shows off the new website to 100+ attendees at the event._
|
||||
|
||||
The goal of this summit was to showcase **a variety of use cases** from companies **running OPA in production.** We received around 20 submissions to the CFP and then selected a handful of talks that demonstrated impressive scale or new applications for OPA in production.
|
||||
|
||||
[During the talk about how Pinterest uses OPA](https://www.youtube.com/watch?v=LhgxFICWsA8), Jeremy Krach and William Fu shared how they architected policy distribution and management for authorization on bare EC2 instances and multi-tenant Kubernetes clusters at Pinterest. Their talk walks through the entire policy lifecycle from authoring to distribution to enforcement. After covering the pipeline, they explained exactly how they offload policy decision-making from services. In terms of volume, their Kafka integration sees the highest traffic. At peak, OPA serves ~450K decisions/second across their clusters (and with caching that increases to ~8.5M decisions/second globally).
|
||||
|
||||
|
||||
|
||||
_William Fu and Jeremy Krach from Pinterest share details on authorization scale at Pinterest._
|
||||
|
||||
[Michael Sorens from Chef](https://www.youtube.com/watch?v=jrrW855xL3s) spoke about how they use OPA to implement IAM in Chef Automate. Michael highlighted OPA's TDD-based approach to policy authoring and explained how they use OPA to implement pre-flight authorization checks that control which UI components are rendered based on user permissions. This use case is becoming more common as OPA is increasingly used higher up in the stack.
|
||||
|
||||
|
||||
|
||||
_Michael speaks about applying TDD to policy._
|
||||
|
||||
You can find other great OPA Summit talks from [Atlassian](https://www.youtube.com/watch?v=nvRTO8xjmrg), [Trip Advisor](https://www.youtube.com/watch?v=X09c1eXvCFM), and [Capital One](https://www.youtube.com/watch?v=vkvWZuqSk5M) on YouTube. We are looking forward to hosting the next OPA Summit in Boston at KubeCon US 2020!
|
||||
|
||||
## Kubernetes: the guardrail enforcement point
|
||||
|
||||
At KubeCon after the day-0 summit, engineers from companies like Yelp, Goldman Sachs, Reddit, Adobe, Google, and Microsoft spoke about how they use OPA.
|
||||
|
||||
There were several excellent sessions about OPA Gatekeeper and admission control use cases in Kubernetes. OPA is used extensively to enforce guardrails over compute, network, and storage resources in Kubernetes and this was reflected at KubeCon.
|
||||
|
||||
Two talks that highlight how **Kubernetes is becoming the defacto standard for managing desired state** were from [Rita Zhang (@ritazzhang) and Ivan Sim (@ihcsim) from Microsoft and Buoyant (respectively)](https://www.youtube.com/watch?v=gMaGVHnvNfs) and [Sandeep Parikh (@crcsmnky) from Google](https://www.youtube.com/watch?v=90RHTBinAFU). This is great for platform administrators because it means they can leverage OPA Gatekeeper to enforce guardrails across not just native Kubernetes resources (e.g., Pods, Services, etc.) but also service mesh resources (e.g., [Linkerd](https://linkerd.io/), [Istio](https://istio.io/)), CI/CD resources (e.g., [tekton.dev](https://tekton.dev/)), cloud resources (e.g., [crossplane.io](https://crossplane.io)), and more.
|
||||
|
||||
Miguel Uzcategui (Goldman Sachs) and OPA co-founder [Tim Hinrichs (CTO of Styra)](https://twitter.com/tlhinrichs) spoke about how [Goldman Sachs uses OPA to do policy-based provisioning in Kubernetes](https://www.youtube.com/watch?v=lYHr_UaHsYQ). They explain how Goldman Sachs implemented Kubernetes controllers that offload decision-making to OPA so that when namespaces are created, resources like quotas, roles, and persistent volumes (and claims) are automatically instantiated (and then re-converged if something changes.) This is important for maintaining strict requirements around security and availability at Goldman Sachs. They also show why OPA is a good fit for this problem (e.g., easier testing, ability to use external context in decision-making, etc.) and how it has performed in production for nearly a year.
|
||||
|
||||
|
||||
|
||||
_Miguel shares results from running the OPA in production for over a year._
|
||||
|
||||
## Outside Kubernetes: App configuration and Microservices
|
||||
|
||||
[@garethr's](https://twitter.com/garethr) talk about [applying OPA policies earlier in application lifecycles](https://www.youtube.com/watch?v=cXfsaE6RKfc) highlighted OPA's general-purpose nature. His talk was filled with examples and demos that show how to use OPA and [conftest](https://github.com/instrumenta/conftest) to validate configuration files (e.g., Pipfiles, Dockerfiles, etc.) and plug into CI/CD systems.
|
||||
|
||||
Finally, on Thursday, [Daniel Popescu](https://twitter.com/danielpopes) and [Ben Plotnick](https://twitter.com/benplotnick) talked about how [Yelp evolved their security infrastructure (using OPA and Envoy)](https://www.youtube.com/watch?v=Z6aN3Smt-9M) as the company transitioned away from a monolith. Their talk highlights how perimeter-based security does not scale to microservice architectures and how development of custom policy languages is challenging. The talk provides a deep dive on how they leverage Envoy and OPA to implement mTLS and access control across a fleet of microservices. They also discuss the gradual migration off their custom policy language by transpiling to OPA.
|
||||
|
||||
|
||||
|
||||
This post only highlights a few of the excellent talks from KubeCon about OPA so if you want to watch more check out [this playlist on YouTube](https://www.youtube.com/playlist?list=PLJHPZt__YGKd4O_X7aEbaABLLz1g6aC56).
|
||||
|
||||
## Conclusions
|
||||
|
||||
KubeCon San Diego demonstrated how many companies run OPA in production for a variety of use cases. After starting the project nearly four years ago it is very exciting to see it fulfilling the original goal of modernizing and enabling policy enforcement across the stack.
|
||||
|
||||
Last year we saw rapid growth in end-user adoption, the launch of OPA Gatekeeper, and promotion to the CNCF Incubating tier. In 2020 we plan to continue investing in performance and usability for the core of the project, new integrations leveraging the recent [WebAssembly compiler feature](/blog/opa-v0-15-1-rego-on-webassembly-81c226c51be4), and better documentation of reference architectures.
|
||||
|
||||
See you all in Amsterdam and Boston!
|
||||
+199
@@ -0,0 +1,199 @@
|
||||
---
|
||||
title: "Rego design principle #1: Syntax should reflect real-world policies"
|
||||
authors: ["timhinrichs"]
|
||||
date: 2020-03-04
|
||||
slug: rego-design-principle-1-syntax-should-reflect-real-world-policies-e1a801ab8bfb
|
||||
---
|
||||
|
||||
|
||||
|
||||
Sometimes people ask why Rego, OPA's policy language, looks or behaves the way it does. Part of the answer is that Rego came about after having built two other general-purpose policy languages, and lessons learned from that process shaped this one. This multi-part post lays out the results of that journey — the key design principles for Rego, why they're important, and how they influenced the language.
|
||||
|
||||
Related posts in the series:
|
||||
|
||||
- [Design principle #2: Embrace Hierarchical Data](/blog/rego-design-principle-2-embrace-hierarchical-data-8a4913bdfea1)
|
||||
- [Design principle #3: Optimize Performance Automatically](/blog/rego-design-principle-3-optimize-performance-automatically-2d29ad3ce96d)
|
||||
|
||||
The first design principle holds that Rego's syntax "should NOT be designed as a general-purpose programming language that reads from disk, writes to network sockets, supports multi-threading, defines custom datastructures, etc."
|
||||
|
||||
## Refresher on OPA
|
||||
|
||||
OPA is a general purpose policy engine that separates policy decisions from the software services that enforce them. It bases decisions on input data, a Rego policy, and optionally external data reflecting real-world state (e.g., on-call schedules or resource ownership).
|
||||
|
||||
These functional requirements ensure that OPA has enough flexibility and generality to make context-aware decisions across a broad range of use-cases: admission control, API authorization, risk-analysis, data-filtering. OPA runs as a lightweight agent or library on the same server as the software service, thereby achieving both the availability and performance needed for policy decision-making in modern, cloud-native computing environments. It has been integrated with over 20 popular software systems and is at the time of writing an incubating project within the Cloud Native Computing Foundation.
|
||||
|
||||
## Natural encoding of real-world policies
|
||||
|
||||
The goal is that Rego should map closely onto plain-language rules and regulations. A useful litmus test: reading Rego aloud should sound close to the source documentation.
|
||||
|
||||
This readability test is especially important because of the broad range of stakeholders who are responsible for policy: developers, operations, security, and compliance. The less of a translation there is from the PDFs and wikis the easier it is to believe that your Rego policies are correct, that operationally you're on solid ground, that your auditors will be convinced the policies do what they should, and that your security vulnerabilities are properly mitigated.
|
||||
|
||||
> A Rego policy is a collection of if statements
|
||||
|
||||
### If statements in Rego
|
||||
|
||||
Nearly every Rego statement functions as an if-statement, but with different proportions than typical programming languages:
|
||||
|
||||
"Programming languages typically have small `if` conditions and relatively large `then` blocks... in Rego the `if` condition is a potentially large block of expressions, and the `then` part is a single expression."
|
||||
|
||||
```rego
|
||||
# Rego example.
|
||||
# An API call is allowed if the method is a GET
|
||||
allow {
|
||||
input.method == "GET"
|
||||
}
|
||||
```
|
||||
|
||||
Multiple statements inside a rule are ANDed; ORs are expressed via multiple rules:
|
||||
|
||||
```rego
|
||||
# Rego example.
|
||||
# An API call is allowed if the method is a GET
|
||||
allow {
|
||||
input.method == "GET"
|
||||
}
|
||||
|
||||
# An API call is allowed if the method is POST and
|
||||
# the user is an admin
|
||||
allow {
|
||||
# Only admins can create new objects
|
||||
input.method == "POST"
|
||||
input.user_is_admin == true
|
||||
}
|
||||
```
|
||||
|
||||
An equivalent JavaScript version, given for contrast:
|
||||
|
||||
```javascript
|
||||
// Not a Rego example. A JavaScript example.
|
||||
function allow() {
|
||||
return allow1() || allow2();
|
||||
}
|
||||
|
||||
function allow1() {
|
||||
return input.method == "GET";
|
||||
}
|
||||
|
||||
function allow2() {
|
||||
return input.method == "POST"
|
||||
&& input.is_admin == true;
|
||||
}
|
||||
```
|
||||
|
||||
Rego "has no need for explicit ANDs and ORs" within a rule, and includes "an explicit NOT operator." A self-documenting example:
|
||||
|
||||
```rego
|
||||
# Rego example.
|
||||
allow {
|
||||
operation_is_create
|
||||
user_is_admin
|
||||
}
|
||||
```
|
||||
|
||||
## Rich policy decisions
|
||||
|
||||
Real-world policies sometimes make a decision as simple as `allow` or `deny` but often they go far beyond that. What about `warn` or `error`? Or what if the decision is a rate-limit (number), a permitted hostname (a string), or the clusters to deploy an application (an array). Since OPA is a general-purpose policy engine (not an authorization engine) it needs to handle a rich collection of policy decisions.
|
||||
|
||||
Because inputs are typically JSON, decisions in Rego can likewise be any JSON type—not just booleans.
|
||||
|
||||
> A Rego decision is a JSON document
|
||||
|
||||
### Rich Policy Decisions in Rego
|
||||
|
||||
`allow` and `deny` are plain variables, not keywords, defaulting to `true`:
|
||||
|
||||
```rego
|
||||
# Rego example.
|
||||
allow = true {
|
||||
operation_is_create
|
||||
user_is_admin
|
||||
}
|
||||
```
|
||||
|
||||
Non-boolean decisions work the same way:
|
||||
|
||||
```rego
|
||||
risk = 100 {
|
||||
input.method == "DELETE"
|
||||
}
|
||||
```
|
||||
|
||||
Partial sets can build up collections, such as error messages:
|
||||
|
||||
```rego
|
||||
deny[msg] {
|
||||
input.method == "DELETE"
|
||||
not user_is_resource_owner
|
||||
msg := "only the owner of a resource may delete it"
|
||||
}
|
||||
```
|
||||
|
||||
## Collaboration
|
||||
|
||||
Real world policies are decided upon by multiple individuals and even teams. The security team might put global requirements in place across all application development teams and each application development team might put policy in place for their app. The k8s cluster administrator puts global policies in place but also delegates policy responsibilities to namespace-level admins. Policy is by its nature a collaborative endeavor, and Rego should recognize and support that.
|
||||
|
||||
At some level collaboration is supported simply because Rego is a text-based policy language (aka "policy-as-code"). Teams can check Rego policies into source control, and use peer-review to manage changes to it. We knew, however, that there are all too many examples where teams want to work more independently than that, e.g. putting global policies in place for an entire cluster and empowering team leads to manage policy for their portion of the cluster. That means that different teams should be able to write their policies independently from each other, and then combine those policies after the fact.
|
||||
|
||||
If different teams write different policies independently, they will inevitably end up with conflicts from time to time (e.g. one allows the decision and the other denys it). And so there must be a way to resolve those conflicts, based on a variety of different factors. Resolving conflicts is not always easy. It may depend on the teams involved, the kind of decision being made, the resource and its attributes, the time of day, and many other factors. Ergo the conflict resolution mechanism must be tantamount to a policy itself. Sometimes languages are designed to avoid the problem of conflict resolution by designing the language to not express conflicts at all. But this approach inevitably leads to problems because when two teams disagree in the real world and there is no way for them to express that disagreement in the policy language, they simply can't write the policy that they truly mean — the language is ambiguous in terms of the author's intent.
|
||||
|
||||
> Rego policies are composable; conflict resolution is a policy itself.
|
||||
|
||||
### Collaboration in Rego
|
||||
|
||||
Each policy lives in a package:
|
||||
|
||||
```rego
|
||||
package microservice.authorization
|
||||
```
|
||||
|
||||
Separate teams can define their own packages:
|
||||
|
||||
```rego
|
||||
package developer
|
||||
allow { … }
|
||||
…
|
||||
```
|
||||
|
||||
```rego
|
||||
package security
|
||||
allow { … }
|
||||
…
|
||||
```
|
||||
|
||||
A combining policy can reference other packages via the `data` keyword:
|
||||
|
||||
```rego
|
||||
package main
|
||||
allow {
|
||||
data.developer.allow
|
||||
data.security.allow
|
||||
}
|
||||
```
|
||||
|
||||
A more nuanced conflict-resolution example, where security's decision takes precedence when it has an opinion:
|
||||
|
||||
```rego
|
||||
package main
|
||||
# allow if the security team allows (and does not deny)
|
||||
allow {
|
||||
data.security.allow
|
||||
not data.security.deny
|
||||
}
|
||||
# allow if the security team has no opinion and
|
||||
# the developer team allows (and does not deny)
|
||||
allow {
|
||||
not data.security.allow
|
||||
not data.security.deny
|
||||
data.developer.allow
|
||||
not data.developer.deny
|
||||
}
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
The post recaps three requirements: policies are mostly if-statements and should read naturally; decisions can be any JSON value rather than just booleans; and composition/conflict-resolution use the same rule mechanics as ordinary policy logic.
|
||||
|
||||
Further reading:
|
||||
|
||||
- [Design principle #2: Embrace Hierarchical Data](/blog/rego-design-principle-2-embrace-hierarchical-data-8a4913bdfea1)
|
||||
- [Design principle #3: Optimize Performance Automatically](/blog/rego-design-principle-3-optimize-performance-automatically-2d29ad3ce96d)
|
||||
+289
@@ -0,0 +1,289 @@
|
||||
---
|
||||
title: "Rego Design Principle #2: Embrace hierarchical data"
|
||||
authors: ["timhinrichs"]
|
||||
date: 2020-03-24
|
||||
slug: rego-design-principle-2-embrace-hierarchical-data-8a4913bdfea1
|
||||
---
|
||||
|
||||
|
||||
|
||||
This is the second part of a blog series on the design principles behind Open Policy Agent's (OPA's) policy language Rego. Previously we described how Rego's [syntax was designed to mirror the structure of real-world policies](/blog/rego-design-principle-1-syntax-should-reflect-real-world-policies-e1a801ab8bfb). In this part of the series, we look at why and how Rego exclusively uses hierarchical data (e.g. JSON and YAML) to represent the raw information it uses to make decisions and to represent the decisions themselves. In the next part of the series, we discuss why and how OPA aims to [optimize the performance of policy evaluation automatically](/blog/rego-design-principle-3-optimize-performance-automatically-2d29ad3ce96d).
|
||||
|
||||
## Quick OPA Refresher
|
||||
|
||||
OPA is designed to offload policy decisions from a broad range of software services. You typically run OPA on the same server as the software needing policy decisions and cajole that software into asking OPA for a policy decision whenever it needs to. As shown in the diagram below, OPA makes a decision using the following pieces of information:
|
||||
|
||||
- **Policy query**. An arbitrary JSON document provided by the service needing a policy decision. Think of the policy query as the concrete information (e.g. user-action-resource) that OPA needs to make a decision about.
|
||||
- **External data.** Any number of JSON documents injected into OPA out-of-band of the policy query that represent what's happening in the real world (e.g. the current resources in a k8s cluster or resource attributes like owner, size, etc.) and that are kept up to date as the world changes.
|
||||
- **Rego policy**. One or more Rego policies. Rego is a custom language purpose-built for expressing policy across any domain.
|
||||
|
||||
The focus of this blog post is to explain why and how we chose to use JSON to represent the policy query, the external data, and even the policy decision itself.
|
||||
|
||||
## JSON is Everywhere
|
||||
|
||||
JSON (or more generally hierarchically structured data) is pervasive throughout the cloud-native ecosystem. Public clouds, kubernetes clusters, No-SQL (and even SQL) databases, service meshes, microservice APIs, and application configuration all ingest and export their state in JSON. Hierarchical data (as opposed to say relational data stored in classic SQL databases) is here to stay, arguably because it is well-suited for modeling many different aspects of software applications and the infrastructure they run on. And further, the prevalence of HTTP/JSON APIs makes JSON a pervasive format for exchanging information.
|
||||
|
||||
What this means for OPA is that it's a near certainty that when a service is asking OPA for a policy decision, it will have some hierarchical data that OPA needs to make the decision.
|
||||
|
||||
For example, maybe it's a JSON Web Token (JWT) that represents the user and her attributes:
|
||||
|
||||
```json
|
||||
{
|
||||
"sub": "1234567890",
|
||||
"name": "Alice Smithsonian",
|
||||
"iat": 1516239022,
|
||||
"groups": ["employee", "billing-manager"]
|
||||
}
|
||||
```
|
||||
|
||||
Or, maybe it's information about the attributes for a pet at a pet store:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "i0779921",
|
||||
"name": "Lassie",
|
||||
"breed": "collie",
|
||||
"owners": [{
|
||||
"first": "Rudd",
|
||||
"last": "Weatherwax"
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
It could also be a description of the configuration for an application running on Kubernetes (here shown in the usual k8s YAML that converts easily to JSON):
|
||||
|
||||
```yaml
|
||||
apiVersion: admission.k8s.io/v1beta1
|
||||
kind: AdmissionReview
|
||||
request:
|
||||
kind:
|
||||
group: extensions
|
||||
kind: Ingress
|
||||
version: v1beta1
|
||||
object:
|
||||
metadata:
|
||||
name: prod
|
||||
labels:
|
||||
costcenter: retail
|
||||
spec:
|
||||
rules:
|
||||
- host: initech.com
|
||||
http:
|
||||
paths:
|
||||
- path: /finance
|
||||
backend:
|
||||
serviceName: banking
|
||||
servicePort: 443
|
||||
- path: /retail
|
||||
backend:
|
||||
serviceName: storefront
|
||||
servicePort: 8080
|
||||
```
|
||||
|
||||
All across the stack, from infrastructure to microservices to the business data stored by an application, JSON is pervasive for representing information. Moreover, even in those areas where JSON data is not pervasive like SQL databases, it is straightforward to convert flat, non-hierarchical data into JSON; whereas, converting JSON into a non-hierarchical data format while possible presents many usability challenges.
|
||||
|
||||
## How OPA interacts with the outside world
|
||||
|
||||
Remember that OPA can consume two sources of data to make policy decisions:
|
||||
|
||||
- the data that the service provides as the policy query
|
||||
- the external data that gets injected into OPA that represents the state of the outside world
|
||||
|
||||
Both of those are arbitrary JSON. OPA does NOT impose any kind of schema or data model on those JSON documents. All OPA knows is that it's a chunk of JSON; it is up to the policy author to understand what that JSON represents in the world and write the policy that makes the appropriate decision.
|
||||
|
||||
We could have designed OPA differently. We could have designed OPA to have a schema or data model for each domain (e.g. k8s, service mesh, databases, applications) and required the outside world to adapt its data to OPA's model.
|
||||
|
||||
For example, suppose OPA required every policy query to have three fields:
|
||||
|
||||
- **username**: a string that represents the user taking an action
|
||||
- **action**: a string that names the action of the user is trying to take
|
||||
- **resource**: a string identifying the resource being acted upon
|
||||
|
||||
This would mean that every application asking OPA for an authorization decision would need to supply exactly those three fields. If the application had the user information stored in the JWT as shown below, it could not just hand that JWT to OPA — it would need to extract the `sub` (subject) value and include it as the `username` value.
|
||||
|
||||
```json
|
||||
{
|
||||
"sub": "1234567890",
|
||||
"name": "Alice Smithsonian",
|
||||
"iat": 1516239022,
|
||||
"groups": ["employee", "billing-manager"]
|
||||
}
|
||||
```
|
||||
|
||||
Imposing a schema or data model would have made building OPA easier because it shifts the burden for integration to the outside world. Every system in the world that wants to integrate with OPA would need to include OPA-specific code that transforms the data to meet OPA's requirements.
|
||||
|
||||
Moreover, the same is true for the external data that OPA uses to make decisions. If OPA imposed a data model on all external data, the system pushing that data into OPA would need to understand OPA's data-model and transform the data from the outside world to match that model.
|
||||
|
||||
Instead, OPA was designed to ingest arbitrary JSON data for both the policy query and external data. This makes integrating with OPA easy: just convert the information into JSON (which every programming language has standard libraries for) and send it across. No need to ETL your data to get it into OPA — any webhook will suffice to integrate OPA. In short…
|
||||
|
||||
> OPA should adapt to data in the outside world, not the other way around
|
||||
|
||||
Ingesting JSON data in whatever form is natural for the outside world is easy, but it does mean that the policy language Rego needs to be flexible enough that people can write policies that adapt to that format. The policy language can't rely on a fixed location for the username or the action, for example. It must be expressive enough that people can write policy that bridges the gap between the world's data model and the format that is best for expressing policy.
|
||||
|
||||
## Rego support for JSON
|
||||
|
||||
The starting point for a Rego policy is (i) an arbitrary JSON object representing the policy query (a.k.a. input) provided by the external software (e.g. an API call, a configuration file, a data element, etc.) and (ii) some number of arbitrary JSON objects representing the state of the world. Neither OPA nor Rego understand what that data means in the real world, but the policy author does. The policy author writes Rego to encode the logic that navigates through those JSON documents and compares them to hard-coded values or other bits of JSON in order to make a decision.
|
||||
|
||||
For example, for a simple HTTP API the input JSON object could be:
|
||||
|
||||
```json
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/dogs/dog123",
|
||||
"user": "alice",
|
||||
"roles": ["customer", "guest"]
|
||||
}
|
||||
```
|
||||
|
||||
As a policy author, I know that this JSON object represents an HTTP API, but Rego doesn't. If I want to allow all GET requests to the root path, I write a simple rule with conditions on the `input` document (a global variable in Rego representing the policy query provided to OPA):
|
||||
|
||||
```rego
|
||||
allow {
|
||||
input.method == "GET"
|
||||
input.path == "/"
|
||||
}
|
||||
```
|
||||
|
||||
This example shows simple equality checks with strings, but in general you might need to break a path like `/dogs/dog123` into multiple pieces, manipulate numbers, inspect the internals of a JWT, etc. The scalar values in JSON often contain information that needs to be extracted or manipulated.
|
||||
|
||||
> Rego must manipulate JSON scalar types: booleans, numbers, strings, and null
|
||||
|
||||
To that end, Rego has [50+ built in functions documented at openpolicyagent.org](https://www.openpolicyagent.org/docs/latest/policy-reference/) that provide all kinds of basic functionality needed to inspect and construct the scalar JSON types.
|
||||
|
||||
Of course, the whole point of supporting JSON is not the scalar types — it's the composite types: arrays and objects. Without those, there's no hierarchy at all.
|
||||
|
||||
There are two key requirements that arise from supporting JSON arrays and objects: the ability to drill down through a hierarchy (which you've already seen via dot notation) and the ability to iterate over elements of a collection (elements of an array or key/value pairs of an object).
|
||||
|
||||
> Rego must navigate through deeply-nested arrays and objects
|
||||
|
||||
Navigating through arrays and objects when you know the exact path is straightforward in Rego. It uses same syntax used by many programming languages: dot-notation and bracket notation.
|
||||
|
||||
For example, suppose the following JSON object is the `input`.
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "i0779921",
|
||||
"name": "Lassie",
|
||||
"breed": "collie",
|
||||
"owners": [{
|
||||
"first": "Rudd",
|
||||
"last": "Weatherwax"
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
You can write all of the following expressions to navigate through this JSON document.
|
||||
|
||||
```rego
|
||||
input.name # "Lassie"
|
||||
input["name"] # "Lassie" x.y is syntactic sugar for x["y"]
|
||||
input.owners[0] # First element of owner's array
|
||||
input.owners[0].first # "Rudd"
|
||||
```
|
||||
|
||||
More interesting is iteration. 99% of Rego statements are simple `if` statements, and iteration is primarily used as a condition in one of those `if` statements.
|
||||
|
||||
For example, say you want to allow an `admin` to perform any operation, and you're given an input that lists all the user's roles.
|
||||
|
||||
```json
|
||||
{
|
||||
"method": "GET",
|
||||
"path": "/dogs/dog123",
|
||||
"user": "alice",
|
||||
"roles": ["customer", "guest"]
|
||||
}
|
||||
```
|
||||
|
||||
You need to write a policy that says the request should be allowed if there is some element of the `roles` array that equals `"admin"`.
|
||||
|
||||
Iteration in Rego uses the keyword `some`. You write an expression testing whether a condition is true and apply `some` to the variables in that expression that you want to iterate over.
|
||||
|
||||
In the admin example, you write the following Rego to check if there is some index `i` of the `roles` array where `input.roles[i]` equals `"admin"`.
|
||||
|
||||
```rego
|
||||
allow {
|
||||
some i
|
||||
input.roles[i] == "admin"
|
||||
}
|
||||
```
|
||||
|
||||
You can apply `some` to many variables at once. For Kubernetes policies this happens all the time. Here is an object that is roughly what Kubernetes hands over for admission control — notice how deeply nested the data is.
|
||||
|
||||
```yaml
|
||||
kind:
|
||||
kind: Ingress
|
||||
group: extensions
|
||||
metadata:
|
||||
name: prod
|
||||
labels:
|
||||
costcenter: retail
|
||||
spec:
|
||||
rules:
|
||||
- host: initech.com
|
||||
http:
|
||||
paths:
|
||||
- path: /finance
|
||||
backend:
|
||||
serviceName: banking
|
||||
servicePort: 443
|
||||
- path: /retail
|
||||
backend:
|
||||
serviceName: storefront
|
||||
servicePort: 8080
|
||||
```
|
||||
|
||||
If you want to deny the creation of this resource whenever there is some `servicePort` that is not 443, you would write the following Rego.
|
||||
|
||||
```rego
|
||||
deny {
|
||||
input.kind.kind == "Ingress"
|
||||
some i,j
|
||||
input.spec.rules[i].http.paths[j].backend.servicePort != 443
|
||||
}
|
||||
```
|
||||
|
||||
While that path to `servicePort` is somewhat long, it is simply the nature of the data. Seeing the path written out in a single line makes it relatively easy to map it back to the real data, which can help the reader understand the intent of the rule.
|
||||
|
||||
In contrast, in traditional programming languages, you need to decompose that JSON path into chunks and dictate exactly the range over which you want to iterate one variable at a time. Here would be the same example in Python.
|
||||
|
||||
```python
|
||||
function deny():
|
||||
return input.kind.kind == "Ingress" and deny_aux()
|
||||
|
||||
function deny_aux():
|
||||
for rule in input.spec.rules:
|
||||
for path in rule.http.paths:
|
||||
if path.backend.servicePort != 443:
|
||||
return true
|
||||
```
|
||||
|
||||
As a reader, to understand what the Python says in terms of the data, you need to reconstruct the JSON path by composing the paths in the `for` loops and `if` statements. The decomposed path approach shown in Python is closer to an implementation of policy than the policy itself.
|
||||
|
||||
Of course, Rego is flexible enough that you can decompose paths if you want.
|
||||
|
||||
```rego
|
||||
deny {
|
||||
input.kind.kind == "Ingress"
|
||||
some i, j
|
||||
rule := input.spec.rules[i]
|
||||
path := rule.http.paths[j]
|
||||
path.backend.servicePort != 443
|
||||
}
|
||||
```
|
||||
|
||||
Having had Rego's ability to iterate in different ways for the last few years, we find that sometimes we decompose paths and sometimes not. Personally, I typically avoid decomposing paths as I find them easier to read when I come back weeks or even days later because I can compare the policy statement more directly to the documentation for that JSON data; and often I don't even need the documentation because the path itself is self-explanatory.
|
||||
|
||||
## Summary
|
||||
|
||||
Rego was designed to express policy over JSON data natively.
|
||||
|
||||
- **Why JSON?** JSON is pervasive in cloud-native environments, which means that the external data and inputs that OPA uses to make policy decisions is easy to come by.
|
||||
- **Rego is designed to adapt to the world around it** — not the other way around. This leads to a low barrier for integrating with OPA, often requiring no OPA-specific code.
|
||||
- **Rego has first-class support for inspecting JSON values.** It has 50+ built in functions for string manipulation, JWT manipulation, network CIDR math, etc. And Rego has first-class support for navigating through deeply-nested arrays and dictionaries.
|
||||
|
||||
OPA was designed to be integrated into a wide array of software systems, and as such ease-of-integration is paramount. Rego's flexibility makes it applicable to a wide variety of use cases, and moreover makes it easy to integrate OPA across the cloud-native stack.
|
||||
|
||||
If you want to know more, check out the other blog posts in the series:
|
||||
|
||||
- [Design principle #1: Syntax should reflect real-world policies](/blog/rego-design-principle-1-syntax-should-reflect-real-world-policies-e1a801ab8bfb)
|
||||
- [Design principle #3: Optimize Performance Automatically](/blog/rego-design-principle-3-optimize-performance-automatically-2d29ad3ce96d)
|
||||
@@ -0,0 +1,52 @@
|
||||
---
|
||||
title: "Rego Playground: New Features"
|
||||
authors: ["tsandall"]
|
||||
date: 2020-03-25
|
||||
slug: rego-playground-new-features-ec0345a73b9e
|
||||
---
|
||||
|
||||
[This time last year we launched the Rego Playground](/blog/the-rego-playground-977566855cec). The playground provides an online interactive environment where users can experiment with and share OPA policies.
|
||||
|
||||
Today, we are excited to release features in the playground that will help new users get up and running with OPA even quicker than before. Let's have a look.
|
||||
|
||||
## Feature: Examples for Kubernetes, Envoy and more
|
||||
|
||||
Anyone who designs user interfaces (or perhaps any software project/product) is probably familiar with the "blank slate" problem: all of the designs assume the system is loaded with data. However, when new users arrive that data does not exist and the system feels _empty_.
|
||||
|
||||
Since the beginning of OPA we have focused on providing detailed documentation so that new users (a) have something to look at and (b) can figure out whether OPA will solve their problems. Of course, this assumes people want (or have time) to read the docs! Ain't nobody got time for that.
|
||||
|
||||
Rather than trying to tell everyone to RTFM, we have decided to preload the playground with a catalogue of examples for common use cases like Kubernetes admission control, API authorization with Envoy and more:
|
||||
|
||||
|
||||
|
||||
The catalogue is searchable and filterable. Over time we plan to continue to curate the catalogue to ensure they demonstrate common use cases and patterns in policy language.
|
||||
|
||||
## Feature: Kick the tires with OPA bundles
|
||||
|
||||
Once you have written a couple policies or modified existing ones, the next thing you often want to see is how OPA can be deployed and have _policies distributed_ to it.
|
||||
|
||||
OPA supports a feature called ["bundles"](https://www.openpolicyagent.org/docs/latest/management/#bundles) that enable policy discovery and distribution. Bundles are just gzipped tarballs containing policy and data files. When bundles are enabled, OPA continually tries to download and activate the latest version of policy and data that control its decision-making. Bundles are designed to be CDN compatible so that policy distribution can scale easily. All you have to do is set up a webserver to host your bundles (or rely on services like AWS S3), however, this is often more work than people want to embark on while they kick the tires.
|
||||
|
||||
To help users get up and running with bundles, we have extended the playground to serve published policies as bundles. All you have to do is click "Publish".
|
||||
|
||||
|
||||
|
||||
Once you publish your policy, the playground displays the steps to:
|
||||
|
||||
- Download and run OPA locally
|
||||
- Configure OPA to use your published policy
|
||||
- Test the policy with the input from the playground
|
||||
|
||||
Any edits to the policy that are published from the same browser window will propagate to OPAs configured to use the playground bundle. This lets you exercise OPA's dynamic policy update capability (aka "hot reloading").
|
||||
|
||||
## Feature: Improved support for context-aware policies
|
||||
|
||||
When software systems query OPA for policy decisions they can supply arbitrary JSON data as _input_. This data drives policy decision-making, however, in many cases, it's not sufficient — more information about the state of the world is required to render a decision. In OPA, we often refer to this information as "context". There are various ways to load context into OPA, however, one of the most common ways is to cache data in-memory alongside the policies.
|
||||
|
||||
When context is cached in-memory it's referenced under the `data` global variable. If you have used OPA for Kubernetes admission control, you may have seen policies that refer to the cached state of the Kubernetes cluster maintained inside of OPA in admission controller deployments (e.g., `data.kubernetes.ingresses`).
|
||||
|
||||
In the initial version of the playground, we did not provide support for loading arbitrary external JSON values under data. This was primarily to keep the UI as simple as possible and also because technically you can just define any JSON values you want _inside of the policy itself — references to JSON defined in the policy are identical to references to raw JSON that would be cached in OPA._
|
||||
|
||||
So while it was possible to experiment with context-aware policies inside the playground, it was a bit non-obvious. In the latest release, there is now an empty "Data" panel (along with "Input" and "Output") that lets you load arbitrary JSON values under data:
|
||||
|
||||
|
||||
+146
@@ -0,0 +1,146 @@
|
||||
---
|
||||
title: "Rego Design Principle #3: Optimize Performance Automatically"
|
||||
authors: ["timhinrichs"]
|
||||
date: 2020-04-07
|
||||
slug: rego-design-principle-3-optimize-performance-automatically-2d29ad3ce96d
|
||||
---
|
||||
|
||||
|
||||
|
||||
This is the third part of a blog series on the design principles behind Open Policy Agent's (OPA's) policy language Rego. Previously, we described how Rego's [syntax was designed to mirror the structure of real-world policies](/blog/rego-design-principle-1-syntax-should-reflect-real-world-policies-e1a801ab8bfb) and how Rego [embraces hierarchical data](/blog/rego-design-principle-2-embrace-hierarchical-data-8a4913bdfea1). In this part of the series, we look at Rego/OPA's commitment to automated performance optimization so that policy authors can focus on writing readable, maintainable policies, and OPA can shoulder the burden of evaluating those policies efficiently.
|
||||
|
||||
## Performance is Important, but Policy Authors Should Be Able to Ignore It
|
||||
|
||||
The goal of OPA is to take policies that people write and automatically enforce, monitor and remediate them. No more relying on people to remember the policies, to understand them or to correctly apply them. OPA helps everyone follow the rules and protects the infrastructure, applications, etc. from security, compliance and operations problems.
|
||||
|
||||
The scale and speed of modern, cloud-native systems means that OPA policies are routinely applied to millions of objects and actions. [Pinterest, at the last Kubecon, described using OPA to make 400k decisions per second (8M with caching)](https://www.youtube.com/watch?v=LhgxFICWsA8), globally across all infrastructure. At that scale, how long it takes to evaluate an OPA policy, how much memory it uses, and how many requests can happen in parallel all matter a great deal.
|
||||
|
||||
Despite the importance of performance one of OPA's design principles is to minimize how much a policy author worries about performance. The policy author should write the logic of her policy so that it is easy to read, maintain, extend, and combine with other peoples' policies. The author shouldn't be forced to complicate the logic of her policy in order to make it perform well.
|
||||
|
||||
> The policy author handles correctness. OPA handles performance.
|
||||
|
||||
In the end, this design principle is all about usability. Policies are easier to write because you only think about correctness and not performance. Policies are easier to read because they're written closer to the way people think about them in the real world. Policies are easier to maintain and extend because you write them each individually and leave global optimizations to OPA. Upgrade OPA and your policies get faster.
|
||||
|
||||
As a point of comparison every programming language aims for good performance too, but OPA's goal is qualitatively more ambitious in the sense that even the global asymptotic runtime of policy evaluation is something that the policy author should be able to ignore. If the clearest way to write/read your policy looks like an O(n¹⁰) algorithm in a traditional language even though it can be done with 10 linear scans, OPA aims to analyze that policy, reformulate it, set up the proper indexing and evaluate it in linear time.
|
||||
|
||||
## How OPA Automates Performance Optimization
|
||||
|
||||
Our approach to realizing this design principle was to base Rego on database query languages and leverage 50 years of R&D on automated performance optimization. In particular:
|
||||
|
||||
**Rego has no side-effects**. Rego always produces the same outputs given the same inputs, and there are no side-effects before, during or after evaluation. Side-effects would include writing to a file or modifying the value of an in-memory object. Side-effects greatly complicate the design of optimization algorithms because those algorithms need to understand how to preserve those side-effects.
|
||||
|
||||
[**Rego rules are unordered**](/blog/orderly-versus-disorderly-policies-717475c23d2f). A Rego policy means the same thing regardless of how the rules are ordered. Since ordering is irrelevant, the meaning of each statement stands on its own and can be optimized on its own.
|
||||
|
||||
In contrast, in languages where ordering matters, every optimization must respect that ordering. Firewall rules are a perennial example, where to understand the impact that the 1000th firewall rule has, you need to understand the impact of the first 999. (Rego does support the 'else' keyword for ordering, but we advise using it sparingly.)
|
||||
|
||||
```rego
|
||||
# Both allow and deny are true.
|
||||
# Order of the statements does not matter.
|
||||
allow = true {
|
||||
input.method == "GET"
|
||||
input.path == "/"
|
||||
}
|
||||
deny = true {
|
||||
input.method == "GET"
|
||||
input.path == "/"
|
||||
}
|
||||
```
|
||||
|
||||
**Rego rules are simple**. Just about every statement in Rego is a conditional variable assignment. A Rego statements assigns a variable to a value if some conditions are true. This uniformity and simplicity help ensure that optimizations apply across all Rego policy statements.
|
||||
|
||||
```rego
|
||||
allow = true { # allow is assigned true if ...
|
||||
input.method == "GET" # method is "GET" AND
|
||||
input.path == "/" # path is "/"
|
||||
}
|
||||
```
|
||||
|
||||
**Rego is designed in layers**. Rego was designed like an onion: the core is syntactically quite simple and highly efficient; the layer above that is syntactically more expressive at the cost of some performance; and so on. The outer layer today is quite expressive but not Turing complete (though in the future we could relax some of the restrictions on the language if we wanted that escape hatch). Below is a diagram showing some of the layers of the language.
|
||||
|
||||
|
||||
|
||||
Of course there are limits to what can be done at all and what has been implemented, but it's a clear design principle — that the language itself should be amenable to deep, automated analysis and global optimizations. At the time of writing, OPA has the following features either built out (GA), in progress (WIP), or planned.
|
||||
|
||||
[**Rego has automatic, multi-dimensional indexing**](/blog/optimizing-opa-rule-indexing-59f03f17caf3) **(GA)**. OPA analyzes rules and automatically constructs a trie that finds the minimal set of applicable rules. For example, the following rules will be represented as the trie shown below.
|
||||
|
||||
```rego
|
||||
# Rules as they appear in a Rego file
|
||||
allow = true {
|
||||
input.method == "POST"
|
||||
input.path == "/pets"
|
||||
something_complex
|
||||
}
|
||||
allow = true {
|
||||
input.method == "GET"
|
||||
input.path == "/pets/dogs"
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
|
||||
[**Partial evaluation**](/blog/partial-evaluation-162750eaf422) **(GA)**. Partial evaluation attempts to compile a policy from one layer into a lower layer (sometimes at the cost of creating additional rules). This could, for example, convert a linear-time policy into a constant-time policy.
|
||||
|
||||
```rego
|
||||
# Before partial evaluation, a data-driven policy that is easy
|
||||
# for people unfamiliar with Rego to contribute to.
|
||||
allow {
|
||||
op = allowed_operations[_]
|
||||
input.method == op.method
|
||||
input.resource == op.resource
|
||||
}
|
||||
allowed_operations = [
|
||||
{"method": "PUT", "resource": "air-conditioner"},
|
||||
{"method": "GET", "resource": "security-camera"},
|
||||
{"method": "POST", "resource": "garage-door"},
|
||||
]
|
||||
```
|
||||
|
||||
After partial evaluation runs we get rules that the indexer can more easily analyze and organize in a trie.
|
||||
|
||||
```rego
|
||||
# After partial evaluation
|
||||
allow {
|
||||
input.method == "PUT"
|
||||
input.resource == "air-conditioner"
|
||||
}
|
||||
allow {
|
||||
input.method == "GET"
|
||||
input.resource == "security-camera"
|
||||
}
|
||||
allow {
|
||||
input.method == "POST"
|
||||
input.resource == "garage-door"
|
||||
}
|
||||
```
|
||||
|
||||
[**Compilation to WebAssembly**](/blog/opa-v0-15-1-rego-on-webassembly-81c226c51be4) **(WIP)**. WebAssembly is a popular general-purpose virtual machine that has implementations in a growing number of languages like Go, Node, JavaScript, and Java. The Rego WebAssembly compiler takes a Rego policy and generates custom WebAssembly code that implements that policy, and like most compiler technology eliminates overhead for the interpreter and provides the opportunity for deep, automated performance tuning.
|
||||
|
||||
**Automated asymptotic analysis (WIP)**. While still a work in progress, the goal of this tool is to identify the run-time complexity of a policy. Which of these three layers does it belong to? If it is a linear-time policy what slice of JSON data is it iterating over?
|
||||
|
||||
For example, in Kubernetes a common policy is to check if all images come from a trusted registry, say `hooli.com`. The complexity analysis will tell you that this policy has complexity `O(input.request.object.spec.containers)`.
|
||||
|
||||
```rego
|
||||
deny {
|
||||
input.request.kind.kind == "Pod"
|
||||
some i
|
||||
image := input.request.object.spec.containers[i].image
|
||||
not startswith(image, "hooli.com/")
|
||||
}
|
||||
```
|
||||
|
||||
**Query optimization (Planned)**. As use cases like audit that require deeper searches over significant amounts of data become more popular, we plan to include optimizations that analyze dependencies, reorder rule evaluation, reorder conditions within rules, shift evaluation of conditions between rules, and the like.
|
||||
|
||||
## Summary
|
||||
|
||||
One of OPA and Rego's design principles is that Rego policy authors should focus on correctness, maintainability and composability. They should not complicate their policy logic to make policy evaluation more efficient. More specifically:
|
||||
|
||||
- OPA aims to optimize both local operations and the global asymptotic runtime
|
||||
- While Rego's syntax looks like a programming language, its semantics is based on database query languages.
|
||||
- The automated optimizations implemented in OPA continue to grow over time.
|
||||
|
||||
While OPA and Rego today have made terrific progress in terms of realizing the goal of automated performance optimization, there's still a lot to do. If you want to help, hop onto Slack or Github and contribute!
|
||||
|
||||
If you want to know more, check out the other blog posts in the series:
|
||||
|
||||
- [Design principle #1: Syntax should reflect real-world policies](/blog/rego-design-principle-1-syntax-should-reflect-real-world-policies-e1a801ab8bfb)
|
||||
- [Design principle #2: Embrace hierarchical data](/blog/rego-design-principle-2-embrace-hierarchical-data-8a4913bdfea1)
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
title: "Open Policy Agent v0.19 Release"
|
||||
authors: ["tsandall"]
|
||||
date: 2020-04-23
|
||||
slug: open-policy-agent-v0-19-release-921d49179440
|
||||
---
|
||||
|
||||
|
||||
|
||||
Last week we released OPA v0.19, containing 63 commits from 12 contributors (of which, 9 were external.) This release includes many important fixes and enhancements, as well as a new Rego parser written in Go that speeds up parsing time by ~100x in most cases. You can find more details on the [GitHub releases](https://github.com/open-policy-agent/opa/releases) page.
|
||||
|
||||
## Community Updates
|
||||
|
||||
Since many in-person events have gone virtual due to the COVID-19 crisis, there have been several virtual events, webinars and podcasts featuring OPA over the last few weeks. Here's a quick roundup:
|
||||
|
||||
- [@LachlanEvenson](https://twitter.com/LachlanEvenson) ("B-grade Bollywood Actor") and Sertaç Özercan (Software Engineer @Azure, OPA Gatekeeper maintainer, [@sozercan](https://twitter.com/sozercan?lang=en)) presented at the CNCF Member Webinar about ensuring compliance in Kubernetes using OPA Gatekeeper. [Slides and recording are here](https://www.cncf.io/webinars/ensuring-compliance-without-sacrificing-development-agility-and-operational-independence-in-k8s-with-opa-gatekeeper/).
|
||||
- Michael Hausenblas (Developer Advocate at AWS, [@mhausenblas](https://twitter.com/mhausenblas)) [did a stream](https://www.youtube.com/watch?v=dlKXFYBngBw) talking about Deprek8 with Steve Wade (K8s consultant & Trainer and Platform Lead at Mettle). You can read more in this [Deprek8 article on Opensource.com](https://opensource.com/article/20/3/deprek8).
|
||||
- Rosemary Wang (Developer Advocate at HashiCorp, [@joatmon08](https://twitter.com/joatmon08)) talked about security & policy for infrastructure as code on OWASP DevSlop! [The stream included live demos of OPA, conftest and more.](https://www.youtube.com/watch?v=KOTXCIN0yE0)
|
||||
- Alex Krause (Software Engineer at QAware, [@alex0ptr](https://twitter.com/alex0ptr)) spoke at the Cloud Native Virtual Summit about cloud compliance with OPA. You can find the [SlideShare presentation slides](https://www.slideshare.net/QAware/cloud-compliance-with-open-policy-agent). The stream is also [available](https://gateway.on24.com/wcc/eh/2010041/lp/2235047/qaware-gmbh-cloud-compliance-with-open-policy-agent) but requires registration.
|
||||
- Kevin Harris (Cloud Architect at Microsoft) talked about how OPA and Kubernetes Admission Control at the [Cyber Tech & Risk Virtual Event.](https://www.youtube.com/watch?v=41Ecd8Uuyvs&feature=youtu.be)
|
||||
- Daniel Mangum (Engineer at Crossplane.io, [@hasheddan](https://twitter.com/hasheddan?lang=en)) [hosted me on The Binding Show to talk about OPA, Crossplane and more.](https://www.youtube.com/watch?v=TaF0_syejXc) I also joined SW Engineering Radio to talk about OPA and distributed policy enforcement in general. You can find the [SW Engineering Radio episode recording](http://hwcdn.libsyn.com/p/b/c/c/bcc49f4be8bc53f1/Episode-406-Torin-Sandall-on-Distributed-Policy-Enforcement_.mp3?c_id=69980306&cs_id=69980306&destination_id=1520171&expiration=1587590198&hwt=48bcf40f509d2271f0952d1e51c23a6d).
|
||||
- [The New Stack published an article](https://thenewstack.io/open-policy-agent-authorization-for-the-cloud) from Tim Hinrichs (co-creator of OPA and founder of Styra, [@tlhinrichs](https://twitter.com/tlhinrichs)) describing various use cases organizations use OPA for today. Tim also recently published a [great series of blog posts about the design principles behind Rego](/blog/rego-design-principle-1-syntax-should-reflect-real-world-policies-e1a801ab8bfb).
|
||||
|
||||
Since KubeCon 2019 in Barcelona, we have asked users to post Q&A style queries on Stack Overflow instead of slack.openpolicyagent.org. The reason is that most **answers posted on Slack are not discoverable**! If you have Q&A style questions (e.g., ["How to test `not deny`?"](https://stackoverflow.com/questions/60083793/rego-testing-how-to-test-not-deny)), try posting on [Stack Overflow and tagging with open-policy-agent](https://stackoverflow.com/questions/tagged/open-policy-agent).
|
||||
|
||||
## Faster Parsing & Better Errors
|
||||
|
||||
The largest change in v0.19 is the new Rego parser, which is written from scratch in Go. Previously, OPA relied on a [generated](https://github.com/open-policy-agent/opa/blob/v0.18.0/ast/rego.peg) [parser](https://github.com/open-policy-agent/opa/blob/v0.18.0/ast/parser.go) that was defined using [PEG (Parsing Expression Grammar [wikipedia])](https://en.wikipedia.org/wiki/Parsing_expression_grammar). Over the years, as the grammar has grown, and larger inputs have been thrown at it, the generated parser became a bottleneck (e.g., it could take about 10x longer to parse an input than compile and evaluate the query.
|
||||
|
||||
Inside OPA we were able to workaround the performance problems with caching, using Go's "encoding/json" package and manually converting to AST ("Abstract Syntax Tree") values when possible, etc. However, new users embedding OPA as a library would (understandably) make mistakes and wonder why performance was poor. The majority of the performance problems in the generated parser were due to a significant amount of heap allocations required to parse any input.
|
||||
|
||||
In addition to performance, we also struggled with usability around parser error messages. If the parser was not able to match an input, you would be presented with an error like "policy.rego:19: no match found". No match? Tell me more!
|
||||
|
||||
Rather than attempt to continue incrementally improving the existing parser, we decided to rewrite it from scratch in Go. The result is a new parser that allocates significantly less memory (which improves performance by approximately 100x in most cases) and has better error messages. One important requirement for the new parser was backwards compatibility — the new parser could not break existing policies OR programs that embed OPA as a library (e.g., the parser APIs and the AST types also had to remain the same). To ensure we did not break existing (valid) policies, we checked for differences in the output of the old and new parser for hundreds of thousands of Rego snippets (which deserves another blog post in the future.) Lastly, we also applied the wonderful [go-fuzz](https://github.com/dvyukov/go-fuzz) project [to the parser](https://github.com/tsandall/fuzz-opa) to help catch crashes and other bugs.
|
||||
|
||||
> Since we no longer have a declarative representation of the language grammar in Go, please refer to the [ENBF grammar in the OPA documentation](https://www.openpolicyagent.org/docs/latest/policy-reference/#grammar) as the authoritative source.
|
||||
|
||||
The chart below shows the difference in performance between the old (v0.18 and earlier) and new (v0.19 and later) parser (log scale):
|
||||
|
||||
|
||||
|
||||
Overall, we are happy with the process. In the future we plan to continue optimizing performance in the parser and looking for ways to improve error messaging and usability.
|
||||
|
||||
## man(1) pages, http.send, and Emacs support
|
||||
|
||||
In addition to the new parser, v0.19 includes dozens of bugfixes and feature enhancements. [@olivierlemasle](https://github.com/olivierlemasle) contributed code to generate OPA man pages from the OPA CLI definitions. The `man` pages are automatically available if you:
|
||||
|
||||
```bash
|
||||
brew install opa
|
||||
```
|
||||
|
||||
|
||||
|
||||
_Install OPA with homebrew and use 'man opa' to learn about it._
|
||||
|
||||
[@jpeach](https://github.com/jpeach) submitted a number of patches that improve testing and support for the [http.send](https://www.openpolicyagent.org/docs/latest/policy-reference/#http) built-in function. For example, policies can now explicitly set TLS server names as well as certificates and keys when invoking the built-in function (previously they could only come from the environment or local files). This is useful if you want to specify those values in data or as local variables inside the policy itself.
|
||||
|
||||
Lastly, the release also includes a pointer to the new [rego-mode](https://github.com/psibi/rego-mode) Emacs package developed by [@psibi](https://github.com/psibi). The package provides syntax highlighting, formatting and more. In the future, the package could be extended to support many of the same features as the OPA extension for VS Code.
|
||||
|
||||
## WebAssembly Update
|
||||
|
||||
At KubeCon 2019 in San Diego we announced [support for compiling OPA policies into WebAssembly (Wasm)](/blog/opa-v0-15-1-rego-on-webassembly-81c226c51be4). Wasm enables OPA policies to execute in new environments like CDNs, service proxies and more without requiring an out-of-process RPC call to query OPA.
|
||||
|
||||
This week we are excited to release further support for Wasm in OPA with the new [golang-opa-wasm](https://github.com/open-policy-agent/golang-opa-wasm) project! This project wraps the [wasmerio/go-ext-wasm](http://wasmerio/go-ext-wasm) runtime library to provide convenient APIs for policy execution and more. The golang-opa-wasm SDK is still work-in-progress but feedback and contributions are welcome.
|
||||
@@ -0,0 +1,99 @@
|
||||
---
|
||||
title: "Open Policy Agent Survey Summary (Spring 2020)"
|
||||
authors: ["tsandall"]
|
||||
date: 2020-05-19
|
||||
slug: open-policy-agent-survey-summary-spring-2020-adaa46e61f0
|
||||
---
|
||||
|
||||
|
||||
|
||||
Last month we surveyed the OPA community to learn more about user adoption and help us plan and improve the OPA project. We received 204 responses (up 175% from the last survey in April 2019) from over 150 organizations, with 91% of respondents indicating they are in some stage of OPA adoption (i.e., experimentation, pre-production, production.) This post highlights what we learned from the survey results.
|
||||
|
||||
## Use Cases and Adoption
|
||||
|
||||
### Most organizations use OPA for multiple use cases
|
||||
|
||||
OPA's general-purpose, domain-agnostic architecture is paying off — 51% of respondent organizations use OPA for at least two use cases, with 29% using it for three or more use cases, such as Kubernetes Admission Control, Microservice API Authorization, Application Authorization, Cloud Security and so on. Similarly, only 15% of respondent organizations indicated they _only_ use OPA for Kubernetes Admission Control.
|
||||
|
||||
This data supports our own experience from working with users — the typical adoption path involves identifying OPA as a good solution for a specific problem. From there, users realize they can apply it elsewhere, across the stack. Going forward we will continue to focus on features and improvements that help solve a broad set of use cases.
|
||||
|
||||
| # of Use Cases | % of Orgs Using OPA |
|
||||
| -------------- | ------------------- |
|
||||
| 1 | 39% |
|
||||
| 2 | 21% |
|
||||
| 3 | 17% |
|
||||
| 4 | 7% |
|
||||
| 5 or more | 6% |
|
||||
|
||||
### Application Authorization is becoming a dominant use case for OPA
|
||||
|
||||
43% of respondent organizations indicated they are in some stage of OPA adoption for Application Authorization. In the next survey, we will dedicate a larger section to this use case as it's clearly becoming another pillar for OPA (the other two being Kubernetes Admission Control and Microservice API Authorization). It's unsurprising that OPA is quickly being adopted for Application Authorization because the policies you need to enforce at the top of the stack are typically more sophisticated than those you enforce at the service/transport layer. Going forward, we will continue to add features that improve support for Application Authorization like data fetching, data filtering and UI preflight checks.
|
||||
|
||||
| Use Case | % of Orgs Using OPA for X |
|
||||
| ------------------------- | ------------------------- |
|
||||
| Kubernetes | 54% |
|
||||
| Application Authorization | 43% |
|
||||
| Microservices | 36% |
|
||||
| Terraform | 25% |
|
||||
| Data Stores | 7% |
|
||||
| Other | 17% |
|
||||
|
||||
### Production usage continues to grow
|
||||
|
||||
56% of respondent organizations use OPA in production for Kubernetes Admission Control and 47% use OPA in production for Microservice API Authorization. This is not particularly new, but it reinforces the fact that OPA is being used to solve policy, authorization and security use cases across the stack, in production. Going forward we will continue to focus on work that improves stability and performance. We also plan to define a support policy that clarifies what to expect in terms of backporting (e.g., we will guarantee to backport fixes to N-M releases when asked; backports to older requests will be best-effort.)
|
||||
|
||||
| Stage | % of Orgs Using OPA |
|
||||
| -------------- | ------------------- |
|
||||
| Production | 47% |
|
||||
| Pre-production | 20% |
|
||||
| Experimenting | 24% |
|
||||
|
||||
### Microservice API Authorization requires scalable policy authoring
|
||||
|
||||
35% of respondent organizations use OPA to enforce (or plan to enforce) policies across more than 100 distinct microservices. This makes sense given that microservice architectures often align with organization boundaries. This implies that policy authoring and distribution need to be scaled across many teams. Going forward we will continue to develop tooling that helps scale the authoring and distribution process (e.g., code generating Rego boilerplate from Open API specifications, the new "opa build" command for producing bundles, etc.).
|
||||
|
||||
| # of Microservices | % of Orgs Using OPA for Microservices |
|
||||
| ------------------ | ------------------------------------- |
|
||||
| 1-10 | 13% |
|
||||
| 11-25 | 24% |
|
||||
| 26-50 | 21% |
|
||||
| 51-100 | 6% |
|
||||
| more than 100 | 35% |
|
||||
|
||||
Policy _and_ runtime portability are important. The survey results showed that 57% of respondents use OPA for more than one Microservice API Authorization use case category (i.e., ingress, egress, or service-to-service) and 45% of respondents use more than one kind of OPA deployment architecture (e.g., library, sidecar, service). Moving forward, we will continue to invest in providing strong support for multiple deployment architectures.
|
||||
|
||||
| Microservices Use Case | % of Orgs Using OPA for Microservices |
|
||||
| ---------------------- | ------------------------------------- |
|
||||
| egress | 40% |
|
||||
| ingress | 68% |
|
||||
| service to service | 81% |
|
||||
|
||||
| OPA Deployment Type | % of Orgs Using OPA for Microservices |
|
||||
| ------------------- | ------------------------------------- |
|
||||
| Go Library | 37% |
|
||||
| Service | 42% |
|
||||
| Sidecar | 65% |
|
||||
|
||||
## OPA Feedback
|
||||
|
||||
In addition to soliciting use case feedback, we also asked users to provide feedback on OPA features, Rego, gaps in the OPA ecosystem and so on. The results were positive and reinforce our effort to improve content on openpolicyagent.org.
|
||||
|
||||
### The Rego playground and testing support should be more prominent
|
||||
|
||||
65% of respondents said they use the Rego playground or features like policy testing that accelerate policy authoring. 10% were not aware that features like policy testing and coverage exist. Ideally, 100% of users would leverage the test framework, so moving forward, we'll focus on making testing more prominent and drive more users to try out playground and VS Code features, like interactive evaluation, that are invaluable during debugging.
|
||||
|
||||
### Rego's learning curve
|
||||
|
||||
52% of respondents said they are comfortable with Rego or "okay" with their skill level. 27% said they need occasional help. 16% say they struggle. **68% say they were able to learn Rego in under a week.** Given that Rego is based on programming language paradigms that are foreign to most developers, these numbers are understandable.
|
||||
|
||||
Going forward, we will continue to invest in answering questions on Stack Overflow and improving examples and documentation on the website. Interestingly, the number of people who struggle with Rego was twice as high among those that use OPA for Kubernetes and Terraform compared to Application Authorization. Perhaps this is (at least partially) due to the complex deeply-nested data structures that policies have to be expressed over within those environments.
|
||||
|
||||
### Examples are the biggest gap in the docs
|
||||
|
||||
The survey asked users: _"how can we improve the tutorials and documentation in OPA?"_ By far, the most common request is for **more policy examples**. Going forward we plan to focus on curating and organizing examples across a range of use cases, as well as building out dedicated sections for specific use cases like [application authorization and IAM](https://github.com/open-policy-agent/opa/issues/2098).
|
||||
|
||||
## Wrap Up
|
||||
|
||||
In the future, we plan to run the survey on a more regular basis with more consistent questions, so that we can compare historical results and observe trends over time. If you filled out the survey, thank you! We know you're excited to receive the t-shirts, and we're working on sending them your way! Due to the current situation, it might take a bit longer than we had expected. If you have not yet filled out the survey, you can still [fill out the survey](https://styra.typeform.com/to/QrYJh8). As always, if you have any questions or additional feedback, we're available on the Slack, GitHub, etc.
|
||||
|
||||
_…even doggos love OPA [credit: [@webbergs](https://twitter.com/webbergs), idea: [@the_dvorkin](https://twitter.com/the_dvorkin)]_
|
||||
@@ -0,0 +1,120 @@
|
||||
---
|
||||
title: "Open Policy Agent 2020, year in review"
|
||||
authors: ["anderseknert"]
|
||||
date: 2021-01-04
|
||||
slug: open-policy-agent-2020-year-in-review-dc25b60308d7
|
||||
---
|
||||
|
||||
|
||||
|
||||
## Introduction
|
||||
|
||||
To call 2020 an eventful year would be an understatement. With a pandemic raging globally, we saw the tech community quickly adapt by moving largely online. Working from home became the new norm as meetings, conferences and other events also moved to digital channels. With so much of our lives now taking place online, the importance of our online platforms became overwhelmingly apparent. As businesses and organizations worked hard to accommodate an ever increasing number of users, both interest and investments in securing their platforms followed.
|
||||
|
||||
2020 saw a big uptake in the adoption of OPA. As the de facto standard for cloud-native authorization, OPA has also found its way into many new and interesting domains. The [list of integrations](https://www.openpolicyagent.org/docs/latest/ecosystem/) has been constantly growing during the year, and many more are being worked on. Hundreds of new open source projects, businesses and organizations have come to rely on OPA as their open source unified policy framework across the stack. And, with the increasing adoption of OPA, we've seen the community grow with it — from the number of contributors to active users on Slack. The trajectory for next year has been set, and it truly looks great — for OPA and its community. Before we look ahead though, let's take some time to look back at what we achieved in 2020.
|
||||
|
||||
## OPA 2020 in numbers
|
||||
|
||||
- ~29 million downloads
|
||||
- ~1500 new Github stars
|
||||
- ~1700 new Slack users, doubled in size!
|
||||
- ~750 commits
|
||||
- 90 contributors
|
||||
- 9 major versions released, 17 point releases
|
||||
- 100th release pushed!
|
||||
- 1000th issue closed!
|
||||
- 10000 VS Code OPA plug-in installs!
|
||||
|
||||
## Notable features and enhancements
|
||||
|
||||
### Management
|
||||
|
||||
Several management capabilities were added. The [bundle signing](https://www.openpolicyagent.org/docs/latest/management/#signing) feature allows verifying the integrity of any bundle processed by OPA. Moreover, the new [bundle persistence](https://github.com/open-policy-agent/opa/releases/tag/v0.24.0) feature enables OPA to store a local copy of bundles received, which may be used in case the server is forced to restart while the configured bundle endpoints are unreachable. To authenticate at remote endpoints, several new credential providers were added, including:
|
||||
|
||||
- [GCP metadata token](https://www.openpolicyagent.org/docs/latest/configuration/#gcp-metadata-token) (thanks [@kelseyhightower](https://github.com/kelseyhightower)!)
|
||||
- [AWS web identity](https://www.openpolicyagent.org/docs/latest/configuration/#using-eks-iam-roles-for-service-account-web-identity-credentials) (thanks [@richicoder](https://github.com/RichiCoder))!)
|
||||
- [OAuth2](https://www.openpolicyagent.org/docs/latest/configuration/#oauth2-client-credentials)
|
||||
|
||||
For example, the following configuration instructs OPA to use the GCP Metadata Server to obtain credentials for downloading bundles from a remote endpoint (which happens to be hosted on Cloud Run):
|
||||
|
||||
```yaml
|
||||
services:
|
||||
cloudrun:
|
||||
url: <BUNDLE_SERVICE_URL>
|
||||
response_header_timeout_seconds: 5
|
||||
credentials:
|
||||
gcp_metadata:
|
||||
audience: <BUNDLE_SERVICE_URL>
|
||||
bundles:
|
||||
authz:
|
||||
service: cloudrun
|
||||
resource: bundles/http/example/authz.tar.gz
|
||||
polling:
|
||||
min_delay_seconds: 60
|
||||
max_delay_seconds: 120
|
||||
```
|
||||
|
||||
Lastly, the decision logger now supports [mutating masks](https://www.openpolicyagent.org/docs/latest/management/#masking-sensitive-data) so that administrators can inject or overwrite (as opposed to just removing) fields inside decision logs.
|
||||
|
||||
### Tooling
|
||||
|
||||
The OPA binary had several new flags and features added. The most notable additions are probably the remodeled `opa build` command for creating bundles and the new `opa bench`/`opa test --bench` for [benchmarking policy evaluation](https://www.openpolicyagent.org/docs/latest/policy-performance/#benchmarking-queries).
|
||||
|
||||
### WebAssembly (Wasm)
|
||||
|
||||
Many improvements for Wasm were incorporated this year. Among the bigger ones we saw the opa build command get support for building Wasm modules from Rego policies, and the addition of an [SDK for Javascript](https://www.openpolicyagent.org/docs/latest/wasm/#javascript-sdk). Plenty of built-ins were implemented natively, and the status for Wasm support is now listed for each on the [policy reference](https://www.openpolicyagent.org/docs/latest/policy-reference/#built-in-functions) page.
|
||||
|
||||
### Performance
|
||||
|
||||
In the performance department we saw many improvements. The [comprehension indexing](https://www.openpolicyagent.org/docs/latest/policy-performance/#comprehension-indexing), which allows O(n) runtime complexity on "group-by" operations, is probably the one to stand out the most. Less visible, but of no less importance, a new parser was introduced, improving the internals of OPA and resulting in a 100x speedup on most .rego files!
|
||||
|
||||
### Rego & Built-in Functions
|
||||
|
||||
Built-in functions error handling [was remodeled](https://github.com/open-policy-agent/opa/releases/tag/v0.25.0) to allow policy authors to gracefully deal with errors like garbled input, which would previously halt policy evaluation in certain cases.
|
||||
|
||||
The new caching options for `http.send` introduced this year vastly increases its utility, allowing it to be used for things like OAuth2 and OpenID Connect metadata retrieval. The raise_error option allows policy authors to deal with errors in communication rather than having policy evaluation halt. Given the number of improvements made to the HTTP client this year — and the number of policies in where it is used — it makes the previous "experimental" label it carried less relevant, and the feature should be considered stable moving forward.
|
||||
|
||||
More than 30 new built-ins were added to OPA in 2020. These include functions for:
|
||||
|
||||
- JSON manipulation: `json.patch`, `json.remove`.
|
||||
- Bitwise operations: `bits.or`, `bits.and`, `bits.negate`, `bits.xor`, `bits.lsh`, `bits.rsh`.
|
||||
- Hex encoding: `hex.encode`, `hex.decode`.
|
||||
- Hashing: `crypto.md5`, `crypto.sha1`, `crypto.sha256`.
|
||||
- And a whole bunch of functions for validating various formats: `json.is_valid`, `yaml.is_valid`, `base64.is_valid`, `regex.is_valid`, `semver.is_valid`.
|
||||
- Other useful built-ins added include `graph.reachable`, `object.get`, `numbers.range` and `semver.compare`.
|
||||
|
||||
The wide variety of these help push OPA as a true general purpose policy engine, further increasing the number of domains where Rego may be used. In addition to all the above, hundreds of bugs were fixed and tons of improvements to documentation were made. It's been an incredible year for the project.
|
||||
|
||||
## Ecosystem and integrations
|
||||
|
||||
### Gatekeeper
|
||||
|
||||
The Gatekeeper project took some great leaps forward this year. In terms of new features this meant, among other things, granular namespace exclusions (narrowing the scope of resources to present for audit, webhooks and sync), Helm 3 support and the Gatekeeper Pod Security Policies being referenced as a serious contender to the Kubernetes provided PSPs.
|
||||
|
||||
In terms of stability, Gatekeeper gained support for multi-pod deployments, completed a CNCF security review and had their first stable non-beta release pushed. Other notable enhancements include semantic logging (getting cluster wide resources in violation of policy from logs), standalone auditing and the dependency on finalizers removed.
|
||||
|
||||
Still in early alpha, we're seeing much anticipated support for mutating webhooks. Surely one thing to look out for in 2021!
|
||||
|
||||
If you haven't already, make sure to check out the [gatekeeper-library](https://github.com/open-policy-agent/gatekeeper-library) repo, which was moved out from the Gatekeeper repository this year and has seen continuous additions and improvements since then.
|
||||
|
||||
### Conftest
|
||||
|
||||
2020 was an exciting year in the development of the Conftest project. Having previously been maintained independently, this year we saw the project included in the larger OPA family of projects.
|
||||
|
||||
As for features, the already versatile tool gained support for a number of new input formats (VCL, XML, EDN, TOML, HOCON, Jsonnet) and a new output format (JUnit). Using conftest for Terraform policies became a common use case, and many improvements for Terraform and the HCL2 format got implemented this year. A new system for native plug-ins was added, as was support for loading data files alongside policies. Keeping conftest up to date on Linux was made easier than ever with both RPM and deb packages being made available. A new documentation site was launched at [https://www.conftest.dev/](https://www.conftest.dev/)
|
||||
|
||||
### OPA Envoy plugin
|
||||
|
||||
The OPA Envoy plugin now also supports the v3 transport API, as well as decoding gRPC payloads. Oh, and the project changed its name too — while Istio is very much still supported it is not limited to that implementation, and OPA Envoy plugin was deemed a better name than the OPA Istio plugin.
|
||||
|
||||
### IntelliJ IDEA plugin
|
||||
|
||||
Of all new integrations and projects worked on in the larger OPA ecosystem, the OPA plug-in for IntelliJ IDEA was probably the most anticipated one. After some time in development, a first release was announced this fall and proved to be well worth waiting for. Not only does it cover the necessities, like syntax highlighting, but integrates features for both evaluating rules as well as for running tests right from inside of the policy editor. If you haven't already, make sure to check it out in the IntelliJ plug-in marketplace or at the project [GitHub page](https://github.com/open-policy-agent/opa-idea-plugin).
|
||||
|
||||
|
||||
|
||||
## Credits
|
||||
|
||||
To all who engaged with or contributed to OPA and its ecosystem in 2020 — whether as [maintainers](https://github.com/open-policy-agent/opa/blob/master/MAINTAINERS.md), [contributors](https://github.com/open-policy-agent/opa/graphs/contributors), [integrators](https://www.openpolicyagent.org/docs/latest/ecosystem/), [adopters](https://github.com/open-policy-agent/opa/blob/master/ADOPTERS.md), users, or by helping others on Slack: You have all contributed not only to OPA but just as much to making this community be such a fun and rewarding place to be. Who knows what 2021 has in store for us? One thing however is for certain — with all the knowledge, skills and creative energy found in the OPA community, it will be no less eventful.
|
||||
|
||||
Thank you!
|
||||
@@ -0,0 +1,117 @@
|
||||
---
|
||||
title: "Introducing OPA for IntelliJ"
|
||||
authors: ["soothseer"]
|
||||
date: 2021-02-02
|
||||
slug: introducing-opa-for-intellij-ecc3e338a93
|
||||
description: "OPA Plugin for IntelliJ IDEA brings Rego language support and an IDE experience to your OPA workflow!"
|
||||
---
|
||||
|
||||
_[OPA Plugin](https://github.com/open-policy-agent/opa-idea-plugin) for IntelliJ IDEA brings Rego language support and an IDE experience to your OPA workflow!_
|
||||
|
||||
[IntelliJ IDEA](https://www.jetbrains.com/idea/) is one of the most popular IDEs for developers, offering built-in support for many programming languages such as Java, Kotlin, and Python. A new OPA plugin extends this support to policies written in the [Rego](https://www.openpolicyagent.org/docs/latest/policy-language/) query language, and it also works with [GoLand](https://www.jetbrains.com/go/) for Go developers. The plugin lets users write and evaluate policies directly inside the IDE.
|
||||
|
||||
|
||||
|
||||
## Syntax Highlighting
|
||||
|
||||
The plugin includes support for rule heads, Rego keywords, function calls, strings, and comments.
|
||||
|
||||
|
||||
|
||||
## OPA Actions Menu
|
||||
|
||||
From the OPA Actions Menu, users can run several key OPA actions on the workspace or the current open file:
|
||||
|
||||
|
||||
|
||||
- Install the OPA binary, if it isn't already installed (the plugin prompts for this when running other actions)
|
||||
- **Format Document**: formats the open `.rego` file using `opa fmt`
|
||||
- **Check Document**: checks the open `.rego` file using `opa check`
|
||||
- **Test the workspace**: finds and runs all tests in the project (rules whose heads are prefixed with `test_`)
|
||||
- **View test coverage** for the workspace
|
||||
- **Display the trace** of selected code, using an `input.json` file found in the project root
|
||||
|
||||
## Run Configurations
|
||||
|
||||
In `.rego` files, every line containing a rule head or package includes a "Run Configuration" launch button in the gutter, allowing users to run `opa eval` or `opa test` directly.
|
||||
|
||||
|
||||
|
||||
## Evaluate Rules
|
||||
|
||||
Running a configuration from a rule head line shows `opa eval` results for that package/rule, using the bundle directory and input file set in the configuration.
|
||||
|
||||
|
||||
|
||||
> Tip: you can use the shortcut `shift + F10` on Windows `^r` on Mac to re-run the last executed `Run Configuration`. It allows you to quickly check how changes affect your policies.
|
||||
|
||||
## Evaluate Packages
|
||||
|
||||
Running a configuration from a package line similarly shows `opa eval` results for the package, using the configured bundle directory and input file.
|
||||
|
||||
## Test Rules and Packages
|
||||
|
||||
Running a configuration from a `test_`-prefixed rule head runs `opa test`, with equivalent functionality available at the package level. Test results are displayed within the IDE.
|
||||
|
||||
|
||||
|
||||
## Coming Soon
|
||||
|
||||
### Eval and Partial Eval results for selected code
|
||||
|
||||
Future updates will let users highlight Rego code and view `eval` or `partial evaluation` results directly in the editor, using an `input.json` file in the project root. Results will appear as formatted JSON, and a profiling action will also be added to the menu.
|
||||
|
||||
## Who is it for?
|
||||
|
||||
The plugin aims to help newcomers to OPA/Rego with language support and quick access to core features, while giving experienced OPA developers a smoother in-IDE workflow.
|
||||
|
||||
## How To Contribute
|
||||
|
||||
The plugin is open source, hosted at [https://github.com/open-policy-agent/opa-idea-plugin](https://github.com/open-policy-agent/opa-idea-plugin). Contributors can file issues or pick up existing ones.
|
||||
|
||||
### Where To Start
|
||||
|
||||
The plugin is built with Kotlin and [Gradle](https://github.com/gradle/gradle). Newcomers to Kotlin are pointed to the official [Language Guide](https://kotlinlang.org/docs/reference/), and the bundled J2K Compiler ([tutorial link](https://kotlinlang.org/docs/tutorials/mixing-java-kotlin-intellij.html#converting-an-existing-java-file-to-kotlin-with-j2k)) can help convert Java boilerplate to Kotlin. The [IntelliJ Platform SDK DevGuide](https://jetbrains.org/intellij/sdk/docs/intro/welcome.html) is recommended for learning plugin development.
|
||||
|
||||
## Project Structure
|
||||
|
||||
A condensed overview of the source tree (full version at the [project's architecture page](https://github.com/open-policy-agent/opa-idea-plugin/blob/master/docs/devel/architecture.md)):
|
||||
|
||||
```
|
||||
opa-idea-plugin/
|
||||
├── gradle.build.kts
|
||||
│ …
|
||||
├── plugin # module to build/run/publish opa-ida-plugin plugin
|
||||
│ ...
|
||||
└── src/main/resources/META-INF/plugin.xml
|
||||
└── plugin.xml
|
||||
├── idea # source code of features only available for IntelliJ IDEA
|
||||
│ ...
|
||||
└── src/main/kotlin/resources/META-INF
|
||||
└── idea-only.xml
|
||||
├── src # source code common to all IDEs
|
||||
├── main
|
||||
├── grammar
|
||||
└── Rego.bnf
|
||||
├── kotlin/.../ideaplugin/
|
||||
├── ide
|
||||
│ ...
|
||||
├── actions
|
||||
└── extensions
|
||||
│ ...
|
||||
└── opa/tool
|
||||
└── OpaActions
|
||||
└── resources
|
||||
└── META-INF
|
||||
└── opa-core.xml
|
||||
└── test
|
||||
├── kotlin/.../ideaplugin/
|
||||
│ ...
|
||||
├── ide
|
||||
└── lang
|
||||
└── resources/.../ideaplugin/
|
||||
```
|
||||
|
||||
## Note of Thanks
|
||||
|
||||
The author credits core contributors Vincent Gramer ([vgramer](https://github.com/vgramer/)), Frankie Cerkvenik ([frankiecerk](https://github.com/frankiecerk)), and Igor Rodzik ([irodzik](https://github.com/irodzik)) for seeding the project, with a full contributor list at [https://github.com/open-policy-agent/opa-idea-plugin/graphs/contributors](https://github.com/open-policy-agent/opa-idea-plugin/graphs/contributors). The plugin draws inspiration from the [IntelliJ Rust](https://github.com/intellij-rust/intellij-rust) project's reference implementation.
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
---
|
||||
title: "Open Policy Agent graduates in the Cloud Native Computing Foundation"
|
||||
authors: ["tsandall", "timhinrichs"]
|
||||
date: 2021-02-04
|
||||
slug: open-policy-agent-graduates-in-the-cloud-native-computing-foundation-f00145202a99
|
||||
---
|
||||
|
||||
|
||||
|
||||
We're excited to announce that Open Policy Agent (OPA) is now a graduated project in the Cloud Native Computing Foundation (CNCF)! Graduation reflects the maturity of the project in terms of adoption, diversity of contributions, community and overall quality.
|
||||
|
||||
The growth over the last year has been phenomenal. The number of users on slack.openpolicyagent.org has grown by 3x (to over 3,600 users) and the number of Docker image downloads surpassed 39M (a 1000% increase!) We attribute much of this growth to the need for a robust policy-as-code solution in the cloud native ecosystem.
|
||||
|
||||
In the last year, the project shipped a number of powerful new features such as signed bundles, improved data fetching capabilities, parser and evaluator optimizations, as well as expanded support for WebAssembly-based execution environments. [OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper) reached GA and added several features including multi-pod scalability, semantic logging, fine-grained metrics, dry-run support, and more. The popular [Conftest](https://github.com/open-policy-agent/conftest) tool (which helps you write tests against structured configuration files) became an official OPA subproject. Finally, support for IntelliJ users landed with the [OPA IDEA](https://github.com/open-policy-agent/opa-idea-plugin) plugin.
|
||||
|
||||
Graduation is a huge milestone for the project, and we wanted to take a moment to thank everyone involved in making OPA a successful, graduated project:
|
||||
|
||||
- First, we'd like to thank the CNCF for their partnership and for all their support over the years. We would particularly like to thank Chris Aniszczyk, Amye Scavarda Perrin, Ihor Dvoretskyi for the excellent support they provided along the way.
|
||||
- We'd also like to thank all of the maintainers and core contributors to the OPA project: Ash Narkar, Boran Seref, Craig Tabita, Gareth Rushgrove, John Reese, Lennard Eijsackers, Max Smythe, Oren Shomron, Patrick East, Rita Zhang, Sertaç Özercan, and Stephan Renatus.
|
||||
- Lastly, we'd like to send a HUGE thank you to the entire OPA community. All new OPA features and partnerships were driven by your feedback and contributions. We also want to recognize certain people whose feedback, contribution, and support has been invaluable: Anders Eknert, Jasper Van der Jeugt, Joe Searcy, and Vincent Gramer.
|
||||
|
||||
Going forward we'll continue to focus on improving all aspects of OPA while solving real problems around policy and authorization in the cloud native ecosystem. We look forward to continuing to work with the amazing OPA community that started nearly five years ago. But, for today, let's all celebrate (socially distanced, of course) together on this great milestone!
|
||||
+109
@@ -0,0 +1,109 @@
|
||||
---
|
||||
title: "Type checking your Rego policies with JSON schema in OPA"
|
||||
authors: ["aavarghese", "mandanavaziri", "tsandall"]
|
||||
date: 2021-03-16
|
||||
slug: type-checking-your-rego-policies-with-json-schema-in-opa-5f7ac4c8a958
|
||||
---
|
||||
|
||||
|
||||
|
||||
_**IBM Research & Styra**_
|
||||
|
||||
The Open Policy Agent (OPA) is an open-source engine that unifies policy enforcement across the cloud native stack. It provides a query language called Rego that lets the user specify policy as code and an engine that evaluates the queries given input data.
|
||||
|
||||
Rego is a powerful declarative language that does not require the user to specify a query strategy. This is achieved by the OPA runtime, leaving users free to reason about policies at a higher level.
|
||||
|
||||
Rego has a gradual type system meaning that types can be partially known statically. For example, an object could have certain fields whose types are known and others that are unknown statically. OPA type checks what it knows statically and leaves the unknown parts to be type checked at runtime. However, the input handed to OPA could by design be any JSON value and hence gradual type-checking has no way of catching policy authoring mistakes, even when the policy author knows the intended schema.
|
||||
|
||||
In this article, we introduce a new feature that enhances OPA's ability to statically type check Rego code by taking into account schemas for input documents. This improves programmer productivity and helps Rego programmers catch errors earlier.
|
||||
|
||||
To achieve this, we enable `opa eval` to take the schema for the input document, specified in JSON Schema format. The input schema is passed with the flag `--schema` (`-s`). Armed with this information, OPA's enhanced type checker can detect bugs stemming from incorrect usage of the input.
|
||||
|
||||
This feature is now available in OPA v0.27.0. Let's check it out!
|
||||
|
||||
|
||||
|
||||
_Rego type checking demo_
|
||||
|
||||
Consider the following Rego code, which assumes as input a Kubernetes admission review. For resources that are Pods, it checks that the image name starts with a specific prefix.
|
||||
|
||||
```rego
|
||||
package kubernetes.admission
|
||||
|
||||
|
||||
deny[msg] {
|
||||
input.request.kind.kinds == "Pod"
|
||||
image := input.request.object.spec.containers[_].image
|
||||
not startswith(image, "hooli.com/")
|
||||
msg := sprintf("image '%v' comes from untrusted registry", [image])
|
||||
}
|
||||
```
|
||||
|
||||
Notice that this code has a typo in it: `input.request.kind.kinds` is undefined and should have been `input.request.kind.kind`.
|
||||
|
||||
Consider the following input document:
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "AdmissionReview",
|
||||
"request": {
|
||||
"kind": {
|
||||
"kind": "Pod",
|
||||
"version": "v1"
|
||||
},
|
||||
"object": {
|
||||
"metadata": {
|
||||
"name": "myapp"
|
||||
},
|
||||
"spec": {
|
||||
"containers": [
|
||||
{ "image": "nginx", "name": "nginx-frontend" },
|
||||
{ "image": "mysql", "name": "mysql-backend" }
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
% opa eval -format pretty -i admission-review.json -d pod.rego
|
||||
[]
|
||||
```
|
||||
|
||||
The empty value returned is indistinguishable from a situation where the input did not violate the policy. This error is therefore causing the policy not to catch violating inputs appropriately.
|
||||
|
||||
If we fix the Rego code and change `input.request.kind.kinds` to `input.request.kind.kind`, then we obtain the expected result:
|
||||
|
||||
```
|
||||
[
|
||||
"image 'nginx' comes from untrusted registry"
|
||||
"image 'mysql' comes from untrusted registry"
|
||||
]
|
||||
```
|
||||
|
||||
With this feature, it is possible to pass a schema to `opa eval`, written in JSON Schema.
|
||||
|
||||
Consider the [Kubernetes admission review input schema](https://github.com/aavarghese/opa-schema-examples/blob/main/kubernetes/admission-schema.json). We can pass this schema to the evaluator as follows:
|
||||
|
||||
```bash
|
||||
% opa eval -format pretty -i admission-review.json -d pod.rego -s admission-schema.json
|
||||
```
|
||||
|
||||
With the erroneous Rego code, we now obtain the following type error:
|
||||
|
||||
```
|
||||
1 error occurred: pod.rego:5: rego_type_error: undefined ref: input.request.kind.kinds
|
||||
input.request.kind.kinds
|
||||
^
|
||||
have: "kinds"
|
||||
want (one of): ["kind" "version"]
|
||||
```
|
||||
|
||||
This indicates the error to the Rego developer right away, without having the need to observe the results of runs on actual data, thereby improving productivity.
|
||||
|
||||
With this new feature, Rego developers will be able to provide JSON Schemas for their input documents and get the most out of static type checking. If you don't have a JSON Schema handy, you can easily obtain one from a sample JSON file using online tools (see links below). In the future, we will add to this feature the support to allow users to specify a directory of schemas and assign schemas to data documents, as well. This will be done via annotations (global, rule-level, rule output) that will indicate what schema is associated with what Rego path expression.
|
||||
|
||||
Happy Rego type checking!
|
||||
|
||||
For more information and limitations, see [documentation](https://www.openpolicyagent.org/docs/policy-language#schema) and [examples](https://github.com/aavarghese/opa-schema-examples/).
|
||||
+188
@@ -0,0 +1,188 @@
|
||||
---
|
||||
title: "Enhanced Type Checking for OPA with JSON Schema Annotations"
|
||||
authors: ["mandanavaziri", "aavarghese", "tsandall"]
|
||||
date: 2021-04-27
|
||||
slug: enhanced-type-checking-for-opa-with-json-schema-annotations-826acb0f575
|
||||
---
|
||||
|
||||
|
||||
|
||||
_**IBM Research & Styra**_
|
||||
|
||||
## What's happened?
|
||||
|
||||
In a [previous Medium blog](/blog/type-checking-your-rego-policies-with-json-schema-in-opa-5f7ac4c8a958), a feature released in [OPA v0.27.0](https://github.com/open-policy-agent/opa/releases/tag/v0.27.0) was introduced that lets OPA's static type checker take JSON schemas for input documents into account, improving how errors from misused data are caught during policy authoring.
|
||||
|
||||
The article explains that `opa eval` can take a schema for the input document via the `--schema(-s)` flag, applied globally across the module, allowing OPA's checker to catch issues like undefined objects.
|
||||
|
||||
## What's new?
|
||||
|
||||
This section covers extending type checking to support **multiple JSON schema files** for both input and data documents.
|
||||
|
||||
Example Rego code (based on a Kubernetes admission review input) is shown with a typo bug:
|
||||
|
||||
```rego
|
||||
package kubernetes.admission
|
||||
|
||||
deny[msg] {
|
||||
input.request.kind.kinds == "Pod" # This line has a typo, should be input.request.kind.kind
|
||||
image := input.request.object.spec.containers[_].image
|
||||
not startswith(image, "hooli.com/")
|
||||
msg := sprintf("image '%v' comes from untrusted registry", [image])
|
||||
}
|
||||
```
|
||||
|
||||
```rego
|
||||
input.request.kind.kinds
|
||||
```
|
||||
|
||||
which should be:
|
||||
|
||||
```rego
|
||||
input.request.kind.kind
|
||||
```
|
||||
|
||||
Running:
|
||||
|
||||
```bash
|
||||
% opa eval --format pretty -i admission-input.json -d policy.rego -s schemas/admission-input.json
|
||||
```
|
||||
|
||||
returns:
|
||||
|
||||
```
|
||||
1 error occurred: policy.rego:4: rego_type_error: undefined ref: input.request.kind.kinds
|
||||
input.request.kind.kinds
|
||||
^
|
||||
have: "kinds"
|
||||
want (one of): ["kind" "version"]
|
||||
```
|
||||
|
||||
The article notes that a similar typo in `input.request.object.spec.containers[_].image` would go undetected, since the admission review schema leaves `input.request.object` generically typed.
|
||||
|
||||
To solve this, `opa eval` now supports a **directory of schema files** via the same `--schema (-s)` flag, while still supporting single schema files. New Rego **Metadata** blocks allow specifying **schema annotations and scope**, improving bug detection for undefined fields.
|
||||
|
||||
An **override feature** is also introduced, described as letting users merge existing schemas and subschemas "for more precise type checking."
|
||||
|
||||
Additionally, schema loading is enabled for `opa eval — bundle`, supporting type checking of data documents across a bundle — useful for "batch type analysis of Rego policies as part of any CI/CD pipelines."
|
||||
|
||||
These features are available in [**OPA v0.28.0**](https://github.com/open-policy-agent/opa/releases/tag/v0.28.0).
|
||||
|
||||
## What's the big deal you say?
|
||||
|
||||
Example policies and schemas are available in the [opa-schema-examples repository](https://github.com/aavarghese/opa-schema-examples).
|
||||
|
||||
The Kubernetes Admission Review example is revisited to demonstrate annotations and schema overriding. The `object` field in an Admission Review can contain any Kubernetes resource, and its [schema](https://github.com/aavarghese/opa-schema-examples/blob/main/kubernetes/schemas/input.json) leaves that field generically typed.
|
||||
|
||||
Annotations associate a Rego expression with an input or data schema loaded via `opa eval -s`, within a given scope.
|
||||
|
||||
Annotations use `METADATA` comment blocks in YAML syntax, where "every line in the block must start at Column 1."
|
||||
|
||||
Example schema directory structure:
|
||||
|
||||
```
|
||||
mySchemasDir/
|
||||
├── input.json
|
||||
└── kubernetes
|
||||
└──────pod.json
|
||||
```
|
||||
|
||||
Loading the schema directory can be done via:
|
||||
|
||||
```bash
|
||||
% opa eval data.kubernetes.admission --format pretty -i opa-schema-examples/kubernetes/input.json -d opa-schema-examples/kubernetes/policy.rego -s opa-schema-examples/kubernetes/mySchemasDir
|
||||
```
|
||||
|
||||
```bash
|
||||
% opa eval data.kubernetes.admission -format pretty -i opa-schema-examples/kubernetes/input.json -b opa-schema-examples/bundle.tar.gz -s opa-schema-examples/kubernetes/mySchemasDir
|
||||
```
|
||||
|
||||
In this example, `input` is associated with the Admission Review schema (`input.json`), and `input.request.object` is set to the Kubernetes Pod schema (`pod.json`), with the second annotation overriding the first. The order of annotations is stated to matter "for overriding to work correctly."
|
||||
|
||||
Notes on schema reference syntax:
|
||||
|
||||
- Relative paths inside `mySchemasDir` are used, omitting the `.json` suffix
|
||||
- The global variable `schema` represents the top level of the directory
|
||||
- `schema.input` is valid; `schema.pod-schema` is invalid due to the hyphen — the correct syntax is `schema["pod-schema"]`
|
||||
|
||||
Combining annotations and overriding allows catching type errors in `input.request.object.spec.containers[_].image`:
|
||||
|
||||
```rego
|
||||
package kubernetes.admission
|
||||
|
||||
# METADATA
|
||||
# scope: rule
|
||||
# schemas:
|
||||
# - input: schema["input"]
|
||||
# - input.request.object: schema.kubernetes["pod"]
|
||||
deny[msg] {
|
||||
input.request.kind.kinds == "Pod" # This line has a typo, should be input.request.kind.kind
|
||||
image := input.request.object.spec.containers[_].images # This line has a typo, should be input.request.object.spec.containers[_].image
|
||||
not startswith(image, "hooli.com/")
|
||||
msg := sprintf("image '%v' comes from untrusted registry", [image])
|
||||
}
|
||||
```
|
||||
|
||||
```
|
||||
2 errors occurred:
|
||||
policy.rego:9: rego_type_error: undefined ref: input.request.kind.kinds
|
||||
input.request.kind.kinds
|
||||
^ have: "kinds"
|
||||
want (one of): ["kind" "version"]
|
||||
policy.rego:10: rego_type_error: undefined ref: input.request.object.spec.containers[_].images
|
||||
input.request.object.spec.containers[_].images
|
||||
^ have: "images"
|
||||
want (one of): ["args" "command" "env" "envFrom" "image" "imagePullPolicy" "lifecycle" "livenessProbe" "name" "ports" "readinessProbe" "resources" "securityContext" "stdin" "stdinOnce" "terminationMessagePath" "terminationMessagePolicy" "tty" "volumeDevices" "volumeMounts" "workingDir"]
|
||||
```
|
||||
|
||||
A second example checks whether an operation is allowed for a user, given an ACL data document. In the first `allow` rule, `input` uses `input.json` and `data.acl` uses `acl-schema.json`; an invalid expression like `data.acl.typo` would trigger a type error.
|
||||
|
||||
```rego
|
||||
package policy
|
||||
|
||||
import data.acl
|
||||
|
||||
default allow = false
|
||||
|
||||
# METADATA
|
||||
# scope: rule
|
||||
# schemas:
|
||||
# - input: schema["input"]
|
||||
# - data.acl: schema["acl-schema"]
|
||||
allow {
|
||||
access = data.acl["alice"]
|
||||
access[_] == input.operation
|
||||
}
|
||||
|
||||
allow {
|
||||
access = data.acl["bob"]
|
||||
access[_] == input.operation
|
||||
}
|
||||
```
|
||||
|
||||
The article clarifies that this annotation "does not constrain other paths under data" — only the type of `data.acl` is statically known.
|
||||
|
||||
The second `allow` rule in the same example has no schema annotations, so it isn't type-checked against any loaded schema. Different rules in the same module can use different input schemas.
|
||||
|
||||
Annotations can also apply at different scopes via the `scope` field in Metadata, defaulting to the following statement if omitted. Supported scope values:
|
||||
|
||||
- **rule** - applies to the individual rule statement
|
||||
- **document** - applies to all of the rules with the same name in the same package
|
||||
- **package** - applies to all of the rules in the package
|
||||
- **subpackages** - applies to all of the rules in the package and all subpackages (recursively)
|
||||
|
||||
More details: [Annotation scopes documentation](https://www.openpolicyagent.org/docs/latest/schemas/#annotation-scopes)
|
||||
|
||||
## What's Next?
|
||||
|
||||
The article closes by noting future plans to extend schema support to additional JSON Schema features such as `additionalProperties`, with more updates promised in upcoming OPA releases.
|
||||
|
||||
Further reading: [OPA schemas documentation](https://www.openpolicyagent.org/docs/latest/schemas/)
|
||||
|
||||
## Links
|
||||
|
||||
- Documentation: [OPA schemas documentation](https://www.openpolicyagent.org/docs/latest/schemas/)
|
||||
- Examples: [opa-schema-examples repository](https://github.com/aavarghese/opa-schema-examples/)
|
||||
- JSON to JSON schema online tool: [jsonschema.net](https://jsonschema.net/)
|
||||
- JSON schema reference: [Understanding JSON Schema reference](http://json-schema.org/understanding-json-schema/reference/index.html)
|
||||
- Related blog: /blog/type-checking-your-rego-policies-with-json-schema-in-opa-5f7ac4c8a958
|
||||
@@ -0,0 +1,114 @@
|
||||
---
|
||||
title: "Community Spotlight — Grant Shively"
|
||||
authors: ["anderseknert"]
|
||||
date: 2021-05-05
|
||||
slug: community-spotlight-grant-shively-12903674c28c
|
||||
---
|
||||
|
||||
|
||||
|
||||
"Community Spotlight" is a new series of blogs where we talk to people in and around the OPA community: users, integrators and contributors. Our first "spotlight" is [Grant Shively](https://www.linkedin.com/in/grant-shively/), principal software engineer at GoDaddy.
|
||||
|
||||
**Okay, so let's start with an introduction.**
|
||||
|
||||
Sure! My name is Grant Shively and I've been working at GoDaddy for the last 13 years. For the last six years or so I've been a principal engineer on what we call our Care Platform, which is essentially our CRM system.
|
||||
|
||||
On the infrastructure side, we've moved from bare metal legacy servers with big monolithic ASP.NET web applications to a cloud-native, microservice architecture. Meanwhile we've transitioned much of the application platform towards Node.js and .NET Core. Recently, I've been doing a lot of work migrating things from our internal cloud to AWS.
|
||||
|
||||
For many years I've been involved in projects where we've needed complex authorization policies. And so we've built all these systems to deal with that, but there's never been a company-wide solution for that kind of thing.
|
||||
|
||||
I've been pushing for an authorization platform for the company for a few years, and last year it finally gained traction. We completed two company-wide initiatives with leaders from each of the major organizations where we all got together to talk about how we wanted to solve for authorization holistically at GoDaddy.
|
||||
|
||||
During the initiatives, we did a buy vs. build analysis where we evaluated different vendors, and open source projects. And that's when I discovered OPA. I watched the [Netflix talk](https://www.youtube.com/watch?v=R6tUNpRpdnY), which showed how they used OPA to solve many of the issues we are currently experiencing.
|
||||
|
||||
And so we did a proof of concept at the end of last year to prove the value and efficacy of the project using OPA, and we got buy-in on it, and now we're here, building out the final multi-tenant solution! We are hoping to have the next team running in production by the end of Q2, and we already have a number of teams lined up to onboard after that.
|
||||
|
||||
**That was a great introduction! So, OPA adoption starting from a platform team, and growing from there?**
|
||||
|
||||
I've definitely evangelized OPA within the company, since I'm a big fan of the project. There's someone from another team helping to contribute to this platform we're building so I guess that makes us two teams at the moment. My team is focused on the systems that support our customer service guides, while the other team is working on our domain control center. Initially, they're looking to implement this authorization platform for some of the high risk scenarios we have around domains, such as transfer of ownership and things like that.
|
||||
|
||||
**You said you were working mainly with .NET and Node.js?**
|
||||
|
||||
Yep. As a company we work in a number of languages, including Python, Go, Node.js, .NET Core, and Java… I think those are most of the blessed languages, but there are probably some other languages used for one-offs as well.
|
||||
|
||||
My team, we were traditionally C#, and then we brought in Node.js during our transition to microservices. With the arrival of .NET Core, we were comfortable with continuing to support C# as well. Personally, after working with both languages for a number of years, I prefer Node.js for implementing our small, single-purpose APIs.
|
||||
|
||||
**You've made quite a few contributions to OPA, which is written in Go. Did you have experience working with that before?**
|
||||
|
||||
No, I hadn't touched it before! Go is definitely approachable. And I think the OPA code base is very clean and logical to explore. You don't have to look too much to figure out where something's at, once you get used to it.
|
||||
|
||||
Go feels like a language that minimizes syntactic sugar. There's really only a couple of "right" ways to do things, I feel. Sometimes it's frustrating too, like with the lack of map/filter/reduce operations. Coming from more functional paradigms it feels frustrating at times.
|
||||
|
||||
**Agreed. In this process of finding OPA, what other alternatives did you consider? Building your own? Some commercial options?**
|
||||
|
||||
We looked at [Athenz](https://github.com/AthenZ/athenz). One of our senior architects had worked on that project and it looked interesting. It did some of the stuff we wanted, but not all. And then we looked at a whole bunch of products from various vendors. We briefly thought about building our own until we ran into OPA. OPA completely negated any reason for us to try and build our own because it did exactly what we needed, and in such an elegant way that it would be very difficult for us to replicate.
|
||||
|
||||
**Compared to the other options you considered, what was the main appeal of OPA? What was it that won you over?**
|
||||
|
||||
One of the things we needed was the ability for multiple engineering teams to work with policy, in a self-service manner, while at the same time leveraging globally-managed policy and data, like authentication token expiration rules and identity attributes.
|
||||
|
||||
Like everybody else, we have a few home-grown systems, and we definitely needed to be able to integrate with those. Also, we have an extremely distributed architecture. Hundreds of AWS accounts, and whatever solution we found was going to need to run in pretty much all of them.
|
||||
|
||||
I knew that there were going to be integration points that would be very difficult for most vendor products. One thing we really didn't want was some sort of centralized authorization API for the whole company. We've been bitten by such architectures in the past. Our apps are highly distributed, and we need our authorization decision points to be highly distributed too, while still being able to manage and distribute policy from a central location. And a lot of the bigger vendors had these products that just felt a little too heavy for large-scale decentralization, you know?.
|
||||
|
||||
**Right.**
|
||||
|
||||
And we really liked Rego. We've talked about this before, but a lot of the vendors are based on [XACML](https://en.wikipedia.org/wiki/XACML), which is just really tedious to work with. Compared to that, Rego was a breath of fresh air. Another factor we considered was the buy-in we saw from some of the bigger companies like Netflix, etc. Rapid adoption of a project is usually a great sign.
|
||||
|
||||
I'm sure I'll have more opinions on Rego once we try and onboard more teams onto our platform. Certain teams at GoDaddy have more operational experience than others, and we'd really like to build a simplified UI policy builder on top of our platform and Rego. That way, they can just fill out some simple input boxes, hit enter, and it publishes a policy for them. That kind of thing.
|
||||
|
||||
**Interesting!**
|
||||
|
||||
Yeah, I'm both looking and not looking forward to tackling that problem, haha!
|
||||
|
||||
**Maybe you don't remember anymore, but did you have any such moments where you got stuck while learning Rego? I know one thing that felt a little odd to me when starting out was how Rego handled undefined values. How rule evaluation just stops when they are encountered.**
|
||||
|
||||
Some of the syntax around iteration was a little confusing at first. I think it's one of those things that makes total sense once you understand it, but it wasn't like anything I'd worked with before.
|
||||
|
||||
The biggest issue though — and we still kind of have it—is understanding how a decision was made, and how to surface that in logs. We have a couple of [open](https://github.com/open-policy-agent/opa/issues/2755) [issues](https://github.com/open-policy-agent/opa/issues/2089) about that, and we have been running OPA internally with a few patches we want to upstream that help address some of this.
|
||||
|
||||
It's really important to our security and engineering teams that we can look at a somewhat human readable, but hopefully not super verbose, way of saying that "this authorization request was approved or denied because of these reasons".
|
||||
|
||||
Another related issue would be how to best propagate obligations up through layers of policy. Some obligations can come from a really low level, like policy checking the claims of a JWT, and if some property doesn't exist, then we want to propagate an obligation up from that lower level rule. Solving some of those problems has felt a little clunky.
|
||||
|
||||
**On the other side, what did you really like?**
|
||||
|
||||
I really liked that you could create custom built-in functions and other things to extend OPAs functionality. That has been really useful to us as we've built things around our requirements, like custom key signing, custom decision logging, and so on.
|
||||
|
||||
There have been a few cases where we needed to extend portions of the OPA code, only to find that they were hardcoded around a particular implementation, without interfaces for us to leverage. I've been very pleased with the responsiveness of and guidance from the OPA maintainers. We've been able to contribute a number of small changes to make the OPA code more flexible for our extensions.
|
||||
|
||||
**On the topic of flexibility, one of the things that really appealed to me was the number of [integrations](https://www.openpolicyagent.org/docs/latest/ecosystem/) available. Even if you're never going to use all of them, it really shows what the general purpose nature of OPA enables.**
|
||||
|
||||
Yeah, that brings to mind another consideration. Netflix talked about how they used OPA for authorization across their entire stack—infrastructure, forward facing code, back end. When I saw OPA, I knew our platform could grow to be much more than just, you know, a simple authorization engine—there are many other policy-based scenarios that don't necessarily have to do with authorization. So that may be a far future thing, but that was also very enticing about OPA.
|
||||
|
||||
**I think some of the flexibility really pays off at scale. Having one unified language to describe policy across the stack, having one place to go for decision logs, and so on.**
|
||||
|
||||
Yep. We're definitely looking at increasing the number of integrations. So many of the systems deployed today work only with Active Directory, and AD group-based authorization, so it'll be interesting to see how we can integrate with systems like that.
|
||||
|
||||
**About integrations. Are there any of the existing ones in the [ecosystem](https://www.openpolicyagent.org/docs/latest/ecosystem/) you are using or are planning to use?**
|
||||
|
||||
I'm only familiar with the [Envoy one](https://github.com/open-policy-agent/opa-envoy-plugin). Oh, and Kubernetes. What else is there? We're just starting to use Envoy, so I think there will likely be some places where we use that integration. And, we'll probably want to look into a Kubernetes integration too at some point, if [AWS EKS](https://aws.amazon.com/eks/) supports that.
|
||||
|
||||
Looking at the list now… Wow, there's really a bunch of them here! I'll need to look into some of these. What we're really doing a lot of is AWS integrations. Kind of in the same vein as the Envoy integration, extending OPA and all that, you know? We're doing the same thing with OPA and AWS, and I hope we'll be able to eventually open source that.
|
||||
|
||||
Things like key signing using the [AWS Key Management Service](https://aws.amazon.com/kms/) (KMS) or shipping decision logs directly to [AWS Kinesis](https://aws.amazon.com/kinesis/) without having to go through HTTP endpoints in between.
|
||||
|
||||
**I know you've mentioned [AWS Lambda](https://aws.amazon.com/lambda/) functions too.**
|
||||
|
||||
Yeah. A lot of our APIs are lambda-based, and we'll obviously want to use this platform we're building for that too. We're currently trying to figure out how to make OPA work as well for on-demand serverless functions as it does for traditional compute environments.
|
||||
|
||||
**Definitely a hot topic! I've seen some questions on that on the OPA Slack as well. What are the challenges there?**
|
||||
|
||||
So, the way OPA currently works is that many of the internal "plugins", like the client downloading bundles, or the one uploading decision logs-they all work on a time-based loop. So you configure them to upload or download or to do whatever they are meant to do at a certain interval, like every thirty seconds or something.
|
||||
|
||||
This doesn't really work for serverless functions though as they don't have continuous compute available; functions freeze after serving a response, so nothing can run in the background. What we'd need in this context is rather something based on [other types of triggers](https://github.com/open-policy-agent/opa/issues/2899). We're waiting for a feature called AWS Lambda Extensions, which should reach general availability in May. They should make it possible to use OPA really efficiently even in a lambda context.
|
||||
|
||||
**Anything you have found missing from OPA or would like to see on the [roadmap](https://docs.google.com/presentation/u/1/d/16QV6gvLDOV3I0_guPC3_19g6jHkEg3X9xqMYgtoCKrs/edit)?**
|
||||
|
||||
I mentioned it before, but yeah, the big thing would be figuring out how to do the tracing in a more efficient way. I have come to understand that the way "full" tracing is done today is apparently pretty expensive. We just need something like [rule level tracing](https://github.com/open-policy-agent/opa/issues/2089), which has been discussed a bit in GitHub issues in the past. With that in OPA proper we wouldn't have a reason to run our own modifications at all, except for plugins.
|
||||
|
||||
**Awesome! Finally, what are your future plans for OPA at GoDaddy?**
|
||||
|
||||
What we're starting out with is authorization for internal systems. Once we nail that and it's working well we're looking to expand it to authorization for customer systems, and at some point possibly our infrastructure too.
|
||||
|
||||
With what OPA can do, I feel like the sky's the limit in terms of what you want to throw into it. We just have to make sure we fully support the platform we're building as we continue growing. I have high hopes for it!
|
||||
@@ -0,0 +1,99 @@
|
||||
---
|
||||
title: "OPA Slack Tune Up"
|
||||
authors: ["peteroneilljr"]
|
||||
date: 2021-08-19
|
||||
slug: opa-slack-tune-up-b3c52492e2fc
|
||||
description: Celebrating the growth of OPA community with a little cleanup.
|
||||
---
|
||||
|
||||
|
||||
|
||||
## Celebrating the growth of OPA community with a little cleanup
|
||||
|
||||
The OPA community now has over 4,600 members in Slack! This is a tremendous milestone and we are so excited to have all of the new members join us. With this explosion of new members, the total number of Slack channels has crept up on us. The OPA team has noticed having so many channels has created confusion for our new and existing members on where to post about specific topics or themes. To make Slack easier to navigate we are rolling out a new set of channel names and descriptions. We hope this new structure will make Slack a bit easier for everyone. However, if you have a suggestion on how to make it better we would love to know.
|
||||
|
||||
## Primary / Default Channels
|
||||
|
||||
Any new members joining the OPA community will want to hang out here to start! Read about the latest announcements, introduce yourself in the community chit-chat and ask questions in the help channel.
|
||||
|
||||
### #announcements
|
||||
|
||||
Previous channel name: #general
|
||||
|
||||
The general channel is now the announcements channel. General was our busiest channel by far! It is now our announcements channel so that big news like releases and community events can stick around longer. New channels have been created for chit-chat and general help questions. The announcements channel is still open for anyone to post, but now think about posting things you want the entire community to know.
|
||||
|
||||
### #chit-chat
|
||||
|
||||
Previous channel name: #random
|
||||
|
||||
Everyone loves a good random channel and the OPA community is no different. Just because we changed the name you shouldn't feel the need to change what you're posting. Drop in your OPA memes and funny web links just like before. But now we also want to include community introductions and general chatter here as well.
|
||||
|
||||
### #help
|
||||
|
||||
Previous channel names:
|
||||
|
||||
- #rego
|
||||
- #openpolicyagent
|
||||
- #questions_and_answers
|
||||
- #feasibility-question
|
||||
|
||||
We want to make it as easy as possible for you to find help while learning about OPA. So we've combined the channels we noticed new members were looking for help into a single channel. We hope that this will make it easier for everyone to know where to go when they need help and where to go when they feel like helping others.
|
||||
|
||||
## Integrations
|
||||
|
||||
For OPA users that have been around for a while you are probably well aware of the Conftest and Gatekeeper projects. These two OPA projects have gained a lot of traction and provide amazing contributions to the OPA community. If you have specific questions about the projects these channels are the best place to go. While the maintainers do hang around these channels, we love seeing community members showing off their OPA knowledge answering questions for each other.
|
||||
|
||||
### #conftest
|
||||
|
||||
Conftest is a must have in your policy toolkit. Write tests against structured configuration files including JSON, YAML, XML, Dockerfile, HCL, and more.
|
||||
|
||||
### #gatekeeper
|
||||
|
||||
Gatekeeper helps you safeguard your Kubernetes clusters by defining OPA-based admission control policies that are enforced via webhooks. Gatekeeper also helps you audit your Kubernetes clusters to detect policy violations.
|
||||
|
||||
## Topics
|
||||
|
||||
Trimmed down from the myriad of channels that existed before, the OPA team has chosen 3 channels that contained the most buzz from the community. Terraform, Envoy and WebAssembly are the 3 topics we noticed everyone likes to chat about. We hope that this buzz continues to grow. Also be on the lookout for programs in the future to be recognized as OPA experts in these areas.
|
||||
|
||||
### #terraform
|
||||
|
||||
[Terraform](https://www.terraform.io/) lets you describe the infrastructure you want and automatically creates, deletes, and modifies your existing infrastructure to match. OPA makes it possible to write policies that test the changes Terraform is about to make before it makes them.
|
||||
|
||||
### #envoy
|
||||
|
||||
[OPA-Envoy](https://github.com/open-policy-agent/opa-envoy-plugin) plugin extends OPA with a gRPC server that implements the [Envoy External Authorization API](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ext_authz_filter.html). You can use this version of OPA to enforce fine-grained, context-aware access control policies with Envoy without modifying your microservices.
|
||||
|
||||
### #wasm
|
||||
|
||||
OPA is able to compile Rego policies into executable Wasm modules that can be evaluated with different inputs and external data. This is not running the OPA server in Wasm, nor is this just cross-compiled Golang code. The compiled Wasm module is a planned evaluation path for the source policy and query.
|
||||
|
||||
## Archived Channels
|
||||
|
||||
- #intros
|
||||
- #bosun
|
||||
- #feedback
|
||||
- #registry
|
||||
- #intellij-extension
|
||||
- #gsoc19
|
||||
|
||||
You may notice some of the lesser used channels have been archived, we picked these channels based on a number of factors such as frequency of posts and average rate of responses. Ultimately we feel consolidating these conversations into the primary channel #chit-chat will increase participation and response rates.
|
||||
|
||||
## Bot Channels
|
||||
|
||||
- #bot-github
|
||||
- #bot-rss
|
||||
|
||||
These are not the bots you're looking for…or maybe they are! Going forward any channels with the `bot-` prefix will be used for channels that include Slack bots or automated tools. Currently, the OPA team uses these channels to keep up to date with external sources like Stack Overflow, Reddit, and GitHub.
|
||||
|
||||
## Wrapping Up
|
||||
|
||||
Whether you're an OPA power user or looking to write your first Rego policy, we want the OPA Slack community to be your home for all things policy related. The OPA team realizes that sometimes new ideas need their own space to flourish. If you're interested in creating a new channel, reach out to @peteroneilljr or @tsandall on Slack and join us in our mission to solve policy enforcement across the stack. In addition to these Slack updates, you should also be on the look out for our new [GitHub Discussions](https://docs.github.com/en/discussions) forum that will officially launch in the next couple of weeks. For a sneak peak check out the link at the bottom of this article!
|
||||
|
||||
|
||||
|
||||
Join the community on:
|
||||
|
||||
- [Slack](https://slack.openpolicyagent.org/)
|
||||
- [Twitter](https://twitter.com/OpenPolicyAgent)
|
||||
- [GitHub](https://github.com/open-policy-agent)
|
||||
- [Discussions](https://github.com/open-policy-agent/feedback/discussions) (Launching Soon!!)
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
title: "Open Policy Agent 2021 Survey Summary"
|
||||
authors: ["tsandall"]
|
||||
date: 2021-08-31
|
||||
slug: open-policy-agent-2021-survey-summary-e749bbd7b824
|
||||
---
|
||||
|
||||
|
||||
|
||||
_…happy OPA 2021 survey from Cal [credit: @eileen_kemp]_
|
||||
|
||||
Last month we surveyed the OPA community to learn more about user adoption and help us plan and improve the project. We received over 300 responses from users across financial services, healthcare, public sector, automotive, cloud technology providers and more. This post highlights some of the survey results.
|
||||
|
||||
## Use Cases and Adoption
|
||||
|
||||
### OPA adoption driven by authorization use cases across the stack
|
||||
|
||||
Like last year, we used the survey to gauge use case adoption among respondents. We're interested in understanding where and why companies are deploying OPA because it helps us steer the project's long-term roadmap in the right direction. This year we asked respondents about the high-level goals they're trying to achieve by using OPA. We found that implementation of internal compliance and governance rules was the most common goal, however, nearly 60% of respondents indicated two or more goals being highly relevant.
|
||||
|
||||
| Goal | % of Respondents |
|
||||
| ------------------------------- | ---------------- |
|
||||
| Internal compliance/governance | 64% |
|
||||
| Operational excellence | 49% |
|
||||
| Implementing end-user IAM | 44% |
|
||||
| External compliance (e.g., PCI) | 28% |
|
||||
|
||||
In terms of use cases (e.g., Kubernetes admission control, Microservice authorization, etc.), the results were similar to the previous year with 50% of respondents indicating they use OPA for two or more use cases:
|
||||
|
||||
| # of Use Cases | % of Respondents |
|
||||
| -------------- | ---------------- |
|
||||
| 1 | 48% |
|
||||
| 2 | 34% |
|
||||
| 3 | 13% |
|
||||
| 4+ | 3% |
|
||||
|
||||
Kubernetes admission control continues to be the most common use case for OPA with 54% of respondents indicating they run OPA or OPA Gatekeeper to enforce various policies on their clusters:
|
||||
|
||||
| Use Case | % of Respondents |
|
||||
| ---------------------------- | ---------------- |
|
||||
| Kubernetes admission control | 54% |
|
||||
| Application authorization | 39% |
|
||||
| Microservice authorization | 39% |
|
||||
| Terraform validation | 25% |
|
||||
| Other | 5% |
|
||||
|
||||
### From experiments to production in 6 months (or less)
|
||||
|
||||
The survey showed the distribution of respondents OPA usage maturity was roughly equal:
|
||||
|
||||
| Stage | % of Respondents |
|
||||
| ------------------- | ---------------- |
|
||||
| Experimentation | 33% |
|
||||
| Pre-production & QA | 32% |
|
||||
| Production | 35% |
|
||||
|
||||
What was more interesting was that about half of respondents indicated they had only been using OPA since January 2021. Of those users, nearly 40% had already reached production. Furthermore, the survey results show that most respondents reached production within 6 months. Beyond that, the percentage of users that are still in experimental stages drops to single digits:
|
||||
|
||||
| | Experimentation | Pre-prod/QA | Production |
|
||||
| -------------- | --------------- | ----------- | ---------- |
|
||||
| < 3 months | 54% | 28% | 14% |
|
||||
| 3-6 months | 22% | 54% | 25% |
|
||||
| 6-12 months | 4% | 38% | 58% |
|
||||
| Over 12 months | 7% | 15% | 76% |
|
||||
|
||||
These results are encouraging and also give us high-level metrics to improve on — ideally the time to production with OPA will continue to decrease as we improve the user experience and harden the project.
|
||||
|
||||
The survey results also highlighted a range of deployment sizes for production users. The following chart breaks down the deployment size responses by use case:
|
||||
|
||||
| Use Case | <10 | 10-50 | 50-200 | >200 |
|
||||
| ---------------------------- | ------ | ----- | ------ | ------- |
|
||||
| Kubernetes admission control | 42% | 31% | 13% | 12% |
|
||||
| Terraform validation | 43% | 25% | 18% | 12% |
|
||||
| Microservice authorization | 37% | 37% | 12% | 11% |
|
||||
| Application authorization | 44% | 32% | 10% | 11% |
|
||||
|
||||
### Policy library adoption is growing
|
||||
|
||||
The survey asked users about various features in OPA and one of the most encouraging bits of information was that policy library adoption is growing within platform authorization use cases, like Kubernetes admission control and Terraform plan validation. Specifically, we found that nearly 60% of Kubernetes admission control users rely on the official [gatekeeper-library](https://github.com/open-policy-agent/gatekeeper-library) policies that implement various best practices as well as PSP. We also found that nearly 30% of users that run OPA to validate Terraform plans rely on various open source policy libraries.
|
||||
|
||||
## OPA Feedback
|
||||
|
||||
In addition to gauging adoption we also used the survey to solicit feedback about the project.
|
||||
|
||||
### Debugging needs some love
|
||||
|
||||
After poring over the feedback comments, we found that the most common area for improvement is _debugging_. As with all surveys, some comments were non-specific, however multiple respondents requested better [tracing modes](https://github.com/open-policy-agent/opa/issues/2089) and explanation presentation formats. Improved [debug output](https://github.com/open-policy-agent/opa/issues/3319) support was another common request, and respondents also mentioned a desire for an [interactive debugger](https://github.com/open-policy-agent/opa/issues/3191) similar to what you find in typical programming languages.
|
||||
|
||||
### SDKs for various languages
|
||||
|
||||
Aside from debugging, the next most common request was better SDK support for OPA in various languages. Several respondents indicated interest in [Wasm-based SDKs](https://www.openpolicyagent.org/docs/latest/wasm/) for OPA, and others requested regular SDKs for Java, NodeJS and other languages. One of the reasons we haven't developed SDKs for OPA yet is because the OPA API is _extremely simple_ (e.g., you can query OPA for decisions with a single HTTP POST request). However, with the Wasm compiler in OPA improving with every release, and the Wasm ecosystem growing rapidly, it feels like it's time to invest into language-specific integration libraries.
|
||||
|
||||
## Wrap Up
|
||||
|
||||
Thanks to everyone who completed the survey! The OPA t-shirts for completing the survey will be shipped soon. If you have not filled out the survey but would like to do so, you can still [complete the OPA survey](https://form.typeform.com/to/pL0jDuyT). As always, if you have questions or feedback, we're available on Slack, GitHub, etc.
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
---
|
||||
title: "Serverless Policy Enforcement: Connecting OPA and AWS Lambda"
|
||||
authors: ["gshively11"]
|
||||
date: 2021-10-05
|
||||
slug: serverless-policy-enforcement-connecting-opa-and-aws-lambda-e624f7176a3
|
||||
---
|
||||
|
||||
|
||||
|
||||
[Open Policy Agent](https://www.openpolicyagent.org/) (OPA) provides policy-based control for cloud native environments. It's commonly used alongside massive projects like Kubernetes and Envoy, and has dozens of other integrations and related projects in its [ecosystem](https://www.openpolicyagent.org/docs/latest/ecosystem/). Recent updates to the project aim to better integrate OPA with serverless architectures and other infrastructure with intermittent compute.
|
||||
|
||||
[AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) is one such serverless solution. When a Lambda function is invoked, its execution environment only stays up until the function responds, at which point the runtime is frozen and all active processes/threads are paused. They are resumed once the next invocation is received, and the cycle repeats. OPA's plugin architecture wasn't designed to handle this freezing process, which resulted in unexpected behavior for things like bundle retrieval and log shipping.
|
||||
|
||||
## Why is Manual Better than Automatic?
|
||||
|
||||
However, thanks to a great new feature released in Open Policy Agent [v0.32.0](https://github.com/open-policy-agent/opa/releases/tag/v0.32.0), plugins can now be triggered manually instead of automatically. Now, you may ask: "Wait, why is manual better than automatic?" Well, in many cases it's not, but for environments like AWS Lambda, it's not only useful, but critical to the stability and functionality of the OPA process.
|
||||
|
||||
First, let's briefly review the default behavior of plugins in OPA. Most of the standard plugins — Discovery, Bundles, and Decision Logs — operate by using a [loop](https://github.com/open-policy-agent/opa/blob/v0.32.0/plugins/logs/plugin.go#L671) running in a [goroutine](https://github.com/open-policy-agent/opa/blob/v0.32.0/plugins/logs/plugin.go#L513) that listens to various signals, one of which is a [timer](https://github.com/open-policy-agent/opa/blob/v0.32.0/plugins/logs/plugin.go#L698). When the timer delay elapses, the plugins will do stuff in the background, e.g. ship logs, check for new bundles to download, etc. OPA calls this a "periodic" trigger.
|
||||
|
||||
For most use cases, periodic triggers are exactly what we need. We don't really care precisely when a plugin is doing something in the background, just that it happens roughly within the interval period we've specified. The non-deterministic nature of periodic plugin triggers works well for a majority of applications. But what happens when that non-determinism becomes a problem? What if we need control over exactly when a plugin both starts and finishes a run of its primary task? Well, now we can get that control, thanks to the addition of [Manual Trigger Support](https://github.com/open-policy-agent/opa/pull/3668).
|
||||
|
||||
## Introducing Manual Trigger support
|
||||
|
||||
Plugins now support an optional "manual" trigger mode that can be set directly in a [plugin's configuration](https://www.openpolicyagent.org/docs/latest/configuration/#bundles). Additionally, if manual triggers are set on the Discovery plugin, all other plugins will inherit that setting. When a plugin's trigger is set to manual, the plugin's background loop will either pause until the [Trigger func](https://github.com/open-policy-agent/opa/blob/v0.32.0/plugins/logs/plugin.go#L639) is called, or it will never start, depending on the plugin. The Trigger func will run a plugin's primary task and return only when the task is complete, or the provided context ends.
|
||||
|
||||
Now that we understand what manual triggers do, let's look at how we can use them in AWS Lambda. Lambda's on-demand compute is excellent for saving on compute costs, but it presents a challenge for code that has indeterminate start and end points, i.e., OPA plugins with periodic triggers. We don't want to interrupt the OPA process by freezing the execution environment when it's in the middle of downloading a bundle or shipping logs. And we don't want to wait around for a background loop to kick off those behaviors. Manual triggers give us the deterministic start and end points we need to properly coordinate OPA plugins in Lambda.
|
||||
|
||||
## Try it for Yourself
|
||||
|
||||
For those that are interested in running OPA in AWS Lambda, GoDaddy has recently [open-sourced a small OPA plugin](https://github.com/godaddy/opa-lambda-extension-plugin) to make this easier, which uses manual triggers to operate OPA as a [Lambda Extension](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-extensions-api.html). Lambda Extensions are deeply integrated into the lifecycle of a Lambda function, allowing you to perform background tasks without impacting the response time of your functions. OPA and Lambda Extensions are a great match, allowing you to plug in OPA's policy enforcement to your serverless infrastructure without complex installation or configuration management.
|
||||
@@ -0,0 +1,152 @@
|
||||
---
|
||||
title: "Introducing the OPA print function"
|
||||
authors: ["anderseknert"]
|
||||
date: 2021-10-29
|
||||
slug: introducing-the-opa-print-function-809da6a13aee
|
||||
---
|
||||
|
||||
|
||||
|
||||
One of the key takeaways from the [Open Policy Agent 2021 Survey](/blog/open-policy-agent-2021-survey-summary-e749bbd7b824), was the need to improve the OPA debugging experience. Simply put, we need to make it easier to know what's going on when policies and rules are evaluated.
|
||||
|
||||
However, whenever someone talks about an "experience," it's rarely a small task and a checkbox to be checked once completed. Rather, it's all the little things that when combined provide a great improvement to the greater goal. If the OPA project used JIRA, it would probably be a safe bet to classify the "improve debugging experience" story as an "epic." With some improvements made, many new ideas and feature requests are likely to emerge along the way, and it would be rather optimistic to think that such a story ever got _done_, in the sense that no new improvements could be made.
|
||||
|
||||
To make things more complicated — and certainly more interesting — the OPA debugging experience isn't isolated to OPA itself. Improving the debugging experience for OPA entails not just looking at where things can be made better in OPA, but just as much in the tools commonly used when authoring Rego policies. These include tools like [VS Code](https://marketplace.visualstudio.com/items?itemName=tsandall.opa), [IntelliJ IDEA](https://plugins.jetbrains.com/plugin/14865-open-policy-agent) and all the [other editors](https://www.openpolicyagent.org/docs/latest/editor-and-ide-support/) commonly used for policy authoring.
|
||||
|
||||
So, where do we start?
|
||||
|
||||
## Debugging with OPA eval
|
||||
|
||||
Evaluating rules and variables has traditionally been done using the aptly named **opa eval** command. Commonly referred to as the "Swiss army knife of OPA," opa eval allows a policy author to quickly evaluate either standalone expressions like:
|
||||
|
||||
```bash
|
||||
opa eval --format raw 1+1
|
||||
```
|
||||
|
||||
Or, more commonly, with policy, data and input provided through command line arguments, and the path to the rule or variable of interest.
|
||||
|
||||
```bash
|
||||
opa eval --data policy.rego --input input.json data.policy.main
|
||||
```
|
||||
|
||||
While truly a versatile tool, debugging with **opa eval** has a couple of drawbacks. Having to create a new file to provide input might feel a little clunky, but hardly a terrible experience. But what if you want to evaluate the value of some variable inside of a rule, a test or a comprehension?
|
||||
|
||||
The [trace](https://www.openpolicyagent.org/docs/latest/policy-reference/#debugging) built-in has to some extent been used for this purpose, but requires additional parameters passed to OPA to actually print something, and while occasionally useful, it was often perceived as somewhat clunky for the purpose of simply printing something.
|
||||
|
||||
Another problem frequently mentioned in the context of debugging is how sometimes **opa eval**, or the **trace** built-in comes back with just… nothing.
|
||||
|
||||
This brings us right into another topic often considered tricky with regards to debugging — OPA's handling of undefined. When OPA encounters undefined values, policy evaluation normally halts. Considering how Rego is a declarative _query_ language, there isn't a whole lot more to do when a query comes back with nothing, just like a query language like SQL wouldn't have a whole lot more to do with an empty resultset. Since Rego rules are often compositions of many statements or other rules, it can sometimes be pretty difficult to tell where the undefined value that halted policy evaluation was introduced.
|
||||
|
||||
What to do?
|
||||
|
||||
## Introducing the print built-in
|
||||
|
||||
To tackle this, OPA v0.34.0 introduces a new **print** function to its ever growing list of [built-ins](https://www.openpolicyagent.org/docs/latest/policy-reference/#built-in-functions). The print function does exactly what you'd expect it to do — prints any provided values to the console. Consider a rule like the one below.
|
||||
|
||||
```rego
|
||||
allow {
|
||||
print("Entering allow")
|
||||
role := input.user.roles[_]
|
||||
print("Found role", role)
|
||||
role == "admin"
|
||||
}
|
||||
```
|
||||
|
||||
Running OPA eval would produce the following result.
|
||||
|
||||
```bash
|
||||
$ opa eval -f raw -d policy.rego -i input.json 'data.policy.allow'
|
||||
Entering allow
|
||||
Found role developer
|
||||
Found role sysadmin
|
||||
Found role dba
|
||||
Found role admin
|
||||
true
|
||||
```
|
||||
|
||||
The **print** function takes any number of arguments (static values, variables, **input**, **data**, etc) and prints each one (separated by whitespace) to the console.
|
||||
|
||||
While simple on the surface, a whole lot of [thought](https://github.com/open-policy-agent/opa/issues/3319) has been put into its design, and unlike other built-in functions (which are often trivial to add into OPA) the print function required changes to the internal compiler. How come?
|
||||
|
||||
## Varargs
|
||||
|
||||
One of the design goals of the new print function was to allow a variable number of values or variables (i.e. varargs) to be passed as arguments, without resorting to the use of an array for the arguments, as is done by **sprintf** and other built-ins. Simply put we wanted something intuitive like:
|
||||
|
||||
```rego
|
||||
print("x", input.x, "y", input.y)
|
||||
```
|
||||
|
||||
To work just as expected. Sounds easy, right? Well, not really.
|
||||
|
||||
One of the more obscure (and hence, not encouraged) features of Rego can be traced back to its Datalog roots. Any built-in function can have it's return value expressed as the last argument to the function. Meaning that:
|
||||
|
||||
```rego
|
||||
x := concat(".", ["a", "b", "c"])
|
||||
```
|
||||
|
||||
Could alternatively be written as:
|
||||
|
||||
```rego
|
||||
concat(".", ["a", "b", "c"], x)
|
||||
```
|
||||
|
||||
With the last argument "reserved" for the return value, how would an implementation of varargs work? The answer was a new type of void function, where there simply is no return value to take into account. Since Rego functions are generally free from side effects, a void type of function hasn't really made sense previously, but with **print** having no purpose other than the desired side effect of printing to the console, adding a void type made sense.
|
||||
|
||||
## Printing undefined
|
||||
|
||||
The next problem to tackle in order for print to work nicely as a debugging tool was how to deal with undefined. Since we can expect print to be used to debug variables from **input** and **data** that might not be defined, it would be kind of a bummer if calling print _itself_ halted policy evaluation! Ideally we'd be able to call the print function and have it print _something_ even if some of the arguments provided pointed at undefined values. That way we could use the function to try and help also with the problem of identifying where in a policy undefined values have been introduced.
|
||||
|
||||
This requirement meant some internal assumptions of how Rego is parsed had to change, and the end result is a print function that prints undefined values as `<undefined>`, without halting policy evaluation.
|
||||
|
||||
```rego
|
||||
allow {
|
||||
print(input.user.email, input.user.roles)
|
||||
input.user.roles[_] == "admin"
|
||||
endswith(input.user.email, "@acmecorp.com")
|
||||
}
|
||||
```
|
||||
|
||||
Evaluating the above allow rule with user.email missing from the input would now output something like this to the console:
|
||||
|
||||
```bash
|
||||
$ opa eval -f raw -d policy.rego -i input.json data.policy.allow
|
||||
<undefined> ["developer", "admin"]
|
||||
```
|
||||
|
||||
## Using print
|
||||
|
||||
OPA supports many different modes of operation, from **opa eval** and **opa test**, to the OPA REPL and of course running as a standalone server. Both **opa eval** and the REPL will always print to the console (stderr, specifically) as expected. When running as a server, OPA will print any output from print function calls at the **info** log level. This makes print useful for debugging at the default info level or below. When configured to run with log level **error** (the generally recommended log level for production), OPA erases any calls to print from policies as they are loaded. Print calls left in the policy at that point will thus not impact performance whatsoever.
|
||||
|
||||
When running **opa test**, the **print** function by default will print to the console on test failures. Should you want to print output also for successful tests, the — verbose (short form -v) will do the trick. One case I've found particularly useful in tests is to use print in combination with the with … as mocking construct, to quickly see what exactly the result of a rule evaluation returns, like:
|
||||
|
||||
```rego
|
||||
test_decison_allowed {
|
||||
result := decision with input as {
|
||||
"user": {
|
||||
"id": "abc123"
|
||||
},
|
||||
"request": {
|
||||
"method": "POST",
|
||||
"path": "/users"
|
||||
}
|
||||
}
|
||||
print(result)
|
||||
print(expectedResult)
|
||||
|
||||
result == expectedResult
|
||||
}
|
||||
|
||||
decision {
|
||||
[_, payload, _] := io.jwt.decode(input.user.token)
|
||||
print(payload)
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
While printing the outcome of rule evaluation in tests like this is valuable, the decision rule _itself_ might be composed of multiple rules, and being able to add a few print lines in those to understand why our result isn't what we expect can help us quickly pinpoint the problem.
|
||||
|
||||
## Wrapping up
|
||||
|
||||
Rego as a policy language isn't a general purpose programming language, and shouldn't be treated as one. However, making the OPA and Rego debugging experience as smooth as possible means we sometimes might need to adapt concepts familiar from the programming languages policy authors normally work with. That the **print** function [pull request](https://github.com/open-policy-agent/opa/pull/3868) added almost 2000 lines of code — half of them however from added test cases! — to the codebase is an interesting case study in how all the "little" details — like backwards compatibility, usability and performance — need to be considered when adding new functionality to a mature open source project like OPA.
|
||||
|
||||
I hope that you'll find the print function a useful addition to OPA, and a small improvement to the OPA/Rego debugging experience. Expect a lot more to come out in this space in future releases, and as always, make your voice heard in the [OPA Slack](https://slack.openpolicyagent.org/) if you have ideas, questions or feature requests you'd like to see incorporated into OPA!
|
||||
@@ -0,0 +1,129 @@
|
||||
---
|
||||
title: "OPA Newsletter: November 2021"
|
||||
sidebar_label: "November Newsletter"
|
||||
authors: ["peteroneilljr"]
|
||||
date: 2021-12-01
|
||||
slug: the-opa-monthly-newsletter-448a995a1f5e
|
||||
---
|
||||
|
||||
_November Edition_
|
||||
|
||||
## Intro
|
||||
|
||||
Hello everyone and welcome to the very first edition of the OPA Monthly Newsletter! We are excited to bring you all the happenings in the OPA ecosystem. You can expect to find a bit of everything in this newsletter, some community updates, a bit of contributor news, a handful of release notes, and any interesting content we've found on the internet this month.
|
||||
|
||||
## Slack Updates
|
||||
|
||||
|
||||
|
||||
Our Slack Org now hosts over 5,150 OPA community members!! The OPA team has been hard at work revamping the space to make it functional and valuable for all of our members. A little while ago you may remember we announced a [Slack Reorg](/blog/opa-slack-tune-up-b3c52492e2fc) to consolidate and update channel names and descriptions. This effort was to give everyone a clear understanding of what's going on and where to go.
|
||||
|
||||
To continue to improve the Slack experience for our members we've added 2 new channels. For everyone interested in contributing to the OPA project you can now hang out in the [#development](https://openpolicyagent.slack.com/archives/C02L1TLPN59) channel to speak directly with other contributors and maintainers.
|
||||
|
||||
We've also added a [#vendor](https://openpolicyagent.slack.com/archives/C02J6LBL6GH) channel to allow members to reach out directly to our rich ecosystem of vendors that are building products on top of OPA. Jump into the channel today and ask questions about how to improve your OPA management.
|
||||
|
||||
## News Highlights
|
||||
|
||||
One of our community members @boranx shared with the community that Conftest has made it into the [Technology Radar by ThoughtWorks](https://www.thoughtworks.com/radar/tools/conftest)
|
||||
|
||||
## GitHub Updates
|
||||
|
||||
The OPA project wouldn't be the same without all of the contributions from the community. As such we would like to send a big thank you to all of the contributors from the v0.34 release.
|
||||
|
||||
- Edward Paget has contributed ([#3826](https://github.com/open-policy-agent/opa/issues/3826) SDK Feat) & ([#3863](https://github.com/open-policy-agent/opa/issues/3863) Bundles Fix)
|
||||
- Kirk Patton a long time contributor added ([#3773](https://github.com/open-policy-agent/opa/issues/3773) Fix for exit statuses)
|
||||
- GitHub User [@0xAP](https://github.com/0xAP) first time contributor added ([#3860](https://github.com/open-policy-agent/opa/issues/3860) Bundles improvement)
|
||||
- Andreas Brehmer first time contributor added ([#3836](https://github.com/open-policy-agent/opa/issues/3836) Fmt fix)
|
||||
- Florian Gasc first time contributor added ([#3879](https://github.com/open-policy-agent/opa/issues/3879) Storage fix)
|
||||
- Omolola Olamide has landed ([#3910](https://github.com/open-policy-agent/opa/issues/3910) Tutorial Updates)
|
||||
|
||||
## Twitter Highlights
|
||||
|
||||
For those not active on Twitter, we've collected some of the highlights and OPA shoutouts here:
|
||||
|
||||
[https://twitter.com/that_tech_tea/status/1451930146835861504](https://twitter.com/that_tech_tea/status/1451930146835861504)
|
||||
|
||||
|
||||
|
||||
[https://twitter.com/nusairat/status/1458815340985520130](https://twitter.com/nusairat/status/1458815340985520130)
|
||||
|
||||
|
||||
|
||||
[https://twitter.com/nmeisenzahl/status/1458419364433117184](https://twitter.com/nmeisenzahl/status/1458419364433117184)
|
||||
|
||||
|
||||
|
||||
Check out the slides and demos that [Nico Meisenzahl](https://twitter.com/nmeisenzahl) created:
|
||||
|
||||
- [enhance-your-compliance-and-governance-with-policy-based-cicd](https://www.slideshare.net/nmeisenzahl/continuous-lifecycle-enhance-your-compliance-and-governance-with-policybased-cicd)
|
||||
- [demo-opa-terraform-validation](https://gitlab.com/nico-meisenzahl/demo-opa-terraform-validation)
|
||||
- [demo-opa-cicd-validation](https://github.com/nmeisenzahl/demo-opa-cicd-validation)
|
||||
|
||||
## Ecosystem Updates
|
||||
|
||||
The OPA Project is always changing, check out the latest updates and features for OPA and some of the sub-projects.
|
||||
|
||||
### [OPA Release v0.35.0](https://github.com/open-policy-agent/opa/releases/tag/v0.35.0)
|
||||
|
||||
- Early Exit Optimization improves performance in many policy types
|
||||
- New net.lookup_ip_addr built-in function to resolve host IP addresses
|
||||
- Massive performance improvement in decision logging compression
|
||||
|
||||
### [OPA Release v0.34.0](https://github.com/open-policy-agent/opa/releases/tag/v0.34.0)
|
||||
|
||||
- A new in operator for checking membership and for iteration
|
||||
- New [print](/blog/introducing-the-opa-print-function-809da6a13aee) function for debugging
|
||||
- New opa inspect command for quickly checking contents of a [bundle](https://www.openpolicyagent.org/docs/latest/management-bundles/)
|
||||
|
||||
### [Gatekeeper Release v3.7.0](https://github.com/open-policy-agent/gatekeeper/releases/tag/v3.7.0)
|
||||
|
||||
- Mutation has graduated to Beta! 🎉
|
||||
- Added ModifySet mutator 📐
|
||||
|
||||
### [Conftest Release v0.28.3](https://github.com/open-policy-agent/conftest/releases/tag/v0.28.3)
|
||||
|
||||
- The OPA [print](/blog/introducing-the-opa-print-function-809da6a13aee) function is now supported in Conftest!
|
||||
|
||||
### [Kube-mgmt Release v3.1.0](https://github.com/open-policy-agent/kube-mgmt/releases/tag/3.1.0)
|
||||
|
||||
- Support extra environment variables in opa and kube-mgmt containers
|
||||
|
||||
## Community Spotlights
|
||||
|
||||
|
||||
|
||||
- The one and only [Developer-Guy](https://github.com/developer-guy) has been working tirelessly to add OPA policy functionality to [Cosign](https://github.com/sigstore/cosign), Check out the [PR](https://github.com/sigstore/cosign/pull/641) to see the awesome work to connect the two projects.
|
||||
|
||||
## What happened this month?
|
||||
|
||||
- Meetup: [OPA London Meetup](https://www.meetup.com/london-opa-meetup/events/281522329)
|
||||
- Meetup: [OPA Stockholm Meetup](https://www.meetup.com/stockholm-opa-meetup/events/281066231/)
|
||||
- Talk: [WTF is Cloud Native talk](https://www.youtube.com/watch?v=RwsyMLyl8O0)
|
||||
- Talk: [API Authorization with Open Policy Agent](https://www.infracloud.io/cloud-native-talks/api-authorization-with-open-policy-agent-opa/)
|
||||
- Blog: [Connecting OPA with AWS Lambda](/blog/serverless-policy-enforcement-connecting-opa-and-aws-lambda-e624f7176a3)
|
||||
- Blog: [Automated Manifest File Validation Using Open Policy Agent and GitHub Actions](https://medium.com/@ravindursr/automated-manifest-file-validation-using-open-policy-agent-and-github-actions-697fa9fd74f0)
|
||||
|
||||
## What's coming up next month?
|
||||
|
||||
A list of community meetings, meetups, and conferences.
|
||||
|
||||
### [OPA Bi-Weekly](https://docs.google.com/document/d/1v6l2gmkRKAn5UIg3V2QdeeCcXMElxsNzEzDkVlWDVg8/edit?usp=sharing)
|
||||
|
||||
- Dec 7th at 10 AM PT
|
||||
- Dec 21st at 10 AM PT
|
||||
|
||||
### [Gatekeeper Weekly](https://docs.google.com/document/d/1A1-Q-1OMw3QODs1wT6eqfLTagcGmgzAJAjJihiO3T48/edit)
|
||||
|
||||
- Dec 2nd, 2 PM PT
|
||||
- Dec 8th, 9 AM PT
|
||||
- Dec 15th, 2 PM PT
|
||||
- Dec 22nd, 9 AM PT
|
||||
|
||||
## Let us know how we did
|
||||
|
||||
This was our very first edition of the OPA Newsletter, we really hope you enjoyed it! While we tried our best to find all the latest and greatest activities in the community we surely missed a lot as well. Want to share some cool content, have an OPA shoutout to make, want to speak at a conference, or host a meetup? Let us know by sending an email to: [opa_newsletter@styra.com](mailto:opa_newsletter@styra.com).
|
||||
|
||||
If you're new to OPA or to the community check out these community resources to get started.
|
||||
|
||||
- Chat with the community on [Slack](https://slack.openpolicyagent.org/)
|
||||
- Ask for help and support on [GitHub Discussions](https://github.com/open-policy-agent/feedback/discussions)
|
||||
@@ -0,0 +1,54 @@
|
||||
---
|
||||
title: "Open Policy Agent 2022 User Survey Summary"
|
||||
authors: ["peteroneilljr"]
|
||||
date: 2022-07-11
|
||||
slug: open-policy-agent-2022-user-survey-summary-370cf0243bb7
|
||||
---
|
||||
|
||||
|
||||
|
||||
It's that time of year again! We have polled the Open Policy Agent (OPA) community to learn a bit more about what members are working on, their goals and how we can improve the project in the future. This year we had over 240 respondents, from various industries ranging from Software, Finance, E-commerce, Security, and more. With this new data set, we can learn if OPA usage has changed from the previous year, what features and tools are utilized the most and how to improve the OPA project as a whole for the community. To start, let's compare last year's survey results to this year's to see how things have changed or remained consistent:
|
||||
|
||||
> [Last Year's Survey](/blog/open-policy-agent-2021-survey-summary-e749bbd7b824)
|
||||
|
||||
## Year-over-year numbers
|
||||
|
||||
Within a couple of percentage points, the number of use cases and respondents' implementation goals show similar results to last year. Of the users that responded that they have over four use cases, 70% of those reporting have used OPA for a year or longer. This shows us that as OPA usage matures in an organization, users gain confidence in adding additional use cases, helping them achieve their higher-level goals.
|
||||
|
||||
|
||||
|
||||
Almost 43% of respondents are in production with their OPA usage. This is a noticeable improvement from last year. With the addition of the Evaluating option, we can assume those users would have chosen experimentation given last year's choices, making the other two possibilities a few percentage points lower than the current year.
|
||||
|
||||
|
||||
|
||||
The last metric we highlighted in last year's survey is the time to production, showing that 40% of users reach production within six months. This year we are seeing about 27% of users in the production phase by this point. However, 50% of the users in this time frame are in a pre-production phase, which is a substantial amount.
|
||||
|
||||
|
||||
|
||||
## Policy libraries
|
||||
|
||||
We've seen a slight increase in usage for the Gatekeeper policy library from 57% to 62%, for the respondents that indicated they're using OPA for Kubernetes Admission Control. However, overall we are seeing 50% of respondents indicating that they are not using any external policy libraries. As policy libraries grow around specific use cases we can expect this number to increase.
|
||||
|
||||
## Feedback
|
||||
|
||||
Last year's request for better debugging tools led to creating two issues, rule-level tracing, and the print function. The Print Function was released in v0.34.0 and happily adopted by the community. Rule-level tracing still needs assistance from the community; perhaps you can help the community and submit a PR?
|
||||
|
||||
As we did with the previous year's survey, we asked the community for feedback to see what improvements would improve their OPA experience. The number one request was for more examples; nearly 33% of respondents asked for examples of specific or complex configurations and tutorials/sample data to go with them. About 12% of respondents asked for more integrations with AWS, such as the AWS CloudFormation integration that came out in June. And another 10% of users asked for additional debugging capabilities.
|
||||
|
||||
## Learning tools
|
||||
|
||||
The official OPA documentation is the most used resource by the community, with over 90% of respondents using it, followed by the Rego Playground at 66%. The OPA docs are consistently evolving and receive updates as new features roll out, but as with most open source projects, we need the community's help to keep the docs up to date. As for the Rego playground, we maintain this tool in the hopes that it helps users debug problems and collaborate on new policies. If you see any way that we can improve it, please let us know by creating a feature request.
|
||||
|
||||
## Monitoring
|
||||
|
||||
One surprising discovery from this year's responses is that 36% of users don't track OPA decisions, and 39% don't monitor their OPA status. While these metrics are accessible via OPA's management APIs, perhaps the docs can be spruced up with some new tutorials on configuring monitoring and logging!
|
||||
|
||||
## Wrap up
|
||||
|
||||
To sum it up, we saw consistency in the implementation goals and number of use cases for OPA with a slight uptick in the overall number of users in production. The utilization of policy libraries seems to have dropped to half of what it was last year. Debugging remains a high-priority area where users wish to see additional improvements, along with more examples and tutorials for the documentation. The OPA Docs and Rego playground take home the gold for most valuable resources, but they could use a few more examples to help community members configure monitoring and logging.
|
||||
|
||||
Thanks for your participation in this year's OPA User Survey. If you've sent us your mailing address, you can expect your t-shirt to arrive in your mailbox soon!
|
||||
|
||||
_photo credit Kayla_
|
||||
|
||||
_Happy OPA 2022 Survey from Charlie!_
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
title: "OPA Newsletter: August 2022"
|
||||
sidebar_label: "August Newsletter"
|
||||
authors: ["peteroneilljr"]
|
||||
date: 2022-08-22
|
||||
slug: august-newsletter-4c177ddc4829
|
||||
---
|
||||
|
||||
_August Edition!_
|
||||
|
||||
August is here and we are feeling the heat! Reply back to this email and let the OPA team know how you're beating the heat this year!
|
||||
|
||||
## Community Updates
|
||||
|
||||
Office Hours and the Bi-weekly meeting have converged to 1 weekly meeting. Each Office Hours is an open format meeting you can use to ask any questions you'd like.
|
||||
|
||||
But now we are reserving the Office Hours Session following a new OPA release to showcase the new features from that release. Join the next session on August 9th to hear about the v0.43.0 release
|
||||
|
||||
[Sign Up for Office Hours](https://calendly.com/peter-styra/opa-office-hours)
|
||||
|
||||
You can also watch the replays from the OPA Office Hours and the Gatekeeper Weekly meetings on YouTube. Check out the release announcement for v0.42.0 below.
|
||||
|
||||
## YouTube Videos 🎥
|
||||
|
||||
Our community has posted lots of good stuff on YouTube this month, check out these cool videos.
|
||||
|
||||
## Feature Release Videos
|
||||
|
||||
## Keywords, Contains and If
|
||||
|
||||
How to incorporate the new keywords **contains** and **if** into your policies.
|
||||
|
||||
[Watch Now](https://youtu.be/dm-JsDQPCZ0)
|
||||
|
||||
## Builtin, Object.subset
|
||||
|
||||
This new builtin allows you to check if a set, array, or object is a subset of another item.
|
||||
|
||||
[Watch Now](https://www.youtube.com/watch?v=indJqrgOSko)
|
||||
|
||||
## Events 📆
|
||||
|
||||
- [Cloud Native Computing Meetup | August 25, 2022](https://www.meetup.com/cloud-native-computing-switzerland/events/283522028/?utm_content=215127640&utm_medium=social&utm_source=twitter&hss_channel=tw-2851142013)
|
||||
- [OPA Office Hours | August 9th, 2022](https://calendly.com/peter-styra/opa-office-hours)
|
||||
|
||||
## Blogs
|
||||
|
||||
- [Open Policy Agent (OPA) For Kubernetes](https://dev.to/thenjdevopsguy/open-policy-agent-opa-for-kubernetes-5895)
|
||||
- [High-Performance OPA](https://dimakorolev.substack.com/p/high-performance-opa)
|
||||
- [How to Shape OPA Data for Policy Performance](https://www.styra.com/blog/how-to-shape-opa-data-for-policy-performance/)
|
||||
|
||||
## Ecosystem
|
||||
|
||||
### [Open Policy Agent v0.43.0](https://github.com/open-policy-agent/opa/releases/tag/v0.43.0)
|
||||
|
||||
- [Large Object Performance Improvements](https://github.com/open-policy-agent/opa/issues/4625)
|
||||
- [GraphQL Tutorial](https://www.openpolicyagent.org/docs/latest/graphql-api-authorization/)
|
||||
|
||||
### [Gatekeeper v3.9.0](https://github.com/open-policy-agent/gatekeeper/releases/tag/v3.9.0)
|
||||
|
||||
- [Constraint schema validation testing](https://github.com/open-policy-agent/gatekeeper/pull/2092)
|
||||
- [Make gatekeeper validate subresources](https://github.com/open-policy-agent/gatekeeper/pull/2054)
|
||||
|
||||
### [Conftest v0.34.0](https://github.com/open-policy-agent/conftest/releases/tag/v0.34.0)
|
||||
|
||||
- [Add parse_config and parse_config_file Rego functions to allow unit testing using config file snippets](https://www.conftest.dev/#writing-unit-tests)
|
||||
|
||||
## Calling all OPA End-Users
|
||||
|
||||
OPA Summit is officially scheduled, are you ready to share your OPA development journey? Let me us know how you're using OPA and we will help you craft a presentation. This event will be colocated at Kubecon in Detroit this October.
|
||||
|
||||
[👉 Speak at OPA Summit 👈](mailto:peteroneilljr@styra.com)
|
||||
|
||||
## Let us know how we did
|
||||
|
||||
The OPA monthly newsletter is built for the OPA community, let us know what you liked or what you wanted to see more of. Reach out using one of the links below.
|
||||
+195
@@ -0,0 +1,195 @@
|
||||
---
|
||||
title: "I have a plan! Exploring the OPA Intermediate Representation (IR) format"
|
||||
authors: ["anderseknert"]
|
||||
date: 2022-10-20
|
||||
slug: i-have-a-plan-exploring-the-opa-intermediate-representation-ir-format-7319cd94b37d
|
||||
---
|
||||
|
||||
|
||||
|
||||
It isn't an overstatement to say that the versatility of Open Policy Agent (OPA) is a key factor in its success. As a general purpose policy engine, OPA needs to handle inputs from a disparate set of systems — Terraform, Kubernetes, CI/CD pipelines or custom applications, to name a few — and deliver decisions in a format understood by that particular system. Providing an agnostic approach to input and output data — anything that is or can be modeled hierarchically in a JSON or YAML document is potentially subject to policy — allows [integrating](https://www.openpolicyagent.org/docs/latest/ecosystem/) OPA with all kinds of systems, applications and tech stacks.
|
||||
|
||||
But how does the data get passed between client application and OPA? Running OPA as a standalone service, and querying OPA for decisions over its [REST API](https://www.openpolicyagent.org/docs/latest/rest-api/), is by far the most common way to integrate OPA, and for good reasons! Running OPA as a separate component provides a nice, unified interface for communication, which commonly involves writing only a few lines of code in most modern programming languages. Additionally, many technologies allow extending the built-in functionality for e.g. authorization by utilizing _webhooks_, which are commonly REST requests with a JSON-encoded payload. Perfect integration point for OPA!
|
||||
|
||||
There are however some scenarios where the standalone REST model proves to be challenging:
|
||||
|
||||
- Resource-constrained environments like embedded systems. While OPA is fairly light-weight, some environments simply don't have the resources required to run a standalone OPA service, or networking capabilities for querying.
|
||||
- Distributed deployments with tight latency budgets, where every millisecond counts.
|
||||
- Environments constrained by other limitations on what type of software can be run, like web browsers.
|
||||
|
||||
To accommodate these requirements, OPA provides a few [alternatives](https://www.openpolicyagent.org/docs/latest/integration/) to the standalone service model:
|
||||
|
||||
- Applications written in Go may integrate directly with the OPA Go API, or through the high-level Go SDK, alleviating the need for OPA to run as a separate service.
|
||||
- Policy may be compiled into Wasm modules, which can then be evaluated in any Wasm runtime. While the most famous Wasm runtime may be that included in web browsers, most programming languages today offer integrations with a Wasm runtime, allowing (at least the policy evaluation parts of) OPA to run "inside" of the application rather than outside of it. Additionally, OPA itself ships with a Wasm runtime, which makes it possible to have OPA pull down bundles including Wasm compiled policy for faster evaluation, and potentially other benefits.
|
||||
|
||||
## Wasm
|
||||
|
||||
Rego policies compiled to Wasm modules offers a flexible, highly performant alternative to "regular" policy evaluation, with runtimes available for a [wide array](https://github.com/appcypher/awesome-wasm-runtimes) of languages, frameworks and platforms. As such, it should be considered an option for any OPA integration where the standalone server model falls short of the requirements. However, as ubiquitous as Wasm runtimes may be, they are not available _everywhere_. Embedded environments, exotic architectures or specialized hardware all constitute examples of environments where we're unlikely to encounter a Wasm runtime. But even with one available, Wasm _itself_ is not without limitations, even by design!
|
||||
|
||||
With the goals of providing a safe, _sandboxed_ environment, originally targeting web browsers, Wasm has several restrictions on what can and can't be done in the confines of the runtime. Interacting with the host system, or for that matter, other host systems — whether through system calls, network requests, or file system operations, is generally prohibited. While the WebAssembly System Interface (WASI) aims to offer an API for this exact purpose, and could potentially be used for certain features of Rego (like the http.send built-in function) in the future, relying on WASI means that a policy evaluated in one runtime might not work in another, as currently only a certain subset of the WASI API is implemented in any given runtime, and Wasm runtimes like those provided by web browsers likely have no interest in supporting interactions with the host system _at all_. Last, while some great progress has been made around WASI recently, it is still nowhere near the maturity of Wasm.
|
||||
|
||||
## Intermediate Representation (IR)
|
||||
|
||||
OPA v0.37.0, released early 2022, brought two major enhancements to OPA: [compiler strict mode](https://www.openpolicyagent.org/docs/latest/strict/) and [delta bundles](https://www.openpolicyagent.org/docs/latest/management-bundles/#delta-bundles). While those two features might have stolen the show of the release, the [changelog](https://github.com/open-policy-agent/opa/releases/tag/v0.37.0) additionally provides us with this:
|
||||
|
||||
> The compile package and the opa build command support a new output format: "plan". It represents a _query plan_, steps needed to take to evaluate a query (with policies). The plan format is a JSON encoding of the intermediate representation (IR) used for compiling queries and policies into Wasm.
|
||||
|
||||
Interesting! Now, what does it mean? As alluded to in the last sentence, the low-level building blocks, or the _evaluation plan_, that eventually becomes Wasm, is now made available for consumption by other implementations. What would another implementation look like? That's up to you! While OPA may provide us with the low-level, step-by-step plan, for the evaluation of a query, it'll be on us to parse and evaluate that plan. Ever wanted to have your policy decisions served right inside of your Python app? Doable. Can't run the OPA server on your tiny microcontroller? You no longer need to. No runtime, no restrictions. What's the catch?
|
||||
|
||||
## Bring Your Own OPA
|
||||
|
||||
Using your programming language of choice to implement the full set of [instructions](https://www.openpolicyagent.org/docs/latest/ir/) included in the intermediate representation format isn't something you'd pull off in just a few hours. A robust implementation is likely going to necessitate quite some effort, and even if you decide to invest the days — or possibly, weeks — required for a greenfield implementation, implementing the IR instructions is only half of the story. OPA provides an impressive number of [built-in functions](https://www.openpolicyagent.org/docs/latest/policy-reference/#built-in-functions), requiring corresponding implementations in the platform you choose to target. You probably won't need every single built-in to accommodate your use case though, so starting with the ones known to be relevant for you is likely a smart idea. Rego modules compiled to Wasm, on the other hand, ship with [native implementations](https://github.com/open-policy-agent/opa/tree/main/wasm) of many of the built-ins. If you plan to build an IR compiler or evaluator in C or C++, leveraging those would give you a head start!
|
||||
|
||||
Another aspect to consider is the management features provided by OPA. Similarly to Wasm, the scope of the IR format is limited to policy _evaluation_. Fetching bundles from remote endpoints, sending decision logs, or providing metrics and status reports is left as an exercise to the implementation. However, just like with the built-in functions, you likely won't need to support the full set of management capabilities shipped with OPA, but can pick and choose the parts that make sense to you. More interestingly, you're free to implement your _own_ management features. Rather than pulling bundles from an S3 bucket, why not stream your permissions data from a Kafka topic? Or build a direct integration against that tool your organization uses for health checks, and so on. As laborious as a custom implementation may be, it opens up for some very interesting opportunities!
|
||||
|
||||
We're getting ahead of ourselves though. Before we dash off to write our own, next generation, OPA implementation in whatever the hottest programming language is these days, we should probably start by getting familiar with the IR format, and how to make sense of evaluation _plans_.
|
||||
|
||||
## Making Plans
|
||||
|
||||
Let's create a simple policy, and build a plan from that. The below policy contains two rules — `is_admin` to check if the "admin" role is included in the list of roles provided in the **input** for a user, and `allow`, which in this case simply is true if `is_admin` is true, but presumably would be extended to include more checks in future iterations of our policy.
|
||||
|
||||
```rego
|
||||
package policy
|
||||
|
||||
import future.keywords.if
|
||||
|
||||
import future.keywords.in
|
||||
|
||||
allow if is_admin
|
||||
|
||||
is_admin if "admin" in input.user.roles
|
||||
```
|
||||
|
||||
Simple enough, right? Let's see what a plan might look like! In order to build one, we'll use the aptly named `opa build` command. This command is used to build [bundles](https://www.openpolicyagent.org/docs/latest/management-bundles/), and the `--target` flag allows us to say that rather than just copying Rego and data files into the bundle, we want OPA to compile either a `plan`, or a `wasm` and put that in the bundle for us too. When building a plan, we'll additionally need to provide an _entrypoint_ — this would be the path to either a package or a rule, from which the plan should be built. The path to the entry points (more than one is allowed) will later be used to query an implementation capable of parsing and evaluating our plan. Let's build a bundle with the plan target, and the entrypoint set to that of our `allow` rule:
|
||||
|
||||
```bash
|
||||
opa build --target plan --entrypoint policy/allow .
|
||||
```
|
||||
|
||||
This will create a `bundle.tar.gz` file in the current directory, with our plan inside of it. Since we're only interested in the plan for now, let's extract it from the bundle:
|
||||
|
||||
```bash
|
||||
tar -zxvf bundle.tar.gz /plan.json
|
||||
```
|
||||
|
||||
## The plan.json file
|
||||
|
||||
We now have a plan to work with! Let's see what's in that plan.json file. The first thing you'll notice is that the plan file contains three top level attributes — `static`, `plans` and `funcs`. The `static` object is fairly straightforward:
|
||||
|
||||
```json
|
||||
{
|
||||
"static": {
|
||||
"strings": [{ "value": "result" }, { "value": "user" }, { "value": "roles" }, { "value": "admin" }],
|
||||
"builtin_funcs": [
|
||||
{
|
||||
"name": "internal.member_2",
|
||||
"decl": {
|
||||
"args": [{ "type": "any" }, { "type": "any" }],
|
||||
"result": { "type": "boolean" },
|
||||
"type": "function"
|
||||
}
|
||||
}
|
||||
],
|
||||
"files": [{ "value": "policy.rego" }]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The `strings` array contains references to all strings included in the plan, and will be referenced whenever needed in evaluation. We'll recognize the "user", "roles" and "admin" strings from our policy, while the "result" string has been added by the plan builder, to be used as a key in the result set from plan evaluation. Thanks, plan builder! The `builtin_funcs` array provides a list of all the built-in functions used in our policy, along with the types expected for their arguments and return values. While "internal.member_2" might look unfamiliar, it's the internal name used for the built-in function representing the `in` operator used in our policy! Finally, the files array contains a list of all files used to build the plan, which in our case is only `policy.rego`.
|
||||
|
||||
The next attribute is the actual _plans_, and here's where things start to turn a bit cryptic. But don't worry, I'll walk you through it!
|
||||
|
||||
```json
|
||||
{
|
||||
"plans": {
|
||||
"plans": [
|
||||
{
|
||||
"name": "policy/allow",
|
||||
"blocks": [
|
||||
{
|
||||
"stmts": [
|
||||
{
|
||||
"type": "CallStmt",
|
||||
"stmt": {
|
||||
"func": "g0.data.policy.allow",
|
||||
"args": [{ "type": "local", "value": 0 }, { "type": "local", "value": 1 }],
|
||||
"result": 2
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "AssignVarStmt",
|
||||
"stmt": {
|
||||
"source": { "type": "local", "value": 2 },
|
||||
"target": 3
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "MakeObjectStmt",
|
||||
"stmt": {
|
||||
"target": 4
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "ObjectInsertStmt",
|
||||
"stmt": {
|
||||
"key": { "type": "string_index", "value": 0 },
|
||||
"value": { "type": "local", "value": 3 },
|
||||
"object": 4
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "ResultSetAddStmt",
|
||||
"stmt": {
|
||||
"value": 4
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Functions, statements, blocks
|
||||
|
||||
For each entrypoint provided, we'll find a corresponding _plan_, which represents the planned evaluation path for that entrypoint. The `name` of the plan is our entrypoint ("policy/allow") and the `blocks` attribute contains the statements to be evaluated in order to "run" the plan. Quite literally in order too, as each statement block and statement will be evaluated in an entirely procedural fashion. Quite a contrast to the Rego code that produced it!
|
||||
|
||||
The first statement is a `CallStmt`, which means we'll need to evaluate the function (i.e. `func`) corresponding to the provided name in the `funcs` object — in our case this has been named "g0.data.policy.allow" (mapped from our `allow` rule) — and we'll take a closer look at the `funcs` part in a minute. The `args` provided to the function is "local" value 0, and "local" value 1. These represent the global **input** and **data** variables that you're likely familiar with from your Rego policies, and "value" in this case is rather the "name" — or pointer — to the values in the local _scope_. Who said naming was one of the hardest problems in computer science? Not the OPA plan compiler!
|
||||
|
||||
## Where the Locals Go
|
||||
|
||||
Since you'll see a lot of references to `local` throughout compiled plans, learning how it's used is imperative to understanding the steps involved in plan execution. When a function is invoked, like our "g0.data.policy.allow" above, a local object is created to represent the inputs to that function. The _statements_ that comprise the function may in turn both read from the local object, as well as write to it, effectively making it a bearer of local _state_.
|
||||
|
||||
If a statement inside of a function involves calling _another_ function, a new, "inner" `local` object will be created for the scope of that function, and the result of the function evaluation will be stored in the "outer" `local`, and so on.
|
||||
|
||||
Back to our `CallStmt`! We now know that it'll be invoked with the input (local 0) and data (local 1) as its arguments. The next attribute in the statement simply says "return": 2, meaning that whatever value is returned by the function should be stored in the next position — i.e. 2 — in the local state. Next up, the `AssignVarStmt` is used to assign the value at position 2 — that's our return value — to a new local value at position 3 (as defined by the "target" attribute). Moving on, we'll see a `MakeObjectStmt` used to create an empty object, which is placed at local value 4. In the next step, a key value pair is inserted into the object (`ObjectInsertStmt`) where the key is the "string_index" at position 0. Remember the `strings` attribute from the `static` object from before? This is it. The first item in that array is "result", so it looks like we're building a result set over here! The value associated with the result is good ol' local value 3, which we might recall was the result of invoking the "g0.data.policy.allow" function. Finally `ResultSetAddStmt` signals that we're done here. We have a result from the plan, and we're now ready to return it.
|
||||
|
||||
## Following Procedure
|
||||
|
||||
What about "g0.data.policy.allow" then? I promised we'd get back to the "funcs" object in a minute, and wow, time really flies when describing procedural instructions of an evaluation plan! A quick glance at the [funcs](https://gist.github.com/anderseknert/d2aec51b950362e156bb413ad312fca3) object reveals that it contains not just the "allow" function, but also our "is_admin" function, here in the form of "g0.data.policy.is_admin". Since the allow function merely mirrors the result of is_admin, let's zoom in on the latter to learn how an implementation would evaluate the statements step by step, and how the state of the local object is updated in (almost) each step along the way. Rather than describing the steps in words, let's use a table for demonstration. In the left column you'll see a simplified version of each statement called, and in the right you'll see the local state after the statement has been applied. Note how each step procedurally builds up the final state, which is eventually returned to the caller. Beautiful, isn't it?
|
||||
|
||||
|
||||
|
||||
_Evaluation plan statements and local state. As Medium does not have embeddable tables, you may find the spreadsheet from the image above in [this Google Sheets document](https://docs.google.com/spreadsheets/d/1pU9J3QkfKwkFOZf7UV_PWPHZspQvHOJxOw_zh4poLoQ/edit?usp=sharing)._
|
||||
|
||||
To learn more about the different statements an evaluator implementation may encounter, consult the [OPA docs](https://www.openpolicyagent.org/docs/latest/ir/) on the topic.
|
||||
|
||||
## Planning Ahead
|
||||
|
||||
Every aspect and instruction of the IR format would be too much for a single blog to cover, but the process of untangling an evaluation plan has hopefully been made clearer by now. While creating a full-fledged "OPA" native to your language or platform of choice might be a huge undertaking, even a basic implementation, with only a handful of built-in functions implemented, gets you surprisingly far towards something that actually feels _usable_. Open source implementations of course would have the benefit of others being able to contribute the parts that make the project usable to them. On the topic of open source implementations, are there any of those out there yet? In fact, there is!
|
||||
|
||||
## Introducing Jarl
|
||||
|
||||
For the past few months, fellow OPA maintainer [Johan Fylling](https://github.com/johanfylling) and I have spent some of our spare time hacking away on an IR implementation for the Java Virtual Machine (JVM) called [Jarl](https://github.com/johanfylling/jarl). A jarl was a chieftain in the age of vikings, and given the viking theme of OPA itself, we figured it would be a good name for the project. And of course, it has that "J" in there too, which seems almost mandatory for JVM-based software.
|
||||
|
||||
We chose to use [Clojure](https://clojure.org/) for our implementation, and while we are both rather novice Clojure coders, it's been a lot of fun to work with! Not only that, but using Clojure means getting access to the broader JVM ecosystem, both in terms of libraries available, and that applications written in Java, Kotlin or Scala will be able to use Jarl eventually. As an added bonus, Clojurescript allows the library to be compiled into Javascript as well, allowing us to target deployments in Node, or web browsers. Quite a versatile platform to build on!
|
||||
|
||||
While we still have a long way to go before Jarl is anywhere near production readiness, we're already at a point where it's usable for evaluation of many types of common policies. As we wanted to ensure conformance against OPA from the start, we ported the OPA [compliance test suite](https://github.com/johanfylling/opa-compliance-test) to the IR format, which has proven to be tremendously useful for testing not just plan evaluation, but also the behavior of built-in function implementations. This code should be useful for anyone building their own implementation, so if that's you, make sure to check it out. As for the built-in functions, we currently have [most of them](https://github.com/johanfylling/jarl/blob/main/doc/builtins.md) ported, but some work remains to be done before we're able to have all tests from OPA pass. Then awaits the management features…
|
||||
|
||||
If the project sounds interesting to you, we'd love to hear from you! Reach out on the OPA Slack, or just try it out and report back on any issues, feature requests or ideas you might have. Or if you'd rather work on your own implementation, we'd love to help you get started.
|
||||
|
||||
## Wrapping Up
|
||||
|
||||
With the introduction of the intermediate representation format, another integration option for OPA has been made available. While it might be a bit of a niche — and certainly not the first choice to consider for most applications — it opens the door for many new and interesting use cases where an OPA integration using the standalone server model might not have been the best fit, or possible at all. Although perhaps not a well-known option until now, I hope this blog may contribute to changing that, and I'm looking forward to seeing how this alternative is leveraged in the future. Interesting times ahead!
|
||||
@@ -0,0 +1,111 @@
|
||||
---
|
||||
title: "OPA Newsletter: November 2022"
|
||||
sidebar_label: "November Newsletter"
|
||||
authors: ["peteroneilljr"]
|
||||
date: 2023-01-25
|
||||
slug: november-newsletter-9f3bbcb29405
|
||||
---
|
||||
|
||||
_November Edition!_
|
||||
|
||||
November has arrived and we are looking forward to the holiday season!
|
||||
|
||||
Thanks to all of the community members that stopped by the booth at Kubecon, it was a pleasure meeting you!
|
||||
|
||||
## User Survey
|
||||
|
||||
We are looking for input from the community to see how everyone is using OPA. Take 5 minutes to fill out this 7 question survey to help out the community!
|
||||
|
||||
[Take the Survey](https://bit.ly/3UaIhWa)
|
||||
|
||||
## Ecosystem Updates
|
||||
|
||||
### [Open Policy Agent v0.46.1](https://github.com/open-policy-agent/opa/releases/tag/v0.46.1)
|
||||
|
||||
- New language feature: refs in rule heads
|
||||
- Entrypoint annotations in rule metadata
|
||||
- New Built-in Function: graphql.schema_is_valid
|
||||
- New Built-in Function: net.cidr_is_valid
|
||||
|
||||
### [Gatekeeper 3.10.0](https://github.com/open-policy-agent/gatekeeper/releases/tag/v3.10.0)
|
||||
|
||||
- Kubernetes v1.25+, removal of Pod Security Policies and migration to Pod Security Admission 🔐
|
||||
- Mutation is promoted to stable 🦠
|
||||
- Introducing Validation of Workload Resources as alpha 🚀
|
||||
- Performance improvements 🏃
|
||||
|
||||
## Contributor Shout Outs
|
||||
|
||||
Thanks to all of the contributors that participated in these releases, the OPA community wouldn't be here without you!
|
||||
|
||||
- @mattfarina
|
||||
- @jaspervdj
|
||||
- @ricardomaraschini
|
||||
- @byronic
|
||||
- @philipaconrad
|
||||
- @pjbgf
|
||||
- @caldwecr
|
||||
- @hzliangbin
|
||||
- @peterchenadded
|
||||
- @phantlantis
|
||||
- @ericjkao
|
||||
- @TheLunaticScripter
|
||||
- @humbertoc-silva
|
||||
- @Juneezee
|
||||
- @vinhph0906
|
||||
- @aholmis
|
||||
- @Joffref
|
||||
- @olegroom
|
||||
- @iamatwork
|
||||
- @fredallen-wk
|
||||
- @bartandacc
|
||||
- @max0ne
|
||||
- @OpenSourceZombie
|
||||
- @JAORMX
|
||||
- @Boojapho
|
||||
- @ethanrange
|
||||
- @stp-bsh
|
||||
- @qa-ship-it
|
||||
- @salaxander
|
||||
- @boatmisser
|
||||
- @gracedo
|
||||
- @meons
|
||||
- @mariusblarsen
|
||||
|
||||
## Community Tools
|
||||
|
||||
### circle-policy-agent
|
||||
|
||||
The policy-agent is essentially a CircleCI-flavored wrapper library around the Open Policy Agent (OPA), which will allow the users to write the policy documents in CircleCI terminology.
|
||||
|
||||
[Star on GitHub](https://github.com/CircleCI-Public/circle-policy-agent)
|
||||
|
||||
### custom-opa-spicedb
|
||||
|
||||
This experiment adds support for querying relations from Authzed / SpiceDB via GRPC to check resource level permissions as custom builtin commands for Open Policy Agent.
|
||||
|
||||
[Star on GitHub](https://github.com/thomasdarimont/custom-opa-spicedb)
|
||||
|
||||
## Videos 🎥
|
||||
|
||||
### Policy as Code with Open Policy Agent — Anders Eknert, Styra
|
||||
|
||||
Should user Alice be allowed to read credit reports? Should a cloud instance be deployable without basic security configuration in place? Should service X be allowed to query the database? Policy defines the rules of our systems, but how do we ensure our policies are enforced consistently in increasingly distributed and diverse tech stacks? In this talk we'll explore the benefits of decoupling policy from our applications, deployment pipelines and platforms, and how Open Policy Agent (OPA) can help unify the way we work with policy across the stack.
|
||||
|
||||
### Securing kubernetes with opa and gatekeeper
|
||||
|
||||
Starts at 3:23:20 as part of the Kubehuddle Edinburgh event.
|
||||
|
||||
## Blogs
|
||||
|
||||
- [I have a plan! Exploring the OPA Intermediate Representation (IR) format](/blog/i-have-a-plan-exploring-the-opa-intermediate-representation-ir-format-7319cd94b37d)
|
||||
- [5 Application Authorization Best Practices for Better Cybersecurity](https://thenewstack.io/5-application-authorization-best-practices-for-better-cybersecurity/)
|
||||
- [Intro to sets in Rego](https://qjuanp.dev/post/introduction-sets-rego-open-policy-agent)
|
||||
- [OPA into WASM](https://inspektor.cloud/blog/evaluating-open-policy-agent-in-rust-using-wasm/)
|
||||
- [Opa for k8s](https://dev.to/thenjdevopsguy/open-policy-agent-opa-for-kubernetes-5895)
|
||||
- [Spring Security Authorization with OPA](https://www.baeldung.com/spring-security-authorization-opa)
|
||||
- [Programming Your Policies: Justin Cormack at QCon San Francisco 2022](https://www.infoq.com/news/2022/10/programming-policy-code/)
|
||||
|
||||
## Let us know how we did
|
||||
|
||||
The OPA monthly newsletter is built for the OPA community, let us know what you liked or what you wanted to see more of. Reach out using one of the links below.
|
||||
@@ -0,0 +1,72 @@
|
||||
---
|
||||
title: "OPA Newsletter: October 2022"
|
||||
sidebar_label: "October Newsletter"
|
||||
authors: ["peteroneilljr"]
|
||||
date: 2023-01-25
|
||||
slug: october-newsletter-6b90cab2dcdb
|
||||
---
|
||||
|
||||
_October Edition!_
|
||||
|
||||
October is here, the leaves are changing colors, and weather is starting to become cool.
|
||||
|
||||
The OPA community will be at KubeCon NA, so don't forget to register for Cloud Native Policy Day with OPA!
|
||||
|
||||
[Register Today!](https://www.styra.com/cloud-native-policy-day-with-opa-2022/)
|
||||
|
||||
## Community Updates
|
||||
|
||||
The OPA community now has over 300 GitHub contributors!
|
||||
|
||||
Our friends over at Postman connected with us to setup an OPA webpage to help our community members explore the OPA API.
|
||||
|
||||
[The OPA API on Postman](https://www.postman.com/openpolicyagent)
|
||||
|
||||
## Ecosystem Updates
|
||||
|
||||
### [Open Policy Agent v0.45.0](https://github.com/open-policy-agent/opa/releases/tag/v0.45.0)
|
||||
|
||||
- **Feature**: Improved Decision Logging with [nd_builtin_cache](https://www.openpolicyagent.org/docs/v0.45.0/management-decision-logs/#masking-sensitive-data)
|
||||
- **New builtin**: [regex.replace](https://www.openpolicyagent.org/docs/v0.45.0/policy-reference/#builtin-regex-regexreplace) for regex-based search/replace on strings
|
||||
- **Optimization**: [object.union_n](https://github.com/open-policy-agent/opa/issues/4985) builtin implementation to use a more efficient merge algorithm
|
||||
|
||||
## Community Tools
|
||||
|
||||
## Capua
|
||||
|
||||
A Kafka Policy engine that will help you validate your resources and artefact creations with style.
|
||||
|
||||
[Like on GitHub](https://github.com/kafka-ops/capua)
|
||||
|
||||
## Videos 🎥
|
||||
|
||||
### rq: Datalog for your shell pipelines
|
||||
|
||||
rq brings the full power of Rego, a Datalog dialect created for Open Policy Agent (OPA) to your shell pipelines. It allows you to easily transform and query data in a variety of commonly used formats using Rego expressions.
|
||||
|
||||
Speaker: Charles Daniels, Backend Software Engineer, Styra Inc.
|
||||
|
||||
## Blogs
|
||||
|
||||
- [How DoorDash Ensures Velocity and Reliability through Policy Automation](https://doordash.engineering/2022/09/20/how-doordash-ensures-velocity-and-reliability-through-policy-automation/)
|
||||
- [Deploying Gatekeeper policies as OCI artifacts, the GitOps way](https://medium.com/google-cloud/deploying-gatekeeper-policies-as-oci-artifacts-the-gitops-way-e1233429ae2)
|
||||
- [Rego — The unified policy language for better policy management](https://amazic.com/rego-the-unified-policy-language-for-better-policy-management/)
|
||||
- [Creating Custom OPA Policies with Azure Policy](https://jfarrell.net/2022/09/16/creating-custom-opa-policies-with-azure-policy/)
|
||||
- [Use OPA Gatekeeper to prohibit specific IAM users from creating resources in a specific Namespace](https://qiita.com/ipppppei/items/07cb80329bd3a9d1732a)
|
||||
- [\[Copy and paste OK\] Procedure for linking Open Policy Agent with Python](http://kakedashi-xx.com:25214/index.php/2021/08/15/post-3024/)
|
||||
|
||||
## Events 📆
|
||||
|
||||
### Cloud Native Policy Day with OPA, Oct 25th
|
||||
|
||||
Cloud Native Policy Day with OPA hosted by Styra, the creators of Open Policy Agent, will bring together the OPA community for a day of sharing and discussing policy-as-code best practices, key learnings and creative use cases for OPA. Project maintainers will be on hand to field 1:1 questions and provide live-coding demos — and you'll see proven real-world implementations from various OPA adopters during each of the sessions.
|
||||
|
||||
Whether you're looking to start down your policy journey, or are an OPA adopter with Rego skills to share, join the community for sharing, learning and socializing.
|
||||
|
||||
Attendees are invited to come for the full day with lunch provided or to stop by just for the sessions that interest them most. To register for the event, add Cloud Native Policy Day with OPA from the co-located event list selections when registering for KubeCon + CloudNativeCon NA 2022 or add it to your existing registration by selecting "modify" on your confirmation page or clicking the "modify" link in your confirmation email.
|
||||
|
||||
[👉 Register Today! 👈](https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/register/)
|
||||
|
||||
## Let us know how we did
|
||||
|
||||
The OPA monthly newsletter is built for the OPA community, let us know what you liked or what you wanted to see more of.
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
title: "OPA Newsletter: September 2022"
|
||||
sidebar_label: "September Newsletter"
|
||||
authors: ["peteroneilljr"]
|
||||
date: 2023-01-25
|
||||
slug: september-newsletter-3266b098c5f5
|
||||
---
|
||||
|
||||
_September Edition!_
|
||||
|
||||
Happy September Everyone! This month's edition is coming in a little late, but don't worry, it's still packed with great information.
|
||||
|
||||
Don't forget to register for Cloud Native Policy Day with OPA! More info at bottom.
|
||||
|
||||
[Register Today!](https://www.styra.com/cloud-native-policy-day-with-opa-2022/)
|
||||
|
||||
## Community Updates
|
||||
|
||||
The Rego Playground now has a "Format" button! 🎉
|
||||
|
||||
This button auto-formats your policy code in the editor, as well as your input/data JSON documents.
|
||||
|
||||
## Ecosystem Updates
|
||||
|
||||
### [Open Policy Agent v0.44.0](https://github.com/open-policy-agent/opa/releases/tag/v0.44.0)
|
||||
|
||||
- security fixes, which mitigate CVE-2022-36085 in OPA itself, and CVE-2022-27664 and CVE-2022-32190 in our Go build tooling.
|
||||
- [Linear performance scaling for sets up into the 500k key range and beyond](https://github.com/open-policy-agent/opa/pull/4999)
|
||||
- [The union builtin is now about 15-30% faster than the equivalent operation in pure Rego.](https://github.com/open-policy-agent/opa/issues/4979)
|
||||
- [This release introduces two new builtins: strings.any_prefix_match, and strings.any_suffix_match.](https://www.openpolicyagent.org/docs/v0.42.0/policy-reference/#builtin-strings-stringsany_prefix_match)
|
||||
|
||||
### [NPM-OPA-WASM v1.8.0](https://github.com/open-policy-agent/npm-opa-wasm/releases/tag/1.8.0)
|
||||
|
||||
- [New Feature: add loadPolicySync by @elliots in #255](https://github.com/open-policy-agent/npm-opa-wasm/pull/255)
|
||||
|
||||
We will discuss these new features in the September 20th Office Hours. Sign up today and send in your questions.
|
||||
|
||||
[Join OPA Office Hours](https://calendly.com/peter-styra/opa-office-hours)
|
||||
|
||||
## Community Tools
|
||||
|
||||
## Goast
|
||||
|
||||
Go AST (Abstract Syntax Tree) based static analysis tool with Rego.
|
||||
|
||||
[Like on GitHub](https://github.com/m-mizutani/goast)
|
||||
|
||||
## Java App with OPA Policies
|
||||
|
||||
Motivation for this code and application was to try to understand and implement the Hexagonal Architecture — also called Port and Adapter Architecture.
|
||||
|
||||
[Test it out](https://github.com/uwegeercken/artikel)
|
||||
|
||||
## OPA Support for Go Fiber
|
||||
|
||||
Open Policy Agent support for Fiber.
|
||||
|
||||
Note: Requires Go 1.16 and above
|
||||
|
||||
[Try it](https://github.com/gofiber/contrib/tree/main/opafiber)
|
||||
|
||||
## Blogs
|
||||
|
||||
Read up on how the OPA community is using OPA.
|
||||
|
||||
- [Control User Access and Permissions in CVAT with Open Policy Agent](https://medium.com/@nikman/control-user-access-and-permissions-in-cvat-with-open-policy-agent-a2abbd09774d)
|
||||
- [What Exposed OPA Servers Can Tell You About Your Applications](https://www.trendmicro.com/en_us/research/22/h/what-exposed-opa-servers-can-tell-you-about-your-applications-.html)
|
||||
- [Using XACML with OPA and Rego: The Best of Both Worlds](https://www.styra.com/blog/using-xacml-with-opa-and-rego-the-best-of-both-worlds/)
|
||||
- [Authorize REST API with OPA (Japanese)](https://christina04.hatenablog.com/entry/opa-rest-api-authorization)
|
||||
- [Controlling Kafka Data Flows using Open Policy Agent](https://opencredo.com/blogs/controlling-kafka-data-flows-using-open-policy-agent/)
|
||||
- [Introduction of Open Policy Agent / Rego to realize Policy as Code](https://tech.isid.co.jp/entry/2021/12/05/Policy_as_Code%E3%82%92%E5%AE%9F%E7%8F%BE%E3%81%99%E3%82%8B_Open_Policy_Agent_/_Rego_%E3%81%AE%E7%B4%B9%E4%BB%8B) (Japanese)
|
||||
- [Collaborating on Access Control Policies with Open Policy Agent](https://zendesk.engineering/collaborating-on-access-control-policies-with-open-policy-agent-fddbc3058359)
|
||||
|
||||
## Events 📆
|
||||
|
||||
### Cloud Native Policy Day with OPA, Oct 25th
|
||||
|
||||
Cloud Native Policy Day with OPA hosted by Styra, the creators of Open Policy Agent, will bring together the OPA community for a day of sharing and discussing policy-as-code best practices, key learnings and creative use cases for OPA. Project maintainers will be on hand to field 1:1 questions and provide live-coding demos — and you'll see proven real-world implementations from various OPA adopters during each of the sessions.
|
||||
|
||||
Whether you're looking to start down your policy journey, or are an OPA adopter with Rego skills to share, join the community for sharing, learning and socializing.
|
||||
|
||||
Attendees are invited to come for the full day with lunch provided or to stop by just for the sessions that interest them most. To register for the event, add Cloud Native Policy Day with OPA from the co-located event list selections when registering for KubeCon + CloudNativeCon NA 2022 or add it to your existing registration by selecting "modify" on your confirmation page or clicking the "modify" link in your confirmation email.
|
||||
|
||||
[Register Today!](https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/register/)
|
||||
|
||||
## Let us know how we did
|
||||
|
||||
The OPA monthly newsletter is built for the OPA community, let us know what you liked or what you wanted to see more of.
|
||||
@@ -0,0 +1,192 @@
|
||||
---
|
||||
title: "Open Policy Agent 2022, Year in Review"
|
||||
authors: ["anderseknert"]
|
||||
date: 2023-01-26
|
||||
slug: open-policy-agent-2022-year-in-review-79324ad54535
|
||||
---
|
||||
|
||||
|
||||
|
||||
It's a new year, and once again it's time to look back and reflect on the year that passed in the world of Open Policy Agent! 2022 was OPA's first full year as a CNCF graduated project, and while it would be easy to think that things would slow down after reaching that point of maturity and recognition, things have rather sped up. In community growth as well as pace of development — both of which we'll take a closer look at here.
|
||||
|
||||
This year, tech communities like ours enjoyed finally getting to meet in person again, and major conferences in our space, like KubeCon / CloudNativeCon, saw thousands of attendees in both Europe and North America. We also saw meetups, hackathons and other smaller gatherings move back from virtual to in-person events. On the topic of events — let's start our review of the year 2022 for Open Policy Agent there.
|
||||
|
||||
## Notable Events
|
||||
|
||||
### 2022 User Survey
|
||||
|
||||
While we queried our nearest OPA instance for policy decisions, we also queried the OPA community to learn about their experience of interacting with the project, documentation and tooling. The [survey results](/blog/open-policy-agent-2022-user-survey-summary-370cf0243bb7) provided us valuable insights into where we might want to spend some extra time and effort in 2023 — richer documentation with more examples was requested by many. We're also seeing a need to better highlight the benefits of the various management capabilities in OPA, like decision logging and monitoring. An interesting trend which continues from 2022 is the increasing number of OPA use cases inside organizations. While infrastructure policies (including Kubernetes) still come out on top, we're seeing more organizations embrace OPA for policy across their whole stack. A standardized way of working with policy was always a goal of the project, so it's really exciting to see more organizations seeing the benefits of this approach!
|
||||
|
||||
### Open Policy Day with OPA
|
||||
|
||||
This year we were excited to see an entire event dedicated to OPA — the Open Policy Day with OPA co-located with KubeCon North America. During the course of the day, attendees got to hear end-user stories on using OPA in production, with speakers from organizations like Nvidia, T-Mobile. Capital One, Chime and Snowflake. If you couldn't attend in person, all the talks are up on [YouTube](https://www.youtube.com/@styra6251/videos)!
|
||||
|
||||
### Conferences and Meetups
|
||||
|
||||
In the cloud-native space, OPA was represented in talks at both KubeCon / CloudNativeCon [Europe](https://www.youtube.com/watch?v=MhyQxIp1H58&t=4s) as well as [North America](https://www.youtube.com/watch?v=RMiovzGGCfI). As the interest in policy as code grew over the year, we saw a number of talks, workshops and events focused on the topic, and many discovered the benefits of unified policy management across tech stacks and organizations.
|
||||
|
||||
Additionally — while OPA has been, and continues to be, a popular topic at cloud-native, security and DevOps themed meetups, it's great to see new meetup groups dedicated entirely to the topic of OPA. Several OPA meetups took place in 2022, and we'd love to see that trend continue in the new year! If you'd like to host your own, [let us know](https://www.openpolicyagent.org/community).
|
||||
|
||||
It was great to see so much buzz around OPA at these events, and much of that is thanks to the amazing work going on within the OPA projects — let's look into what's been going on there next!
|
||||
|
||||
## New Features
|
||||
|
||||
OPA and Rego saw a record number of new features added in 2022.
|
||||
|
||||
### New Keywords
|
||||
|
||||
A few new keywords made a big difference both to the aesthetics of Rego, as well as its functionality. First off, the new [`every` keyword](https://www.openpolicyagent.org/docs/v0.38.1/policy-language/#every-keyword) elegantly helps solve the problem of expressing "[for all](https://www.openpolicyagent.org/docs/latest/#for-some-and-for-all)" type of queries:
|
||||
|
||||
```rego
|
||||
import future.keywords.every
|
||||
|
||||
only_dev_servers {
|
||||
some site in sites
|
||||
site.name == "dev"
|
||||
|
||||
every server in site.servers {
|
||||
endswith(server.name, "-dev")
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Next, the new `if` keyword helps policy authors express their rules in the same way as they normally should be read — as conditional assignments: "allow is true if conditions x, y and z are true". As an added bonus, the `if` keyword allows skipping the braces around single-line rule bodies, leading to rule constructs that read more like plain English:
|
||||
|
||||
```rego
|
||||
allow {
|
||||
"admin" in input.user.roles
|
||||
}
|
||||
```
|
||||
|
||||
May now be written as:
|
||||
|
||||
```rego
|
||||
allow if "admin" in input.user.roles
|
||||
```
|
||||
|
||||
Similarly, the new `contains` keyword allows set-building partial rules to be expressed as "the set **contains** x if conditions x, y and z are true":
|
||||
|
||||
```rego
|
||||
deny[message] {
|
||||
input.user.security_clearance_level < 2
|
||||
message := "Security clearance level 2 or higher required"
|
||||
}
|
||||
```
|
||||
|
||||
May now be written as:
|
||||
|
||||
```rego
|
||||
deny contains message if {
|
||||
input.user.security_clearance_level < 2
|
||||
message := "Security clearance level 2 or higher required"
|
||||
}
|
||||
```
|
||||
|
||||
### Refs in Rule Heads
|
||||
|
||||
It is said that one of the hardest things in computer science is naming things… "[refs in rule heads](https://github.com/open-policy-agent/opa/releases/tag/v0.46.1)" seems to support that claim! Don't let the name intimidate you though, this is a great addition to Rego! Dynamically creating deeply nested objects would previously often require the use of nested packages, with each package provided in its own file. This is no longer the case, as nesting can now be expressed in one place:
|
||||
|
||||
```rego
|
||||
package policy
|
||||
|
||||
claims.user.name := concat(" ", [input.user.first_name, input.user.last_name])
|
||||
|
||||
request.method := input.request.method
|
||||
request.valid := validate(input.request)
|
||||
```
|
||||
|
||||
Evaluating the policy package will now provide something like the below result:
|
||||
|
||||
```json
|
||||
{
|
||||
"claims": {
|
||||
"user": {
|
||||
"name": "John Doe"
|
||||
}
|
||||
},
|
||||
"request": {
|
||||
"method": "PUT",
|
||||
"valid": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Built-in Function Metadata and Metadata Introspection
|
||||
|
||||
Another exciting addition to Rego this year was the introduction of [metadata annotations](https://www.openpolicyagent.org/docs/latest/annotations/). Previously, policy authors would have to devise custom methods and formats for annotating their policies, packages and rules with metadata in the form of comments. Native support for annotations now provides a unified way not only for authoring structured annotations, but also for parsing them using either the "opa inspect" command, programmatically from Go, or even from Rego policies themselves via the new built-in rego.metadata functions. Having a standardized way of annotating packages and rules with metadata should benefit both human consumers as well as tooling.
|
||||
|
||||
### Testing
|
||||
|
||||
A major advantage of treating policy as code is that code is testable. Unit testing provides effective guardrails around policies and rules, and allows frequent updates without the risk of breaking things. While test-driven development has been considered a best practice for Rego since the start, one feature that many found missing was the ability to mock functions. [Function mocking](https://www.openpolicyagent.org/docs/latest/policy-testing/#data-and-function-mocking) allows replacing built-in functions, like http.send, or custom functions, with different implementations, commonly free of side-effects, all using the same `with` keyword familiar to test authors.
|
||||
|
||||
### Policy and Data Distribution
|
||||
|
||||
Policy is commonly only half of the equation when OPA makes decisions — having access to up-to-date _data_ related to users, endpoints or resources is often just as important. Data tends to be more dynamic in nature than policy, and certain types of deployments require a **lot** of data. The combination of huge datasets and frequently changing data would previously require continuous transfer of all required data in a bundle — even when only a few attributes had been updated. The introduction of [delta bundles](https://www.openpolicyagent.org/docs/latest/management-bundles/#delta-bundles) solves this by providing a new bundle type containing only the changes to data made since the last fetch, i.e. the _delta_. A much-awaited feature, and one that will help ensure OPA covers even the most complex of use cases going forward.
|
||||
|
||||
Another introduction this year was support for [OCI](https://opencontainers.org/) [bundle registries](https://www.openpolicyagent.org/docs/latest/management-bundles/#oci-registry). In an increasingly containerized world, distribution of applications is no longer the only use case for containers, and providing policy and data using the same channels as you would for e.g. Docker images, simplify things considerably in many environments.
|
||||
|
||||
### Disk Storage
|
||||
|
||||
While delta bundles may solve the problem of _distributing_ large volumes of data, that data must eventually still be stored somewhere for OPA to make use of it. Up until this year, only a single option for storage was provided: in-memory. While this is normally the best place to store it for fast access, large datasets distributed in-memory across a large number of running instances quickly adds up, and many are those who learnt the hard way that the old slogan of "memory is cheap" isn't always true at scale. The [disk based storage](https://www.openpolicyagent.org/docs/latest/configuration/#disk-storage) option now provides a balance between performance and costs, and is a welcome addition to OPA in many types of integrations.
|
||||
|
||||
### Compiler Strict mode
|
||||
|
||||
The Topdown compiler in OPA was enhanced with a new [strict mode](https://www.openpolicyagent.org/docs/latest/strict/) option, allowing policy authors to catch common mistakes before deploying their policy to production. Unused imports and variables, or use of deprecated built-in functions — now all flagged by the compiler with strict mode turned on. Together with JSON schema-based [type checking](https://www.openpolicyagent.org/docs/latest/schemas/), developers are provided some powerful tools to ensure the robustness of their policy. Guard rails around the guard rails!
|
||||
|
||||
### Intermediate Representation (IR)
|
||||
|
||||
The OPA project has the ambitious goal of standardizing policy across the full stack, because of this it's sometimes necessary to consider alternative deployment models to the traditional standalone service. This year we saw OPA provide a new [intermediate representation](/blog/i-have-a-plan-exploring-the-opa-intermediate-representation-ir-format-7319cd94b37d) format, allowing custom implementations to parse and execute evaluation plans. An implementation for the JVM and Javascript has already been made available with the [Jarl](https://github.com/borgeby/jarl) project, and hopefully we'll see more to follow in 2023!
|
||||
|
||||
## Built-in functions
|
||||
|
||||
20 new [built-in functions](https://www.openpolicyagent.org/docs/latest/policy-reference/#built-in-functions) got added in 2022 — more than any year before! We saw new functions in almost every existing category, and OPA's exciting new [GraphQL](https://www.openpolicyagent.org/docs/latest/graphql-api-authorization/) capabilities create a category of their own.
|
||||
|
||||
**GraphQL**: `graphql.is_valid`, `graphql.parse`, `graphql.parse_and_verify`, `graphql.parse_query`, `graphql.parse_schema`, `graphql.schema_is_valid`
|
||||
|
||||
**Strings**: `indexof_n`, `regex.replace`, `strings.any_prefix_match`, `strings.any_suffix_match`
|
||||
|
||||
**Objects**: `object.subset`, `object.union_n`, `object.keys`
|
||||
|
||||
**Crypto**: `crypto.hmac.md5`, `crypto.hmac.sha1`, `crypto.hmac.sha256`, `crypto.hmac.sha512`
|
||||
|
||||
**Misc**: `providers.aws.sign_req`, `net.cidr_is_valid`, `graph.reachable_paths`
|
||||
|
||||
## Performance Improvements
|
||||
|
||||
Some improvements are more understated, though no less noteworthy. As a mature software project, deployed in production in thousands of organizations across the world, OPA needs to be both robust and performant. This year saw the following exciting improvements in OPA performance:
|
||||
|
||||
- The — optimize flag now works for more commands (previously only available for `opa build`)
|
||||
- Lazy objects optimization allows delaying evaluation of attributes until needed
|
||||
- Built-in function optimizations for `object.get`, `in`, `object.union_n`, and others
|
||||
- Two new highly optimized built-in functions for doing prefix and suffix matching _en masse_: `strings.any_prefix_match` and `strings.any_suffix_match`
|
||||
- Internal optimizations to set element addition, object insertion and set union.
|
||||
|
||||
## Ecosystem and integrations
|
||||
|
||||
### Gatekeeper
|
||||
|
||||
A whole lot of great things landed in the Gatekeeper project this year! Following recent developments, Gatekeeper was made compatible with the Kubernetes v1.25 shift from Pod Security Policies to Pod Security Admission. On the topic of Kubernetes workloads, the new [Validation of Workload Resources](https://open-policy-agent.github.io/gatekeeper/website/docs/expansion) feature allows writing rules that apply to any Pod spec, whether deployed as a standalone pod or embedded in a parent resource, like a Deployment. Similarly, Gatekeeper now also allows validating subresources. The [external data feature](https://open-policy-agent.github.io/gatekeeper/website/docs/externaldata), which, as the name implies, allows Gatekeeper to interface with various external data sources for validation and mutation, moved to beta this year, and the next feature to do so is [Gator](https://open-policy-agent.github.io/gatekeeper/website/docs/gator), which allows testing of Gatekeeper ConstraintTemplates and Constraints in a local environment. Finally, the mutation feature is now considered stable.
|
||||
|
||||
In addition to all the features listed above, Gatekeeper is now faster than ever before! Some of the most notable improvements include reduced time for template compilation, adding and evaluating constraints, a whopping ~20X reduction in persistent audit memory usage, reduced request duration for policies with replicated data and reduced CPU time when adding data to OPA storage.
|
||||
|
||||
The ecosystem around Gatekeeper also saw improvements this year, where the most notable ones were the new Gatekeeper Policies [website](https://open-policy-agent.github.io/gatekeeper-library/website), and the inclusion of Gatekeeper policies on [ArtifactHub](https://artifacthub.io/packages/search?repo=gatekeeper-policies).
|
||||
|
||||
### Conftest
|
||||
|
||||
Conftest saw a rapid pace of development this year, with 12 releases pushed — from version 0.29.0, and ending in version 0.37.0! The project added support for policy authoring using a number of additional file types — like env, hcl, jsonc, CycloneDX, and SPDX — as input. Possibly even more exciting is the addition of several new built-in functions exclusive to Conftest, like parse_config, parse_config_file, and parse_combined_config. These functions all allow policy authors to pull in configuration to test from inside of a policy, allowing a greater deal of flexibility in how config tests are executed.
|
||||
|
||||
The tooling around Conftest improved as well: when using the `--version` flag, the version of OPA used by Conftest will now also be displayed. Additionally, the new `--quiet` flag allows excluding anything but errors in the output, which should help in quickly identifying issues.
|
||||
|
||||
### Integrations
|
||||
|
||||
OPA would not be what it is without its massive ecosystem of tools, integrations and useful and fun projects. The year started out with some great news in the infrastructure space, with AWS opening up for the possibility of externalizing compliance checks of CloudFormation templates via hooks, and it did not take long for the [AWS CloudFormation hook for OPA](https://github.com/StyraInc/opa-aws-cloudformation-hook) to arrive on the scene. Later this year, Hashicorp announced [support for OPA](https://developer.hashicorp.com/terraform/cloud-docs/policy-enforcement) in their Terraform Cloud offering. A [Pulumi](https://github.com/pulumi/pulumi-policy-opa) integration was also added to the [ecosystem](https://www.openpolicyagent.org/docs/latest/ecosystem/). The message seems clear — the tool to use for infrastructure as code (IaC) compliance is OPA, and the language to define IaC policies is Rego!
|
||||
|
||||
Outside of the infrastructure space, we saw a number of interesting integrations being built by the community, like [Alfred](https://github.com/dolevf/Open-Policy-Agent-Alfred), a Rego Playground you can self host, a [CircleCI](https://circleci.com/docs/config-policy-management-overview/) integration for CI/CD pipeline policies, [fig](https://github.com/open-policy-agent/contrib/tree/main/opa_fig_autocomplete) support for command line auto-completion goodness, [self-sovereign identity](https://docs.walt.id/v/ssikit/ssi-kit/open-policy-agent) (SSI) integrations, and even policy-driven access to remote systems via [SansShell](https://github.com/Snowflake-Labs/sansshell). OPA-powered policy enforcement even made it to the desktop this year, with the CISA-developed [ScubaGear](https://github.com/cisagov/ScubaGear/) project using Rego for validating M365 tenant configurations!
|
||||
|
||||
Finally, a much awaited addition to the OPA ecosystem — the [Rego Style Guide](https://github.com/StyraInc/rego-style-guide) now offers policy authors a comprehensive set of rules and best practices for authoring Rego.
|
||||
|
||||
For a more comprehensive list of OPA integrations, check out the OPA [ecosystem page](https://www.openpolicyagent.org/docs/latest/ecosystem/), and the [Awesome OPA](https://github.com/anderseknert/awesome-opa) list.
|
||||
|
||||
## Credits
|
||||
|
||||
None of the above would have been made possible without the amazing community around OPA. In 2022, we saw an incredible number of people contribute to the project in all imaginable ways — code, documentation, bug reports, support discussions, integrations and tools. OPA is being used more and more widely around the world, and in different domains. With this growth, it'd be easy to overlook the huge effort in growing a community to support the project. We know how hard the community has worked to get to where we are and for that we are immensely grateful. Thank you all for getting us to where we are today and laying the foundation for another fantastic year with Open Policy Agent.
|
||||
@@ -0,0 +1,163 @@
|
||||
---
|
||||
title: "Open Policy Agent 2023, Year in Review"
|
||||
authors: ["anderseknert"]
|
||||
date: 2023-12-20
|
||||
slug: open-policy-agent-2023-year-in-review-4c12df22e351
|
||||
---
|
||||
|
||||
|
||||
|
||||
As 2023 draws to a close, the time has come to reflect on another important year for Open Policy Agent (OPA). Now more than two years deep into CNCF Graduated status, OPA continues to see accelerated growth in production deployments — and across a diverse range of use cases. Such use cases demand both performance and stability, while user base growth depends on learning resources and ease of use. This year, the OPA community has worked hard and delivered on all fronts, for new and experienced users alike. This post takes time to share how this was achieved; highlight prominent events and updates; celebrate input from the wider community and set the scene for a historic year of OPA in 2024.
|
||||
|
||||
## OPA 'Away From Keyboard'
|
||||
|
||||
While OPA users and maintainers predominantly collaborate online, there were a good number of occasions where OPA existed very much in the physical realm this year too.
|
||||
|
||||
KubeCon EU enabled a few OPA events in Amsterdam early this summer. For the first time ever, an OPA-themed ContribFest session was held, where OPA, [Conftest](https://www.conftest.dev) and [OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper) maintainers worked with new contributors to the different OPA projects. In Amsterdam we also saw an OPA meet-up where speakers from [Miro](https://medium.com/miro-engineering/how-miro-leverages-open-policy-agent-to-implement-authorization-as-a-service-763f08469e5), [Bankdata](https://www.bankdata.dk) and [Styra](http://styra.com) presented. At this KubeCon EU there were four OPA talks:
|
||||
|
||||
- [The Compliance Business Case for Kubernetes in the EU: Anders Eknert](https://www.youtube.com/watch?v=XoWf4QcSbDw)
|
||||
- [Open Policy Agent. (OPA) Intro & Deep Dive — Charlie Egan, Rita Zhang](https://www.youtube.com/watch?v=6RNp3m_THw4)
|
||||
- [Calling OPA from eBPF, Through WASM, in the Kernel? You've Gone Mad! — Nandor Kracser](https://www.youtube.com/watch?v=JSKNch6piyY)
|
||||
- [Scratching an Itch: Running Policy in Hard to Reach Places with WASM & OPA — Charlie Egan](https://www.youtube.com/watch?v=BdeBhukLwt4)
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Rolling forward a few months, OPA also had a strong presence in Chicago at KubeCon NA. KubeCon is a huge event and it was great to get so many eyes on OPA as part of the graduated projects update in the keynote session. On top of that, the OPA kiosk in the project pavilion was an important meeting place for maintainers and users at the event. Discussions covered all sorts of use cases from authorization of applications, Kubernetes admission, IAC policy and beyond. Don't forget to check out the [OPA project update](https://www.youtube.com/watch?v=wJkjsvVpj_Q) from the conference's maintainer track.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## From Strength to Strength
|
||||
|
||||
OPA grows in so many different ways each year it's sometimes hard to know how to quantify it. Here are some highlighted figures which illustrate OPA's trajectory as we enter 2024.
|
||||
|
||||
**2700 Contributors**. [Nearly 3000](https://opa.devstats.cncf.io/) people have helped make OPA into the project it is today. Contributors help make OPA better by making changes to docs and code; by participating in GitHub discussions and by filing bugs. What's equally impressive is how these contributors are from over 450 different companies. OPA is a general purpose, domain agnostic policy engine so it's vital the project is guided by such a varied contributor base.
|
||||
|
||||
**9 years** spent by users reading the documentation on the OPA website. This year OPA contributors worked hard and made over [160](https://gist.github.com/charlieegan3/533ca9795787265c8b536b32fd8e2c8b) updates to the docs; and so it's reassuring to look back at the end of the year and see just how many users benefited from the hard work.
|
||||
|
||||
**2000 Go repositories build on OPA**. [Integrating with OPA](https://www.openpolicyagent.org/docs/latest/integration/) has always been a priority so it's fantastic to see that just so many different projects are adding policy functionality in this way. With the [OPA SDK](https://pkg.go.dev/github.com/open-policy-agent/opa/sdk), it's possible to bring all the best parts of OPA right into your Go application making it a powerful tool when standardizing your policy as code stack.
|
||||
|
||||
**1.5 million Playground Runs.** The [Rego Playground](https://play.openpolicyagent.org) is for every OPA user, it's there as a learning tool, as a collaborative scratch pad and now also integrates the output from [Regal](https://docs.styra.com/regal), the new linter for Rego. On average, every 20s someone clicks the 'Evaluate' button on the playground, all day long, all year long. One of the major uses of the playground is for users and maintainers collaborating on support in the OPA Slack, if you're interested in getting help within your team or on the Slack, creating a minimal example on the playground is a place to start.
|
||||
|
||||
It's not just OPA's community that's moving forward in leaps and bounds, OPA itself has been keeping pace and has received loads of great updates this year too. Let's dig into that now.
|
||||
|
||||
## New Features
|
||||
|
||||
### General references in rule heads
|
||||
|
||||
The single most important addition to Rego this year was arguably general references in rule heads. Simply put, it is now possible to include variables in rule names (or "references"), making it possible to build complex, nested map structures which would previously require multiple rules distributed over several packages.
|
||||
|
||||
Example using dynamic policy composition to collect informative notices from all "rules" policies, and have them organized by category and title.
|
||||
|
||||
```rego
|
||||
grouped_notices[category][title] contains notice if {
|
||||
some category, title
|
||||
rules_to_run[category][title]
|
||||
|
||||
some notice in data.rules[category][title].notices
|
||||
}
|
||||
```
|
||||
|
||||
Output would be a nested structure, as expected:
|
||||
|
||||
```json
|
||||
{
|
||||
"grouped_notices": {
|
||||
"testing": {
|
||||
"file-missing-test-suffix": ["ignored"]
|
||||
},
|
||||
"custom": {
|
||||
"naming-convention": ["ignored"],
|
||||
"one-liner-rule": ["obsolete", "ignored"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For more examples and information, see the [OPA docs](https://www.openpolicyagent.org/docs/latest/policy-language/#rule-heads-containing-references) on the topic.
|
||||
|
||||
### Default keyword on functions
|
||||
|
||||
The default keyword has been around since forever, and is considered idiomatic for scenarios where a "fallback" value is needed, should rule evaluation fail in other rules sharing the same name. A long requested feature has been to extend support for default to cover custom functions, and 2023 was the year it [happened](https://www.openpolicyagent.org/docs/latest/policy-language/#default-keyword).
|
||||
|
||||
```rego
|
||||
package functions
|
||||
|
||||
default first_name(_) := "unknown"
|
||||
|
||||
first_name(full_name) := split(full_name, " ")[0]
|
||||
```
|
||||
|
||||
### New built-in functions
|
||||
|
||||
Seven new built-in functions were added to Rego this year. The [json.verify_schema](https://www.openpolicyagent.org/docs/latest/policy-reference/#builtin-object-jsonverify_schema) and [json.match_schema](https://www.openpolicyagent.org/docs/latest/policy-reference/#builtin-object-jsonmatch_schema) functions are both recent additions for evaluating policy against JSON schemas — a use case that's been increasingly common in recent times. The [time.format](https://www.openpolicyagent.org/docs/latest/policy-reference/#builtin-time-timeformat) function will help policy authors present dates and time using either a custom format, or one of the supported constants for [datetime formats](https://www.openpolicyagent.org/docs/latest/policy-reference/#timestamp-parsing) that were also added this year. Three new crypto functions were added: [crypto.hmac.equal](https://www.openpolicyagent.org/docs/latest/policy-reference/#builtin-crypto-cryptohmacequal), [crypto.x509.parse_keypair](https://www.openpolicyagent.org/docs/latest/policy-reference/#builtin-crypto-cryptox509parse_keypair), and [crypto.parse_private_keys](https://www.openpolicyagent.org/docs/latest/policy-reference/#builtin-crypto-cryptoparse_private_keys). Finally, the new [numbers.range_step](https://www.openpolicyagent.org/docs/latest/policy-reference/#builtin-numbers-numbersrange_step) function, which works just as numbers.range, but with a configurable step value.
|
||||
|
||||
### Package scoped annotations
|
||||
|
||||
A metadata annotation using the scope of package is now truly [scoped to the package](https://github.com/open-policy-agent/opa/releases/tag/v0.50.0), and not just the file in which it is declared. This allows for some interesting opportunities to separate metadata declarations from "implementing" packages, and build things like lightweight frameworks leveraging Rego's metadata annotations. Additionally, it'll allow defining package scoped annotations for "packages" created via general references in rule heads, where the package scope isn't directly allowed on the rule itself.
|
||||
|
||||
### Debugging
|
||||
|
||||
The Swiss army knife of OPA also known as "opa eval" got a new flag to help debugging policy this year. Using the `--show-builtin-errors` flag, policy authors may now get a list of all errors produced by built-in functions as part of evaluation, making it much faster to identify certain types of problems.
|
||||
|
||||
### Performance
|
||||
|
||||
OPA keeps getting faster, and in 2023 we saw some great improvements in this area. The [json.patch](https://www.openpolicyagent.org/docs/latest/policy-reference/#builtin-object-jsonpatch) built-in function was remodeled entirely and now performs extremely well even when provided with a huge list of changes. Performance isn't entirely in the hands of OPA though. Some [optimizations](https://www.openpolicyagent.org/docs/latest/policy-performance/) can only be performed at the level of an actual Rego policy, and OPA provides several tools to help policy authors with this. The [profiler](https://www.openpolicyagent.org/docs/latest/policy-performance/#profiling) (`opa eval --profile`) is one such tool, and from this year it'll now also include the number of generated expressions in evaluation. This helps policy authors better understand why some expressions are evaluated more times than what one might expect.
|
||||
|
||||
### Server
|
||||
|
||||
Several improvements to the server (and by extension, the [OPA SDK](https://www.openpolicyagent.org/docs/latest/integration/#integrating-with-the-go-sdk)) landed in OPA this year. Bundle fetching now works with [AWS Signing Version 4A](https://www.openpolicyagent.org/docs/latest/configuration/#aws-signature), allowing bundles hosted on AWS to be distributed across different geographical regions. Also, a new shorthand format for quickly running the server pointed at a remote bundle was [introduced](https://github.com/open-policy-agent/opa/releases/tag/v0.50.0). The OCI downloader saw several [new authentication methods](https://www.openpolicyagent.org/docs/v0.52.0/configuration/#using-private-image-from-oci-repositories) added. Finally, instance [labels](https://www.openpolicyagent.org/docs/v0.52.0/management-discovery/#limitations) may now be added via discovery allowing for greater flexibility in runtime re-configuration of long running OPAs.
|
||||
|
||||
### Monitoring
|
||||
|
||||
Given the large number of OPA instances running in production, having a good story around monitoring is essential. The [status API](https://www.openpolicyagent.org/docs/latest/management-status/#status-service-api) provides a way for any OPA deployed to report its current status to a centralized control plane or monitoring system. In 2023, several new metrics got added to the status reports, including most notably the request count for unauthorized calls to the OPA REST API when an [authentication/authorization policy](https://www.openpolicyagent.org/docs/latest/security/#authentication-and-authorization) is in use, as well as errors that might have happened in [decision logging](https://www.openpolicyagent.org/docs/latest/management-decision-logs/).
|
||||
|
||||
### Security
|
||||
|
||||
2023 was the year the OPA docker images finally made rootless the default, and the special "-rootless" images that previously existed for this purpose are now obsolete. If you're still using them, make sure to remove the suffix from the image on your next version upgrade!
|
||||
|
||||
## Ecosystem
|
||||
|
||||
### Gatekeeper
|
||||
|
||||
The OPA Gatekeeper project had a busy 2023, with many improvements landing this year. The external data feature allows users to connect with external data sources as part of policy evaluation. This year it gained support for caching of responses from external data providers for both audit and admission. A new AssignImage mutator which enables mutation of image registry or tag was also made available. The new PubSub feature (currently in alpha) enables users to subscribe to pubsub services to consume a large number of audit violations. Additionally, observability statistics for admission, audit and gator CLI are now available!
|
||||
|
||||
Speaking of the Gator CLI — the tool now prints violating object names on test output, and additionally supports trace and image flags. It may now also be provided an AdmissionReview object for verification.
|
||||
|
||||
Using the new (experimental) Kubernetes Native Validation feature, users can now write CEL (Common Expression Language) based rules in addition to Rego rules in constraint templates, similar to Kubernetes ValidatingAdmissionPolicy. Finally, the ExpansionTemplate feature, which enables validation of workload resources, has graduated to beta.
|
||||
|
||||
### Conftest
|
||||
|
||||
The Conftest project saw many improvements around tooling this year. A new `--strict` flag was added to the verify and test commands, which will enforce additional safety checks on the policies such as unused arguments, duplicate imports, [and more](https://www.openpolicyagent.org/docs/latest/policy-language/#strict-mode). Two more flags got added: the `--quiet` flag to the verify command which will silence success notifications and only show errors, and a `--config` flag which allows users to specify where the config file to be tested lives. Test results may now also be emitted in a format compatible with Azure DevOps. On the topic of formats, a new input format was added to the already long list of supported ones, and [textproto](https://protobuf.dev/reference/protobuf/textformat-spec/) files may now be targeted for policy evaluation too. Finally, the Confest Docker images now also support both the `linux/amd64` and `linux/arm64` platforms.
|
||||
|
||||
### OPA Ecosystem
|
||||
|
||||
One goal of the OPA project is to build a domain agnostic policy engine. Being domain agnostic is achieved by simultaneously building generic core policy functionality, while also supporting a range of out-of-the-box integrations for different use cases. This year, the wider OPA community has wholeheartedly delivered on the latter and listed 22 new integrations on the website. The OPA Ecosystem also has a new home as a top level page, where integrations can be browsed by category, [check it out](http://openpolicyagent.org/ecosystem/)!
|
||||
|
||||
|
||||
|
||||
Most new ecosystem additions this year have been with other open source tools, generally adding policy functionality to a larger tool or leaning on Rego to provide a solid foundation for a domain-specific policy tool. Some notable examples include:
|
||||
|
||||
- Source Code Management: [Reposaur](https://www.openpolicyagent.org/integrations/reposaur/), a repository compliance tool; and [Legitify](https://www.openpolicyagent.org/integrations/legitify/), a repository security configuration scanner.
|
||||
- Supply Chain Security: [dependency-management-data](https://www.openpolicyagent.org/integrations/dependency-management-data/), helps understand software dependency posture; and [Enterprise Contract](https://www.openpolicyagent.org/integrations/enterprise-contract/) verifies supply chain security artifacts with Rego policy.
|
||||
- Infrastructure CD checks: [Torque](https://www.openpolicyagent.org/integrations/torque/), [Spinnaker](https://www.openpolicyagent.org/integrations/spinnaker-pipeline/) integrate Rego-based checks for continuous deployment while [ccbr](https://www.openpolicyagent.org/integrations/wirelesssecuritylab/) and [BrainIAC](https://www.openpolicyagent.org/integrations/carbonetes/) support a range of checks on existing IAC codebases.
|
||||
- Extending Authorization with OPA: The data orchestration tool [Alluxio](https://www.openpolicyagent.org/integrations/alluxio/) now also supports delegation of permissions to OPA.
|
||||
|
||||
[Digger](https://www.openpolicyagent.org/integrations/digger/), an open source CI/CD orchestrator for Terraform both integrates OPA for user RBAC and leaning into existing tooling by leveraging OPA project [conftest](http://conftest.dev) for IAC policy.
|
||||
|
||||
Meanwhile, other integrations went deeper and applied Rego in previously unexplored ways. [regocpp](https://www.openpolicyagent.org/integrations/regocpp/) is a cutting-edge project from collaborators at Microsoft that aims to bring Rego to other environments, natively. Based on C++, regocpp supports a number of Rego built-ins and the grammar as of v0.55.0.
|
||||
|
||||
The aforementioned linter, [Regal](https://www.openpolicyagent.org/integrations/regal/) also pushes the boundaries of where Rego can be used to write policies. Using the JSON representation of the Rego abstract syntax tree, this project implements a range of [linting rules](https://docs.styra.com/regal/category/rules)… in Rego! Regal has already been deployed by a number of open source Rego policy libraries and now supports over 60 rules. Integrated with the [Rego Playground](https://play.openpolicyagent.org) the linter is already available to everyone. There's no doubt that this will be a great tool for OPA learners and long-timers alike while continuing to help [scale](https://thenewstack.io/scaling-open-source-community-by-getting-closer-to-users/) the OPA community.
|
||||
|
||||
If you're interested in listing your OPA integration or project, please see [the instructions](https://github.com/open-policy-agent/opa/tree/main/docs#opa-ecosystem) or stop by the #ecosystem channel in the [OPA slack](https://communityinviter.com/apps/openpolicyagent/signup) if you have any questions.
|
||||
|
||||
## Thanks
|
||||
|
||||
2023 was an exciting year for OPA and its community. With so many projects using, integrating or extending OPA for all sorts of use cases — and so many users helping to contribute in all sorts of ways — this community is truly a great place to be. Thank you all who helped make it so! Your efforts are seen and appreciated.
|
||||
|
||||
There's a lot of great stuff lined up for next year already, so buckle up, and let's `import future.2024`!
|
||||
|
||||
Special thanks to Charlie Egan, Rita Zhang and John Reese for having helped contribute to this blog.
|
||||
@@ -0,0 +1,181 @@
|
||||
---
|
||||
title: "OPA 1.0 is coming. Here's what you need to know."
|
||||
authors: ["anderseknert"]
|
||||
date: 2024-03-15
|
||||
slug: opa-1-0-is-coming-heres-what-you-need-to-know-c8fb0d258368
|
||||
---
|
||||
|
||||
|
||||
_Towards Open Policy Agent 1.0_
|
||||
|
||||
December 28th marked the 8th anniversary of the first commit in the Open Policy Agent project. 5000+ commits from more than 400 contributors later, we're starting to prepare for OPA 1.0.
|
||||
|
||||
Following the rules of semantic versioning, one would be excused to think of 1.0 as the first "stable" version. That's not really the case for this project. Since the first public releases of OPA, great care has been taken to ensure new changes don't break existing policy. Thousands of organizations have come to rely on OPA for policy enforcement across the whole stack — often for critical production use cases.
|
||||
|
||||
While many **features** have been added to OPA and the Rego language since 2015, few have ever been removed. All this means that almost any policy written eight, five or three years ago still evaluates using the very latest version OPA, just as it did when it was written! But to keep adding features without ever being able to remove things that might not have worked out as we imagined — or simply didn't age well — also comes at a price. An increased cost of maintenance for sure, but more importantly, old ideas, language constructs and built-in functions all add to the cost of learning Rego, compared to having policy look and act consistently using only modern equivalent features.
|
||||
|
||||
With OPA 1.0, we're aiming to fix this.
|
||||
|
||||
## TL;DR
|
||||
|
||||
If you have a busy day ahead and want to get right to something actionable — here's what you can do. From OPA [v0.59.0](https://github.com/open-policy-agent/opa/releases/tag/v0.59.0) and onwards, you can start to prepare for the changes in the upcoming 1.0 release following these steps:
|
||||
|
||||
- Use import rego.v1 in each of your Rego files. This replaces all future.keywords imports from previous versions, and is all you need to import until OPA 1.0. Use opa fmt --rego-v1 to format your policy with automatic additions of OPA v1.0 constructs, like if, contains and more. This will also replace any future.keywords imports with the rego.v1 import.
|
||||
- Use opa check --rego-v1 to ensure your policy is compatible with "Rego 1.0" mode.
|
||||
|
||||
We'll get back to practical concerns by the end of the blog, but before that, let's see what changes are planned for the first major OPA version.
|
||||
|
||||
### Changes to Rego coming in OPA 1.0
|
||||
|
||||
**Note: Below follows a non-comprehensive list subject to change.** While these features are planned — and most of them even implemented already! — updates may be made before the final 1.0 release. I'll do my best to keep this blog post up to date, as we will with the OPA 1.0 documentation.
|
||||
|
||||
#### The future is now — no more import future.keywords
|
||||
|
||||
Several keywords (in, every, _if_ and contains*)* have been added to OPA since the start. Introducing new keywords means there's always a risk that existing policy might break in case identifiers, like rule names or variables, have been named in a way that clashes with the new keywords. In order to prevent this, access to these keywords have required an import of future.keywords. We now live in that future.
|
||||
|
||||
|
||||
|
||||
OPA v1.0 makes import future.keywords a no-op, as all keywords are now made available everywhere. In the time before OPA 1.0 is released, the new rego.v1 should be used in place of future.keywords imports.
|
||||
|
||||
#### The if keyword made mandatory
|
||||
|
||||
The if keyword helps explain the "inverted if … then" nature of Rego rules, and makes rules easier to read. Additionally, any rule body with only a single expression, like:
|
||||
|
||||
```rego
|
||||
allow {
|
||||
"admin" in user.roles
|
||||
}
|
||||
```
|
||||
|
||||
May with the help of if be expressed as a one-liner, with the curly brackets removed:
|
||||
|
||||
```rego
|
||||
allow if "admin" in user.roles
|
||||
```
|
||||
|
||||
OPA 1.0 makes if a natural — and mandatory — part of every rule's anatomy.
|
||||
|
||||
**Tip:** Use the opa fmt --rego-v1 utility in OPA v0.59.0+ to automatically rewrite all of your rules with if added to the rule head.
|
||||
|
||||
#### The contains keyword made mandatory
|
||||
|
||||
The contains keyword helps express multi-value (or as they're often called, partial) rules — i.e. rules that build a _set_ of values. It also helps avoid ambiguities around certain classes of rules, like the fairly recently introduced "nested" rule type:
|
||||
|
||||
```rego
|
||||
users.names contains name if {
|
||||
# ...
|
||||
name := sprintf("%s, %s", [first_name, last_name])
|
||||
}
|
||||
```
|
||||
|
||||
The contains keyword was already mandatory for nested rules. OPA v1.0 makes the use of this consistent across all multi-value rules by making contains a requirement.
|
||||
|
||||
**Tip:** Use the opa fmt --rego-v1 utility in OPA v0.59.0+ to automatically rewrite all of your multi-value rules with contains added to the rule head.
|
||||
|
||||
#### Strict mode made (mostly) the default
|
||||
|
||||
Most of the rules that have existed in OPA [strict mode](https://www.openpolicyagent.org/docs/latest/policy-language/#strict-mode) will be made the default in OPA 1.0. This will help users catch mistakes early, and have them fixed right away. If you have been running opa check --strict as part of your policy build pipeline, you're already in the clear here.
|
||||
|
||||
The rules from strict mode that will be made default in OPA 1.0 are:
|
||||
|
||||
#### No duplicate imports
|
||||
|
||||
Duplicate imports should realistically not be a problem in any repo, and the fix is simply to have them removed.
|
||||
|
||||
```rego
|
||||
package policy
|
||||
|
||||
import data.authz
|
||||
import data.authz # this is now an error
|
||||
```
|
||||
|
||||
#### No deprecated built-in functions
|
||||
|
||||
Deprecated built-in functions will be removed in OPA 1.0, and most of them are trivial to replace using a single line of Rego, or a different built-in function.
|
||||
|
||||
```rego
|
||||
package policy
|
||||
|
||||
# simply change to use `true in {input.foo, input.bar}`
|
||||
# using `in` additionally has the benefit that it can be used
|
||||
# to check for any type of value, and not just boolean "true"
|
||||
a := any([input.foo, input.bar])
|
||||
|
||||
# change to use `every` keyword, e.g.
|
||||
# every x in [input.foo, input.bar] {
|
||||
# x == true
|
||||
# }
|
||||
# just like `in` may be used for much more, `every` can be
|
||||
# used to evaluate complex expressions
|
||||
e := all([input.foo, input.bar])
|
||||
|
||||
# simply use the minus (`-`) operator instead, e.g.
|
||||
# s3 := s1 - s2
|
||||
s3 := set_diff(s1, s2)
|
||||
|
||||
# simply change to use regex.match instead
|
||||
r := re_match(..)
|
||||
|
||||
# simply change to use net.cidr_intersects
|
||||
n := net.cidr_overlap(..)
|
||||
|
||||
# cast_array, cast_set, cast_string, cast_boolean, cast_null, cast_object
|
||||
# use the "is_x" equivalent built-in function in their place
|
||||
s := is_string("yes")
|
||||
```
|
||||
|
||||
#### input and data now reserved keywords
|
||||
|
||||
OPA 1.0 prohibits the use of input or data as identifiers:
|
||||
|
||||
```rego
|
||||
# this is not allowed in OPA 1.0
|
||||
input := "overloaded"
|
||||
|
||||
# and neither is this
|
||||
data := {}
|
||||
```
|
||||
|
||||
Overloading input has mostly been common in tests. Do note that with input as `{..}` remains valid. If you're using assignment to input however, (`input := {..}`), you'll just need to change the name to something like inp instead.
|
||||
|
||||
### Other changes coming in OPA 1.0
|
||||
|
||||
See the [1.0 tag](https://github.com/open-policy-agent/opa/issues?q=is%3Aissue+label%3A1.0+) in the OPA backlog for a list of all issues related to OPA 1.0. Do note though that not all issues marked 1.0 might be picked for inclusion, and new issues may pop up before the release!
|
||||
|
||||
Other notable changes include:
|
||||
|
||||
#### import rego.v1
|
||||
|
||||
As previously mentioned — beginning with OPA v0.59.0 a new handy import to help with the 1.0 transition is made available. By adding import rego.v1 to a Rego policy, you can tell OPA to treat the policy just as it will handle it once version 1.0 is released.
|
||||
|
||||
- Since rego.v1 implies all the (no longer) future keywords, the importing future.keywords is no longer needed when import rego.v1 is present, and will in fact be an error.
|
||||
- Just as in OPA 1.0, the use of if and contains will be enforced
|
||||
- The strict mode requirements brought in OPA 1.0 will be checked automatically
|
||||
|
||||
#### Bind server to localhost interface by default
|
||||
|
||||
In OPA 1.0, the server will bind to the localhost interface by default, and not 0.0.0.0 (all interfaces). This change is needed in order to avoid accidentally exposing OPA to the internet, which while uncommon (as OPA normally runs behind firewalls and gateways) still happens, and we should aim to provide a secure default. Should you still want to bind against 0.0.0.0, or some other interface, you can use the --addr flag of the opa run command, like opa run --server --addr 0.0.0.0:8181. The impact of this change is expected to be small, but good to keep in mind.
|
||||
|
||||
### Documentation
|
||||
|
||||
The OPA docs have been updated to cover much of what's mentioned in this blog in greater detail. It also covers more of the technical reasons some of these changes are needed. See the docs on [OPA 1.0](https://www.openpolicyagent.org/docs/latest/opa-1/) for more information.
|
||||
|
||||
### How to prepare
|
||||
|
||||
As we covered in the TL;DR section, we're providing a number of tools to help with the transition starting from OPA v0.59.0 already. These tools will likely be extended and improved in following releases, but starting to use them today will ensure as smooth transition as possible.
|
||||
|
||||
#### To summarize
|
||||
|
||||
- OPA 1.0 planned for release this year, including some backwards incompatible changes
|
||||
- Starting now, you should use import rego.v1 in all of your policies (this replaces future.keywords) imports
|
||||
- Use opa check --rego-v1 for testing compliance against 1.0
|
||||
- Use opa fmt --rego-v1 to have your Rego code updated for 1.0 compliance
|
||||
- Run the OPA server with the --v1-compatible flag for OPA 1.0 compliance
|
||||
|
||||
Also worth pointing out — following guides like the [Rego Style Guide](https://docs.styra.com/opa/rego-style-guide), and using tools like [Regal](https://docs.styra.com/regal), is an excellent way to ensure not just compliance with future changes to Rego, but that your current policy repo is continuously kept in the best possible condition.
|
||||
|
||||
If you have any questions, concerns or would like to provide feedback around the upcoming 1.0 release, or the tools made available to help you transition smoothly — don't hesitate to reach out using any of the below channels:
|
||||
|
||||
- The OPA community's [discussion board](https://github.com/open-policy-agent/community/)
|
||||
- The [OPA Slack](https://slack.openpolicyagent.org/)
|
||||
- The OPA project's [backlog](https://github.com/open-policy-agent/opa/issues) (to file an issue or feature request)
|
||||
+37
@@ -0,0 +1,37 @@
|
||||
---
|
||||
title: "Announcing OPA 1.0: A New Standard for Policy as Code"
|
||||
authors: ["charlie"]
|
||||
date: 2024-12-20
|
||||
slug: announcing-opa-1-0-a-new-standard-for-policy-as-code-a6d8427ee828
|
||||
---
|
||||
|
||||
|
||||
|
||||
We are excited to announce OPA 1.0, a milestone release consolidating an improved developer experience for the future of Policy as Code. After nearly 10 years of innovations and contributions from over 450 developers, OPA 1.0 is finally here. The release makes new functionality designed to simplify policy writing and improve the language's consistency the default. This release marks the beginning of a new era for our project and represents a robust foundation for Policy as Code projects in the years ahead.
|
||||
|
||||
Since the project's [CNCF graduation](https://www.cncf.io/announcements/2021/02/04/cloud-native-computing-foundation-announces-open-policy-agent-graduation/) at the start of 2021, our work has been focused on Rego's developer experience and consistency. While much of the new functionality has been around for some time, OPA 1.0 will make new features the default — warranting the [SemVer](https://semver.org/) bump to `v1.0.0`. At this time, we would like to take the chance to highlight the following key changes to the defaults in Rego v1:
|
||||
|
||||
- Using `if` for all rule definitions and `contains` for multi-value rules is now mandatory, not just when using the `rego.v1` import.
|
||||
- Other new keywords (`every`, `in`) are available without any imports.
|
||||
- Previously requirements that were only run in "strict mode" (like `opa check --strict`) are now the default. Duplicate imports and imports which shadow each other are no longer allowed.
|
||||
- OPA 1.0 comes with a range of backwards compatibility features to aid your migrations, please see the [v0 compatibility guide](https://www.openpolicyagent.org/docs/latest/v0-compatibility/) if you must continue to support v0 Rego.
|
||||
|
||||
Many users have already started using the new syntax (we started adding the new keywords three years ago back in v0.34.0) but for those who haven't; in order to get the best of OPA 1.0; and to remain abreast of follow on updates; users are encouraged to upgrade as soon as possible (using the backwards compatibility functionality if required). Most users will be able to update their Rego quickly as the process can be largely automated using the Rego tools already built into OPA. The process can be summarized as follows:
|
||||
|
||||
- Download and install an [OPA 1.0 binary](https://github.com/open-policy-agent/opa/releases/tag/v1.0.0) before running the following commands.
|
||||
- `opa check --v0-v1`: Find parser and compiler errors that might be present in old code.
|
||||
- `opa check --v0-v1 --strict`: Find problems in Rego code no longer permitted in OPA 1.0.
|
||||
- `opa fmt --write --v0-v1`: Automatically update code to the OPA 1.0 syntax.
|
||||
- `regal lint`: using Regal, the linter for Rego is also recommended to find bugs and performance issues.
|
||||
|
||||
For users looking for more detailed information, please see the following resources: [OPA Documentation on upgrading to 1.0](http://openpolicyagent.org/docs/latest/v0-upgrade/); the [Renovating Rego](https://www.styra.com/blog/renovating-rego/) post for older projects which digs into the above process in detail; the [Maintainer Track presentation](https://youtu.be/QuotLxFb2f4?feature=shared&t=800) from KubeCon NA 2024 for a video overview.
|
||||
|
||||
Those [integrating with OPA's Go packages](https://www.openpolicyagent.org/docs/latest/integration/) — both via the SDK and the low-level Rego package — are encouraged to update their applications to use the new v1 packages. This is also documented in the [upgrading documentation](http://openpolicyagent.org/docs/latest/v0-upgrade/) and is a straightforward process. In some cases, users or integrators will need to support both v0 and v1 Rego simultaneously in the same application. This is generally only applicable to those offering OPA as part of a managed offering where the Rego is controlled by end users. Those who do have this use case, please review the [v0 compatibility guide](https://www.openpolicyagent.org/docs/latest/v0-compatibility/) to review the most suitable option for your application.
|
||||
|
||||
One last thing, while OPA 1.0 is primarily about consolidating the Rego developer experience, the release also comes with some significant improvements to performance. Check the [release notes](https://github.com/open-policy-agent/opa/releases/tag/v1.0.0) to dig into the details.
|
||||
|
||||
And that's a wrap! OPA 1.0 is here, it's a milestone release for our project and we're proud of the effort this release represents. We're excited for you to upgrade and experience these improvements firsthand. As you do, please report any issues via [Slack](http://slack.openpolicyagent.org/) or [GitHub Discussions](https://github.com/orgs/open-policy-agent/discussions), and feel free to open bug reports or feature requests to help shape future releases. Also, don't forget to use the [Regal language server](https://docs.styra.com/regal/editor-support) to ensure your Rego code is efficient & error-free too.
|
||||
|
||||
We want to extend our gratitude to our incredible community for always testing the latest versions, reporting bugs, contributing code and supporting fellow community members on their Rego journeys. We're not done yet and look forward to working together for years to come.
|
||||
|
||||
_Happy holidays — the OPA team_
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
---
|
||||
title: "Introducing Swift OPA: Native Policy Evaluation for Swift"
|
||||
authors: ["shomron"]
|
||||
date: 2025-05-14
|
||||
slug: introducing-swift-opa-native-policy-evaluation-for-swift-d5136c8a662e
|
||||
---
|
||||
|
||||
Exciting news — Swift OPA, a new way to integrate OPA natively within Swift applications and services, has now been released.
|
||||
|
||||
|
||||
|
||||
Swift OPA builds on the robust foundation of OPA's [Intermediate Representation](/blog/i-have-a-plan-exploring-the-opa-intermediate-representation-ir-format-7319cd94b37d) (IR) plans. Introduced in 2022, the OPA toolchain gained the ability to compile Rego policy into IR, precisely defining the concrete steps necessary to evaluate the policy. Swift OPA then interprets these IR plans, allowing you to leverage OPA's rich policy natively within Swift applications and services.
|
||||
|
||||
Until today, integrating OPA with a Swift application or service required one of the following approaches:
|
||||
|
||||
- Inter-process communication to a standalone OPA instance
|
||||
- Evaluating Rego policy compiled into a Web Assembly (WASM) binary
|
||||
|
||||
Each of these approaches represent different ways of using OPA. For most of the OPA project's history, it's been used with a client/server architecture where communication takes place over the network. The embedded approach taken by WASM, on the other hand, avoids this step to enable in-process evaluation.
|
||||
|
||||
With its native language support, Swift OPA works more closely to the embedded approach described: evaluation occurs within the process boundary of your service or application. When developing Swift applications, this reduces latency, as well as complexity and operational overhead by simplifying builds and deployments.
|
||||
|
||||
## Current Status
|
||||
|
||||
Swift OPA development is currently focused on expanding support for OPA's built-in functions, starting with over 80 built-ins and a roadmap to add many more in the future. A rigorous conformance test suite has also been published to GitHub. A community discussion has been initiated about how to best publish the transformed data used by the test suite, with the goal of ensuring consistent behavior across different language implementations of OPA including Swift OPA.
|
||||
|
||||
It's exciting to add Swift support to the OPA community, and you're invited to start exploring Swift OPA today.
|
||||
|
||||
## How to get involved
|
||||
|
||||
You can explore Swift OPA now at [https://github.com/open-policy-agent/swift-opa](https://github.com/open-policy-agent/swift-opa), and join the community on the Open Policy Agent Slack channel: [#swift-opa](https://openpolicyagent.slack.com/archives/C08PCS1KJ48). Your feedback and ideas are welcome - and there are plenty of opportunities to contribute to the project!
|
||||
+45
@@ -0,0 +1,45 @@
|
||||
---
|
||||
title: "Note from Teemu, Tim, and Torin to the Open Policy Agent community"
|
||||
authors: ["timhinrichs"]
|
||||
date: 2025-08-20
|
||||
slug: note-from-teemu-tim-and-torin-to-the-open-policy-agent-community-2dbbfe494371
|
||||
---
|
||||
|
||||
|
||||
|
||||
Today we're excited to announce that the creators of Open Policy Agent (along with many team members from Styra) have joined Apple to continue our decade-long mission of delivering an open source solution to unifying policy across the cloud-native stack.
|
||||
|
||||
Apple is an enthusiastic OPA user, utilizing it as a key component of its authorization infrastructure to manage a vast portfolio of global-scale cloud services. Today's announcement demonstrates Apple's commitment to the OPA project by making a larger investment in the technology and the community.
|
||||
|
||||
Open Policy Agent has a rich and vibrant community of end-user organizations, vendors, and individuals each contributing ideas, integrations, docs, and code so anyone in the world can use OPA to enforce the policies they care about. We've been fortunate to see such a community grow over the last decade, and look forward to continuing our contributions to the project as active community members.
|
||||
|
||||
We've compiled a set of FAQs below to address questions.
|
||||
|
||||
**What does this mean for the Open Policy Agent project?**
|
||||
Open Policy Agent remains a CNCF graduated open source project and there are no changes to the project governance or licensing.
|
||||
|
||||
**What does this mean for Open Policy Agent maintainers?**
|
||||
There is no change to the list of maintainers, except for an organization change from Styra to Apple for the maintainers that are joining Apple.
|
||||
|
||||
**What will happen to the tools I use under the Styra GitHub?**
|
||||
We've initiated the community process for these repositories to be included in the CNCF OPA GitHub organization with the goal of deeper collaboration with the open source community:
|
||||
|
||||
- Styra's commercial distribution of OPA, EOPA: an optimized version of OPA designed for data heavy workloads with data-filtering functionality that was previously only available to enterprise customers.
|
||||
- OPA Control Plane: a new control plane for OPA capable of building bundles from git and additional datasources and deploying them to S3 on AWS, GCP, and Azure.
|
||||
- SDKs: SDKs for integrating with OPA including TypeScript, React, UCAST-Prisma, C#, ASP.NET, Java, and Springboot.
|
||||
- Regal: a linter for OPA's policy language Rego (partly written in Rego itself).
|
||||
|
||||
Other tools will be evaluated for contribution in the future and remain publicly available for the community.
|
||||
|
||||
**What will happen to the OPA website and Rego Playground?**
|
||||
The OPA website continues to remain available and managed by the CNCF and broader OPA community. The Rego Playground continues to be operated by Styra with no changes to current functionality.
|
||||
|
||||
**What is the planned roadmap for OPA?**
|
||||
We're excited to continue the development of OPA with the same monthly release schedule. The 2025 OPA roadmap includes the following categories of work:
|
||||
|
||||
- Language extensions (and/or/not, keywords in refs, string interpolation, partial-set-functions, ellipsis)
|
||||
- Type checking improvements
|
||||
- Tooling improvements (streaming OPA test results, debugger attach to trace, rule tracing)
|
||||
- Partial evaluation improvements (redundant expression elimination, 'in' handling)
|
||||
- Performance (multiple expression indexing, faster loading of dependency-free bundles)
|
||||
- Decision API and logging (congestion back-pressure, logging metadata, logging to disk)
|
||||
@@ -0,0 +1,59 @@
|
||||
tsandall:
|
||||
name: Torin Sandall
|
||||
image_url: /img/blog/authors/tsandall.png
|
||||
page: true
|
||||
|
||||
anderseknert:
|
||||
name: Anders Eknert
|
||||
image_url: /img/blog/authors/anderseknert.jpeg
|
||||
page: true
|
||||
|
||||
timhinrichs:
|
||||
name: Tim Hinrichs
|
||||
image_url: /img/blog/authors/timhinrichs.png
|
||||
page: true
|
||||
|
||||
peteroneilljr:
|
||||
name: Peter O'Neill
|
||||
image_url: /img/blog/authors/peteroneilljr.png
|
||||
page: true
|
||||
|
||||
charlie:
|
||||
name: Charlie Egan
|
||||
image_url: /img/blog/authors/charlie.jpeg
|
||||
page: true
|
||||
|
||||
shomron:
|
||||
name: Oren Shomron
|
||||
image_url: /img/blog/authors/shomron.png
|
||||
page: true
|
||||
|
||||
gshively11:
|
||||
name: Grant Shively
|
||||
image_url: /img/blog/authors/gshively11.png
|
||||
page: true
|
||||
|
||||
aavarghese:
|
||||
name: Ansu Varghese
|
||||
image_url: /img/blog/authors/aavarghese.png
|
||||
page: true
|
||||
|
||||
mandanavaziri:
|
||||
name: Mandana Vaziri
|
||||
image_url: /img/blog/authors/mandanavaziri.png
|
||||
page: true
|
||||
|
||||
soothseer:
|
||||
name: Asad Ali
|
||||
image_url: /img/blog/authors/soothseer.jpeg
|
||||
page: true
|
||||
|
||||
lperkins:
|
||||
name: Luc Perkins
|
||||
image_url: /img/blog/authors/lperkins.jpeg
|
||||
page: true
|
||||
|
||||
ashumania:
|
||||
name: Ash Narkar
|
||||
image_url: /img/blog/authors/ashumania.jpeg
|
||||
page: true
|
||||
@@ -129,3 +129,164 @@ datasources
|
||||
OCP's
|
||||
subnets
|
||||
bool
|
||||
Abrahamson
|
||||
Amye
|
||||
Anders
|
||||
Andreas
|
||||
Aniszczyk
|
||||
Ansu
|
||||
architected
|
||||
Athenz
|
||||
Atlassian
|
||||
Authzed
|
||||
backport
|
||||
backporting
|
||||
backports
|
||||
Bankdata
|
||||
benchstat
|
||||
Boran
|
||||
Brehmer
|
||||
bugfixes
|
||||
Ceph
|
||||
Cerkvenik
|
||||
colocated
|
||||
Composability
|
||||
Contribfest
|
||||
Crossplane
|
||||
Cyber
|
||||
Cybersecurity
|
||||
datastructures
|
||||
defacto
|
||||
Dockerfile
|
||||
Dockerfiles
|
||||
doggos
|
||||
DSLs
|
||||
Dvoretskyi
|
||||
eg
|
||||
Egan
|
||||
Eijsackers
|
||||
Ekenstam
|
||||
Eknert
|
||||
Emscripten
|
||||
ext_authz
|
||||
facto
|
||||
Fastly
|
||||
Florian
|
||||
Fmt
|
||||
Fu
|
||||
func
|
||||
Gasc
|
||||
Goast
|
||||
goroutine
|
||||
Gradle
|
||||
Gramer
|
||||
gzipped
|
||||
hackathons
|
||||
haha
|
||||
hardcoded
|
||||
Hausenblas
|
||||
heterogenous
|
||||
Hinrichs
|
||||
Hockin
|
||||
http
|
||||
ide
|
||||
Ihor
|
||||
Jeugt
|
||||
Kinesis
|
||||
Krach
|
||||
Kracser
|
||||
Krause
|
||||
Kubecon
|
||||
Kubehuddle
|
||||
kubernetes
|
||||
Kubernetes's
|
||||
learnings
|
||||
Lennard
|
||||
Linkerd
|
||||
Mandana
|
||||
Mangum
|
||||
Meisenzahl
|
||||
microcontroller
|
||||
Minio
|
||||
mutator
|
||||
Nandor
|
||||
Narkar
|
||||
Nason's
|
||||
nd_builtin_cache
|
||||
Nico
|
||||
Nvidia
|
||||
Olamide
|
||||
Omolola
|
||||
onwards
|
||||
Orgs
|
||||
Paget
|
||||
Parikh
|
||||
parse_config
|
||||
parse_config_file
|
||||
PDFs
|
||||
performant
|
||||
Pinterest
|
||||
Pipfiles
|
||||
Plotnick
|
||||
Popescu
|
||||
preload
|
||||
QAware
|
||||
QCon
|
||||
Renatus
|
||||
Rodzik
|
||||
rq
|
||||
Rushgrove
|
||||
Sandall
|
||||
sandboxed
|
||||
Sandeep
|
||||
Scavarda
|
||||
Searcy
|
||||
Seref
|
||||
serverless
|
||||
Shively
|
||||
Shomron
|
||||
shoutout
|
||||
shoutouts
|
||||
siloed
|
||||
Smythe
|
||||
Sorens
|
||||
Springboot
|
||||
src
|
||||
Strebel
|
||||
Styra
|
||||
Styra's
|
||||
subnet
|
||||
subproject
|
||||
subresources
|
||||
Tabita
|
||||
tcp
|
||||
Teemu
|
||||
Tekton
|
||||
toolchains
|
||||
Torin
|
||||
transpiling
|
||||
unorderedness
|
||||
Urvil
|
||||
Uzcategui
|
||||
Varghese
|
||||
Vaziri
|
||||
webserver
|
||||
wikipedia
|
||||
Zhang
|
||||
|
||||
# Usernames
|
||||
aavarghese
|
||||
anderseknert
|
||||
ashumania
|
||||
frankiecerk
|
||||
irodzik
|
||||
lperkins
|
||||
mandanavaziri
|
||||
nmeisenzahl
|
||||
nusairat
|
||||
peteroneilljr
|
||||
soothseer
|
||||
that_tech_tea
|
||||
timhinrichs
|
||||
tsandall
|
||||
vgramer
|
||||
|
||||
@@ -21,7 +21,7 @@ For a step-by-step walkthrough of evaluating a Rego policy _partially_, see [Eva
|
||||
|
||||
The translation of data policies into queries (like SQL WHERE clauses) is driven by _partial evaluation (PE)_ of a Rego query.
|
||||
|
||||
> With partial evaluation, callers specify that certain inputs or pieces of data are _unknown_. OPA evaluates as much of the policy as possible without touching parts that depend on unknown values.<sup>[1](https://blog.openpolicyagent.org/partial-evaluation-162750eaf422)</sup>
|
||||
> With partial evaluation, callers specify that certain inputs or pieces of data are _unknown_. OPA evaluates as much of the policy as possible without touching parts that depend on unknown values.<sup>[1](/blog/partial-evaluation-162750eaf422)</sup>
|
||||
|
||||
The _unknown_ values that remain during partial evaluation represent the pieces of data that represent your filter.
|
||||
|
||||
|
||||
@@ -70,13 +70,13 @@ directly.
|
||||
|
||||
### Use indexed statements
|
||||
|
||||
The linear-time fragment ensures that the cost of evaluation is no larger than the size of the policy. OPA lets you write non-linear policies, because sometimes you need to, and because sometimes it's convenient. The blog on [partial evaluation](https://blog.openpolicyagent.org/partial-evaluation-162750eaf422) describes one mechanism for converting non-linear policies into linear policies.
|
||||
The linear-time fragment ensures that the cost of evaluation is no larger than the size of the policy. OPA lets you write non-linear policies, because sometimes you need to, and because sometimes it's convenient. The blog on [partial evaluation](/blog/partial-evaluation-162750eaf422) describes one mechanism for converting non-linear policies into linear policies.
|
||||
|
||||
But as the size of the policy grows, the cost of evaluation grows with it. Sometimes the policy can grow large enough that even the linear-fragment fails to meet the performance budget.
|
||||
|
||||
In the linear fragment, OPA includes special algorithms that **index rules efficiently**, sometimes making evaluation constant-time, even as the policy grows. The more effective the indexing is the fewer rules need to be evaluated.
|
||||
|
||||
Here is an example policy from the [rule-indexing blog](https://blog.openpolicyagent.org/optimizing-opa-rule-indexing-59f03f17caf3) giving the details for these algorithms. See the rest of this section for details on indexed statements.
|
||||
Here is an example policy from the [rule-indexing blog](/blog/optimizing-opa-rule-indexing-59f03f17caf3) giving the details for these algorithms. See the rest of this section for details on indexed statements.
|
||||
|
||||
```rego
|
||||
package indexed
|
||||
@@ -1015,7 +1015,7 @@ For high-performance use cases:
|
||||
|
||||
- Write your policies to minimize iteration and search.
|
||||
- Use objects instead of arrays when you have a unique identifier for the elements of the array.
|
||||
- Consider [partial-evaluation](https://blog.openpolicyagent.org/partial-evaluation-162750eaf422) to compile non-linear policies to linear policies.
|
||||
- Write your policies with indexed statements so that [rule-indexing](https://blog.openpolicyagent.org/optimizing-opa-rule-indexing-59f03f17caf3) is effective.
|
||||
- Consider [partial-evaluation](/blog/partial-evaluation-162750eaf422) to compile non-linear policies to linear policies.
|
||||
- Write your policies with indexed statements so that [rule-indexing](/blog/optimizing-opa-rule-indexing-59f03f17caf3) is effective.
|
||||
- Use the profiler to help identify portions of the policy that would benefit the most from improved performance.
|
||||
- Use the benchmark tools to help get real world timing data and detect policy performance changes.
|
||||
|
||||
@@ -148,4 +148,4 @@ appropriate for all situations.
|
||||
#### Further Reading
|
||||
|
||||
- [print-or-trace-call](/projects/regal/rules/testing/print-or-trace-call) Regal Rule
|
||||
- [Introducing the OPA print function](https://blog.openpolicyagent.org/introducing-the-opa-print-function-809da6a13aee)
|
||||
- [Introducing the OPA print function](/blog/introducing-the-opa-print-function-809da6a13aee)
|
||||
|
||||
@@ -9,7 +9,7 @@ groups:
|
||||
- [Policy API](#policy-api) - manage policy loaded into the OPA instance.
|
||||
- [Data API](#data-api) - evaluate rules and retrieve data.
|
||||
- [Query API](#query-api) - execute ad hoc queries.
|
||||
- [Compile API](#compile-api) - access Rego's [Partial Evaluation](https://blog.openpolicyagent.org/partial-evaluation-162750eaf422) and data filtering functionality.
|
||||
- [Compile API](#compile-api) - access Rego's [Partial Evaluation](/blog/partial-evaluation-162750eaf422) and data filtering functionality.
|
||||
- [Health API](#health-api) - access instance operational health information.
|
||||
- [Config API](#config-api) - view instance configuration.
|
||||
- [Status API](#status-api) - view instance [status](./management-status) state.
|
||||
@@ -1430,10 +1430,10 @@ The [Compile API](#compile-api) allows you to partially evaluate Rego queries
|
||||
and obtain a simplified version of the policy. This is most useful when building
|
||||
integrations where policy logic is to be translated and evaluated in another
|
||||
environment. For example,
|
||||
[this post](https://blog.openpolicyagent.org/write-policy-in-opa-enforce-policy-in-sql-d9d24db93bf4)
|
||||
[this post](/blog/write-policy-in-opa-enforce-policy-in-sql-d9d24db93bf4)
|
||||
on the OPA blog shows how SQL can be generated based on Compile API output.
|
||||
For more details on Partial Evaluation in OPA, please refer to
|
||||
[this blog post](https://blog.openpolicyagent.org/partial-evaluation-162750eaf422).
|
||||
[this blog post](/blog/partial-evaluation-162750eaf422).
|
||||
|
||||
Note that non-determinstic builtins (like `http.send`) are _not evaluated_ during PE.
|
||||
You can change that by providing `nondeterminsticBuiltins: true` in your payload options.
|
||||
|
||||
@@ -47,7 +47,22 @@ const baseUrl = "/";
|
||||
breadcrumbs: false,
|
||||
sidebarPath: require.resolve("./src/lib/sidebars.js"),
|
||||
},
|
||||
blog: false,
|
||||
blog: {
|
||||
path: "blog",
|
||||
routeBasePath: "blog",
|
||||
blogTitle: "OPA Blog",
|
||||
blogDescription: "News, releases, and community updates from the Open Policy Agent project.",
|
||||
postsPerPage: 10,
|
||||
blogSidebarTitle: "OPA Blog Posts",
|
||||
blogSidebarCount: "ALL",
|
||||
showReadingTime: false,
|
||||
feedOptions: {
|
||||
type: ["rss", "atom"],
|
||||
title: "Open Policy Agent Blog",
|
||||
},
|
||||
onInlineTags: "ignore",
|
||||
onUntruncatedBlogPosts: "ignore",
|
||||
},
|
||||
theme: {
|
||||
customCss: require.resolve("./src/css/custom.css"),
|
||||
},
|
||||
@@ -146,7 +161,7 @@ const baseUrl = "/";
|
||||
{ to: "/support", label: "Support" },
|
||||
{ to: "/community", label: "Community" },
|
||||
{ to: "/survey", label: "Survey" },
|
||||
{ href: "https://blog.openpolicyagent.org/", label: "Blog" },
|
||||
{ to: "/blog", label: "Blog" },
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -679,6 +694,7 @@ The Linux Foundation has registered trademarks and uses trademarks. For a list o
|
||||
markdownExportPlugin,
|
||||
],
|
||||
clientModules: [
|
||||
require.resolve("./src/lib/gtagFallback.js"),
|
||||
require.resolve("./src/lib/playground.js"),
|
||||
require.resolve("./src/lib/kapa.js"),
|
||||
],
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
// Google's gtag script can be missing locally (blocked by an ad-blocker, or
|
||||
// not injected outside production builds), which makes the official
|
||||
// @docusaurus/plugin-google-gtag client module throw "window.gtag is not a
|
||||
// function" and trigger the full-screen dev error overlay. Stub it out so a
|
||||
// missing gtag just logs instead of crashing the page.
|
||||
if (typeof window !== "undefined" && typeof window.gtag !== "function") {
|
||||
window.gtag = (...args) => {
|
||||
console.log("[gtag stub]", ...args);
|
||||
};
|
||||
}
|
||||
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 22 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 56 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 6.3 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 29 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 8.1 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 80 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 77 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 65 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 59 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 128 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 3.9 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 63 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 26 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 130 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 27 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 25 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 47 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 24 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 80 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 90 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 40 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 22 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 4.4 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 147 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 82 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 103 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 24 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 103 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 44 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 44 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 23 KiB |
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user