ewilliams28/releases
1
0
Fork 0
mirror of https://github.com/open-policy-agent/opa.git synced 2026-08-12 19:32:48 -06:00
Code Issues Packages Projects Releases Wiki Activity

docs: Deploy a new OPA website under /new (#7534)

Browse Source 
* New site v1

Signed-off-by: Charlie Egan <charlie@styra.com>

* Add feature and language pages

Signed-off-by: Charlie Egan <charlie@styra.com>

* tidy data generation

Signed-off-by: Charlie Egan <charlie@styra.com>

* Import docs

Signed-off-by: Charlie Egan <charlie@styra.com>

* WIP

Signed-off-by: Charlie Egan <charlie@styra.com>

* Update content d65888c14f

Signed-off-by: Charlie Egan <charlie@styra.com>

* Refactor site to support any base path

Signed-off-by: Charlie Egan <charlie@styra.com>

* Use base URL for nav icons

Signed-off-by: Charlie Egan <charlie@styra.com>

* Refactor locations of community and support logos

Signed-off-by: Charlie Egan <charlie@styra.com>

* More asset refactoring

Signed-off-by: Charlie Egan <charlie@styra.com>

* Move k8s docs into correct location

Signed-off-by: Charlie Egan <charlie@styra.com>

* Build the sidebar

Signed-off-by: Charlie Egan <charlie@styra.com>

* Fix CLI formatting

Signed-off-by: Charlie Egan <charlie@styra.com>

* Fix broken links

Signed-off-by: Charlie Egan <charlie@styra.com>

* Tidy nav with some dividers

Signed-off-by: Charlie Egan <charlie@styra.com>

* Deploy under /new

Signed-off-by: Charlie Egan <charlie@styra.com>

* Deploy new site on preview too

Signed-off-by: Charlie Egan <charlie@styra.com>

* docs: Link to OPA repo for GH nav link

Signed-off-by: Charlie Egan <charlie@styra.com>

* Updates to homepage

Signed-off-by: Charlie Egan <charlie@styra.com>

* docs: Add a logo grid to homepage

Signed-off-by: Charlie Egan <charlie@styra.com>

* docs: Update homepage to have improved layout

Signed-off-by: Charlie Egan <charlie@styra.com>

* docs: Update homepage rego example

Signed-off-by: Charlie Egan <charlie@styra.com>

* docs: Update homepage logo list

Signed-off-by: Charlie Egan <charlie@styra.com>

* docs: Update PlaygroundExample to used ?? operator

Signed-off-by: Charlie Egan <charlie@styra.com>

* docs: Add a simple search

We might want to come back to this later to make the results better but
this is as good as the current search.

Signed-off-by: Charlie Egan <charlie@styra.com>

* docs: Add a docs archive page

Signed-off-by: Charlie Egan <charlie@styra.com>

* docs: Revise homepage content and use demo video

Signed-off-by: Charlie Egan <charlie@styra.com>

* docs: correct archive version list

Signed-off-by: Charlie Egan <charlie@styra.com>

* docs: Only support light mode

This is something we can come back to when we have time to review the
graphics for multi mode

Signed-off-by: Charlie Egan <charlie@styra.com>

* docs: Link is available by default

Signed-off-by: Charlie Egan <charlie@styra.com>

* docs: Remove blue hero color, and use smaller logo grid

Signed-off-by: Charlie Egan <charlie@styra.com>

* docs: Correct ecosystem feature links

Signed-off-by: Charlie Egan <charlie@styra.com>

* docs: Drop loadEcosystem

This is no longer needed as ecosystem pages are now loaded like other
pages (no special asset handling).

Signed-off-by: Charlie Egan <charlie@styra.com>

* docs: Update content to f308f61

This branch duplicates content to reformat some of it for the new site.
This commit ports in some of the new changes.

https://github.com/open-policy-agent/opa/compare/f7b002e17...f308f612b

Signed-off-by: Charlie Egan <charlie@styra.com>

* docs: Update slack logos

Use one with a transparent background instead.

Signed-off-by: Charlie Egan <charlie@styra.com>

* docs: Use styled components, clear custom css

This keeps custom css to a minimum. Used for side by side and for hiding
the codeAPI text.

Signed-off-by: Charlie Egan <charlie@styra.com>

* docs: Update archive to show 1.4.2 release

Signed-off-by: Charlie Egan <charlie@styra.com>

---------

Signed-off-by: Charlie Egan <charlie@styra.com>
This commit is contained in:
Charlie Egan
2025-05-12 10:36:10 +01:00
committed by GitHub
parent f308f612b6
commit 53ab5b7a80
453 changed files with 56341 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Makefile
+2 -2
View File
@@ -463,10 +463,10 @@ release-ci: push-image push-manifest-list-latest
endif
.PHONY: netlify-prod
netlify-prod: clean docs-clean build docs-production-build
netlify-prod: clean docs-clean build docs-production-build docs-new-site
.PHONY: netlify-preview
netlify-preview: clean docs-clean build docs-live-blocks-install-deps docs-live-blocks-test docs-dev-generate docs-preview-build
netlify-preview: clean docs-clean build docs-live-blocks-install-deps docs-dev-generate docs-preview-build docs-new-site
# Kept for compatibility. Use `make fuzz` instead.
.PHONY: check-fuzz
docs/Makefile
+10 -1
View File
@@ -88,4 +88,13 @@ preview-build:
--buildDrafts \
--buildFuture \
--ignoreCache
make live-blocks-inject
# this is not compatible with node 22
# make live-blocks-inject
.PHONY: new-site
new-site:
cd new && \
npm ci && \
npx docusaurus build && \
mkdir -p ../website/public/new && \
cp -r build/. ../website/public/new
docs/new/.gitignore
+2
View File
@@ -0,0 +1,2 @@
.docusaurus
build
docs/new/Makefile
+2
View File
@@ -0,0 +1,2 @@
dev:
npx docusaurus start
docs/new/docs/aws-cloudformation-hooks.md
+508
View File
@@ -0,0 +1,508 @@
---
title: AWS CloudFormation Hooks
---
[AWS CloudFormation Hooks](https://docs.aws.amazon.com/cloudformation-cli/latest/userguide/hooks.html) allows users to
verify AWS infrastructure components defined in AWS CloudFormation
[templates](https://aws.amazon.com/cloudformation/resources/templates/), like S3 Buckets or EC2 instances, prior to
deployment. This is done via **hooks**. Hooks are composed of custom code running in an AWS Lambda function, which is
invoked before a resource is created, updated or deleted.
AWS currently supports hooks written in either Java or Python, and provides a
[sample repository](https://github.com/aws-cloudformation/aws-cloudformation-samples), which includes example hooks
written in both languages. Since we'd rather use OPA for this purpose, we'd need some code to process the requests
handled by the hook and send them forward to OPA for policy decisions via its
[REST API](https://www.openpolicyagent.org/docs/latest/rest-api/) using
the [OPA AWS CloudFormation Hook](https://github.com/StyraInc/opa-aws-cloudformation-hook).
## Goals
This tutorial shows how to deploy an AWS CloudFormation Hook that forwards requests to OPA for policy decisions,
allowing us to use policy to determine whether a request to create, update or delete a resource should be
allowed or denied. We'll learn how to author policies that take the input structure of CloudFormation Templates into
account, and some special considerations to be aware of in this environment.
In addition, this tutorial shows how we can leverage dynamic policy composition to group and structure our policies in a
way that follows the domain to which they apply.
## Prerequisites
In order to complete this tutorial, the following prerequisites needs to be met:
- An AWS account, with permissions to deploy resources via AWS CloudFormation, and valid credentials available to the CLI commands
- The [AWS CLI](https://aws.amazon.com/cli/) (`aws`) tool
- The [CloudFormation CLI](https://docs.aws.amazon.com/cloudformation-cli/latest/userguide/what-is-cloudformation-cli.html) (`cfn`) tool
- Docker
- OPA server running at an endpoint reachable by the AWS Lambda function, either within the same AWS environment, or
elsewhere. While developing your CloudFormation policies, a good option is to run OPA locally, but exposed to the
public via a service like [tunnelmole](https://tunnelmole.com/docs), an open source tunneling tool or [ngrok](https://ngrok.com/),
a popular closed source tunneling tool.
## Steps
### 1. Install the CloudFormation Hook
To start out, clone the OPA AWS CloudFormation Hook repository:
```shell
git clone https://github.com/StyraInc/opa-aws-cloudformation-hook.git
cd opa-aws-cloudformation-hook
```
To install (but not activate) the hook provided in this repository into your AWS account, cd into the `hooks` directory
and run:
```shell
cd hooks
cfn submit --set-default
```
When the command above is finished (this may take several minutes), you should see output similar to this:
```
Successfully submitted type. Waiting for registration with token '16697881-de36-45b8-8bc4-d9744431fa82' to complete.
Registration complete.
{
'ProgressStatus': 'COMPLETE',
'Description': 'Deployment is currently in DEPLOY_STAGE of status COMPLETED',
'TypeArn': 'arn:aws:cloudformation:eu-north-1:687803501377:type/hook/Styra-OPA-Hook',
...
}
```
### 2. Configure the OPA AWS CloudFormation Hook
The hook is now installed but needs to be configured for your environment. First, copy the value of the `TypeArn`
attribute from the JSON output of the above command, and store it in an environment variable:
```shell
export HOOK_TYPE_ARN="arn:aws:cloudformation:eu-north-1:687803501377:type/hook/Styra-OPA-Hook"
```
Next, set the AWS region and the URL to use for calling OPA:
```shell
export AWS_REGION="eu-north-1"
export OPA_URL="https://cfn-opa.example.com"
```
**(OPTIONAL):** If you want to use a bearer token to authenticate against OPA, provide an ARN pointing to the AWS Secret
containing the token:
```shell
export OPA_AUTH_TOKEN_SECRET="arn:aws:secretsmanager:eu-north-1:687803501377:secret:opa-cfn-token-l26bHK"
```
With the configuration variables set, push the configuration to AWS (remove `opaAuthTokenSecret` if you don't intend to
use it):
```shell
aws cloudformation --region "$AWS_REGION" set-type-configuration \
--configuration "{\"CloudFormationConfiguration\":{\"HookConfiguration\":{\"TargetStacks\":\"ALL\",\"FailureMode\":\"FAIL\",\"Properties\":{\"opaUrl\": \"$OPA_URL\",\"opaAuthTokenSecret\":\"$OPA_AUTH_TOKEN_SECRET\"}}}}" \
--type-arn $HOOK_TYPE_ARN
```
The hook is now installed, configured and activated!
### 3. Learn the Domain
Before we proceed to write our first policy, let's take a closer look at the data we'll be working with.
#### AWS CloudFormation Templates
A template file is commonly a YAML or JSON file, describing a set of AWS resources. While a template may describe
multiple resources, the hook will send each resource for validation separately. Important to note here is that the
resource presented to the hook will be shown **exactly** as provided in the template file. The hook does not perform any
type preprocessing, such as adding default values where missing, or providing auto-generated names. Policy authors must
hence take into account that even "obvious" attributes like name might not be present in the resource provided for
evaluation. As an example, a template to deploy an S3 Bucket with default attributes may be as minimal as this:
```yaml
Resources:
ExampleS3Bucket:
Type: AWS::S3::Bucket
```
For more information on templates, see the
[AWS User Guide](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-guide.html) on that topic.
#### Input and Response Format
The OPA configured to receive requests from the CFN hook will have its input provided in this format:
```json
{
"action": "CREATE",
"hook": "Styra::OPA::Hook",
"resource": {
"id": "MyS3Bucket",
"name": "AWS::S3::Bucket",
"type": "AWS::S3::Bucket",
"properties": {
"Tags": [{ "Key": "Owner", "Value": "Platform Team" }],
"BucketName": "platform-bucket-1"
}
}
}
```
- The `action` is either `CREATE`, `UPDATE` or `DELETE`
- The `id` is the key of the resource, as provided in the template
- The `type` is divided by "resource domain" and the specific type, so e.g. the S3 domain may contain `Bucket`,
`BucketPolicy`, and so on.
The hook expects the response to contain a boolean `allow` attribute, and a list of (potential) `violations`:
```json
{
"allow": false,
"violations": [
"bucket must not be public",
"bucket name must follow naming standard"
]
}
```
Any request denied will be logged in [AWS CloudWatch](https://aws.amazon.com/cloudwatch/) for the same account.
### 4. Write a CloudFormation Hook Policy
With knowledge of the domain and the data model, we're ready to write our first CloudFormation Hook policy. Since we'll
have a single OPA endpoint servicing requests for all types of resources, we'll use the
[default decision](./configuration/#miscellaneous) policy, which by default queries the `system.main` rule. Let's add a
simple policy to block an S3 Bucket unless it has an `AccessControl` attribute set to `Private`:
```live:example/system:module
package system
main := {
"allow": count(deny) == 0,
"violations": deny,
}
deny contains msg if {
bucket_create_or_update
not bucket_is_private
msg := sprintf("S3 Bucket %s 'AccessControl' attribute value must be 'Private'", [input.resource.id])
}
bucket_create_or_update if {
input.resource.type == "AWS::S3::Bucket"
input.action in {"CREATE", "UPDATE"}
}
bucket_is_private if {
input.resource.properties.AccessControl == "Private"
}
```
Since we know that CloudFormation Templates may contain only the bare minimum of information, we can't assume that there
will be an `AccessControl` attribute present in the input at all. Using negation of boolean rules inside of our `deny`
rules help alleviate the problem of values potentially being undefined. Compare to the following deny rule, which might
look correct at a first glance:
```live:example/fail:module
deny contains msg if {
bucket_create_or_update
input.resource.properties.AccessControl != "Private"
msg := sprintf("S3 Bucket %s 'AccessControl' attribute value must be 'Private'", [input.resource.id])
}
```
This rule would work fine as long as there _is_ an `AccessControl` attribute present in the input resource, but would
fail (i.e. not evaluate) as soon as the property was missing, leading to the resource being allowed! Using helper rules
and negation is a good way to work with data that might or might not be present, and results in more readable policies,
too.
:::danger
Surprisingly, boolean values from CloudFormation Templates are provided to the hook in the form of **strings** (i.e.
"true" and "false"). Policy authors must take this into account, and explicitly check for the value of these
attributes. An example S3 bucket policy might for example want to check that public ACLs are blocked:
```live:example/boolean_fail:module:read_only
# Wrong: will allow both "true" and "false" values as both are considered "truthy"
block_public_acls if {
input.resource.properties.PublicAccessBlockConfiguration.BlockPublicAcls
}
```
```live:example/boolean_correct:module:read_only
# Correct: will allow only when property set to "true"
block_public_acls if {
input.resource.properties.PublicAccessBlockConfiguration.BlockPublicAcls == "true"
}
```
:::
### 5. Policy Enforcement Testing
With the above policy loaded into OPA, we may proceed to try it out. Let's deploy the minimal S3 Bucket from the
previous template example. Save the below minimal template to a file called `s3bucket.yaml`:
```yaml
Resources:
ExampleS3Bucket:
Type: AWS::S3::Bucket
```
Since our S3 bucket doesn't have an `AccessControl` attribute, it should be denied by the hook. We
deploy a template by creating a **stack**:
```shell
aws cloudformation create-stack --stack-name cfn-s3 --template-body file://s3bucket.yaml
```
The output of the above command will simply be a confirmation that the stack was deployed. It won't tell us whether the
deployment was successful or not. In order to know that, we'll need to check the stack events:
```shell
aws cloudformation describe-stack-events --stack-name cfn-s3
```
The output of the above command will be a list of all events associated with the `cfn-s3` stack. Among the events, you
should now find an item describing that the hook denied the request, and its reason for doing so:
```json
{
"StackEvents": [
{
"StackId": "arn:aws:cloudformation:eu-north-1:55523455647:stack/cfn-s3/4f605f70-b1ca-12ec-b4d8-0a63e869dfee",
"EventId": "ExampleS3Bucket-c243efd6-bfe7-3f10-8304-a0e40fe5f6f4",
"StackName": "cfn-s3",
"LogicalResourceId": "ExampleS3Bucket",
"PhysicalResourceId": "",
"ResourceType": "AWS::S3::Bucket",
"Timestamp": "2022-03-31T08:41:15.946000+00:00",
"ResourceStatus": "CREATE_IN_PROGRESS",
"HookType": "Styra::OPA::Hook",
"HookStatus": "HOOK_COMPLETE_FAILED",
"HookStatusReason": "Hook failed with message: S3 Bucket ExampleS3Bucket 'AccessControl' attribute value must be 'Private'",
"HookInvocationPoint": "PRE_PROVISION",
"HookFailureMode": "FAIL"
}
]
}
```
Congratulations! You've just successfully enforced your first CloudFormation Hook policy using OPA. Let's update the
template so that it passes our policy requirement:
**s3bucket.yaml**
```yaml
Resources:
ExampleS3Bucket:
Type: AWS::S3::Bucket
Properties:
AccessControl: Private
```
Even though our stack did not create an S3 bucket (as the change got rolled back), the **stack** still exists.
In order to try again, we'll first need to delete the existing stack:
```shell
aws cloudformation delete-stack --stack-name cfn-s3
```
Now, let's try again:
```shell
aws cloudformation create-stack --stack-name cfn-s3 --template-body file://s3bucket.yaml
```
Checking the output of `aws cloudformation describe-stack-events --stack-name cfn-s3` once more will now show that the
resource was created. Do note that this could take up to a minute, so if you don't see it immediately, rerun the command
a bit later.
```json
{
"StackEvents": [
{
"StackId": "arn:aws:cloudformation:eu-north-1:55523455647:stack/cfn-s3/4f605f70-b1ca-12ec-b4d8-0a63e869dfee",
"EventId": "e20fdfa0-b0d0-11ec-b669-0e70f1f560a6",
"StackName": "cfn-s3",
"LogicalResourceId": "cfn-s3",
"PhysicalResourceId": "arn:aws:cloudformation:eu-north-1:55523455647:stack/cfn-s3/cf418141-b0d9-11bc-b421-0a1244c68dd1",
"ResourceType": "AWS::CloudFormation::Stack",
"Timestamp": "2022-03-31T08:59:33.392000+00:00",
"ResourceStatus": "CREATE_COMPLETE"
}
]
}
```
Note: once our stack is successfully deployed, we can use the `update-stack` command after we've made changes to our
templates:
```shell
aws cloudformation update-stack --stack-name cfn-s3 --template-body file://s3bucket.yaml`
```
## Further Improvements
### Dynamic Policy Composition
Having a single policy file for all rules will quickly become unwieldy. Could we improve this somehow? One way of doing
that would be to use dynamic policy composition, where a single main policy acts as a "router", and forwards queries to
other packages based on attributes from the input. A natural attribute to use for CloudFormation templates might for
example be the resource type, allowing us to group our policies by the resource type they're meant to act on. Let's
take a look at what such a main policy might look like:
**main.rego**
```live:example/router:module
# METADATA
# description: |
# Dynamic routing to policy based in input.resource.type,
# aggregating the deny rules found in all policies with a
# matching package name
#
package system
main := {
"allow": count(violations) == 0,
"violations": violations,
}
# METADATA
# description: |
# Main routing logic, simply converting input.resource.type, e.g.
# AWS::S3::Bucket to data.aws.s3.bucket and returning that document.
#
# By default, only input.action == "CREATE" | "UPDATE" will be routed
# to the data.aws.s3.bucket document. If handling "DELETE" actions is
# desirable, one may create a special policy for that by simply appending
# "delete" to the package name, e.g. data.aws.s3.bucket.delete
#
route := document(lower(component), lower(type)) if {
["AWS", component, type] = split(input.resource.type, "::")
}
violations contains msg if {
# Aggregate all deny rules found in routed document
some msg in route.deny
}
#
# Basic input validation to avoid having to do this in each resource policy
#
violations contains "Missing input.resource" if {
not input.resource
}
violations contains "Missing input.resource.type" if {
not input.resource.type
}
violations contains "Missing input.resource.id" if {
not input.resource.id
}
violations contains "Missing input.action" if {
not input.action
}
#
# Helpers
#
document(component, type) := data.aws[component][type] if {
input.action != "DELETE"
}
document(component, type) := data.aws[component][type].delete if {
input.action == "DELETE"
}
```
The above policy will invoke the `route` rule to determine which package should be evaluated based on the
`input.resource.type`, transforming a value such as `AWS::S3::Bucket` into a call to the `data.aws.s3.bucket` package,
where each rule named `deny` will be evaluated, and the result aggregated into the final decision.
Since most of our policies will only deal with `CREATE` or `UPDATE` actions, we'd rather want to avoid having to check
for this in all of our rules. Instead, we'll have the router append `.delete` to the package name for `DELETE`
operations, so that a request to delete e.g. an S3 bucket would invoke the `data.aws.s3.bucket.delete` package (if it
exists).
Additionally, we'll also do some simple input validation at this stage, so that we may avoid doing so in our resource
specific policies.
We can now modify our original policy to verify S3 bucket resources only:
```live:example/bucket:module
package aws.s3.bucket
deny contains sprintf("S3 Bucket %s 'AccessControl' attribute value must be 'Private'", [input.resource.id]) if {
not bucket_is_private
}
bucket_is_private if {
input.resource.properties.AccessControl == "Private"
}
```
Note how we no longer need the `bucket_create_or_update` rule, as that is already asserted by the main policy.
Quite an improvement in terms of readability, and a good foundation for further policy authoring. If you'd like to see
more examples of policy utilizing this pattern, check out the
[policy directory](https://github.com/StyraInc/opa-aws-cloudformation-hook/tree/main/policy) in the OPA AWS
CloudFormation Hook repo.
### OPA Authentication via AWS Secrets
#### OPA Configuration
Since the OPA server does not run inside the AWS Lambda, it is a good idea to require authentication to access its REST
API, as described in the OPA [documentation](https://www.openpolicyagent.org/docs/latest/security/#authentication-and-authorization).
A simple authz policy for checking the bearer token might look something like this:
**authz.rego**
```live:example/authz:module
package system.authz
default allow := false
allow if {
input.identity == "my_secret_token"
}
```
Once created, remember to pass the appropriate flags to `opa run` to enable authentication / authorization:
```shell
opa run --server --authentication=token --authorization=basic .
```
#### OPA AWS CloudFormation Hook Configuration
If configured to use a bearer token for authenticating against OPA (by setting the `OPA_AUTH_TOKEN_SECRET` environment
variable as described in the section on configuring the hook), the hook will try to fetch the token from the
secret provided in the `opaAuthTokenSecret` (ARN) configuration attribute. Note that the token should be provided
as a plain string in the secret (i.e. the `SecretString`) and not wrapped in a JSON object.
In order to fetch the token, the hook will need to be permitted to perform the `secretsmanager:GetSecretValue`
operation. Note that the hook will **only** read the secret provided by `opaAuthTokenSecret`, but it's recommended
to limit the `HookTypePolicy` on the IAM role to the specific secret accessed, i.e. the same ARN provided in
`opaAuthTokenSecret`. Example `HookTypePolicy` to allow the hook access to a specific secret:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "secretsmanager:GetSecretValue",
"Resource": "arn:aws:secretsmanager:eu-north-1:673240551671:secret:opa-cfn-token-l26bHK"
}
]
}
```
If you aren't planning to use bearer tokens for authentication, you may remove the permission entirely.
docs/new/docs/cli.md
Executable
+1195
View File
File diff suppressed because it is too large Load Diff
docs/new/docs/comparison-to-other-systems.md
+499
View File
@@ -0,0 +1,499 @@
---
title: Comparison to Other Systems
---
Often the easiest way to understand a new language is by comparing
it to languages you already know. Here we show how policies from
several existing policy systems can be implemented with the Open
Policy Agent.
## Role-based access control (RBAC)
Role-based access control (RBAC) is pervasive today for authorization.
To use RBAC for authorization, you write down two different kinds of
information.
- Which users have which roles
- Which roles have which permissions
Once you provide RBAC with both those assignments, RBAC tells you
how to make an authorization decision. A user is authorized for
all those permissions assigned to any of the roles she is assigned to.
For example, we might have the following user/role assignments:
| User | Role |
| ------- | ------------- |
| `alice` | `engineering` |
| `alice` | `webdev` |
| `bob` | `hr` |
And the following role/permission assignments:
| Role | Permission | Resource |
| ------------- | ---------- | ------------- |
| `engineering` | `read` | `server123` |
| `webdev` | `write` | `server123` |
| `webdev` | `read` | `server123` |
| `hr` | `read` | `database456` |
In this example, RBAC makes the following authorization decisions:
| User | Operation | Resource | Decision |
| ------- | --------- | ------------- | -------------------------------------------------------- |
| `alice` | `read` | `server123` | `allow` because `alice` is in `engineering` |
| `alice` | `write` | `server123` | `allow` because `alice` is in `webdev` |
| `bob` | `read` | `database456` | `allow` because `bob` is in `hr` |
| `bob` | `read` | `server123` | `deny` because `bob` is not in `engineering` or `webdev` |
With OPA, you can write the following snippets to implement the
example RBAC policy shown above.
```live:rbac:module:openable
package rbac.authz
# user-role assignments
user_roles := {
"alice": ["engineering", "webdev"],
"bob": ["hr"],
}
# role-permissions assignments
role_permissions := {
"engineering": [{"action": "read", "object": "server123"}],
"webdev": [{"action": "read", "object": "server123"},
{"action": "write", "object": "server123"}],
"hr": [{"action": "read", "object": "database456"}],
}
# logic that implements RBAC.
default allow := false
allow if {
# lookup the list of roles for the user
roles := user_roles[input.user]
# for each role in that list
r := roles[_]
# lookup the permissions list for role r
permissions := role_permissions[r]
# for each permission
p := permissions[_]
# check if the permission granted to r matches the user's request
p == {"action": input.action, "object": input.object}
}
```
```live:rbac:query:hidden
allow
```
As you can see, querying the `allow` rule with the following input
```live:rbac:input
{
"user": "bob",
"action": "read",
"object": "server123"
}
```
Results in the response you'd expect.
```live:rbac:output
```
### RBAC Separation of duty (SOD)
Separation of duty (SOD) refers to the idea that there are certain
combinations of permissions that no one should have at the same time.
For example, no one should be able to both create payments and approve payments.
In RBAC, that means there are some pairs of roles that no one should be
assigned simultaneously. For example, any user assigned both of the roles
in each pair below would violate SOD.
- create-payment and approve-payment
- create-vendor and pay-vendor
OPA's API does not yet let you enforce SOD by rejecting improper role-assignments,
but it does let you express SOD constraints and ask for all SOD violations,
as shown below. (Here we assume the statements below are added to the RBAC
statements above.)
```live:rbac/sod:module:openable
# Pairs of roles that no user can be assigned to simultaneously
sod_roles := [
["create-payment", "approve-payment"],
["create-vendor", "pay-vendor"],
]
# Find all users violating SOD
sod_violation contains user if {
some user
# grab one role for a user
role1 := user_roles[user][_]
# grab another role for that same user
role2 := user_roles[user][_]
# check if those roles are forbidden by SOD
sod_roles[_] == [role1, role2]
}
```
(For those familiar with SOD, this is the static version since SOD violations
happen whenever a user is assigned two conflicting roles. The dynamic version of SOD allows
a single user to be assigned two conflicting roles but requires that the same user not
utilize those roles on the same transaction, which is out of scope for this document.)
## Attribute-based access control (ABAC)
With attribute-based access control, you make policy decisions using the
attributes of the users, objects, and actions involved in the request.
It has three main components:
- Attributes for users
- Attributes for objects
- Logic dictating which attribute combinations are authorized
For example, we might know the following attributes for our users
- alice
- joined the company 15 years ago
- is a trader
- bob
- joined the company 5 years ago
- is an analyst
We would also have attributes for the objects, in this case stock ticker symbols.
- MSFT
- is sold on NASDAQ
- sells at $59.20 per share
- AMZN
- is sold on NASDAQ
- sells at $813.64 per share
An example ABAC policy in english might be:
- Traders may purchase NASDAQ stocks for under $2M
- Traders with 10+ years experience may purchase NASDAQ stocks for under $5M
OPA supports ABAC policies as shown below.
```live:abac:module:openable
package abac
# User attributes
user_attributes := {
"alice": {"tenure": 15, "title": "trader"},
"bob": {"tenure": 5, "title": "analyst"},
}
# Stock attributes
ticker_attributes := {
"MSFT": {"exchange": "NASDAQ", "price": 59.20},
"AMZN": {"exchange": "NASDAQ", "price": 813.64},
}
default allow := false
# all traders may buy NASDAQ under $2M
allow if {
# lookup the user's attributes
user := user_attributes[input.user]
# check that the user is a trader
user.title == "trader"
# check that the stock being purchased is sold on the NASDAQ
ticker_attributes[input.ticker].exchange == "NASDAQ"
# check that the purchase amount is under $2M
input.amount <= 2000000
}
# traders with 10+ years experience may buy NASDAQ under $5M
allow if {
# lookup the user's attributes
user := user_attributes[input.user]
# check that the user is a trader
user.title == "trader"
# check that the stock being purchased is sold on the NASDAQ
ticker_attributes[input.ticker].exchange == "NASDAQ"
# check that the user has at least 10 years of experience
user.tenure > 10
# check that the purchase amount is under $5M
input.amount <= 5000000
}
```
```live:abac:query:hidden
allow
```
```live:abac:input
{
"user": "alice",
"ticker": "MSFT",
"action": "buy",
"amount": 1000000
}
```
Querying the `allow` rule with the input above returns the following answer:
```live:abac:output
```
In OPA, there's nothing special about users and objects. You can attach
attributes to anything. And the attributes can themselves be structured JSON objects
and have attributes on attributes on attributes, etc. Because OPA was designed to work
with arbitrarily nested JSON data, it supports incredibly rich ABAC policies.
## Amazon Web Services IAM
Amazon Web Services (AWS) lets you create policies that can be attached to users, roles, groups,
and selected resources. You write `allow` and `deny` statements to enforce which users/roles can/can't
execute which API calls on which resources under certain conditions.
By default all API access requests are implicitly denied (i.e., not allowed). Policy statements
can explicitly allow or deny API requests. If a request is both allowed and denied, it is always denied.
Let's assume that the following [customer managed policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html#customer-managed-policies) is defined in AWS:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "FirstStatement",
"Effect": "Allow",
"Action": ["iam:ChangePassword"],
"Resource": "*"
},
{
"Sid": "SecondStatement",
"Effect": "Allow",
"Action": "s3:ListAllMyBuckets",
"Resource": "*"
},
{
"Sid": "ThirdStatement",
"Effect": "Allow",
"Action": [
"s3:List*",
"s3:Get*"
],
"Resource": [
"arn:aws:s3:::confidential-data",
"arn:aws:s3:::confidential-data/*"
]
}
]
}
```
And the above policy is attached to principal alice in AWS using
[attach-user-policy](https://docs.aws.amazon.com/cli/latest/reference/iam/attach-user-policy.html) API.
In OPA, you write each of the AWS `allow` statements as a separate statement, and you
expect the input to have `principal`, `action`, and `resource` fields.
```live:iam:module:openable
package aws
default allow := false
# FirstStatement
allow if {
principals_match
input.action == "iam:ChangePassword"
}
# SecondStatement
allow if {
principals_match
input.action == "s3:ListAllMyBuckets"
}
# ThirdStatement
# Use helpers to handle implicit OR in the AWS policy.
# Below all of the 'principals_match', 'actions_match' and 'resources_match' must be true.
allow if {
principals_match
actions_match
resources_match
}
# principals_match is true if input.principal matches
principals_match if {
input.principal == "alice"
}
# actions_match is true if input.action matches one in the list
actions_match if {
# iterate over the actions in the list
actions := ["s3:List.*", "s3:Get.*"]
action := actions[_]
# check if input.action matches an action
regex.globs_match(input.action, action)
}
# resources_match is true if input.resource matches one in the list
resources_match if {
# iterate over the resources in the list
resources := ["arn:aws:s3:::confidential-data", "arn:aws:s3:::confidential-data/.*"]
resource := resources[_]
# check if input.resource matches a resource
regex.globs_match(input.resource, resource)
}
```
```live:iam:input
{
"principal": "alice",
"action": "ec2:StartInstance",
"resource": "arn:aws:ec2:::instance/i78999879"
}
```
Querying `allow` with the input above returns the following answer:
```live:iam:query:hidden
allow
```
```live:iam:output
```
## XACML
eXtensible Access Control Markup Language (XACML) was designed to express security policies: allow/deny decisions using attributes of users, resources, actions, and the environment.
The following policy says that users from the organization Curtiss or Packard who are US or Great Britain nationals and who work on DetailedDesign or Simulation are permitted access to documents about NavigationSystems.
```xml
<Policy xmlns="urn:oasis:names:tc:xacml:3.0:core:schema:wd-17" PolicyId="urn:curtiss:ba:taa"
Version="1.1"
RuleCombiningAlgId="urn:oasis:names:tc:xacml:3.0:rule-combining-algorithm:deny-unless-permit">
<Description>Policy for Business Authorization category TAA-1.1</Description>
<Target />
<Rule RuleId="Rule for NavigationSystems" Effect="Permit">
<Target>
<AnyOf>
<AllOf>
<Match MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">NavigationSystem</AttributeValue>
<AttributeDesignator
Category="urn:oasis:names:tc:xacml:3.0:attribute-category:resource"
AttributeId="urn:curtiss:names:tc:xacml:1.0:resource:Topics"
DataType="http://www.w3.org/2001/XMLSchema#string" MustBePresent="true" />
</Match>
</AllOf>
</AnyOf>
<AnyOf>
<AllOf>
<Match MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">Packard</AttributeValue>
<AttributeDesignator
Category="urn:oasis:names:tc:xacml:1.0:subject-category:access-subject"
AttributeId="http://schemas.tscp.org/2012-03/claims/OrganizationID"
DataType="http://www.w3.org/2001/XMLSchema#string" MustBePresent="true" />
</Match>
</AllOf>
<AllOf>
<Match MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">Curtiss</AttributeValue>
<AttributeDesignator
Category="urn:oasis:names:tc:xacml:1.0:subject-category:access-subject"
AttributeId="http://schemas.tscp.org/2012-03/claims/OrganizationID"
DataType="http://www.w3.org/2001/XMLSchema#string" MustBePresent="true" />
</Match>
</AllOf>
</AnyOf>
<AnyOf>
<AllOf>
<Match MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">GB</AttributeValue>
<AttributeDesignator
Category="urn:oasis:names:tc:xacml:1.0:subject-category:access-subject"
AttributeId="http://schemas.tscp.org/2012-03/claims/Nationality"
DataType="http://www.w3.org/2001/XMLSchema#string" MustBePresent="true" />
</Match>
</AllOf>
<AllOf>
<Match MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">US</AttributeValue>
<AttributeDesignator
Category="urn:oasis:names:tc:xacml:1.0:subject-category:access-subject"
AttributeId="http://schemas.tscp.org/2012-03/claims/Nationality"
DataType="http://www.w3.org/2001/XMLSchema#string" MustBePresent="true" />
</Match>
</AllOf>
</AnyOf>
<AnyOf>
<AllOf>
<Match MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">DetailedDesign</AttributeValue>
<AttributeDesignator
Category="urn:oasis:names:tc:xacml:1.0:subject-category:access-subject"
AttributeId="http://schemas.tscp.org/2012-03/claims/Work-Effort"
DataType="http://www.w3.org/2001/XMLSchema#string" MustBePresent="true" />
</Match>
</AllOf>
<AllOf>
<Match MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">Simulation</AttributeValue>
<AttributeDesignator
Category="urn:oasis:names:tc:xacml:1.0:subject-category:access-subject"
AttributeId="http://schemas.tscp.org/2012-03/claims/Work-Effort"
DataType="http://www.w3.org/2001/XMLSchema#string" MustBePresent="true" />
</Match>
</AllOf>
</AnyOf>
</Target>
</Rule>
</Policy>
```
The same statement is shown below in OPA. Here the inputs are assumed to be
roughly the same as for XACML: attributes of users, actions, and resources.
```live:xacml:module:openable
package xacml
# METADATA
# title: urn:curtiss:ba:taa:taa-1.1
# description: Policy for Business Authorization category TAA-1.1
default permit := false
permit if {
# Check that resource has a "NavigationSystem" entry
input.resource["NavigationSystem"]
# Check that organization is one of the options
input.user.organization in ["Packard", "Curtiss"]
# Check that nationality is one of the options
input.user.nationality in ["GB", "US"]
# Check that work_effort is one of the options
input.user.work_effort in ["DetailedDesign", "Simulation"]
}
```
```live:xacml:input
{
"user": {
"name": "alice",
"organization": "Packard",
"nationality": "GB",
"work_effort": "DetailedDesign"
},
"resource": {
"NavigationSystem": true
},
"action": {
"name": "read"
}
}
```
Querying `permit` with the input above returns the following answer:
```live:xacml:query:hidden
permit
```
```live:xacml:output
```
docs/new/docs/configuration.md
+1158
View File
File diff suppressed because it is too large Load Diff
docs/new/docs/contrib-adding-builtin-functions.md
+185
View File
@@ -0,0 +1,185 @@
---
title: Adding Built-in Functions
---
[Built-in Functions](./policy-reference/#built-in-functions)
can be added inside the `topdown` package.
Built-in functions may be upstreamed if they are generally useful and provide functionality that would be
impractical to implement natively in Rego (e.g., CIDR arithmetic). Implementations should avoid third-party
dependencies. If absolutely necessary, consider importing the code manually into the `internal` package.
:::info
Read more about extending OPA with custom built-in functions in go [here](./extensions#custom-built-in-functions-in-go).
:::
Adding a new built-in function involves the following steps:
1. [Declare and register](#declare-and-register) the function
2. [Implement](#implement) the function
3. [Test](#test) the function
4. [Document](#document) the function
5. [Add a capability](#add-a-capability) for the function
## Example
The following example adds a simple built-in function, `repeat(string, int)`, that returns a given string repeated a given number of times.
### Declare and Register
In `ast/builtins.go`, we declare the structure of our built-in function with a `Builtin` struct instance:
```go
var Repeat = &Builtin{
Name: "repeat", // The name of the function
Description: "Returns, as a string, the given string repeated the given number of times.",
Decl: types.NewFunction(
types.Args( // The built-in takes two arguments, where ..
types.Named("str", types.S).Description("string to repeat"), // named string argument
types.Named("count", types.N).Description("how often to repeat `str`"), // named number argument
),
types.Named("output", types.S).Description("the repetitions"), // The return type is a string.
),
Categories: category("strings"), // the category the built-in belongs to
}
```
To register the new built-in function, we locate the `DefaultBuiltins` array in `ast/builtins.go`, and add the `Builtin` instance to it:
```go
var DefaultBuiltins = [...]*Builtin{
...
Repeat,
...
}
```
### Implement
In the `topdown` package, we locate a suitable source file for our new built-in function, or add a new file, as appropriate.
In this example, we introduce a new source file, `topdown/repeat.go`:
```go
package topdown
import (
"fmt"
"strings"
"github.com/open-policy-agent/opa/v1/ast"
"github.com/open-policy-agent/opa/v1/topdown/builtins"
)
// implements topdown.BuiltinFunc
func builtinRepeat(_ BuiltinContext, operands []*ast.Term, iter func(*ast.Term) error) error {
// Get the first argument as a string, returning an error if it's not the correct type.
str, err := builtins.StringOperand(operands[0].Value, 1)
if err != nil {
return err
}
// Get the second argument as an int, returning an error if it's not the correct type or not a positive value.
count, err := builtins.IntOperand(operands[1].Value, 2)
if err != nil {
return err
} else if count < 0 {
// Defensive check, strings.Repeat(...) will panic for count<0
return fmt.Errorf("count must be a positive integer")
}
// Return a string by invoking the given iterator function
return iter(ast.StringTerm(strings.Repeat(string(str), count)))
}
func init() {
RegisterBuiltinFunc(ast.Repeat.Name, builtinRepeat)
}
```
In the above code, `builtinRepeat` implements the `topdown.BuiltinFunc` function type.
The call to `RegisterBuiltinFunc(...)` in `init()` adds the built-in function to the evaluation engine; binding the implementation to `ast.Repeat` that was registered in [an earlier step](#declare-and-register).
### Test
All built-in function implementations must include a test suite.
Test cases for built-in functions are written in YAML and located under `test/cases/testdata/v1`.
We create two new test cases (one positive, expecting a string output; and one negative, expecting an error) for our built-in function:
```yaml
cases:
- note: repeat/positive
query: data.test.p = x
modules:
- |
package test
p := repeated if {
repeated := repeat(input.str, input.count)
}
input: { "str": "Foo", "count": 3 }
want_result:
- x: FooFooFoo
- note: repeat/negative
query: data.test.p = x
modules:
- |
package test
p := repeated if {
repeated := repeat(input.str, input.count)
}
input: { "str": "Foo", "count": -3 }
strict_error: true
want_error_code: eval_builtin_error
want_error: "repeat: count must be a positive integer"
```
The above test cases can be run separate from all other tests through: `go test ./topdown -v -run 'TestRego/v1/repeat'`
See [test/cases/testdata/helloworld](https://github.com/open-policy-agent/opa/blob/main/test/cases/testdata/helloworld)
for a more detailed example of how to implement tests for your built-in functions.
:::info
Note: We can manually test our new built-in function by [building](./contrib-development#getting-started)
and running the `eval` command. E.g.: `$./opa_<OS>_<ARCH> eval 'repeat("Foo", 3)'`
:::
### Document
All built-in functions will automatically be documented in `docs/content/policy-reference.md` under an appropriate subsection.
For this example, we'll get an entry for our new function under the `Strings` section.
### Add a capability
:::info
Read more about extending the default capabilities list for built-ins [here](./deployments/#built-ins).
:::
One of the security features of OPA is [capabilities](./deployments/#capabilities) checks on policies, allowing users to restrict which built-in functions will be available to policies at runtime.
To ensure that our new `repeat` function will be available to callers, we'll need to add it to the `capabilities.json` file at the root of the repo.
We can have this entry auto-generated for us by running `make generate`.
After running `make generate` we should see a new JSON object entry in the list under the `"builtins"` key:
```json
...
{
"name": "repeat",
"decl": {
"args": [
{
"type": "string"
"type": "number"
}
],
"result": {
"type": "string"
},
"type": "function"
}
},
...
```
docs/new/docs/contrib-code.md
+189
View File
@@ -0,0 +1,189 @@
---
title: Contributing Code
---
## Code Contributions
If you are contributing code, please consider the following:
- Most changes should be accompanied with tests.
- All commits must be signed off (see next section).
- Related commits must be squashed before they are merged.
- All tests must pass and there must be no warnings from the `make check` target.
If you are new to Go, consider reading
[Effective Go](https://golang.org/doc/effective_go.html) and
[Go Code Review Comments](https://github.com/golang/go/wiki/CodeReviewComments)
for guidance on writing idiomatic Go code.
When you implement new features in OPA, consider whether the
types/functions you are adding need to be exported. Prefer
unexported types and functions as much as possible.
If you need to share logic across multiple OPA packages, consider
implementing it inside of the
`github.com/open-policy-agent/opa/internal` package. The `internal`
package is not visible outside of OPA.
Avoid adding third-party dependencies (vendoring). OPA is designed to be minimal,
lightweight, and easily embedded. Vendoring may make features _easier_ to
implement however they come with their own cost for both OPA developers and
OPA users (e.g., vendoring conflicts, security, debugging, etc.)
### Commit Messages
Commit messages should explain _why_ the changes were made and should probably
look like this:
```
Description of the change in 50 characters or less
More detail on what was changed. Provide some background on the issue
and describe how the changes address the issue. Feel free to use multiple
paragraphs but please keep each line under 72 characters or so.
```
If your changes are related to an open issue (bug or feature), please include
the following line at the end of your commit message:
```
Fixes #<ISSUE_NUMBER>
```
If the changes are isolated to a specific OPA package or directory please
include a prefix on the first line of the commit message with the following
format:
```
<package or directory path>: <description>
```
For example, a change to the `ast` package:
```
ast: Fix X when Y happens
<Details...>
Fixes: #123
Signed-off-by: Random J Developer <random@developer.example.org>
```
or a change in the OPA website content (found in `./docs/content`):
```
docs/website: Add X to homepage for Y
<Details...>
Fixes: #456
Signed-off-by: Random J Developer <random@developer.example.org>
```
### Developer Certificate Of Origin
The OPA project requires that contributors sign off on changes submitted to OPA
repositories.
The [Developer Certificate of Origin (DCO)](https://developercertificate.org/)
is a simple way to certify that you wrote or have the right to submit the code
you are contributing to the project.
The DCO is a standard requirement for Linux Foundation and CNCF projects.
You sign-off by adding the following to your commit messages:
```
This is my commit message
Signed-off-by: Random J Developer <random@developer.example.org>
```
Git has a `-s` command line option to do this automatically.
```sh
git commit -s -m 'This is my commit message'
```
You can find the full text of the DCO here: https://developercertificate.org/
:::info
**Note:** If using AI or machine learning tools to assist in the authoring
of OPA patches, you must ensure the code you produce is compliant with the
DCO requirements, and OPA's license. All commits in your patch _must_ be signed
off by a human author.
The OPA maintainers reserve the right to request additional information about
patches and reject PRs where code origin cannot be verified.
:::
### Code Review
Before a Pull Request is merged, it will undergo code review from other members
of the OPA community. In order to streamline the code review process, when
amending your Pull Request in response to a review, do not squash your changes
into relevant commits until it has been approved for merge. This allows the
reviewer to see what changes are new and removes the need to wade through code
that has not been modified to search for a small change.
When adding temporary patches in response to review comments, consider
formatting the message subject like one of the following:
- `Fixup into commit <commit ID> (squash before merge)`
- `Fixed changes requested by @username (squash before merge)`
- `Amended <description> (squash before merge)`
The purpose of these formats is to provide some context into the reason the
temporary commit exists, and to label it as needing squashed before a merge
is performed.
It is worth noting that not all changes need be squashed before a merge is
performed. Some changes made as a result of review stand well on their own,
independent of other commits in the series. Such changes should be made into
their own commit and added to the PR.
If your Pull Request is small though, it is acceptable to squash changes during
the review process. Use your judgement about what constitutes a small Pull
Request. If you aren't sure, send a message to the OPA slack or post a comment
on the Pull Request.
### Vulnerability scanning
On each Pull Request, a series of tests will be run to ensure that the code
is up to standard. Part of this process is also to run vulnerability scanning
on the code and on the generated container image.
[Trivy](https://aquasecurity.github.io/trivy/) is used to run the aforementioned
vulnerability scanning. To install, follow the [installation instructions](https://aquasecurity.github.io/trivy/v0.29.2/getting-started/installation/).
To run the vulnerability scanning, on the code-base, run the following command:
```bash
$ trivy fs .
```
To run the vulnerability scanning on the container image, run the following command:
```bash
$ trivy image <Image tag>
```
If the tool catches any false positives, it's recommended to appropriately document them
in the `.trivyignore` file.
## Contribution process
Small bug fixes (or other small improvements) can be submitted directly via a
[Pull Request](https://github.com/open-policy-agent/opa/pulls) on GitHub.
You can expect at least one of the OPA maintainers to respond quickly.
Before submitting large changes, please open an issue on GitHub outlining:
- The use case that your changes are applicable to.
- Steps to reproduce the issue(s) if applicable.
- Detailed description of what your changes would entail.
- Alternative solutions or approaches if applicable.
Use your judgement about what constitutes a large change. If you aren't sure,
send a message in
[#contributors](https://openpolicyagent.slack.com/archives/C02L1TLPN59) on Slack
or submit [an issue on GitHub](https://github.com/open-policy-agent/opa/issues).
docs/new/docs/contrib-development.md
+201
View File
@@ -0,0 +1,201 @@
---
title: Development
---
OPA is written in the [Go](https://golang.org) programming language.
If you are not familiar with Go we recommend you read through the [How to Write Go Code](https://golang.org/doc/code.html) article to familiarize yourself with the standard Go development environment.
Requirements:
- Git
- GitHub account (if you are contributing)
- Go (version 1.15+ is supported though older versions are likely to work)
- GNU Make
- Python3, pip, yamllint (if linting YAML files manually)
## Getting Started
After forking the repository and creating a [clone from your fork](https://docs.github.com/en/get-started/quickstart/contributing-to-projects),
just run `make`. This will:
- Build the OPA binary.
- Run all of the tests.
- Run all of the static analysis checks.
If the build was successful, a binary will be produced in the top directory (`opa_<OS>_<ARCH>`).
Verify the build was successful with `./opa_<OS>_<ARCH> run`.
You can re-build the project with `make build`, execute all of the tests
with `make test`, and execute all of the performance benchmarks with `make perf`.
For quicker development-test iteration, you may use `make test-short` during development,
and only run `make test` before submitting your changed. This avoids running the slowest
tests and normally completes under a minute (compared to the several minutes required to run
the full test suite).
The static analysis checks (e.g., `go fmt`, `golint`, `go vet`) can be run
with `make check`.
> To correct any imports or style errors run `make fmt`.
## Workflow
### Fork, clone, create a branch
Go to [https://github.com/open-policy-agent/opa](https://github.com/open-policy-agent/opa) and fork the repository
into your account by clicking the "Fork" button.
Clone the fork to your local machine:
```bash
git clone git@github.com/<GITHUB USERNAME>/opa.git opa
cd opa
git remote add upstream https://github.com/open-policy-agent/opa.git
```
Create a branch for your changes.
```bash
git checkout -b somefeature
```
### Developing your change
Develop your changes and regularly update your local branch against upstream,
for example by rebasing:
```bash
git fetch upstream
git rebase upstream/main
```
> Be sure to run `make check` before submitting your pull request. You
> may need to run `go fmt` on your code to make it comply with standard Go
> style.
> For YAML files, you may need to run the `yamllint` tool on the
> `test/cases/testdata` folder to make sure any new tests are well-formatted.
> If you have Docker available, you can run `make check-yaml-tests` to
> run `yamllint` on the tests without installing any Python dependencies.
### Submission
Commit changes and push to your fork.
```bash
git commit -s
git push origin somefeature
```
> Make sure to use a [good commit message](./contrib-code/#commit-messages).
Now, submit a Pull Request from your fork.
See the official [GitHub Documentation](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request-from-a-fork)
for instructions to create the request.
> Hint: You should be prompted to with a "Compare and Pull Request" button
> that mentions your new branch on [https://github.com/open-policy-agent/opa](https://github.com/open-policy-agent/opa)
Once your Pull Request has been reviewed and signed off please squash your
commits. If you have a specific reason to leave multiple commits in the
Pull Request, please mention it in the discussion.
> If you are not familiar with squashing commits, see [the following blog post for a good overview](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html).
## Benchmarks
Several packages in this repository implement benchmark tests. To execute the
benchmarks you can run `make perf` in the top-level directory. We use the Go
benchmarking framework for all benchmarks.
## Dependencies
OPA is a Go module [https://github.com/golang/go/wiki/Modules](https://github.com/golang/go/wiki/Modules)
and dependencies are tracked with the standard [go.mod](https://github.com/open-policy-agent/opa/blob/main/go.mod) file.
We also keep a full copy of the dependencies in the [vendor](https://github.com/open-policy-agent/opa/tree/main/vendor)
directory. All `go` commands from the [Makefile](https://github.com/open-policy-agent/opa/blob/main/Makefile) will enable
module mode by setting `GO111MODULE=on GOFLAGS=-mod=vendor` which will also
force using the `vendor` directory.
To update a dependency ensure that `GO111MODULE` is either on, or the repository
qualifies for `auto` to enable module mode. Then simply use `go get ..` to get
the version desired. This should update the [go.mod](https://github.com/open-policy-agent/opa/blob/main/go.mod) and (potentially)
[go.sum](https://github.com/open-policy-agent/opa/blob/main/go.sum) files. After this you _MUST_ run `go mod vendor` to ensure
that the `vendor` directory is in sync.
Example workflow for updating a dependency:
```bash
go get -u github.com/sirupsen/logrus@v1.4.2 # Get the specified version of the package.
go mod tidy # (Somewhat optional) Prunes removed dependencies.
go mod vendor # Ensure the vendor directory is up to date.
```
If dependencies have been removed ensure to run `go mod tidy` to clean them up.
### Tool Dependencies
Sometimes we use some tools which are versioned and vendored
with OPA as dependencies. For now, we have none, but any we use in the future
should go in [tools.go](https://github.com/open-policy-agent/opa/blob/main/tools.go).
More details on the pattern: [https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md](https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md)
Update these the same way as any other Go package. Ensure that any build script
only uses `go run ./vendor/<tool pkg>` to force using the correct version.
### Go
If you need to update the version of Go used to build OPA you must update these
files in the root of this repository:
- `.go-version`- which is used by the Makefile and CI tooling. Put the exact go
version that OPA should use.
## Refactoring and Style Fixes
If you've found some code that you think would benefit from a refactoring — either by making it more readable or more
performant, that's great! Some things should however be considered before you submit such a change:
- Avoid mixing bug fixes and feature PRs with refactorings or style fixes. These PRs are generally difficult to review.
Instead, split your work up in multiple, separate PRs. If a refactoring is "needed" for a feature, at least ensure to
split the two into separate commits.
- If you intend to work on a larger refactoring project, make sure to first create an issue for discussion. Sometimes
things are the way they are for a reason, even when it's not immediately obvious.
- Ensure that there are tests covering the code subject to change.
## CI Configuration
OPA uses Github Actions defined in the [.github/workflows](https://github.com/open-policy-agent/opa/tree/main/.github/workflows)
directory.
### Github Action Secrets
The following secrets are used by the Github Action workflows:
| Name | Description |
| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| S3_RELEASE_BUCKET | AWS S3 Bucket name to upload `edge` release binaries to. Optional -- If not provided the release upload steps are skipped. |
| AWS_ACCESS_KEY_ID | AWS credentials required to upload to the configured `S3_RELEASE_BUCKET`. Optional -- If not provided the release upload steps are skipped. |
| AWS_SECRET_ACCESS_KEY | AWS credentials required to upload to the configured `S3_RELEASE_BUCKET`. Optional -- If not provided the release upload steps are skipped. |
| DOCKER_IMAGE | Full docker image name (with org) to tag and publish OPA images. Optional -- If not provided the image defaults to `openpolicyagent/opa`. |
| DOCKER_WASM_BUILDER_IMAGE | Full docker image name (with org) to tag and publish WASM builder images. Optional -- If not provided the image defaults to `openpolicyagent/opa-wasm-builder`. |
| DOCKER_USER | Docker username for uploading release images. Will be used with `docker login`. Optional -- If not provided the image push steps are skipped. |
| DOCKER_PASSWORD | Docker password or API token for the configured `DOCKER_USER`. Will be used with `docker login`. Optional -- If not provided the image push steps are skipped. |
| SLACK_NOTIFICATION_WEBHOOK | Slack webhook for sending notifications. Optional -- If not provided the notification steps are skipped. |
| TELEMETRY_URL | URL to inject at build-time for OPA version reporting. Optional -- If not provided the default value in OPA's source is used. |
| NETLIFY_BUILD_HOOK_URL | URL to trigger Netlify (openpolicyagent.org) deploys after release. Optional -- If not provided the Netlify steps are skipped. |
### Periodic Workflows
Some of the Github Action workflows are triggered on a schedule, and not included in the
post-merge, pull-request, etc actions. These are reserved for time consuming or potentially
non-deterministic jobs (race detection tests, fuzzing, etc).
Below is a list of workflows and links to their status:
| Workflow | Description |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ |
| [![Nightly](https://github.com/open-policy-agent/opa/workflows/Nightly/badge.svg?branch=main)](https://github.com/open-policy-agent/opa/actions?query=workflow%3A"Nightly") | Runs once per day at 8:00 UTC. |
docs/new/docs/contrib-docs.md
+66
View File
@@ -0,0 +1,66 @@
---
title: Contributing Docs
---
Contributing to our docs is one of the best ways to get started contributing to the OPA project. The OPA docs are one of the first places most people go for help, so any changes are impactful for the community. Thanks in advance for contributing to the docs.
## Sub-project docs
Docs for the OPA sub-projects each have their own home. Check out their docs to see how to contribute.
- [Gatekeeper](https://open-policy-agent.github.io/gatekeeper/website/docs/) docs for Kubernetes Admission Control
- [Conftest](https://www.conftest.dev/) docs for validating your structured configuration data, like YAML and HCL files
## Docs File Structure
- [devel/](https://github.com/open-policy-agent/opa/blob/main/docs/devel) - Developer documentation for OPA (not part of the website)
- [website/](https://github.com/open-policy-agent/opa/blob/main/docs/website) - This directory contains all of the Markdown, HTML, Sass/CSS, and other assets needed to build the [openpolicyagent.org](https://openpolicyagent.org) website. See the section below for steps to build the site and test documentation changes locally. This content is not versioned for each release, it is common scaffolding for the website.
- [content/](https://github.com/open-policy-agent/opa/blob/main/docs/content) - The raw OPA documentation can be found under the directory. This content is versioned for each release and should have all images and code snippets alongside the markdown content files.
- [content/integrations, content/organizations, content/softwares](https://github.com/open-policy-agent/opa/blob/main/docs/content) - the source for data used to generate the [OPA Ecosystem](https://www.openpolicyagent.org/docs/latest/ecosystem/)
## Markdown Page Structure
Each page starts with a Frontmatter block that provides the necessary metadata to format the Docs website. For example, the frontmatter for this page is as follows:
```YAML
---
title: Contributing Docs #Title of the page
kind: contrib #Section the page belongs to
weight: 2 #Order the page should appear in the left hand nav
---
```
After the Frontmatter block, each page uses standard markdown formatting. For a primer on Markdown, check out the [GitHub Markdown Guide](https://docs.github.com/en/github/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax).
## Create a local branch
To get started, fork the OPA repository and create a local branch for your Docs changes. Check out the [Development Guide](./contrib-development/#fork-clone-create-a-branch) if you need some help setting this up.
## Update Existing Docs
Navigate to the [content/](https://github.com/open-policy-agent/opa/blob/main/docs/content) folder in your local environment. Each top level item in the documentation nav will have an associated markdown file in the content folder. Locate the file you wish to update and confirm the title in the Frontmatter matches. Once you've located the correct page, edit the markdown page as necessary.
## Adding New Pages
In the case where you want to add a topic that doesn't fit nicely into any of the existing pages, it may make sense to add a new page. Create a markdown file in the content folder and add the appropriate Frontmatter heading. Aside from the title, you will need to specify the `kind` (section in the docs) and `weight` (order the page appears).
## Test your changes
Once you have made your updates, the next step is to test that they look as expected. To test your changes, generate a local preview with your updated files and preview them with Netlify.
Summary of steps:
1. Install dependencies: [Hugo](https://github.com/open-policy-agent/opa/tree/main/docs#installing-hugo), [NodeJS](https://nodejs.org), and [Netlify CLI](https://www.netlify.com/products/dev/)
1. Build artifacts: `make build`
1. Generate HTML files: `make docs-serve-local` (_This can take up to 5 min_)
1. Start preview server: `netlify dev`
For detailed instructions on setting up local preview, and remote previews with Netlify, check out the docs [README page](https://github.com/open-policy-agent/opa/blob/main/docs/README.md#how-to-edit-and-test).
## Submit a Pull Request
Once you've tested your changes and you're happy with how they look, commit them to your branch and open a pull request. If this is your first time opening a pull request with the OPA repository, check out the [Contributing Guide](./contributing). Once your PR has been received a Netlify preview will be automatically created, check the PR for a unique link.
## Having trouble
Reach out in the [#contributors](https://openpolicyagent.slack.com/archives/C02L1TLPN59) channel to ask for help.
docs/new/docs/contributing.md
+62
View File
@@ -0,0 +1,62 @@
---
title: How to contribute
---
Thanks for your interest in contributing to the Open Policy Agent project!
:::info
Most of the discussions about OPA take place on Slack, if you haven't already,
you can [sign up here](https://slack.openpolicyagent.org/).
:::
## I'd like to help OPA users
Most users ask in the [#help](https://openpolicyagent.slack.com/archives/CBR63TK2A)
channel in Slack, hang out in there and see if there are any questions you
can help with.
You might also want to keep an eye on the
[OPA Discussion Board](https://github.com/orgs/open-policy-agent/discussions).
## I'd like to contribute code
If you have found a bug and would like to work on a fix, we still encourage you
file a [GitHub Issue](https://github.com/open-policy-agent/opa/issues) first
to talk about the problem and the solution you have in mind.
Similarly, if you have an idea for a new feature, we encourage you to file an
issue to discuss the feature before you start working on it too.
If you want to chat to the maintainers before opening an issue or about anything
else, head over to
[#contributors](https://openpolicyagent.slack.com/archives/C02L1TLPN59) in
Slack.
If you want to contribute code check out the
[development reference](./contrib-development/) for pointers on how to get
started. Please note we have some restrictions around the use of AI tooling
which are documented here.
## I'd like to help improve the documentation
Great! Please see our [documentation page](./contrib-docs) for more details.
## I have an OPA project or talk I'd like to share
Awesome! For OPA-based projects, we have our [Ecosystem page](../ecosystem/).
This is a great place to showcase your project. See
[the instructions](https://github.com/open-policy-agent/opa/tree/main/docs#opa-ecosystem)
here to get it listed.
If you have a talk or blog you'd like to share please feel free to post in:
- [#ecosystem](https://openpolicyagent.slack.com/archives/C02J6LBL6GH) in Slack.
- [Ecosystem Discussions](https://github.com/orgs/open-policy-agent/discussions/categories/ecosystem) on GitHub.
## I'm interested in something else...
Sounds interesting, we'd love to hear all about it,
[sign up for our Slack](https://slack.openpolicyagent.org/) and
drop a message in the
[#contributors](https://openpolicyagent.slack.com/archives/C02L1TLPN59)
channel.
docs/new/docs/debugging/debugging-dap.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 805 KiB

docs/new/docs/debugging/index.md
+134
View File
@@ -0,0 +1,134 @@
---
title: Debugging OPA
---
This section outlines the various tools and techniques that can be used to debug OPA, both as a component in a
distributed system and as a policy engine evaluating the Rego language.
# Debugging Rego Policies
Debugging Rego is crucial to ensuring OPA functions properly, as its policy
engine evaluates policies written in Rego. Effective Rego debugging helps
identify and resolve issues that impact OPA’s performance and behavior. A strong
grasp of Rego debugging is key to ensuring the correct functioning of OPA as a
whole.
## Live Debugging
Using the [Debug Adapter](https://docs.styra.com/regal/debug-adapter)
based on [OPA's Debug API](https://github.com/open-policy-agent/opa/blob/main/debug/README.md),
supported clients are now able to live debug Rego line by line. Breakpoints,
variable inspection and print statements are all supported.
Read more about the supported editors of this debugging method in the
[Regal Editor Support Page](https://docs.styra.com/regal/editor-support).
![Debugging Rego in VS Code](debugging-dap.gif)
## OPA REPL and Playground
Often it can take a few tries to get a Rego policy correct, the OPA REPL and Playground are great tools for
reducing the feedback loop when debugging policies.
The REPL can be run locally and loaded with the policy and data files you are working on:
```shell
opa run [policy-files] [data-files]
```
The [Rego Playground](http://play.openpolicyagent.org) is a web-based Rego development environment that can be
used to test policies with different inputs and data. If you are interested in asking for help in the
[OPA Slack](https://inviter.co/opa), the playground is a great way to share your policy and data with
others.
## Using the `print` Built-in Function
The `print` built-in function can be used to print values to stdout, this can be useful for checking values
during policy evaluation as well as seeing how many times a particular line of code is executed.
See the [print function documentation](./policy-reference/#debugging) for more details on how to use
the `print` built-in function in different contexts.
## Performance Profiling
Sometimes the issue isn't the correctness of the policy but rather the performance. The
[Policy Performance](./policy-performance) section of the documentation outlines various techniques for
profiling and optimizing Rego policies.
## Ecosystem Projects
<EcosystemEmbed feature="debugging-rego">
Here are some projects in the OPA ecosystem that can help with debugging Rego policies.
</EcosystemEmbed>
# Debugging OPA Instances in Distributed Systems
Debugging problems in distributed systems poses a number of challenges. Since OPA is commonly deployed in a distributed
fashion, as part of a larger platform, it is helpful to understand the various tools and techniques available for
debugging OPA in these environments.
## OPA Logs
OPA logs are a great place to start when debugging issues. The logs can be used to understand what OPA is doing
at any given time. Common issues such as failing to load in policy or data bundles will be shown here.
You can also enable debug logging to get more detailed information about what OPA is doing with `--log-level debug`.
This is documented in the [CLI documentation](./cli/#options-10) for `opa run`.
### Decision Logging
When OPA responds to a query, it is making a decision based on the policy and data that it has loaded. With the default
logging configuration, these are not logged in detail to the OPA logs. However, it is possible to enable console decision
logging by setting the following in OPA's config file:
```yaml
decision_logs:
console: true
```
It might be preferable to send these logs to an HTTP endpoint or other system, to learn more about decision logging,
take a look at the [Decision Logging documentation](./management-decision-logs).
## Metrics, Health and Status APIs
Like other cloud-native tools, OPA exposes `/metrics` and `/health` endpoints that can be used to understand the
state of an OPA instance at any given time.
- `/metrics` - exposes Prometheus metrics about the OPA instance's memory use, bundle loading and HTTP requests.
Read more in the [Metrics documentation](./monitoring).
- `/health` - shows information about the instance's readiness to serve requests, there are options available to also
show information about the loading of bundles and other plugins. Read more about the endpoint in the
[Health API documentation](./rest-api/#health-api).
- `/status` - is a JSON formatted endpoint that shows both health and metrics information. Read more in
[Status API documentation](./rest-api/#status-api).
## Manually Querying OPA
In distributed systems, it's common that an OPA instance is being invoked by another service, sometimes it can be helpful
to isolate the OPA instance and query it directly. This can be done using the [REST API](./rest-api).
For example, to get a snapshot of the data that OPA has loaded, you can use the following command:
```shell
curl --silent https://$OPA_HOSTNAME/v1/data
```
Or to manually evaluate a policy rule with some input:
```shell
curl -X POST https://$OPA_HOSTNAME/v0/data/example_package/example_rule -d '{"foo": "bar"}'
```
## Load a Production Bundle Locally
Sometimes there are too many moving parts in a distributed system to debug an issue effectively on a live system.
In these cases, it can be helpful to load a bundle into a local OPA instance and debug the issue there.
You can quickly start an OPA instance with a remote bundle using the following command:
```shell
opa run -s https://example.com/bundles/bundle.tar.gz
```
If you need to configure the OPA instance with other options, you can use a config file to
make more detailed configurations. Read more in the [Configuration documentation](./configuration) documentation.
docs/new/docs/deployments.md
+553
View File
@@ -0,0 +1,553 @@
---
title: Deployment
---
This document helps you get OPA up and running in different deployment
environments. You should read this document if you are planning to deploy OPA.
## Docker
Docker makes OPA easy to deploy in different types of environments.
This section explains how to use the official OPA Docker images. If this is your
first time deploying OPA and you plan to use one of the Docker images, we
recommend you review this section to familiarize yourself with the basics.
OPA releases are available as images on Docker Hub.
- [openpolicyagent/opa](https://hub.docker.com/r/openpolicyagent/opa/)
### Running with Docker
If you start OPA outside of Docker without any arguments, it prints a list of
available commands. By default, the official OPA Docker image executes the `run`
command which starts an instance of OPA as an interactive shell. This is nice
for development, however, for deployments, we want to run OPA as a server.
The `run` command accepts a `--server` (or `-s`) flag that starts OPA as a
server. See `--help` for more information on other arguments. The most important
command line arguments for OPA's server mode are:
- `--addr` to set the listening address (default: `localhost:8181`).
- `--log-level` (or `-l`) to set the log level (default: `"info"`).
- `--log-format` to set the log format (default: `"json"`).
By default, OPA listens for normal HTTP connections on `localhost:8181`. To make
OPA listen for HTTPS connections, see [Security](./security).
We can run OPA as a server using Docker:
```bash
docker run -p 8181:8181 openpolicyagent/opa \
run --server --log-level debug --addr=0.0.0.0:8181
```
:::info
We have to use `--addr` here to bind to all interfaces to ensure OPA is
accessible from outside the container. This is not necessary when running OPA
in other environments.
More information can be found in the
[security documentation](./security/#interface-binding).
:::
Test that OPA is available:
```
curl -i localhost:8181/
```
#### Logging
OPA logs to stderr and the level can be set with `--log-level/-l`. The default log level is `info` which causes OPA to log request/response information.
```
{"client_addr":"[::1]:64427","level":"debug","msg":"Received request.","req_body":"","req_id":1,"req_method":"GET","req_params":{},"req_path":"/v1/data","time":"20.7.13-11T18:22:18-08:00"}
{"client_addr":"[::1]:64427","level":"debug","msg":"Sent response.","req_id":1,"req_method":"GET","req_path":"/v1/data","resp_bytes":13,"resp_duration":0.392554,"resp_status":200,"time":"20.7.13-11T18:22:18-08:00"}
```
If the log level is set to `debug` the request and response message bodies will be logged. This is useful for development however it can be expensive in production.
```
{"addrs":[":8181"],"insecure_addr":"","level":"info","msg":"First line of log stream.","time":"2019-05-08T17:25:26-07:00"}
{"level":"info","msg":"Starting decision log uploader.","plugin":"decision_logs","time":"2019-05-08T17:25:26-07:00"}
{"client_addr":"[::1]:63902","level":"info","msg":"Received request.","req_body":"","req_id":1,"req_method":"GET","req_params":{},"req_path":"/v1/data","time":"2019-05-08T17:25:41-07:00"}
{"client_addr":"[::1]:63902","level":"info","msg":"Sent response.","req_id":1,"req_method":"GET","req_path":"/v1/data","resp_body":"{\"decision_id\":\"f4b41501-2408-4a14-8269-1c1085abeda4\",\"result\":{}}","resp_bytes":66,"resp_duration":2.545972,"resp_status":200,"time":"2019-05-08T17:25:41-07:00"}
```
The default log format is json and intended for production use. For more human readable
formats use "json-pretty" or "text".
> **Note:** The `text` log format is not performance optimized or intended for production use.
#### Volume Mounts
By default, OPA does not include any data or policies.
The simplest way to load data and policies into OPA is to provide them via the
file system as command line arguments. When running inside Docker, you can
provide files via volume mounts.
```bash
docker run -v $PWD:/example openpolicyagent/opa eval -d /example 'data.example.greeting'
```
**policy.rego**:
```live:docker_hello_world:module:read_only
package example
greeting := msg {
info := opa.runtime()
hostname := info.env["HOSTNAME"] # Docker sets the HOSTNAME environment variable.
msg := sprintf("hello from container %q!", [hostname])
}
```
#### More Information
For more information on OPA's command line, see `--help`:
```
docker run openpolicyagent/opa run --help
```
### Tagging
The Docker Hub repository contains tags for every release of OPA. For more
information on each release see the [GitHub Releases](https://github.com/open-policy-agent/opa/releases) page.
The "latest" tag refers to the most recent release. The latest tag is convenient
if you want to quickly try out OPA however for production deployments, we
recommend using an explicit version tag.
Development builds are also available on Docker Hub. For each version the
`{version}-dev` tag refers the most recent development build for that version.
The `edge` tag refers to the current `main` branch of OPA. Useful for testing
unreleased features. It is not recommended to use `edge` for production deployments.
The version information is contained in the OPA executable itself. You can check
the version with the following command:
```bash
docker run openpolicyagent/opa version
```
## Kubernetes
### Kicking the Tires
This section shows how to quickly deploy OPA on top of Kubernetes to try it out.
> If you are interested in using OPA to enforce admission control policies in
> Kubernetes, see the [Kubernetes Admission Control Tutorial](./kubernetes/tutorial).
> These steps assume Kubernetes is deployed with
> [minikube](https://github.com/kubernetes/minikube). If you are using a different
> Kubernetes provider, the steps should be similar. You may need to use a
> different Service configuration at the end.
First, create a ConfigMap containing a test policy.
In this case, the policy file does not contain sensitive information so it's
fine to store as a ConfigMap. If the file contained sensitive information, then
we recommend you store it as a Secret.
**example.rego**:
```live:k8s_deployment_hello_world:module:read_only
package example
greeting := msg {
info := opa.runtime()
hostname := info.env["HOSTNAME"] # Kubernetes sets the HOSTNAME environment variable.
msg := sprintf("hello from pod %q!", [hostname])
}
```
```bash
kubectl create configmap example-policy --from-file example.rego
```
Next, create a Deployment to run OPA. The ConfigMap containing the policy is
volume mounted into the container. This allows OPA to load the policy from
the file system.
**deployment-opa.yaml**:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: opa
labels:
app: opa
spec:
replicas: 1
selector:
matchLabels:
app: opa
template:
metadata:
labels:
app: opa
name: opa
spec:
containers:
- name: opa
image: openpolicyagent/opa:{{< current_docker_version >}}
ports:
- name: http
containerPort: 8181
args:
- "run"
- "--ignore=.*" # exclude hidden dirs created by Kubernetes
- "--server"
- "/policies"
volumeMounts:
- readOnly: true
mountPath: /policies
name: example-policy
volumes:
- name: example-policy
configMap:
name: example-policy
```
```bash
kubectl create -f deployment-opa.yaml
```
At this point OPA is up and running. Create a Service to expose the OPA API so
that you can query it:
**service-opa.yaml**:
```yaml
kind: Service
apiVersion: v1
metadata:
name: opa
labels:
app: opa
spec:
type: NodePort
selector:
app: opa
ports:
- name: http
protocol: TCP
port: 8181
targetPort: 8181
```
```bash
kubectl create -f service-opa.yaml
```
Get the URL of OPA using `minikube`:
```bash
OPA_URL=$(minikube service opa --url)
```
Now you can query OPA's API:
```bash
curl $OPA_URL/v1/data
```
OPA will respond with the greeting from the policy (the pod hostname will differ):
```json
{
"result": {
"example": {
"greeting": "hello from pod \"opa-78ccdfddd-xplxr\"!"
}
}
}
```
### Readiness and Liveness Probes
OPA exposes a `/health` API endpoint that you can configure Kubernetes
[Readiness and Liveness Probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/)
to call. For example:
```yaml
containers:
- name: opa
image: openpolicyagent/opa:{{< current_docker_version >}}
ports:
- name: http
containerPort: 8181
args:
- "run"
- "--ignore=.*" # exclude hidden dirs created by Kubernetes
- "--server"
- "/policies"
volumeMounts:
- readOnly: true
mountPath: /policies
name: example-policy
livenessProbe:
httpGet:
path: /health
scheme: HTTP # assumes OPA listens on localhost:8181
port: 8181
initialDelaySeconds: 5 # tune these periods for your environment
periodSeconds: 5
readinessProbe:
httpGet:
path: /health?bundle=true # Include bundle activation in readiness
scheme: HTTP
port: 8181
initialDelaySeconds: 5
periodSeconds: 5
```
See the [Health API](./rest-api#health-api) documentation for more detail on the `/health` API endpoint.
## HTTP Proxies
OPA uses the standard Go [net/http](https://golang.org/pkg/net/http/) package
for outbound HTTP requests that download bundles, upload decision logs, etc. In
environments where an HTTP proxy is required, you can configure OPA using the
pseudo-standard `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` environment
variables.
## CPU and Memory Requirements
For more information see the [Resource Utilization section on the Policy Performance page](./policy-performance#resource-utilization).
## Operational Readiness and Failure Modes
Depending on how you deploy OPA, it may or may not have policies available as soon as it starts up. If OPA starts making decisions without any policies, it will return `undefined` as an answer to all policy queries. This can be problematic because even though OPA returns a response, it has not actually returned the decision dictated by policy.
For example, without loading any policies into OPA whatsoever, a policy query will return the answer `undefined`, which via the HTTP API is represented as an empty JSON object `{}`.
```
$ opa run -s
$ curl localhost:8181/v1/data/foo/bar
{}
```
In contrast, when policies are loaded, OPA is operationally ready for policy queries, and the answer is defined, the answer is a JSON object of the form `{"result": ...}`
```
$ opa run foo.rego -s
$ curl localhost:8181/v1/data/foo/bar
{"result": 7}
```
However, it is possible that even though policies have been loaded the policy response is still `undefined` because the policy makes no decision for the given inputs.
```
$ opa run foo.rego -s
$ curl localhost:8181/v1/data/baz
{}
```
Just because OPA has returned an answer for a policy query, that does not indicate that it was operationally ready for that query. Moreover, the operational readiness of OPA cannot be ascertained from the query response, as illustrated above. Two issues must therefore be addressed: how to know when OPA is operationally ready for policy queries and how to make a decision before OPA is ready.
### Ensuring Operational Readiness
The relevance of the discussion above depends on how you have chosen to deploy policies into OPA.
If you deploy policies to OPA on disk (e.g. volume mounting into the OPA container on Kubernetes), then OPA will only start answering policy queries once all the policies are successfully loaded. In this case, it is impossible for OPA to answer policy queries before it has loaded policy, so the discussion above is a non-issue.
On the other hand, if you use the [Bundle service](./management-bundles) OPA will start up without any policies and immediately start downloading a bundle. But even before the bundle has successfully downloaded, OPA will answer policy queries if asked (which is in every case except the bootstrap case the right thing to do). For this reason, OPA provides a `/health` [API](./rest-api/#health-api) that verifies that the server is operational and optionally that a bundle has been successfully activated. As long as no policy queries are routed to OPA until the `/health` API verifies that OPA is operational. The recommendation is to ensure the `/health` API indicates that OPA is operational before routing policy queries to it.
Finally, you might choose to push policies into OPA via its [REST API](./rest-api/#create-or-update-a-policy). In this case, there is no way for OPA to know whether it has a complete policy set, and so the decision as to when to route policy queries to OPA must be handled by whatever software is pushing policies into OPA.
### Making Decisions before OPA is Ready
The mechanisms discussed above ensure that OPA is not asked to answer policy queries before it is ready to do so. But from the perspective of the software needing decisions, until OPA is operational, the software must make a decision on its own. Typically there are two choices:
- fail-open: if OPA does not provide a decision, then treat the decision as allowed.
- fail-closed: if OPA does not provide a decision, then treat the decision as denied.
The choices are more varied if the policy is not making an allow/deny decision, but often there is some analog to fail-open and fail-closed. The key observation is that this logic is entirely the responsibility of the software asking OPA for a policy decision. Despite the fact that what to do when OPA is unavailable is technically a policy question, it is one that we cannot rely on OPA to answer. The right logic can depend on many factors including the likelihood of OPA not making a decision and the cost of allowing or denying a request incorrectly.
In Kubernetes admission control, for example, the Kubernetes admin can choose whether to fail-open or fail-closed, leaving the decision up to the user. And often this is the correct way to build an integration because it is unlikely that there is a universal solution. For example, running an OPA-integration in a development environment might require fail-open, but running exactly the same integration in a production environment might require fail-closed.
## Capabilities
OPA now supports a _capabilities_ check on policies. The check allows callers to restrict the [built-in](./policy-reference/#built-in-functions) functions that policies may depend on. If the policies passed to OPA require built-ins not listed in the capabilities structure, an error is returned. The capabilities check is currently supported by the `check` and `build` sub-commands and can be accessed programmatically on the `ast.Compiler` structure. The OPA repository includes a set of capabilities files for previous versions of OPA in the [capabilities](https://github.com/open-policy-agent/opa/tree/main/capabilities) folder.
For example, given the following policy:
```rego
package example
deny["missing semantic version"] {
not valid_semantic_version_tag
}
valid_semantic_version_tag {
semver.is_valid(input.version)
}
```
We can check whether it is compatible with different versions of OPA:
```bash
# OK!
$ opa build ./policies/example.rego --capabilities ./capabilities/v0.22.0.json
# ERROR!
$ opa build ./policies/example.rego --capabilities ./capabilities/v0.21.1.json
```
### Built-ins
The 'build' command can validate policies against a configurable set of OPA capabilities. The capabilities define the built-in functions and other language features that policies may depend on. For example, the following capabilities file only permits the policy to depend on the "plus" built-in function ('+'):
```json
{
"builtins": [
{
"name": "plus",
"infix": "+",
"decl": {
"type": "function",
"args": [
{
"type": "number"
},
{
"type": "number"
}
],
"result": {
"type": "number"
}
}
}
]
}
```
The following command builds a directory of policies ('./policies') and validates them against `capability-built-in-plus.json`:
```bash
opa build ./policies --capabilities ./capability-built-in-plus.json
```
### Network
When passing a capabilities definition file via `--capabilities`, one can restrict which hosts remote schema definitions can be retrieved from. For example, a `capabilities.json` containing the json below would disallow fetching remote schemas from any host but "kubernetesjsonschema.dev". Setting `allow_net` to an empty array would prohibit fetching any remote schemas.
**capabilities.json**
```json
{
"builtins": [ ... ],
"allow_net": [ "kubernetesjsonschema.dev" ]
}
```
Not providing a capabilities file, or providing a file without an `allow_net` key, will permit fetching remote schemas from any host.
Note that the metaschemas [http://json-schema.org/draft-04/schema](http://json-schema.org/draft-04/schema), [http://json-schema.org/draft-06/schema](http://json-schema.org/draft-06/schema), and [http://json-schema.org/draft-07/schema](http://json-schema.org/draft-07/schema), are always available, even without network access.
Similarly, the `allow_net` capability restricts what hosts the `http.send` built-in function may send requests to, and what hosts the `net.lookup_ip_addr` built-in function may resolve IP addresses for.
### Features
Some features of OPA can be toggled on and off through the `features` list:
```json
{
"features": [
"rule_head_ref_string_prefixes",
"rule_head_refs",
"rego_v1_import"
]
}
```
Features present in the list are enabled, while features not present are disabled. The following features are available:
- `rule_head_ref_string_prefixes`: Enables the use of a [reference in place of name](./policy-language/#rule-heads-containing-references) in the head of rules. This is a subset of `rule_head_refs`, and only covers references where all terms are primitive types, or where only the last element of the ref (the key in the generated object or set) is allowed to be a variable.
- `rule_head_refs`: Enables general support for [references in rule heads](./policy-language/#rule-heads-containing-references), including [variables at arbitrary locations](./policy-language/#variables-in-rule-head-references). This feature also covers the functionality of `rule_head_ref_string_prefixes`.
- `rego_v1_import`: enables use of the `rego.v1` import.
### Future keywords
:::info
It is recommended to use the `rego.v1` import instead of `future.keywords` imports, as this will ensure that your policy is compatible with the future release of [OPA v1.0](./v0-upgrade/)
If the `rego.v1` import is present in a module, then `future.keywords` and `future.keywords.*` import is implied, and not allowed.
:::
The availability of future keywords in an OPA version can also be controlled using the capabilities file:
```json
{
"future_keywords": ["in"]
}
```
With these capabilities, the future import `future.keywords.in` would be available. See [the documentation
of the membership and iteration operator for details](./policy-language/#membership-and-iteration-in).
### Wasm ABI compatibility
A specific OPA version's capabilities file shows which Wasm ABI versions it is capable of evaluating:
```json
{
"wasm_abi_versions": [
{
"version": 1,
"minor_version": 1
},
{
"version": 1,
"minor_version": 2
}
]
}
```
This snippet would allow for evaluating bundles containing Wasm modules of the ABI version 1.1 and 1.2.
See [the ABI version docs](./wasm/#abi-versions) for details.
### Building your own capabilities JSON
Use the following JSON structure to build more complex capability checks.
```json
{
"builtins": [
{
"name": "name", // REQUIRED: Unique name of built-in function, e.g., <name>(arg1,arg2,...,argN)
"infix": "+", // OPTIONAL: Unique name of infix operator. Default should be unset.
"decl": { // REQUIRED: Built-in function type declaration.
"type": "function", // REQUIRED: states this is a function
"args": [ // REQUIRED: List of types to be passed in as an argument: any, number, string, boolean, object, array, set.
{
"type": "number"
},
{
"type": "number"
}
],
"result": { // REQUIRED: The expected result type.
"type": "number"
}
}
}
],
"allow_net": [ // OPTIONAL: allow_net is an array of hostnames or IP addresses, that an OPA instance is allowed to connect to.
"mycompany.com",
"database.safe"
],
"future_keywords": ["in"]
}
```
docs/new/docs/docker-authorization.md
+440
View File
@@ -0,0 +1,440 @@
---
title: Docker
---
Docker’s out-of-the-box authorization model is all or nothing. But many users
require finer-grained access control and Docker’s plugin infrastructure allows
us to do so.
This is an excellent opportunity to see how to policy enable an existing
service.
## Goals
This tutorial helps you get started with OPA and introduces you to core concepts
in OPA.
> Policy enabling an application decouples the policy implementation from the
> business logic so that administrators can define policy without changing the
> application while still keeping up with the size, complexity, and dynamic
> nature of modern applications.
For the purpose of this tutorial, we want to use OPA to enforce a policy that
prevents users from running insecure containers.
This tutorial illustrates two key concepts:
1. OPA policy definition is decoupled from the implementation of the service
(in this case Docker). The administrator is empowered to define and manage
policies without requiring changes to any of the apps.
2. Both the data relevant to policy and the policy definitions themselves can
change rapidly.
## Prerequisites
This tutorial requires:
- Docker Engine 18.06.0-ce or newer
- Docker API version 1.38 or newer
- `root` or `sudo` access
- Nginx, or any capable [bundle](https://www.openpolicyagent.org/docs/latest/management-bundles/) server
The tutorial has been tested on the following platforms:
- Ubuntu 20.04 (64-bit)
If you are using a different distro, OS, or architecture, the steps will be the
same. However, there may be slight differences in the commands you need to run.
## Steps
Several of the steps below require `root` or `sudo` access. When you are
modifying files under `/etc/docker` or signalling the Docker daemon to
restart, you will need root access.
### 1. Create an empty policy definition that will allow all requests.
**authz.rego**:
```live:docker_authz:module:read_only
package docker.authz
allow := true
```
This policy defines a single rule named `allow` that always produces the
decision `true`. Once all the components are running, we will come back to
the policy.
### 2. Create policy bundle and OPA configuration.
For the purpose of this example, we are going to use [Nginx](https://www.openpolicyagent.org/docs/latest/management-bundles/#nginx)
to serve bundles from the same machine Docker is running on.
With nginx running, simply build the policy bundle placed into the nginx web root directory.
```shell
opa build --bundle --output /var/www/html/bundle.tar.gz .
```
Next, create an OPA configuration file pointing to the bundle.
```yaml
services:
authz:
url: http://localhost
bundles:
authz:
service: authz
resource: bundle.tar.gz
# Optional - Print decisions in the Docker logs. Configure a remote service for production use cases.
decision_logs:
console: true
```
Save the above file as `opa-config.yaml`. We'll need to place this somewhere where the plugin can find it.
The `/etc/docker` directory will be mounted as `/opa` in the container running the plugin, so let's create a
sub-directory for our configuration file there.
```shell
sudo mkdir -p /etc/docker/config
sudo mv opa-config.yaml /etc/docker/config/
```
### 3. Install the opa-docker-authz plugin.
Install the `opa-docker-authz` plugin and point it to the config file just created.
```shell
docker plugin install --alias opa-docker-authz ghcr.io/open-policy-agent/opa-docker-authz:v0.10 opa-args="-config-file /opa/config/opa-config.yaml"
```
You need to configure the Docker daemon to use the plugin for authorization.
```shell
cat > /etc/docker/daemon.json <<EOF
{
"authorization-plugins": ["opa-docker-authz"]
}
EOF
```
Signal the Docker daemon to reload the configuration file.
```shell
kill -HUP $(pidof dockerd)
```
### 4. Run a simple Docker command to make sure everything is still working.
```shell
docker ps
```
If everything is set up correctly, the command should exit successfully. You can
expect to see log messages from OPA and the plugin.
### 5. Test that the policy definition is working.
Let’s modify our policy to **deny** all requests:
**authz.rego**:
```live:docker_authz_deny_all:module:read_only
package docker.authz
allow := false
```
Rebuild the bundle and save it in the Nginx document root directory.
```shell
opa build --bundle --output /var/www/html/bundle.tar.gz .
```
In OPA, rules defines the content of documents. Documents be boolean values
(true/false) or they can represent more complex structures using arrays,
objects, strings, etc.
In the example above we modified the policy to always return `false` so that
requests will be rejected.
```shell
docker ps
```
The output should be:
```shell
Error response from daemon: authorization denied by plugin opa-docker-authz: request rejected by administrative policy
```
To learn more about how rules define the content of documents, see: [How Does OPA Work?](../#overview)
With this policy in place, users will not be able to run any Docker commands. Go
ahead and try other commands such as `docker run` or `docker pull`. They will
all be rejected.
Now let's change the policy so that it's a bit more useful.
### 6. Update the policy to reject requests with the unconfined [seccomp](https://en.wikipedia.org/wiki/Seccomp) profile:
**authz.rego**:
```live:docker_authz_deny_unconfined:module:openable
package docker.authz
default allow := false
allow if {
not deny
}
deny if {
seccomp_unconfined
}
seccomp_unconfined if {
# This expression asserts that the string on the right-hand side is equal
# to an element in the array SecurityOpt referenced on the left-hand side.
input.Body.HostConfig.SecurityOpt[_] == "seccomp:unconfined"
}
```
Again, rebuild the bundle and save it in the Nginx document root directory.
```shell
opa build --bundle --output /var/www/html/bundle.tar.gz .
```
The plugin queries the `allow` rule to authorize requests to Docker. The `input`
document is set to the attributes passed from Docker.
```live:docker_authz_deny_unconfined:query:hidden
allow
```
```live:docker_authz_deny_unconfined:input
{
"AuthMethod": "",
"Body": {
"AttachStderr": true,
"AttachStdin": false,
"AttachStdout": true,
"Cmd": null,
"Domainname": "",
"Entrypoint": null,
"Env": [],
"HostConfig": {
"AutoRemove": false,
"Binds": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceWriteIOps": null,
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"CapAdd": null,
"CapDrop": null,
"Cgroup": "",
"CgroupParent": "",
"ConsoleSize": [
0,
0
],
"ContainerIDFile": "",
"CpuCount": 0,
"CpuPercent": 0,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpuShares": 0,
"CpusetCpus": "",
"CpusetMems": "",
"DeviceCgroupRules": null,
"Devices": [],
"DiskQuota": 0,
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IOMaximumBandwidth": 0,
"IOMaximumIOps": 0,
"IpcMode": "",
"Isolation": "",
"KernelMemory": 0,
"Links": null,
"LogConfig": {
"Config": {},
"Type": ""
},
"MaskedPaths": null,
"Memory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": -1,
"NanoCpus": 0,
"NetworkMode": "default",
"OomKillDisable": false,
"OomScoreAdj": 0,
"PidMode": "",
"PidsLimit": 0,
"PortBindings": {},
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyPaths": null,
"ReadonlyRootfs": false,
"RestartPolicy": {
"MaximumRetryCount": 0,
"Name": "no"
},
"SecurityOpt": null,
"ShmSize": 0,
"UTSMode": "",
"Ulimits": null,
"UsernsMode": "",
"VolumeDriver": "",
"VolumesFrom": null
},
"Hostname": "",
"Image": "hello-world",
"Labels": {},
"NetworkingConfig": {
"EndpointsConfig": {}
},
"OnBuild": null,
"OpenStdin": false,
"StdinOnce": false,
"Tty": false,
"User": "",
"Volumes": {},
"WorkingDir": ""
},
"Headers": {
"Content-Length": "1470",
"Content-Type": "application/json",
"User-Agent": "Docker-Client/18.06.1-ce (linux)"
},
"Method": "POST",
"Path": "/v1.38/containers/create",
"User": ""
}
```
For the input above, the value of `allow` is:
```live:docker_authz_deny_unconfined:output
```
> Many of the examples in the documentation are interactive. Try editing the
> input above by setting `Body.HostConfig.SecurityOpt` to
> `["seccomp:unconfined"]`.
### 7. Test the policy is working by running a simple container:
```shell
docker run hello-world
```
Now try running the same container but disable seccomp (which should be
prevented by the policy):
```shell
docker run --security-opt seccomp:unconfined hello-world
```
Congratulations! You have successfully prevented containers from running without
seccomp!
The rest of the tutorial shows how you can grant fine-grained access to specific
clients.
### <a name="identify-user"></a> 8. Identify the user in Docker requests.
> Back up your existing Docker configuration, just in case. You can replace your
> original configuration after you are done with the tutorial.
```shell
mkdir -p ~/.docker
cp ~/.docker/config.json ~/.docker/config.json~
```
To identify the user, include an HTTP header in all of the requests sent to the
Docker daemon:
```shell
cat >~/.docker/config.json <<EOF
{
"HttpHeaders": {
"Authz-User": "bob"
}
}
EOF
```
> Docker does not currently provide a way to authenticate clients. But in Docker
> 1.12, clients can be authenticated using TLS and there are plans to include
> other means of authentication. For the purpose of this tutorial, we assume that
> an authentication system is place.
### 9. Update the policy to include basic user access controls.
```live:docker_authz_users:module:read_only,openable
package docker.authz
default allow := false
# allow if the user is granted read/write access.
allow if {
user_id := input.Headers["Authz-User"]
user := users[user_id]
not user.readOnly
}
# allow if the user is granted read-only access and the request is a GET.
allow if {
user_id := input.Headers["Authz-User"]
users[user_id].readOnly
input.Method == "GET"
}
# users defines permissions for the user. In this case, we define a single
# attribute 'readOnly' that controls the kinds of commands the user can run.
users := {
"bob": {"readOnly": true},
"alice": {"readOnly": false},
}
```
### 10. Attempt to run a container.
Because the configured user is `"bob"`, the request is rejected:
```shell
docker run hello-world
```
### 11. Change the user to "alice" and re-run the container.
```shell
cat > ~/.docker/config.json <<EOF
{
"HttpHeaders": {
"Authz-User": "alice"
}
}
EOF
```
Because the configured user is `"alice"`, the request will succeed:
```shell
docker run hello-world
```
That's it!
docs/new/docs/editor-and-ide-support.md
+35
View File
@@ -0,0 +1,35 @@
---
title: Editor and IDE Support
---
OPA can be integrated into editors and IDEs to provide features like syntax highlighting, query
evaluation, policy coverage, and more.
## Integrations
| Editor | Link | Note |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ |
| Visual Studio Code | [marketplace.visualstudio.com/items?itemName=tsandall.opa](https://marketplace.visualstudio.com/items?itemName=tsandall.opa) | Supports Language Server and Debug Adapter |
| Neovim | Syntax highlighting [tree-sitter-rego](https://github.com/FallenAngel97/tree-sitter-rego), Language server [nvim-lspconfig](https://github.com/neovim/nvim-lspconfig/blob/master/doc/server_configurations.md#regal), Debugger [nvim-dap](https://github.com/mfussenegger/nvim-dap) + [nvim-dap-rego](https://github.com/rinx/nvim-dap-rego) | Supports Language Server and Debug Adapter |
| Zed | [github.com/StyraInc/zed-rego](https://github.com/StyraInc/zed-rego) | Supports Language Server |
| IntelliJ IDEA | [github.com/open-policy-agent/opa-idea-plugin](https://github.com/open-policy-agent/opa-idea-plugin) | |
| Vim | [github.com/tsandall/vim-rego](https://github.com/tsandall/vim-rego) | |
| Emacs | [github.com/psibi/rego-mode](https://github.com/psibi/rego-mode) | |
| Nano | [github.com/scopatz/nanorc](https://github.com/scopatz/nanorc) | |
| Sublime Text | [github.com/open-policy-agent/opa/tree/main/misc/syntax/sublime](https://github.com/open-policy-agent/opa/tree/main/misc/syntax/sublime) | |
| TextMate | [github.com/open-policy-agent/opa/tree/main/misc/syntax/textmate](https://github.com/open-policy-agent/opa/tree/main/misc/syntax/textmate) | |
:::info
**Your editor missing? Built a Rego integration for your editor?** Drop us a
message on [Slack](https://inviter.co/opa)
We also have our [Ecosystem page](/ecosystem/). This is a great place to
showcase your project. See
[these instructions](https://github.com/open-policy-agent/opa/tree/main/docs#opa-ecosystem)
to get it listed.
:::
## Rego Playground
The Rego Playground provides a great editor to get started with OPA and share
policies. Try it out at
[play.openpolicyagent.org](https://play.openpolicyagent.org/).
docs/new/docs/envoy/_category_.yaml
+4
View File
@@ -0,0 +1,4 @@
position: 16
label: "Envoy"
collapsible: true
collapsed: true
docs/new/docs/envoy/assets/envoy-ext-authz-flow.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 34 KiB

docs/new/docs/envoy/debugging.md
+84
View File
@@ -0,0 +1,84 @@
---
title: Debugging Tips
sidebar_position: 6
---
This page provides some pointers that could assist in addressing issues encountered while using the
OPA-Envoy plugin. If none of these tips work, feel free to join
[our slack](https://inviter.co/opa) and ask for help.
## Debugging Performance Issues
### Benchmarking Queries
The `opa bench` command evaluates a Rego query multiple times and reports metrics. You can also profile your polices using
`opa eval` to understand expression evaluation time. More information on improving policy performance can be found [here](https://www.openpolicyagent.org/docs/latest/policy-performance/).
### Analyzing Decision Logs
The OPA-Envoy plugin logs every decision that it makes. These logs contain lots of useful information including metrics like
gRPC server handler time and Rego query evaluation time which can help in measuring the OPA-Envoy plugin's performance.
To enable local console logging of decisions see [this](https://www.openpolicyagent.org/docs/latest/management-decision-logs/#local-decision-logs).
### Envoy External Authorization Filter Configuration
Envoy's External authorization gRPC service configuration uses either Envoy’s in-built gRPC client, or the Google C++ gRPC client.
From the [benchmarking](./performance#opa-benchmarks) results, lower latency numbers are seen while using Envoy’s gRPC client versus Google's. Experimenting
with the gRPC service configuration may help in improving performance.
The filter configuration also has a `status_on_error` field that can be used to indicate a network error between the filter
and the OPA-Envoy plugin. The default status on such an error is HTTP `403 Forbidden`. Changing the default value of this
field will help uncover potential network issues as `403 Forbidden` is also generated when a request is denied.
## Interacting with the gRPC server
This section provides examples of interacting with the Envoy External Authorization gRPC server using the [grpcurl](https://github.com/fullstorydev/grpcurl) tool.
- List all services exposed by the server
```bash
$ grpcurl -plaintext localhost:9191 list
```
Output:
```bash
envoy.service.auth.v2.Authorization
envoy.service.auth.v3.Authorization
grpc.reflection.v1alpha.ServerReflection
```
- Invoke a v3 Check RPC on the server
```bash
$ grpcurl -plaintext -d '
{
"attributes": {
"request": {
"http": {
"method": "GET",
"path": "/api/v1/products"
}
}
}
}' localhost:9191 envoy.service.auth.v3.Authorization/Check
```
Output:
```json
{
"status": {
},
"okResponse": {
"headers": [
{
"header": {
"key": "x-ext-auth-allow",
"value": "yes"
}
}
]
}
}
```
docs/new/docs/envoy/index.md
+85
View File
@@ -0,0 +1,85 @@
---
title: Overview & Architecture
---
[Envoy](https://www.envoyproxy.io/docs/envoy/latest/intro/what_is_envoy) is a
L7 proxy and communication bus designed for large modern service oriented
architectures. Envoy (v1.7.0+) supports an [External Authorization filter](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ext_authz_filter.html)
which calls an authorization service to check if the incoming request is
authorized or not.
This feature makes it possible to delegate authorization decisions to an
external service and also makes the request context available to the service. The request context contains information
such as the source of a network activity, destination of a network activity, the network request (eg. http request).
All this information can be used by the external service to make an informed decision about the fate of the
incoming request received by Envoy.
## What is OPA-Envoy Plugin?
[OPA-Envoy](https://github.com/open-policy-agent/opa-envoy-plugin) plugin extends OPA with a gRPC server that
implements the [Envoy External Authorization API](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ext_authz_filter.html).
You can use this version of OPA to enforce fine-grained, context-aware access control policies with Envoy _without_
modifying your microservice.
## How does it work?
In addition to the Envoy sidecar, your application pods will include an OPA-Envoy
sidecar. When Envoy receives API requests destined for your
microservice, it checks with OPA to decide if the request should be allowed.
Evaluating policies locally with Envoy is preferable because it
avoids introducing a network hop (which has implications on performance and
availability) in order to perform the authorization check.
![Envoy External Authorization Flow](./assets/envoy-ext-authz-flow.png)
> 💡 The OPA-Envoy plugin is frequently deployed in Kubernetes environments as a sidecar container however it can also
> be used in other environments as a standalone process running next to Envoy.
## Configuration
The OPA-Envoy plugin supports the following configuration fields:
| Field | Required | Description |
| --------------------------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `plugins["envoy_ext_authz_grpc"].addr` | No | Set listening address of Envoy External Authorization gRPC server. This must match the value configured in the Envoy config. Default: `:9191`. |
| `plugins["envoy_ext_authz_grpc"].path` | No | Specifies the hierarchical policy decision path. The policy decision can either be a `boolean` or an `object`. If boolean, `true` indicates the request should be allowed and `false` indicates the request should be denied. If the policy decision is an object, it **must** contain the `allowed` key set to either `true` or `false` to indicate if the request is allowed or not respectively. It can optionally contain a `headers` field to send custom headers to the downstream client or upstream. An optional `body` field can be included in the policy decision to send a response body data to the downstream client. Also an optional `http_status` field can be included to send a HTTP response status code to the downstream client other than `403 (Forbidden)`. Default: `envoy/authz/allow`. |
| `plugins["envoy_ext_authz_grpc"].dry-run` | No | Configures the Envoy External Authorization gRPC server to unconditionally return an `ext_authz.CheckResponse.Status` of `google_rpc.Status{Code: google_rpc.OK}`. Default: `false`. |
| `plugins["envoy_ext_authz_grpc"].enable-reflection` | No | Enables gRPC server reflection on the Envoy External Authorization gRPC server. Default: `false`. |
| `plugins["envoy_ext_authz_grpc"].proto-descriptor` | No | Set the path to a pb that enables the capability to decode the raw body to the parsed body. Default: turns this capability off. |
| `plugins["envoy_ext_authz_grpc"].grpc-max-recv-msg-size` | No | Set the max message size in bytes the gRPC server can receive. Defaults to 4MB. |
| `plugins["envoy_ext_authz_grpc"].grpc-max-send-msg-size` | No | Set the max message size in bytes the gRPC server can send. Defaults to 2048MB. |
| `plugins["envoy_ext_authz_grpc"].skip-request-body-parse` | No | Specifies if the plugin should skip parsing the input request body. Default: `false`. |
If the configuration does not specify the `path` field, `envoy/authz/allow` will be considered as the default policy
decision path. `data.envoy.authz.allow` will be the name of the policy decision to query in the default case.
The `dry-run` parameter is provided to enable you to test out new policies. You can set `dry-run: true` which will
unconditionally allow requests. Decision logs can be monitored to see what "would" have happened. This is especially
useful for initial integration of OPA or when policies undergo large refactoring.
The `enable-reflection` parameter registers the Envoy External Authorization gRPC server with reflection. After enabling
server reflection, a command line tool such as [grpcurl](https://github.com/fullstorydev/grpcurl) can be used to invoke
RPC methods on the gRPC server. See [Interacting with the gRPC server](./envoy/debugging#interacting-with-the-grpc-server)
section for more details.
Providing a file containing a protobuf descriptor set allows the plugin to decode gRPC message payloads.
So far, only unary methods using uncompressed protobuf-encoded payloads are supported.
The protoset can be generated using `protoc`, e.g. `protoc --descriptor_set_out=protoset.pb --include_imports`.
## Additional Resources
See the following pages on [envoyproxy.io](https://www.envoyproxy.io/) for more
information on external authorization:
- [External Authorization](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ext_authz_filter.html)
to learn about the External Authorization filter.
- [Network](https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/network_filters/ext_authz_filter#config-network-filters-ext-authz)
and [HTTP](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/ext_authz_filter#config-http-filters-ext-authz)
for details on configuring the External Authorization filter.
## Ecosystem Projects
<EcosystemEmbed feature="envoy">
Here are some projects relating to Envoy and OPA from the OPA ecosystem.
</EcosystemEmbed>
docs/new/docs/envoy/performance.md
+258
View File
@@ -0,0 +1,258 @@
---
title: Performance
sidebar_position: 6
---
This page provides some guidance and best practices around benchmarking the performance of the OPA-Envoy plugin in order
to give users an idea of the overhead of using the plugin. It describes an example setup to perform the benchmarks, different
benchmarking scenarios and important metrics that should be captured to understand the impact of the OPA-Envoy plugin.
### Benchmark Setup
#### Sample App
The first component of the setup features a simple Go app which provides information about employees in a company. It
exposes a `/people` endpoint to `get` and `create` employees. The app's source code can be found [here](https://github.com/ashutosh-narkar/go-test-server).
#### Envoy
Next, is the Envoy proxy that runs alongside the example application. The Envoy configuration below defines an external authorization
filter `envoy.ext_authz` for a gRPC authorization server. The config uses Envoy’s in-built gRPC client which
is a minimal custom implementation of gRPC to make the external gRPC call.
```yaml
static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 8000
filter_chains:
- filters:
- name: envoy.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: service
http_filters:
- name: envoy.ext_authz
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
transport_api_version: V3
with_request_body:
max_request_bytes: 8192
allow_partial_message: true
failure_mode_allow: false
grpc_service:
envoy_grpc:
cluster_name: opa-envoy
timeout: 0.5s
- name: envoy.filters.http.router
clusters:
- name: service
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
load_assignment:
cluster_name: service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8080
- name: opa-envoy
connect_timeout: 1.25s
type: strict_dns
lb_policy: round_robin
http2_protocol_options: {}
load_assignment:
cluster_name: opa-envoy
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 9191
admin:
access_log_path: "/dev/null"
address:
socket_address:
address: 0.0.0.0
port_value: 8001
layered_runtime:
layers:
- name: static_layer_0
static_layer:
envoy:
resource_limits:
listener:
example_listener_name:
connection_limit: 10000
overload:
global_downstream_max_connections: 50000
```
#### OPA-Envoy Plugin
Now let's deploy OPA as an External Authorization server. Below is a sample configuration for the OPA-Envoy container:
```yaml
containers:
- image: openpolicyagent/opa:{{< current_opa_envoy_docker_version >}}
imagePullPolicy: IfNotPresent
name: opa
resources:
requests:
memory: "64Mi"
cpu: "1m"
limits:
memory: "128Mi"
cpu: "2m"
args:
- "run"
- "--server"
- "--addr=localhost:8181"
- "--diagnostic-addr=0.0.0.0:8282"
- "--set=plugins.envoy_ext_authz_grpc.addr=:9191"
- "--set=plugins.envoy_ext_authz_grpc.path=envoy/authz/allow"
- "--ignore=.*"
- "/policy/policy.rego"
livenessProbe:
httpGet:
path: /health?plugins
port: 8282
readinessProbe:
httpGet:
path: /health?plugins
port: 8282
```
> 💡 Consider specifying CPU and memory resource requests and limits for the OPA and other containers to prevent
> deployments from resource starvation.
> You can also start OPA with the [`GOMAXPROCS`](https://golang.org/pkg/runtime)environment variable to limit the number of
> cores that OPA can consume.
>
> 💡 The OPA-Envoy plugin can be configured to listen on a UNIX Domain Socket. A complete example of such a setup
> can be found [here](https://github.com/open-policy-agent/opa-envoy-plugin/tree/main/examples/envoy-uds).
### Load Generator And Measurement Tool
Consider using a load generator and measurement tool that measures latency from the end user’s perspective and reports
latency as the percentiles of a distribution, e.g. `p50` (median), `p99`, `p999` etc. As example
implementation of such a tool can be found [here](https://github.com/ashutosh-narkar/stress-opa-envoy).
### Benchmark Scenarios
Following are some scenarios to perform benchmarks on. The results could be used to compare OPA-Envoy plugin's
latency and resource consumption with the baseline (no-opa) case for instance.
- **App Only**
In this case, requests are sent directly to the application ie. no Envoy and OPA in the request path.
- **App and Envoy**
In this case, OPA is not included in the request path but Envoy is (ie. [Envoy External Authorization API](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ext_authz_filter.html) disabled).
- **App, Envoy and OPA (NOP policy)**
In this case, performance measurements are observed with [Envoy External Authorization API](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ext_authz_filter.html) enabled. This means
Envoy will make a call to OPA on every incoming request with the below NOP policy loaded into OPA.
```live:nop_example:module:read_only
package envoy.authz
default allow := true
```
- **App, Envoy and OPA (RBAC policy)**
In this case, performance measurements are observed with [Envoy External Authorization API](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ext_authz_filter.html) enabled and
a sample real-world RBAC policy as shown below loaded into OPA.
```live:rbac_example:module:read_only
package envoy.authz
import input.attributes.request.http as http_request
default allow := false
allow {
roles_for_user[r]
required_roles[r]
}
roles_for_user[r] {
r := user_roles[user_name][_]
}
required_roles[r] {
perm := role_perms[r][_]
perm.method == http_request.method
perm.path == http_request.path
}
user_name := parsed {
[_, encoded] := split(http_request.headers.authorization, " ")
[parsed, _] := split(base64url.decode(encoded), ":")
}
user_roles := {
"alice": ["guest"],
"bob": ["admin"]
}
role_perms := {
"guest": [
{"method": "GET", "path": "/people"},
],
"admin": [
{"method": "GET", "path": "/people"},
{"method": "POST", "path": "/people"},
],
}
```
- **App, Envoy and OPA (Header Injection policy)**
This scenario is similar to the previous one expect the policy decision is an object which contains optional
response headers. An example of such a policy can be found [here](../envoy/primer#example-policy-with-object-response).
### Measurements
This section describes some metrics that should help to measure the cost of the OPA-Envoy plugin in terms of
CPU and memory consumed as well as latency added.
- `End-to-end Latency` is the latency measured from the end user’s perspective. This includes time spent on the network,
in the application, in OPA and so on. The sample [load tester tool](https://github.com/ashutosh-narkar/stress-opa-envoy)
shows how to measure this metric.
- `OPA Evaluation` is the time taken to evaluate the policy.
- `gRPC Server Handler` is the total time taken to prepare the input for the policy, evaluate the policy (`OPA Evaluation`)
and prepare the result. Basically this is time spent by the OPA-Envoy plugin to process the request. OPA's [metrics](https://pkg.go.dev/github.com/open-policy-agent/opa/metrics)
package provides helpers to measure both `gRPC Server Handler` and `OPA Evaluation` time.
- `Resource utilization` refers to the CPU and memory usage of the OPA-Envoy container. `kubectl top` utility can be
leveraged to measure this.
### Features
The sample OPA-Envoy deployment described [previously](#opa-envoy-plugin), does not utilize OPA's [decision logs](https://www.openpolicyagent.org/docs/latest/management-decision-logs/)
management API that enables periodic reporting of decision logs to remote HTTP servers or local console. Decision logging
can be enabled by updating the OPA-Envoy configuration, and the guidance provided on this page can be used to
gather benchmark results.
docs/new/docs/envoy/primer.md
+534
View File
@@ -0,0 +1,534 @@
---
title: Policy Primer via Examples
sidebar_position: 1
---
This page covers how to write policies for the content of the requests that are passed to OPA by Envoy's
[External Authorization filter](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ext_authz_filter.html).
## Writing Policies
Let's start with an example policy that restricts access to an endpoint based on a user's role and permissions.
```live:bool_example:module:openable
package envoy.authz
import input.attributes.request.http
default allow := false
allow if {
is_token_valid
action_allowed
}
is_token_valid if {
token.valid
now := time.now_ns() / 1000000000
token.payload.nbf <= now
now < token.payload.exp
}
action_allowed if {
http.method == "GET"
token.payload.role == "guest"
glob.match("/people/*", ["/"], http.path)
}
action_allowed if {
http.method == "GET"
token.payload.role == "admin"
glob.match("/people/*", ["/"], http.path)
}
action_allowed if {
http.method == "POST"
token.payload.role == "admin"
glob.match("/people", ["/"], http.path)
lower(input.parsed_body.firstname) != base64url.decode(token.payload.sub)
}
token := {"valid": valid, "payload": payload} if {
[_, encoded] := split(http.headers.authorization, " ")
[valid, _, payload] := io.jwt.decode_verify(encoded, {"secret": "secret"})
}
```
The first line `package envoy.authz` declaration gives the (hierarchical) name `envoy.authz` to the rules in the
remainder of the policy. If the OPA-Envoy [configuration](../#configuration) does not specify the `path`
field, `envoy/authz/allow` will be considered as the default policy decision path. `data.envoy.authz.allow` will be the
name of the policy decision to query in the default case.
The above policy uses the `io.jwt.decode_verify` builtin function to parse and verify the JWT containing
information about the user making the request. It uses other builtins like `glob.match`, `lower`, `base64url.decode` etc.
OPA has 150+ builtins detailed at [openpolicyagent.org/docs/policy-reference](../policy-reference).
The dot notation seen in multiple places in the policy for ex. `input.parsed_body.firstname` simply descends through
the hierarchy to access the requested value. The dot (.) operator never throws any errors; if the path does not exist
the value of the expression is `undefined`.
```live:bool_example:query:hidden
data.envoy.authz.allow
```
Sample input received by OPA is shown below:
```live:bool_example:input
{
"attributes": {
"request": {
"http": {
"method": "GET",
"path": "/people/",
"headers": {
"authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyb2xlIjoiZ3Vlc3QiLCJzdWIiOiJZV3hwWTJVPSIsIm5iZiI6MTUxNDg1MTEzOSwiZXhwIjoxNjQxMDgxNTM5fQ.K5DnnbbIOspRbpCr2IKXE9cPVatGOCBrBQobQmBmaeU"
}
}
}
}
}
```
With the input value above, the answer is:
```live:bool_example:output
```
## Example Policy with Additional Controls
The `allow` variable in the above policy returns a `boolean` decision to indicate whether a request should be allowed or not.
If you want, you can also control the HTTP status sent to the upstream or downstream client, along with the response body, and the response headers. Response metadata can also be set for consumption by the next Envoy filter. To do that, you can write rules like the ones below to fill in values for variables with the following types:
- `headers` is an object whose keys are strings and values are strings. In case the request is denied, the object represents the HTTP response headers to be sent to the downstream client. If the request is allowed, the object represents additional request headers to be sent to the upstream.
- `response_headers_to_add` is an object whose keys are strings and values are strings. It defines the HTTP response headers to be sent to the downstream client when a request is allowed.
- `request_headers_to_remove` is an array of strings which describes the HTTP headers to remove from the original request before dispatching it to the upstream when a request is allowed.
- `body` is a string which represents the response body data sent to the downstream client when a request is denied.
- `status_code` is a number which represents the HTTP response status code sent to the downstream client when a request is denied.
- `dynamic_metadata` is an object whose keys are strings and values can be booleans, strings, numbers, arrays, or objects. It will set the `DynamicMetadata` in the `CheckResponse` returned by the `opa-envoy-plugin` and can be consumed elsewhere in the envoy filter chain.
- `query_parameters_to_set` is an object whose keys are strings and values can be strings or arrays of strings. It defines the query parameters to be added or modified in the request before dispatching it to the upstream when a request is allowed. When a value is an array, it represents multiple values for the same parameter key.
```live:obj_example:module:openable
package envoy.authz
import input.attributes.request.http
default allow := false
allow if {
is_token_valid
action_allowed
}
headers["x-ext-auth-allow"] := "yes"
headers["x-validated-by"] := "security-checkpoint"
request_headers_to_remove := ["one-auth-header", "another-auth-header"]
response_headers_to_add["x-foo"] := "bar"
query_parameters_to_set = {
"user-role": token.payload.role,
"tags": ["main-flow", "auth-enabled"]
}
status_code := 200 if {
allow
} else := 401 if {
not is_token_valid
} else := 403
body := "Authentication Failed" if status_code == 401
body := "Unauthorized Request" if status_code == 403
dynamic_metadata := {"foo": "bar"}
is_token_valid if {
token.valid
now := time.now_ns() / 1000000000
token.payload.nbf <= now
now < token.payload.exp
}
action_allowed if {
http.method == "GET"
token.payload.role == "guest"
glob.match("/people/*", ["/"], http.path)
}
action_allowed if {
http.method == "GET"
token.payload.role == "admin"
glob.match("/people/*", ["/"], http.path)
}
action_allowed if {
http.method == "POST"
token.payload.role == "admin"
glob.match("/people", ["/"], http.path)
lower(input.parsed_body.firstname) != base64url.decode(token.payload.sub)
}
token := {"valid": valid, "payload": payload} if {
[_, encoded] := split(http.headers.authorization, " ")
[valid, _, payload] := io.jwt.decode_verify(encoded, {"secret": "secret"})
}
```
```live:obj_example:query:hidden
data.envoy.authz
```
Sample input received by OPA is shown below:
```live:obj_example:input
{
"attributes": {
"request": {
"http": {
"method": "GET",
"path": "/people",
"headers": {
"authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyb2xlIjoiZ3Vlc3QiLCJzdWIiOiJZV3hwWTJVPSIsIm5iZiI6MTUxNDg1MTEzOSwiZXhwIjoxNjQxMDgxNTM5fQ.K5DnnbbIOspRbpCr2IKXE9cPVatGOCBrBQobQmBmaeU"
}
}
}
}
}
```
With the input value above, the value of all the variables in the package are:
```live:obj_example:output
```
## Output Document
When Envoy receives a policy decision, it expects a JSON object with the following fields:
- `allowed` (required): a boolean deciding whether or not the request is allowed
- `headers` (optional): an object mapping a string header name to a string header value (e.g. key "x-ext-auth-allow" has value "yes")
- `response_headers_to_add` (optional): an object mapping a string header name to a string header value
- `request_headers_to_remove` (optional): is an array of string header names
- `http_status` (optional): a number representing the HTTP status code
- `body` (optional): the response body
- `dynamic_metadata` (optional): an object representing dynamic metadata to be consumed by the next Envoy filter.
- `query_parameters_to_remove` (optional): is an array containing the names of string query parameters to be removed.
- `query_parameters_to_set` (optional): an object mapping parameter names to values (string) or arrays of values (for multiple values with the same key)
To construct that output object using the policies demonstrated in the last section, you can use the following Rego snippet. Notice that we are using partial object rules so that any variables with undefined values simply have no key in the `result` object.
```rego
result["allowed"] := allow
result["headers"] := headers
result["response_headers_to_add"] := response_headers_to_add
result["request_headers_to_remove"] := request_headers_to_remove
result["body"] := body
result["http_status"] := status_code
result["dynamic_metadata"] := dynamic_metadata
result["query_parameters_to_remove"] := query_parameters_to_remove
result["query_parameters_to_set"] = query_parameters_to_set
```
For a single user, including this snippet in your normal policy is fine, but when you have multiple teams writing policies, you will typically pull this bit of boilerplate into a wrapper package, so your teams can focus on writing the policies shown in the previous sections.
## Input Document
In OPA, `input` is a reserved, global variable whose value is the request sent by the Envoy External Authorization filter
to OPA. The OPA-Envoy plugin supports both [v2](https://www.envoyproxy.io/docs/envoy/latest/api-v2/service/auth/v2/external_auth.proto#service-auth-v2-checkrequest)
and [v3](https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/auth/v3/external_auth.proto#service-auth-v3-checkrequest)
versions of the `CheckRequest` which is used to pass the request to OPA.
For v3 requests, the [specified JSON mapping for protobuf](https://developers.google.com/protocol-buffers/docs/proto3#json)
is used for making the incoming `envoy.service.auth.v3.CheckRequest` available in `input`.
It differs from the encoding used for v2 requests. In v3, all keys are lower camelcase.
Also, needless nesting of `oneof` values is removed.
For example, source address data that looks like this in v2,
```
"source": {
"address": {
"Address": {
"SocketAddress": {
"PortSpecifier": {
"PortValue": 59052
},
"address": "127.0.0.1"
}
}
}
}
```
becomes, in v3,
```
"source": {
"address": {
"socketAddress": {
"address": "127.0.0.1",
"portValue": 59052
}
}
}
```
The following table shows the rego code for common data, in v2 and v3:
| information | rego v2 | rego v3 |
| --------------------- | ------------------------------------------------------------------------------------ | -------------------------------------------------------------- |
| `source address` | `input.attributes.source.address.Address.SocketAddress.address` | `input.attributes.source.address.socketAddress.address` |
| `source port` | `input.attributes.source.address.Address.SocketAddress.PortSpecifier.PortValue` | `input.attributes.source.address.socketAddress.portValue` |
| `destination address` | `input.attributes.destination.address.Address.SocketAddress.address` | `input.attributes.destination.address.socketAddress.address` |
| `destination port` | `input.attributes.destination.address.Address.SocketAddress.PortSpecifier.PortValue` | `input.attributes.destination.address.socketAddress.portValue` |
| `dynamic metadata` | `input.attributes.metadata_context.filter_metadata` | `input.attributes.metadataContext.filterMetadata` |
Due to those differences, it's important to know which version is used when writing policies.
Thus, this information is passed into the OPA evaluation under `input.version`, where you'll either
find, for v2,
```live:v2_sample:module:read_only
input.version == { "ext_authz": "v2", "encoding": "encoding/json" }
```
or, for v3,
```live:v3_sample:module:read_only
input.version == { "ext_authz": "v3", "encoding": "protojson" }
```
To have Envoy use the v3 version of the service, the `http_filters` entry in the Envoy configuration should look
like below (minimal version):
```yaml
http_filters:
- name: envoy.ext_authz
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
transport_api_version: V3
grpc_service:
google_grpc: # or envoy_grpc
target_uri: "127.0.0.1:9191"
```
### Example Input
<details>
<summary>Example v3 Input</summary>
```json
{
"attributes": {
"source": {
"address": {
"socketAddress": {
"address": "172.17.0.1",
"portValue": 61402
}
}
},
"destination": {
"address": {
"socketAddress": {
"address": "172.17.06",
"portValue": 8000
}
}
},
"request": {
"time": "2020-11-20T09:47:47.722473Z",
"http": {
"id": "13519049518330544501",
"method": "POST",
"headers": {
":authority": "192.168.99.206:30164",
":method": "POST",
":path": "/people?lang=en",
"accept": "*/*",
"authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyb2xlIjoiYWRtaW4iLCJzdWIiOiJZbTlpIiwibmJmIjoxNTE0ODUxMTM5LCJleHAiOjE2NDEwODE1Mzl9.WCxNAveAVAdRCmkpIObOTaSd0AJRECY2Ch2Qdic3kU8",
"content-length": "41",
"content-type": "application/json",
"user-agent": "curl/7.54.0",
"x-forwarded-proto": "http",
"x-request-id": "7bca5c86-bf55-432c-b212-8c0f1dc999ec"
},
"host": "192.168.99.206:30164",
"path": "/people?lang=en",
"protocol": "HTTP/1.1",
"body": "{\"firstname\":\"Charlie\", \"lastname\":\"Opa\"}",
"size": 41
}
},
"metadataContext": {}
},
"parsed_body": { "firstname": "Charlie", "lastname": "Opa" },
"parsed_path": ["people"],
"parsed_query": { "lang": ["en"] },
"truncated_body": false,
"version": {
"encoding": "protojson",
"ext_authz": "v3"
}
}
```
</details>
<details>
<summary>Example v2 Input</summary>
```json
{
"attributes": {
"source": {
"address": {
"Address": {
"SocketAddress": {
"PortSpecifier": {
"PortValue": 61402
},
"address": "172.17.0.1"
}
}
}
},
"destination": {
"address": {
"Address": {
"SocketAddress": {
"PortSpecifier": {
"PortValue": 8000
},
"address": "172.17.0.6"
}
}
}
},
"request": {
"http": {
"id": "13519049518330544501",
"method": "POST",
"headers": {
":authority": "192.168.99.206:30164",
":method": "POST",
":path": "/people?lang=en",
"accept": "*/*",
"authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyb2xlIjoiYWRtaW4iLCJzdWIiOiJZbTlpIiwibmJmIjoxNTE0ODUxMTM5LCJleHAiOjE2NDEwODE1Mzl9.WCxNAveAVAdRCmkpIObOTaSd0AJRECY2Ch2Qdic3kU8",
"content-length": "41",
"content-type": "application/json",
"user-agent": "curl/7.54.0",
"x-forwarded-proto": "http",
"x-request-id": "7bca5c86-bf55-432c-b212-8c0f1dc999ec"
},
"host": "192.168.99.206:30164",
"path": "/people?lang=en",
"protocol": "HTTP/1.1",
"body": "{\"firstname\":\"Charlie\", \"lastname\":\"Opa\"}",
"size": 41
}
}
},
"parsed_body": { "firstname": "Charlie", "lastname": "Opa" },
"parsed_path": ["people"],
"parsed_query": { "lang": ["en"] },
"truncated_body": false,
"version": {
"encoding": "encoding/json",
"ext_authz": "v2"
}
}
```
</details>
The `parsed_path` field in the input is generated from the `path` field in the HTTP request which is included in the
Envoy External Authorization `CheckRequest` message type. This field provides the request path as a string array which
can help policy authors perform pattern matching on the HTTP request path. The below sample policy allows anyone to
access the path `/people`.
```live:parsed_path_example:module:read_only
package envoy.authz
default allow := false
allow if input.parsed_path == ["people"]
```
The `parsed_query` field in the input is also generated from the `path` field in the HTTP request. This field provides
the HTTP URL query as a map of string array. The below sample policy allows anyone to access the path
`/people?lang=en&id=1&id=2`.
```live:parsed_query_example:module:read_only
package envoy.authz
default allow := false
allow if {
input.parsed_path == ["people"]
input.parsed_query.lang == ["en"]
input.parsed_query.id == ["1", "2"]
}
```
The `parsed_body` field in the input is generated from the `body` field in the HTTP request which is included in the
Envoy External Authorization `CheckRequest` message type. This field contains the deserialized JSON request body which
can then be used in a policy as shown below.
```live:parsed_body_example:module:read_only
package envoy.authz
default allow := false
allow if {
input.parsed_body.firstname == "Charlie"
input.parsed_body.lastname == "Opa"
}
```
The `truncated_body` field in the input represents if the HTTP request body is truncated. The body is considered to be
truncated, if the value of the `Content-Length` header exceeds the size of the request body.
If `skip-request-body-parse: true` is specified in the OPA-Envoy [configuration](../#configuration), then
the `parsed_body` and `truncated_body` fields will be omitted from the input.
## Example with JWT payload passed from Envoy
Envoy can be configured to pass validated JWT payload data into the `ext_authz` filter with `metadata_context_namespaces`
and `payload_in_metadata`.
### Example Envoy Configuration
```yaml
http_filters:
- name: envoy.filters.http.jwt_authn
typed_config:
"@type": type.googleapis.com/envoy.config.filter.http.jwt_authn.v2alpha.JwtAuthentication
providers:
example:
payload_in_metadata: verified_jwt
<...>
- name: envoy.ext_authz
config:
metadata_context_namespaces:
- envoy.filters.http.jwt_authn
<...>
```
### Example OPA Input
This will result in something like the following dictionary being added to `input.attributes` (some common fields have
been excluded for brevity):
```json
"metadata_context": {
"filter_metadata": {
"envoy.filters.http.jwt_authn": {
"verified_jwt": {
"email": "alice@example.com",
"exp": 1569026124,
"name": "Alice"
}
}
}
}
```
docs/new/docs/envoy/tutorial-gloo-edge.md
+341
View File
@@ -0,0 +1,341 @@
---
title: "Tutorial: Gloo Edge"
sidebar_position: 4
---
[Gloo Edge](https://docs.solo.io/gloo-edge/latest/) is an Envoy based API Gateway that provides a Kubernetes CRD to manage Envoy configuration for performing traffic management and routing.
Gloo Edge allows creation of a [Custom External Auth Service](https://docs.solo.io/gloo-edge/master/guides/security/auth/custom_auth/) that implements the Envoy spec for an [External Authorization Server](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ext_authz_filter.html).
The purpose of this tutorial is to show how OPA could be used with Gloo Edge to apply security policies for upstream services.
## Prerequisites
This tutorial requires Kubernetes 1.14 or later. To run the tutorial locally, we recommend using [minikube](https://minikube.sigs.k8s.io/docs/start/) in version v1.0+ with Kubernetes 1.14+.
The tutorial also requires [Helm](https://helm.sh/docs/intro/install/) to install Gloo Edge on a Kubernetes cluster.
## Steps
### 1. Start Minikube
```bash
minikube start
```
### 2. Setup and Configure Gloo Edge
```bash
helm repo add gloo https://storage.googleapis.com/solo-public-helm
helm upgrade --install --namespace gloo-system --create-namespace gloo gloo/gloo
kubectl config set-context $(kubectl config current-context) --namespace=gloo-system
```
Ensure all the pods are running using `kubectl get pod` command.
### 3. Create Virtual Service and Upstream
[Virtual Services](https://docs.solo.io/gloo-edge/latest/introduction/architecture/concepts/#virtual-services) define a set of route rules, security configuration, rate limiting, transformations, and other core routing capabilities supported by Gloo Edge.
[Upstreams](https://docs.solo.io/gloo-edge/latest/introduction/architecture/concepts/#upstreams) define destinations for routes.
Save the configuration as **vs.yaml**.
```yaml
apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
name: httpbin
spec:
static:
hosts:
- addr: httpbin.org
port: 80
---
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
name: httpbin
spec:
virtualHost:
domains:
- "*"
routes:
- matchers:
- prefix: /
routeAction:
single:
upstream:
name: httpbin
namespace: gloo-system
options:
autoHostRewrite: true
```
```bash
kubectl apply -f vs.yaml
```
### 4. Test Gloo
For simplification port-forwarding will be used. Open another terminal and execute.
```bash
kubectl port-forward deployment/gateway-proxy 8080:8080
```
The `VirtualService` created in the previous step forwards requests to http://httpbin.org
Let's test that Gloo works properly by running the below command in the first terminal.
```bash
curl -XGET -Is localhost:8080/get | head -n 1
HTTP/1.1 200 OK
curl http -XPOST -Is localhost:8080/post | head -n1
HTTP/1.1 200 OK
```
### 5. Define an OPA Policy
The following OPA policy will work as follows:
- Alice is granted a **guest** role and can perform `GET` requests.
- Bob is granted an **admin** role and can perform `GET` and `POST` requests.
**policy.rego**
```live:example:module:openable
package envoy.authz
import input.attributes.request.http as http_request
default allow := false
allow if {
is_token_valid
action_allowed
}
is_token_valid if {
token.valid
now := time.now_ns() / 1000000000
token.payload.nbf <= now
now < token.payload.exp
}
action_allowed if {
http_request.method == "GET"
token.payload.role == "guest"
}
action_allowed if {
http_request.method == "GET"
token.payload.role == "admin"
}
action_allowed if {
http_request.method == "POST"
token.payload.role == "admin"
}
token := {"valid": valid, "payload": payload} if {
[_, encoded] := split(http_request.headers.authorization, " ")
[valid, _, payload] := io.jwt.decode_verify(encoded, {"secret": "secret"})
}
```
A sample input can be seen below using Alice's token, Alice should be able to `GET` but not `POST`
```live:example:input
{
"attributes": {
"request": {
"http": {
"method": "GET",
"headers": {
"authorization": "Bearer eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCJ9.eyJleHAiOiAyMjQxMDgxNTM5LCAibmJmIjogMTUxNDg1MTEzOSwgInJvbGUiOiAiZ3Vlc3QiLCAic3ViIjogIllXeHBZMlU9In0.Uk5hgUqMuUfDLvBLnlXMD0-X53aM_Hlziqg3vhOsCc8"
}
}
}
}
}
```
With the input value above, the answer is:
```live:example:output
```
Next we build an OPA bundle.
```bash
opa build policy.rego
```
And now we serve the OPA bundle created above using Nginx.
```bash
docker run --rm --name bundle-server -d -p 8888:80 -v ${PWD}:/usr/share/nginx/html:ro nginx:latest
```
### 6. Setup OPA-Envoy
Create a deployment as shown below and save it in **deployments.yaml**
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: opa
labels:
app: opa
spec:
replicas: 1
selector:
matchLabels:
app: opa
template:
metadata:
labels:
app: opa
spec:
containers:
- name: opa
image: openpolicyagent/opa:{{< current_opa_envoy_docker_version >}}
volumeMounts:
- readOnly: true
mountPath: /policy
name: opa-policy
args:
- "run"
- "--server"
- "--addr=0.0.0.0:8181"
- "--set=services.default.url=http://host.minikube.internal:8888"
- "--set=bundles.default.resource=bundle.tar.gz"
- "--set=plugins.envoy_ext_authz_grpc.addr=0.0.0.0:9191"
- "--set=plugins.envoy_ext_authz_grpc.path=envoy/authz/allow"
- "--set=decision_logs.console=true"
- "--set=status.console=true"
- "--ignore=.*"
volumes:
- name: opa-policy
```
```bash
kubectl apply -f deployments.yaml
```
Ensure all pods are running using `kubectl get pod` command.
Next, define a Kubernetes `service` for OPA-Envoy. This is required to create a DNS record and thereby create a Gloo `Upstream` object.
**service.yaml**
```yaml
apiVersion: v1
kind: Service
metadata:
name: opa
spec:
selector:
app: opa
ports:
- name: grpc
protocol: TCP
port: 9191
targetPort: 9191
```
**Note**: Since the name of the service port is `grpc`, `Gloo` will understand that traffic should be routed using HTTP2 protocol.
`kubectl apply -f service.yaml`
### 7. Configure Gloo Edge to use OPA
To use OPA as a custom auth server, we need to add the `extauth` attribute as described below:
**gloo.yaml**
```yaml
global:
extensions:
extAuth:
extauthzServerRef:
name: gloo-system-opa-9191
namespace: gloo-system
```
To apply it, run the following command:
```bash
helm upgrade --install --namespace gloo-system --create-namespace -f gloo.yaml gloo gloo/gloo
```
Configure Gloo Edge routes to perform authorization via configured extauth before regular processing.
**vs-patch.yaml**
```yaml
spec:
virtualHost:
options:
extauth:
customAuth: {}
```
Then apply the patch to our `VirtualService` as shown below:
```bash
kubectl patch vs httpbin --type=merge --patch "$(cat vs-patch.yaml)"
```
### 8. Exercise the OPA Policy
Before we exercise the policy, for convenience sake, we will want to store Alice and Bob's tokens in environment variables as such:
```bash
export ALICE_TOKEN="eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCJ9.eyJleHAiOiAyMjQxMDgxNTM5LCAibmJmIjogMTUxNDg1MTEzOSwgInJvbGUiOiAiZ3Vlc3QiLCAic3ViIjogIllXeHBZMlU9In0.Uk5hgUqMuUfDLvBLnlXMD0-X53aM_Hlziqg3vhOsCc8"
export BOB_TOKEN="eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCJ9.eyJleHAiOiAyMjQxMDgxNTM5LCAibmJmIjogMTUxNDg1MTEzOSwgInJvbGUiOiAiYWRtaW4iLCAic3ViIjogIlltOWkifQ.5qsm7rRTvqFHAgiB6evX0a_hWnGbWquZC0HImVQPQo8"
```
Now let's verify that OPA only allows **Alice** to perform `GET` requests.
```bash
curl -XGET -Is -H "Authorization: Bearer $ALICE_TOKEN" localhost:8080/get
HTTP/1.1 200 OK
```
And with a `POST` request, we get:
```bash
curl http -XPOST -Is -H "Authorization: Bearer $ALICE_TOKEN" localhost:8080/post
HTTP/1.1 403 Forbidden
```
And for **Bob**, we should be able to `GET` and `POST`:
```bash
curl -XGET -Is -H "Authorization: Bearer $BOB_TOKEN" localhost:8080/get
HTTP/1.1 200 OK
```
And the `POST`:
```bash
curl http -XPOST -Is -H "Authorization: Bearer $BOB_TOKEN" localhost:8080/post
HTTP/1.1 200 OK
```
Check OPA's decision logs to view the inputs received by OPA from Gloo Edge and the results generated by OPA.
```bash
kubectl logs deployment/opa -n gloo-system
```
## Wrap Up
Congratulations for finishing the tutorial!
This tutorial showed how you can use OPA with [Gloo Edge](https://docs.solo.io/gloo-edge/latest/) to apply security policies for upstream services and how to create and test a policy that would allow `GET` or `POST` requests based on your user role.
docs/new/docs/envoy/tutorial-istio.md
+217
View File
@@ -0,0 +1,217 @@
---
title: "Tutorial: Istio"
sidebar_position: 3
---
[Istio](https://istio.io/latest/) is an open source service mesh for managing the different microservices that make
up a cloud-native application. Istio provides a mechanism to use a service as an external authorizer with the
[AuthorizationPolicy API](https://istio.io/latest/docs/tasks/security/authorization/authz-custom/).
This tutorial shows how Istio's AuthorizationPolicy can be configured to delegate authorization decisions to OPA.
## Prerequisites
This tutorial requires Kubernetes 1.20 or later. To run the tutorial locally ensure you start a cluster with Kubernetes
version 1.20+, we recommend using [minikube](https://kubernetes.io/docs/getting-started-guides/minikube) or
[KIND](https://kind.sigs.k8s.io/).
The tutorial also requires Istio v1.19.0 or later. It assumes you have Istio deployed on top of Kubernetes.
See Istio's [Quick Start](https://istio.io/docs/setup/kubernetes/install/kubernetes/) page to get started.
If you are using an earlier version of Istio (1.9+), you will have to customize the `AuthorizationPolicy` in the
`quick_start.yaml` file to use the `security.istio.io/v1beta1` API version instead of `security.istio.io/v1`.
## Steps
### 1. Install OPA-Envoy
```bash
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/opa-envoy-plugin/main/examples/istio/quick_start.yaml
```
The `quick_start.yaml` manifest defines the following resources:
- AuthorizationPolicy to direct authorization checks to the OPA-Envoy sidecar. See `kubectl -n {$NAMESPACE} get authorizationpolicy ext-authz` for details.
- ServiceEntry to allow Istio to find the OPA-Envoy sidecars. See `kubectl -n {$NAMESPACE} get serviceentry opa-ext-authz-grpc-local` for details.
- Kubernetes namespace (`opa-istio`) for OPA-Envoy control plane components.
- Kubernetes admission controller in the `opa-istio` namespace that automatically injects the OPA-Envoy sidecar into pods in namespaces labelled with `opa-istio-injection=enabled`.
- OPA configuration file and an OPA policy into ConfigMaps in the namespace where the app will be deployed, e.g., `default`.
The following is the example OPA policy:
- alice is granted a **guest** role and can perform a `GET` request to `/productpage`.
- bob is granted an **admin** role and can perform a `GET` to `/productpage` and `/api/v1/products`.
```live:example:module:openable
package istio.authz
default allow := false
allow if {
input.parsed_path[0] == "health"
input.attributes.request.method == "GET"
}
allow if {
some user_role in _user_roles[_user_name]
some permission in _role_permissions[user_role]
permission.method == input.attributes.request.http.method
permission.path == input.attributes.request.http.path
}
# Underscore prefix used only to signal that rules and functions are
# intended to be referenced only within the same policy, i.e. "private".
# It has no special meaning to OPA.
_user_name := parsed if {
[_, encoded] := split(input.attributes.request.http.headers.authorization, " ")
[parsed, _] := split(base64url.decode(encoded), ":")
}
_user_roles := {
"alice": ["guest"],
"bob": ["admin"],
}
_role_permissions := {
"guest": [{"method": "GET", "path": "/productpage"}],
"admin": [
{"method": "GET", "path": "/productpage"},
{"method": "GET", "path": "/api/v1/products"},
],
}
```
OPA is configured to query for the `data.istio.authz.allow`
decision. If the response is `true` the operation is allowed, otherwise the
operation is denied. Sample input received by OPA is shown below:
```live:example:query:hidden
data.istio.authz.allow
```
```live:example:input
{
"attributes": {
"request": {
"http": {
"method": "GET",
"path": "/productpage",
"headers": {
"authorization": "Basic YWxpY2U6cGFzc3dvcmQ="
}
}
}
}
}
```
With the input value above, the answer is:
```live:example:output
```
An example of the complete input received by OPA can be seen [here](https://github.com/open-policy-agent/opa-envoy-plugin/tree/main/examples/istio#example-input).
> In typical deployments the policy would either be built into the OPA container
> image or it would be fetched dynamically via the [Bundle API](https://www.openpolicyagent.org/docs/latest/bundles/). ConfigMaps are
> used in this tutorial for test purposes.
### 2. Configure the mesh to define the external authorizer
Edit the mesh configmap with `kubectl edit configmap -n istio-system istio` and define the external provider:
```yaml
data:
mesh: |-
# Add the following lines to define the ServiceEntry previously created as an external authorizer:
extensionProviders:
- name: opa-ext-authz-grpc
envoyExtAuthzGrpc:
service: opa-ext-authz-grpc.local
port: 9191
```
See [the Istio Docs for AuthorizationPolicy](https://istio.io/latest/docs/tasks/security/authorization/authz-custom/#define-the-external-authorizer) for
more details.
The format of the service value is `[<Namespace>/]<Hostname>`. The specification
of `<Namespace>` is required only when it is insufficient to unambiguously resolve
a service in the service registry. See also the [configuration documentation](https://istio.io/latest/docs/reference/config/istio.mesh.v1alpha1/#MeshConfig-ExtensionProvider-EnvoyExternalAuthorizationGrpcProvider).
Example: `opa-ext-authz-grpc.foo.svc.cluster.local` or
`bar/opa-ext-authz-grpc.local`.
### 3. Enable automatic injection of the Istio Proxy and OPA-Envoy sidecars in the namespace where the app will be deployed, e.g., `default`
```bash
kubectl label namespace default opa-istio-injection="enabled"
kubectl label namespace default istio-injection="enabled"
```
### 4. Deploy the BookInfo application and make it accessible outside the cluster
```bash
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/bookinfo/platform/kube/bookinfo.yaml
```
```bash
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/bookinfo/networking/bookinfo-gateway.yaml
```
### 5. Set the `SERVICE_HOST` environment variable in your shell to the public IP/port of the Istio Ingress gateway
Run this command in a new terminal window to start a Minikube tunnel that sends traffic to your Istio Ingress Gateway:
```
minikube tunnel
```
Check that the Service shows an `EXTERNAL-IP`:
```bash
kubectl -n istio-system get service istio-ingressgateway
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-ingressgateway LoadBalancer 10.98.42.178 127.0.0.1 15021:32290/TCP,80:30283/TCP,443:32497/TCP,31400:30216/TCP,15443:30690/TCP 5s
```
**minikube:**
```bash
export SERVICE_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
```
For other platforms see the [Istio documentation on determining ingress IP and ports.](https://istio.io/docs/tasks/traffic-management/ingress/#determining-the-ingress-ip-and-ports)
### 6. Exercise the OPA policy
Check that **alice** can access `/productpage` **BUT NOT** `/api/v1/products`.
```bash
curl --user alice:password -i http://$SERVICE_HOST/productpage
curl --user alice:password -i http://$SERVICE_HOST/api/v1/products
```
Check that **bob** can access `/productpage` **AND** `/api/v1/products`.
```bash
curl --user bob:password -i http://$SERVICE_HOST/productpage
curl --user bob:password -i http://$SERVICE_HOST/api/v1/products
```
## Wrap Up
Congratulations for finishing the tutorial !
This tutorial showed how Istio's [AuthorizationPolicy API](https://istio.io/latest/docs/tasks/security/authorization/authz-custom/)
can be configured to use OPA as an External authorization service.
This tutorial also showed a sample OPA policy that returns a `boolean` decision
to indicate whether a request should be allowed or not.
More details about the tutorial can be seen
[here](https://github.com/open-policy-agent/opa-envoy-plugin/tree/main/examples/istio).
docs/new/docs/envoy/tutorial-standalone-envoy.md
+561
View File
@@ -0,0 +1,561 @@
---
title: "Tutorial: Standalone Envoy"
sidebar_position: 2
---
The tutorial shows how Envoy’s External
[authorization filter](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ext_authz_filter.html)
can be used with OPA as an authorization service to enforce security policies over API requests
received by Envoy. The tutorial also covers examples of authoring custom
policies over the HTTP request body.
## Overview
In this tutorial we'll see how to use OPA as an External
Authorization service for the Envoy proxy. We'll do this by:
- Running a local Kubernetes cluster
- Creating a simple authorization policy in Rego and serving it via the Bundle API
- Deploying a sample application with Envoy and OPA sidecars
- Run some sample requests to see the policy in action
Note that other than the HTTP client and bundle server, all components
are co-located in the same pod.
## Running a local Kubernetes cluster
To start a local Kubernetes cluster to run our demo, we'll be using
[kind](https://kind.sigs.k8s.io/).
:::info
If you haven't used `kind` before, you can find installation instructions
in the [project documentation](https://kind.sigs.k8s.io/#installation-and-usage).
:::
Create a cluster with the following command:
```shell
$ kind create cluster --name opa-envoy --image kindest/node:v1.27.3
Creating cluster "opa-envoy" ...
✓ Ensuring node image (kindest/node:v1.27.3) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
...
```
Once the cluster is created, make sure your `kubectl` context is set to connect
to the new cluster:
```shell
$ kubectl cluster-info --context kind-opa-envoy
Kubernetes control plane is running at ...
CoreDNS is running at ...
...
```
Listing the cluster nodes, should show something like this:
```shell
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
opa-envoy-control-plane Ready control-plane 2m35s v1.27.3
```
## Creating & Serving our Policy Bundle
This tutorial assumes you have some Rego knowledge, in summary the policy below does the following:
- Checks that the JWT token is valid
- Checks that the action is allowed based on the token payload `role` and the request path
- Guests have read-only access to the `/people` endpoint, admins can create users too as long as the
name is not the same as the admin's name.
```rego
# policy.rego
package envoy.authz
import input.attributes.request.http as http_request
default allow := false
allow if {
is_token_valid
action_allowed
}
is_token_valid if {
token.valid
now := time.now_ns() / 1000000000
token.payload.nbf <= now
now < token.payload.exp
}
action_allowed if {
http_request.method == "GET"
token.payload.role == "guest"
glob.match("/people", ["/"], http_request.path)
}
action_allowed if {
http_request.method == "GET"
token.payload.role == "admin"
glob.match("/people", ["/"], http_request.path)
}
action_allowed if {
http_request.method == "POST"
token.payload.role == "admin"
glob.match("/people", ["/"], http_request.path)
lower(input.parsed_body.firstname) != base64url.decode(token.payload.sub)
}
token := {"valid": valid, "payload": payload} if {
[_, encoded] := split(http_request.headers.authorization, " ")
[valid, _, payload] := io.jwt.decode_verify(encoded, {"secret": "secret"})
}
```
Create a file called `policy.rego` with the above content and store it in a ConfigMap:
```shell
kubectl create configmap authz-policy --from-file policy.rego
```
Now that the policy is stored in a ConfigMap, we can spin up an HTTP server to make it
available as a Bundle to OPA when it's making decisions for our application:
```yaml
# bundle-server.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: bundle-server
labels:
app: bundle-server
spec:
replicas: 1
selector:
matchLabels:
app: bundle-server
template:
metadata:
labels:
app: bundle-server
spec:
initContainers:
- name: opa-builder
image: openpolicyagent/opa:latest
args:
- "build"
- "--bundle"
- "/opt/policy/"
- "--output"
- "/opt/output/bundle.tar.gz"
volumeMounts:
- name: index
mountPath: /opt/output/
- name: policy
mountPath: /opt/policy/
containers:
- name: bundle-server
image: nginx:1.25
ports:
- containerPort: 80
name: http
volumeMounts:
- name: index
mountPath: /usr/share/nginx/html
volumes:
- name: index
emptyDir: {}
- name: policy
configMap:
name: authz-policy
---
apiVersion: v1
kind: Service
metadata:
name: bundle-server
spec:
selector:
app: bundle-server
ports:
- protocol: TCP
port: 80
targetPort: http
```
Create a file called `bundle-server.yaml` with the above content and apply it to the cluster:
```shell
kubectl apply -f bundle-server.yaml
```
Once the deployment is running, we can check that the bundle is available by running:
```shell
kubectl port-forward service/bundle-server 8080:80
```
Before checking that the bundle has been generated correctly and is available to download:
```shell
$ curl -I localhost:8080/bundle.tar.gz
HTTP/1.1 200 OK
...
```
You may now exit the port-forwarding session, the bundle server will only be accessed
from inside the cluster from now on.
## Deploying an application with Envoy and OPA sidecars
In this tutorial, we are manually configuring the Envoy proxy sidecar to intermediate
HTTP traffic from clients and our application. Envoy will consult OPA to
make authorization decisions for each request by sending `CheckRequest` messages over
a gRPC connection.
We will use the following Envoy configuration to achieve this. In summary, this
configures Envoy to:
- Listen on port `8000` for HTTP traffic
- Consult OPA for authorization decisions at 127.0.0.1:9191 & deny failing requests
- Forward requests to the application at 127.0.0.1:8080 if ok.
```yaml
# envoy.yaml
static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 8000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: service
http_filters:
- name: envoy.ext_authz
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
transport_api_version: V3
with_request_body:
max_request_bytes: 8192
allow_partial_message: true
failure_mode_allow: false
grpc_service:
google_grpc:
target_uri: 127.0.0.1:9191
stat_prefix: ext_authz
timeout: 0.5s
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: service
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
load_assignment:
cluster_name: service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8080
admin:
access_log_path: "/dev/null"
address:
socket_address:
address: 0.0.0.0
port_value: 8001
layered_runtime:
layers:
- name: static_layer_0
static_layer:
envoy:
resource_limits:
listener:
example_listener_name:
connection_limit: 10000
overload:
global_downstream_max_connections: 50000
```
Create a `ConfigMap` containing the above configuration by running:
```shell
kubectl create configmap proxy-config --from-file envoy.yaml
```
Our application will be configured using a `Deployment` and `Service`.
There are a few things to note:
- the pods have an `initContainer` that configures the `iptables` rules to
redirect traffic to the Envoy proxy.
- the `demo-test-server` container is a simple user store using in-memory state.
- the `envoy` container is configured to use the `proxy-config` `ConfigMap` we
created earlier.
- The OPA container is configured to download policy bundles from
the in-cluster bundle server (`bundle-server.default.svc.cluster.local`).
```yaml
# app.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
name: example-app
labels:
app: example-app
spec:
replicas: 1
selector:
matchLabels:
app: example-app
template:
metadata:
labels:
app: example-app
spec:
initContainers:
- name: proxy-init
image: openpolicyagent/proxy_init:v8
# Configure the iptables bootstrap script to redirect traffic to the
# Envoy proxy on port 8000. Envoy will be running as 1111, and port
# 8282 will be excluded to support OPA health checks.
args: ["-p", "8000", "-u", "1111", "-w", "8282"]
securityContext:
capabilities:
add:
- NET_ADMIN
runAsNonRoot: false
runAsUser: 0
containers:
- name: app
image: openpolicyagent/demo-test-server:v1
ports:
- containerPort: 8080
- name: envoy
image: envoyproxy/envoy:v1.26.3
volumeMounts:
- readOnly: true
mountPath: /config
name: proxy-config
args:
- "envoy"
- "--config-path"
- "/config/envoy.yaml"
env:
- name: ENVOY_UID
value: "1111"
- name: opa
image: openpolicyagent/opa:latest-envoy
args:
- "run"
- "--server"
- "--addr=localhost:8181"
- "--diagnostic-addr=0.0.0.0:8282"
- "--set=services.default.url=http://bundle-server"
- "--set=bundles.default.resource=bundle.tar.gz"
- "--set=plugins.envoy_ext_authz_grpc.addr=:9191"
- "--set=plugins.envoy_ext_authz_grpc.path=envoy/authz/allow"
- "--set=decision_logs.console=true"
- "--set=status.console=true"
- "--ignore=.*"
livenessProbe:
httpGet:
path: /health?plugins
scheme: HTTP
port: 8282
initialDelaySeconds: 5
periodSeconds: 5
readinessProbe:
httpGet:
path: /health?plugins
scheme: HTTP
port: 8282
initialDelaySeconds: 1
periodSeconds: 3
volumes:
- name: proxy-config
configMap:
name: proxy-config
---
apiVersion: v1
kind: Service
metadata:
name: example-app
spec:
selector:
app: example-app
ports:
- protocol: TCP
port: 80
targetPort: 8080
```
Deploy the application and Kubernetes Service to the cluster with:
```shell
kubectl apply -f app.yaml
```
Check that everything is working by listing the pod (make sure that
all three pods are running ok).
```shell
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
bundle-server-5d7bfffdb6-bgn86 1/1 Running 0 1m
example-app-74b4bc88-5d4wh 3/3 Running 0 1m
```
## See the Policy in Action
Run a shell inside the cluster to use for testing. We will use this in-cluster
shell for the rest of the tutorial.
```shell
kubectl run curl --restart=Never -it --rm --image curlimages/curl:8.1.2 -- sh
```
Set two tokens for two users, Alice and Bob with different permissions.
As defined by our policy:
```shell
export ALICE_TOKEN="eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCJ9.eyJleHAiOiAyMjQxMDgxNTM5LCAibmJmIjogMTUxNDg1MTEzOSwgInJvbGUiOiAiZ3Vlc3QiLCAic3ViIjogIllXeHBZMlU9In0.Uk5hgUqMuUfDLvBLnlXMD0-X53aM_Hlziqg3vhOsCc8"
export BOB_TOKEN="eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCJ9.eyJleHAiOiAyMjQxMDgxNTM5LCAibmJmIjogMTUxNDg1MTEzOSwgInJvbGUiOiAiYWRtaW4iLCAic3ViIjogIlltOWkifQ.5qsm7rRTvqFHAgiB6evX0a_hWnGbWquZC0HImVQPQo8"
```
### Listing People
Send a request to list people. This should succeed for both Alice and Bob.
```shell
curl -i -H "Authorization: Bearer $ALICE_TOKEN" http://example-app/people
```
```
HTTP/1.1 200 OK
content-type: application/json
date: Tue, 18 Jul 2023 15:22:25 GMT
content-length: 96
x-envoy-upstream-service-time: 14
server: envoy
[{"id":"1","firstname":"John","lastname":"Doe"},{"id":"2","firstname":"Jane","lastname":"Doe"}]
```
And for Bob:
```shell
curl -i -H "Authorization: Bearer $BOB_TOKEN" http://example-app/people
```
```
HTTP/1.1 200 OK
...omitted...
```
### Creating People
Send a request to create a new user. This should fail for Alice but not Bob:
```shell
curl -i -H "Authorization: Bearer $ALICE_TOKEN" \
-d '{"firstname":"Foo", "lastname":"Bar"}' -H "Content-Type: application/json" \
-X POST http://example-app/people
```
```
HTTP/1.1 403 Forbidden
date: Tue, 18 Jul 2023 15:25:28 GMT
server: envoy
content-length: 0
```
And for Bob, the request is permitted and the user is saved with an ID
```shell
curl -i -H "Authorization: Bearer $BOB_TOKEN" \
-d '{"firstname":"Foo", "lastname":"Bar"}' -H "Content-Type: application/json" \
-X POST http://example-app/people
```
```
HTTP/1.1 200 OK
content-type: application/json
date: Tue, 18 Jul 2023 15:28:20 GMT
content-length: 51
x-envoy-upstream-service-time: 11
server: envoy
{"id":"498081","firstname":"Foo","lastname":"Bar"}
```
### Creating People: Conflict
Our policy also blocks users from creating users with the same name, test that
functionality with this request:
```shell
curl -i -H "Authorization: Bearer $BOB_TOKEN" \
-d '{"firstname":"Bob", "lastname":"Bar"}' -H "Content-Type: application/json" \
-X POST http://example-app/people
```
```
HTTP/1.1 403 Forbidden
date: Tue, 18 Jul 2023 15:31:48 GMT
server: envoy
content-length: 0
```
## Shutting Down
Exit the in-cluster shell by typing `exit`.
Delete the cluster by running:
```shell
$ kind delete cluster --name opa-envoy
Deleting cluster "opa-envoy" ...
Deleted nodes: ["opa-envoy-control-plane"]
```
## Wrap Up
Congratulations on finishing the tutorial !
This tutorial showed how to use OPA as an External authorization service to
enforce custom policies by leveraging Envoy’s External authorization filter.
This tutorial also showed a sample OPA policy that returns a `boolean` decision
to indicate whether a request should be allowed or not.
Envoy's external authorization filter allows optional response headers and body
to be sent to the downstream client or upstream. An example of a rule that
returns an object that not only indicates if a request is allowed or not but
also provides optional response headers, body and HTTP status that can be sent
to the downstream client or upstream can be seen
[here](https://github.com/open-policy-agent/opa-envoy-plugin#example-policy-with-object-response).
docs/new/docs/extensions.md
+538
View File
@@ -0,0 +1,538 @@
---
title: Extending OPA
sidebar_position: 8
---
OPA can be extended with custom built-in functions and plugins that
implement functionality like support for new protocols. This page explains how
to customize and extend OPA in different ways.
## Custom Built-in Functions in Go
Read this section if you want to extend OPA with custom built-in functions.
:::info
This section assumes you are embedding OPA as a library and executing policies
via the `github.com/open-policy-agent/opa/rego` package. If you are NOT embedding OPA
as a library and instead want to customize the OPA runtime, read this section
anyway because it provides useful information on implementing built-in functions.
For a complete example that shows how to add custom built-in functions to the
OPA runtime, see the [Adding Built-in Functions to the OPA Runtime](#adding-built-in-functions-to-the-opa-runtime) appendix.
:::
OPA supports built-in functions for simple operations like string manipulation
and arithmetic as well as more complex operations like JWT verification and
executing HTTP requests. If you need to to extend OPA with custom built-in
functions for use cases or integrations that are not supported out-of-the-box
you can supply the function definitions when you prepare queries.
Using custom built-in functions involves providing a declaration and
implementation. The declaration tells OPA the function's type signature and the
implementation provides the callback that OPA can execute during query
evaluation.
To get started you need to import three packages:
```
import "github.com/open-policy-agent/opa/ast"
import "github.com/open-policy-agent/opa/types"
import "github.com/open-policy-agent/opa/rego"
```
The `ast` and `types` packages contain the types for declarations and runtime
objects passed to your implementation. Here is a trivial example that shows the
process:
```golang
r := rego.New(
rego.Query(`x = hello("bob")`),
rego.Function1(
&rego.Function{
Name: "hello",
Decl: types.NewFunction(types.Args(types.S), types.S),
},
func(_ rego.BuiltinContext, a *ast.Term) (*ast.Term, error) {
if str, ok := a.Value.(ast.String); ok {
return ast.StringTerm("hello, " + string(str)), nil
}
return nil, nil
}),
)
query, err := r.PrepareForEval(ctx)
if err != nil {
// handle error.
}
```
At this point you can execute the `query`:
```golang
rs, err := query.Eval(ctx)
if err != nil {
// handle error.
}
// Do something with result.
fmt.Println(rs[0].Bindings["x"])
```
If you executed this code you the output would be:
```live:trivial:query:read_only
"hello, bob"
```
The example above highlights a few important points.
- The `rego` package includes variants of `rego.Function1` for accepting
different numbers of operands (e.g., `rego.Function2`, `rego.Function3`, etc.)
- The `rego.Function#Name` struct field specifies the operator that queries can
refer to.
- The `rego.Function#Decl` struct field specifies the function's type signature.
In the example above the function accepts a string and returns a string.
- The function indicates it's undefined by returning `nil` for the first return
argument.
Let's look at another example. Imagine you want to expose GitHub repository
metadata to your policies. One option is to implement a custom built-in
function to fetch the data for specific repositories on-the-fly.
```golang
r := rego.New(
rego.Query(`github.repo("open-policy-agent", "opa")`),
rego.Function2(
&rego.Function{
Name: "github.repo",
Decl: types.NewFunction(types.Args(types.S, types.S), types.A),
Memoize: true,
Nondeterministic: true,
},
func(bctx rego.BuiltinContext, a, b *ast.Term) (*ast.Term, error) {
// see implementation below.
},
),
)
```
Built-in function names can include `.` characters. Consider namespacing your
built-in functions to avoid collisions. This declaration indicates the function
accepts two strings and returns a value of type `any`. The `any` type is the
union of all types in Rego.
:::info
`types.S` and `types.A` are shortcuts for constructing Rego types. If you need
to define use-case specific types (e.g., a list of objects that have fields
`foo`, `bar`, and `baz`, you will need to construct them using the `types`
packages APIs.)
:::
The declaration also sets `rego.Function#Memoize` to true to enable memoization
across multiple calls in the same query. If your built-in function performs I/O,
you should enable memoization as it ensures function evaluation is
deterministic.
Since this built-in could have non-deterministic results, depending on network
conditions, the declaration also sets `rego.Function#Nondeterministic` to true.
This provides basic safety information to the runtime, so that the function
isn't accidentally run during bundle builds or partial evaluation. If your
builtin can have non-deterministic results, you should mark it appropriately
to avoid surprises.
The implementation wraps the Go standard library to perform HTTP requests to
GitHub's API:
```golang
func(bctx rego.BuiltinContext, a, b *ast.Term) (*ast.Term, error) {
var org, repo string
if err := ast.As(a.Value, &org); err != nil {
return nil, err
} else if err := ast.As(b.Value, &repo); err != nil {
return nil, err
}
req, err := http.NewRequest("GET", fmt.Sprintf("https://api.github.com/repos/%v/%v", org, repo), nil)
if err != nil {
return nil, err
}
resp, err := http.DefaultClient.Do(req.WithContext(bctx.Context))
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf(resp.Status)
}
v, err := ast.ValueFromReader(resp.Body)
if err != nil {
return nil, err
}
return ast.NewTerm(v), nil
}
```
The implementation is careful to use the context passed to the built-in function
when executing the HTTP request. See the appendix at the end of this page for
the complete example.
:::danger
Custom built-in functions **must not** be used for effecting changes in
external systems as OPA does not guarantee that the statement will be executed due
to automatic performance optimizations that are applied during policy evaluation.
:::
## Custom Plugins for OPA Runtime
Read this section if you want to customize or extend the OPA runtime/executable
with custom behaviour.
OPA defines a plugin interface that allows you to customize certain behaviour
like decision logging or add new behaviour like different query APIs. To
implement a custom plugin you must implement two interfaces:
- [`Factory`](https://pkg.go.dev/github.com/open-policy-agent/opa/plugins#Factory)
to instantiate your plugin.
- [`Plugin`](https://pkg.go.dev/github.com/open-policy-agent/opa/plugins#Plugin)
to provide your plugin behavior.
You can register your factory with OPA by calling
[`RegisterPlugin`](https://pkg.go.dev/github.com/open-policy-agent/opa/runtime#RegisterPlugin)
inside your main function.
### Plugin Status
The plugin may (optionally) report its current status to the plugin Manager via the `plugins.Manager#UpdatePluginStatus`
API.
:::info
If no status is provided the plugin is assumed to be working OK.
:::
Typically the plugin should report `StatusNotReady` at creation time and update to `StatusOK` (or `StatusErr`) when
appropriate.
### Putting It Together
The example below shows how you can implement a custom [Decision Logger](./management-decision-logs)
that writes events to a stream (e.g., stdout/stderr).
```golang
import (
"encoding/json"
"github.com/open-policy-agent/opa/plugins/logs"
)
const PluginName = "println_decision_logger"
type Config struct {
Stderr bool `json:"stderr"` // false => stdout, true => stderr
}
type PrintlnLogger struct {
manager *plugins.Manager
mtx sync.Mutex
config Config
}
func (p *PrintlnLogger) Start(ctx context.Context) error {
p.manager.UpdatePluginStatus(PluginName, &plugins.Status{State: plugins.StateOK})
return nil
}
func (p *PrintlnLogger) Stop(ctx context.Context) {
p.manager.UpdatePluginStatus(PluginName, &plugins.Status{State: plugins.StateNotReady})
}
func (p *PrintlnLogger) Reconfigure(ctx context.Context, config interface{}) {
p.mtx.Lock()
defer p.mtx.Unlock()
p.config = config.(Config)
}
// Log is called by the decision logger when a record (event) should be emitted. The logs.EventV1 fields
// map 1:1 to those described in https://www.openpolicyagent.org/docs/latest/management-decision-logs
func (p *PrintlnLogger) Log(ctx context.Context, event logs.EventV1) error {
p.mtx.Lock()
defer p.mtx.Unlock()
w := os.Stdout
if p.config.Stderr {
w = os.Stderr
}
bs, err := json.Marshal(event)
if err != nil {
p.manager.UpdatePluginStatus(PluginName, &plugins.Status{State: plugins.StateErr})
return nil
}
_, err = fmt.Fprintln(w, string(bs))
if err != nil {
p.manager.UpdatePluginStatus(PluginName, &plugins.Status{State: plugins.StateErr})
}
return nil
}
```
Next, implement a factory function that instantiates your plugin:
```golang
import (
"github.com/open-policy-agent/opa/plugins"
"github.com/open-policy-agent/opa/util"
)
type Factory struct{}
func (Factory) New(m *plugins.Manager, config interface{}) plugins.Plugin {
m.UpdatePluginStatus(PluginName, &plugins.Status{State: plugins.StateNotReady})
return &PrintlnLogger{
manager: m,
config: config.(Config),
}
}
func (Factory) Validate(_ *plugins.Manager, config []byte) (interface{}, error) {
parsedConfig := Config{}
return parsedConfig, util.Unmarshal(config, &parsedConfig)
}
```
Finally, register your factory with OPA and call `cmd.RootCommand.Execute`. The
latter starts OPA and does not return.
```golang
import (
"github.com/open-policy-agent/opa/cmd"
"github.com/open-policy-agent/opa/runtime"
)
func main() {
runtime.RegisterPlugin(PluginName, Factory{})
if err := cmd.RootCommand.Execute(); err != nil {
fmt.Println(err)
os.Exit(1)
}
}
```
At this point you can build an OPA executable including your plugin.
```
go build -o opa++
```
Define an OPA configuration file that will use your plugin:
**opa-config.yaml**:
```yaml
decision_logs:
plugin: println_decision_logger
plugins:
println_decision_logger:
stderr: false
```
Start OPA with the configuration file:
```bash
./opa++ run --server --config-file opa-config.yaml
```
Exercise the plugin via the OPA API:
```
curl localhost:8181/v1/data
```
If everything worked you will see the Go struct representation of the decision
log event written to stdout.
The source code for this example can be found
[here](https://github.com/open-policy-agent/contrib/tree/main/decision_logger_plugin_example).
:::info
If there is a mask policy set (see [Decision Logger](./management-decision-logs)
for details) the `Event` received by the demo plugin will potentially be different
than the example documented.
:::
## Setting the OPA Runtime Version
The OPA runtime version is set statically at build-time. The following global variables
are exported by the `github.com/open-policy-agent/opa/version` package and can be
set at build-time:
| Name | Description |
| ----------- | ------------------------------------------------------- |
| `Version` | Human-readable/semantic version of the OPA runtime. |
| `Vcs` | Git SHA that the OPA runtime was built from. |
| `Timestamp` | Date/time when the OPA runtime was built. |
| `Hostname` | Hostname of the system where the OPA runtime was built. |
These values can be set on the command-line when building OPA from source:
```
go build \
-ldflags=" \
-X github.com/open-policy-agent/opa/v1/version.Version=MY_VERSION\
-X github.com/open-policy-agent/opa/v1/version.Vcs=MY_COMMIT_HASH \
-X github.com/open-policy-agent/opa/v1/version.Hostname=MY_HOSTNAME \
-X github.com/open-policy-agent/opa/v1/version.Timestamp=MY_TIMESTAMP" \
-o opa++
```
## Appendix
### Custom Built-in Function in Go
```golang
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"net/http"
"github.com/open-policy-agent/opa/ast"
"github.com/open-policy-agent/opa/rego"
"github.com/open-policy-agent/opa/types"
)
func main() {
r := rego.New(
rego.Query(`github.repo("open-policy-agent", "opa")`),
rego.Function2(
&rego.Function{
Name: "github.repo",
Decl: types.NewFunction(types.Args(types.S, types.S), types.A),
Memoize: true,
Nondeterministic: true,
},
func(bctx rego.BuiltinContext, a, b *ast.Term) (*ast.Term, error) {
var org, repo string
if err := ast.As(a.Value, &org); err != nil {
return nil, err
} else if err := ast.As(b.Value, &repo); err != nil {
return nil, err
}
req, err := http.NewRequest("GET", fmt.Sprintf("https://api.github.com/repos/%v/%v", org, repo), nil)
if err != nil {
return nil, err
}
resp, err := http.DefaultClient.Do(req.WithContext(bctx.Context))
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf(resp.Status)
}
v, err := ast.ValueFromReader(resp.Body)
if err != nil {
return nil, err
}
return ast.NewTerm(v), nil
},
),
)
rs, err := r.Eval(context.Background())
if err != nil {
log.Fatal(err)
} else if len(rs) == 0 {
fmt.Println("undefined")
} else {
bs, _ := json.MarshalIndent(rs[0].Expressions[0].Value, "", " ")
fmt.Println(string(bs))
}
}
```
### Adding Built-in Functions to the OPA Runtime
```golang
package main
import (
"fmt"
"net/http"
"os"
"github.com/open-policy-agent/opa/ast"
"github.com/open-policy-agent/opa/cmd"
"github.com/open-policy-agent/opa/rego"
"github.com/open-policy-agent/opa/types"
)
func main() {
rego.RegisterBuiltin2(
&rego.Function{
Name: "github.repo",
Decl: types.NewFunction(types.Args(types.S, types.S), types.A),
Memoize: true,
Nondeterministic: true,
},
func(bctx rego.BuiltinContext, a, b *ast.Term) (*ast.Term, error) {
var org, repo string
if err := ast.As(a.Value, &org); err != nil {
return nil, err
} else if err := ast.As(b.Value, &repo); err != nil {
return nil, err
}
req, err := http.NewRequest("GET", fmt.Sprintf("https://api.github.com/repos/%v/%v", org, repo), nil)
if err != nil {
return nil, err
}
resp, err := http.DefaultClient.Do(req.WithContext(bctx.Context))
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf(resp.Status)
}
v, err := ast.ValueFromReader(resp.Body)
if err != nil {
return nil, err
}
return ast.NewTerm(v), nil
},
)
if err := cmd.RootCommand.Execute(); err != nil {
fmt.Println(err)
os.Exit(1)
}
}
```
docs/new/docs/external-data/assets/data-bundle.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 24 KiB

docs/new/docs/external-data/assets/data-input.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 23 KiB

docs/new/docs/external-data/assets/data-jwt.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 66 KiB

docs/new/docs/external-data/assets/data-pull.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 20 KiB

docs/new/docs/external-data/assets/data-push.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 24 KiB

docs/new/docs/external-data/index.md
+209
View File
@@ -0,0 +1,209 @@
---
title: External Data
sidebar_position: 6
---
OPA was designed to let you make context-aware authorization and policy decisions by injecting external data that describes what is happening in the world and then writing policy using that data. OPA has a cache or replica of that data, just as OPA has a cache/replica of policy; OPA is not designed to be the source of truth for either.
This document describes options for replicating data into OPA. The content of the data does not matter, but the size, frequency of update, and consistency constraints all do impact which kind of data replication to employ. You should prefer earlier options in the list to later options, but in the end the right choice depends on your situation.
## Option 1: JWT Tokens
[JSON Web Tokens (JWTs)](https://tools.ietf.org/html/rfc7519) allow you to securely transmit JSON data between software systems and are usually produced during the authentication process. You can set up authentication so that when the user logs in you create a JWT with that user's attributes (or any other data as far as OPA is concerned). Then you hand that JWT to OPA and use OPA's specialized support for JWTs to extract the information you need to make a policy decision.
### Flow
The following diagram shows this process in more detail.
1. User logs in to an authentication system, e.g. LDAP/AD/etc.
1. The user is given a JWT token encoding group membership and other user attributes stored in LDAP/AD
1. The user provides that JWT token to an OPA-enabled software system for authentication
1. The OPA-enabled software system includes that token as part of the usual `input` to OPA.
1. OPA decodes the JWT token and uses the contents to make policy decisions.
![JWT flow](./assets/data-jwt.png)
### Updates
The JWT only gets refreshed when the user authenticates; how often that happens is up to the TTL included in the token. When user-attribute information changes, those changes will not be seen by OPA until the user authenticates and gets a new JWT.
### Size Limitations
JWTs have a limited size in practice, so if your organization has too many user attributes you may not be able to fit all the required information into a JWT.
### Security
- OPA includes primitives to verify the signature of JWT tokens.
- OPA let's you check the TTL.
- OPA has support for making HTTP requests during evaluation, which could be used to check if a JWT has been revoked. Though if you're connecting to a remote system on every policy decision anyway, you should think about whether connecting to the authentication system directly is more appropriate (see below).
## Option 2: Overload `input`
Often policies require external data that's not available to the authentication system, ruling out JWTs. The calling system can include external data as part of `input` (necessitating of course that the policy is written accordingly).
For example, suppose your policy says that only a file's owner may delete it. The authentication system does not track resource-ownership, but the system responsible for files certainly does.
The file-ownership system may be the one that is asking for an authorization decision from OPA. It already knows which file is being operated on and who the owner is, so it can hand OPA the file-owner as part of OPA's `input`. This can be dangerous in that it ties the integration of OPA to the policy, but often it's sufficient to have the file-ownership system hand over all the file's metadata.
### Flow
1. OPA-enabled software gathers relevant metadata (and caches it for subsequent requests)
1. OPA-enabled software sends `input` to OPA including the external data
1. Policy makes decisions based on external data included in `input`
![Input flow](./assets/data-input.png)
### Updates
External data gets updated as frequently as the OPA-enabled software updates it. Often some of that data is local to the OPA-enabled software, and sometimes it is remote. The remote data is usually cached for performance and hence is as updated as the caching strategy allows.
### Size Limitations
Size limitations are rarely a problem for OPA in this approach because it only sees the metadata for 1 request at a time. However, the cache of remote data that the OPA-enabled service creates will have a limit that the developer controls.
### Security
This approach is as secure as the connection between the OPA-enabled service and OPA itself, under the assumption that the OPA-enabled service gathers the appropriate metadata securely. That is, using external data with this approach is as secure as using OPA in the first place.
### Recommended usage: Local, Dynamic data
This approach is valuable when the data changes fairly frequently and/or when the cost of making decisions using stale data is high. It works especially well when the external data is local to the system asking for authorization decisions. It can work in the case of remote data as well, but there is more coupling of the system to OPA because the system is hardcoded to fetch the data needed by the policy (and only that data).
## Option 3: Bundle API
When external data changes infrequently and can reasonably be stored in memory all at once, you can replicate that data in bulk via OPA's bundle feature. The bundle feature periodically downloads policy bundles from a centralized server, which can include data as well as policy. Every time OPA gets updated policies, it gets updated data too. You must implement the bundle server and integrate your external data into the bundle server--OPA does NOT help with that--but once it is done, OPA will happily pull the data (and policies) out of your bundle server.
### Flow
Three things happen independently with this kind of data integration.
- A. OPA-enabled software system asks OPA for policy decisions
- B. OPA downloads new policy bundles including external data
- C. Bundle server replicates data from source of truth
![Bundle flow](./assets/data-bundle.png)
### Updates
The lag between a data update and OPA having the update is the sum of the lag for an update between data replication and the central bundle server and the lag for an update between the central bundle server and OPA. So if data replication happens every 5 minutes, and OPA pulls a new bundle every 2 minutes, then the total maximum lag is 7 minutes.
### Size limitations
OPA stores the entire datasource at once in memory. Obviously this can be a problem with large external data sets. Because the centralized server handles both policy and data it can prune data to just that which is needed for the policies.
<!-- **Security**
* Don't expose OPA's API except through localhost
* Assuming LDAP/AD is the only context needed, use OPA's authentication/authorization to disable all of OPA's APIs except those needed by the OPA-enabled service. -->
### Recommended usage: Static, Medium-sized data
This approach is more flexible than the JWT and `input` cases above because you can include an entirely new data source at the bundle server without changing the authentication service or the OPA-enabled service. You are also guaranteed that the policy and its corresponding data always arrive at the same time, making the policy-data consistency perfect.
The drawback is that the consistency of the data with the source of truth is worse than the `input` case and could be better or worse than the consistency for the JWT case (because JWTs only get updated on login). One feature currently under design is a delta-based bundle protocol, which could improve the data consistency model significantly by lowering the cost of frequent updates. But as it stands this approach is ideal when the data is relatively static and the data fits into memory.
### Ecosystem Projects
<EcosystemEmbed feature="opa-bundles">
Loading policy and data via Bundles is an important part of the OPA API. A number of
ecosystem projects make use of this functionality to share code and keep data
up-to-date.
</EcosystemEmbed>
## Option 4: Push Data
Another way to replicate external data in its entirety into OPA is to use OPA's API for injecting arbitrary JSON data. You can build a replicator that pulls information out of the external data source and pushes that information in OPA through its API. This approach is similar in most respects to the bundle API, except it lets you optimize for update latency and network traffic.
### Flow
Three things happen independently with this kind of data replication.
- A. OPA-enabled software system asks OPA for policy decisions
- B. Data replicator pushes data into OPA
- C. Data replicator replicates data from source of truth
Depending on the replication scheme, B and C could be tied together so that every update the data replicator gets from the source of truth it pushes into OPA, but in general those could be decoupled depending on the desired network load, the changes in the data, and so on.
![Push flow](./assets/data-push.png)
### Updates
The total lag between the external data source being updated and OPA being updated is the sum of the lag for an update between the data source and the synchronizer plus the lag for an update between the synchronizer and OPA.
### Size limitations
The entirety of the external data source is stored in memory, which can obviously be a problem with large external data sources. But unlike the bundle API, this approach does allow updates to data.
<!--
**Security**
* Use mutual TLS to ensure only the synchronizer can use OPA's API to change the LDAP/AD data.
* Ensure OPA's policy rejects requests without sufficient data so that an OPA restart that wipes out memory does not leave the OPA-enabled service vulnerable. -->
### Recommended usage: Dynamic, Medium-sized data
This approach is very similar to the bundle approach except it updates the data stored in OPA with deltas instead of an entire snapshot at a time. Because the data is updated as deltas, this approach is well-suited for data that changes frequently. It assumes the data can fit entirely in memory and so is well-suited to small and medium-sized data sets.
### Ecosystem Projects
<EcosystemEmbed feature="external-data-realtime-push">
Some OPA Ecosystem projects support pushing data into OPA.
</EcosystemEmbed>
## Option 5: Pull Data during Evaluation
OPA includes functionality for reaching out to external servers during evaluation. This functionality handles those cases where there is too much data to synchronize into OPA, JWTs are ineffective, or policy requires information that must be as up to date as possible.
That functionality is implemented using built-in functions such as [`http.send`](https://www.openpolicyagent.org/docs/latest/policy-reference/#http). Check the docs for the latest instructions.
### Current limitations
- Credentials needed for the external service can either be hardcoded into policy or pulled from the environment.
- The built-in functions do not implement any retry logic.
### Flow
The key difference here is that every decision requires contacting the external data source. If that service or the network connection is slow or unavailable, OPA may not be able to return a decision.
1. OPA-enabled service asks OPA for a decision
1. During evaluation OPA asks the external data source for additional information
![Pull flow](./assets/data-pull.png)
### Updates
External data is perfectly fresh. There is no lag between an update to the external data and when OPA sees that update.
### Size limitations
Only the data actually needed by the policy is pulled from the external data source. There is no need for a replicator to figure out what data the policy will need before execution.
### Performance and Availability
Latency and availability of decision-making are dependent on the network. This approach may still be superior to running OPA on a remote server entirely because a local OPA can make some decisions without going over the network--those decisions that do not require information from the remote data server.
### Recommended usage: Highly Dynamic or Large-sized data
If the data is too large to fit into memory, or it changes too frequently to cache it inside of OPA, the only real option is to fetch the data on demand. The `input` approach fetches data on demand as well, but puts the burden on the OPA-enabled service to fetch the necessary data (and to know what data is necessary).
The downside to pulling data on demand is reduced performance and availability because of the network, which can be mitigated via caching. In the `input` case, caching is under the control of the OPA-enabled service and can therefore be tailored to fit the properties of the data. In the `http.send` case, caching is largely under the control of the remote service that sets HTTP response headers to indicate how long the response can be cached for. It is crucial in this approach for the OPA-enabled service to handle the case when OPA returns no decision.
### Ecosystem Projects
<EcosystemEmbed feature="external-data-realtime">
Loading data at evaluation time has been an area of focus for some projects in the OPA community.\
</EcosystemEmbed>
## Summary
| Approach | Perf/Avail | Limitations | Recommended Data |
| --------------- | -------------------- | ------------------------------------------------------- | ---------------- |
| JWT | High | Updates only when user logs back in | User attributes |
| Input | High | Coupling between service and OPA | Local, dynamic |
| Bundle | High | Updates to policy/data at the same time. Size an issue. | Static, medium |
| Push | High | Control data refresh rate. Size an issue. | Dynamic, medium |
| Evaluation Pull | Dependent on network | Perfectly up to date. No size limit. | Dynamic or large |
## Ecosystem Projects
<EcosystemEmbed feature="external-data">
Here are some projects that integrate with OPA to provide external data.
</EcosystemEmbed>
docs/new/docs/faq.md
+447
View File
@@ -0,0 +1,447 @@
---
title: Frequently Asked Questions
---
## How do I make user attributes stored in LDAP/AD available to OPA for making decisions?
[This best-practice guide](./external-data) explains three options: JSON Web Tokens, synchronization with LDAP/AD, and calling into LDAP/AD during policy evaluation.
## How does OPA do conflict resolution? {#conflict-resolution}
In Rego (OPA's policy language), you can write statements that both allow and
deny a request, such as
```live:conflict_resolution:module:read_only
package foo
allow { input.name == "alice" }
deny { input.name == "alice" }
```
Neither `allow` nor `deny` are keywords in Rego so if you want to treat them
as contradictory, you control which one takes precedence explicitly. When you ask for
a policy decision from OPA, you specify both the policy name (`foo`) and the
virtual document that names the decision within foo. Typically in this scenario,
you create a virtual document called `authz` and define it so that `allow`
overrides `deny` or vice versa. Then when asking for a policy decision, you
ask for `foo/authz`.
```live:conflict_resolution_deny_by_default:module:read_only
# deny everything by default
default authz := false
# deny overrides allow
authz {
allow
not deny
}
```
If instead you want to resolve conflicts using a first-match strategy (where
the first statement applicable makes the decision), see the FAQ entry on
[statement order](#statement-order).
## Does Statement Order Matter? {#statement-order}
The order in which statements occur does not matter in Rego. Reorder any two statements
and the policy means exactly the same thing. For example, the following two statements
mean the same thing whichever order you write them in.
```live:unordered:module:openable
package unordered
ratelimit := 4 if input.name == "alice"
ratelimit := 5 if input.name == "bob"
```
```live:unordered:input
{
"name": "bob"
}
```
```live:unordered:query:hidden
ratelimit
```
```live:unordered:output
```
Sometimes, though, you want the statement order to matter. For example, you might put more specific statements before more general statements so that the more specific statements take precedence (e.g. for [conflict resolution](#conflict-resolution)). Rego lets you do that using the `else` keyword. For example, if you want to make the first statement above take precedence, you would write the following Rego.
```live:ordered:module:openable
package ordered
ratelimit := 4 if {
input.owner == "bob"
} else := 5 if {
input.name == "alice"
}
```
```live:ordered:input
{
"name": "alice",
"owner": "bob"
}
```
```live:ordered:query:hidden
ratelimit
```
```live:ordered:output
```
## Which Equality Operator Should I Use?
Rego supports three kinds of equality: assignment (`:=`), comparison (`==`), and unification `=`. We recommend using assignment (`:=`) and comparison (`==`) whenever possible for policies that are easier to read and write.
```live:equality:query:read_only
# Assignment: declare local variable x and give it value 7
# If x appears before this statement in the rule, compiler throws error.
x := 7
y := {"a", "b", "c"}
# Comparison: check if two values are the same.
# Do not assign variables--variables must be "safe".
x == 7
x == y
y == [1, 2, [3]]
# Unification: assign variables to values that make the
# equality true
x = 7 # causes x to be assigned 7
[x, 2] = [3, y] # x is assigned 3 and y is assigned 2
```
## Collaboration Using Import
OPA lets multiple teams contribute independent policies that you can then combine to make an overall decision. Each team writes their policy in a separate `package`, then you write one more policy that imports all the teams policies and makes a decision.
For example, suppose there is a network team, a storage team, and a compute team. Suppose they each write their own policy:
```live:collab_compute:module:read_only
package compute
allow { ... }
```
```live:collab_network:module:read_only
package network
allow { ... }
```
```live:collab_storage:module:read_only
package storage
allow { ... }
```
Now the cloud team, who is in charge of the overall decision, writes another policy that combines the decisions for each of the team policies. In the example below, all 3 teams must allow for the overall decision to be allowed.
```live:collab_main:module:read_only
package main
import data.compute
import data.storage
import data.network
# allow if all 3 teams allow
allow {
compute.allow
storage.allow
network.allow
}
```
The cloud team could have a more sophisticated scheme for combining policies, e.g. using just the compute policy for compute-only resources or requiring the compute policy to allow the compute-relevant portions of resource. Remember that `allow` is not special--it is just another boolean that the policy author can use to make decisions.
## Functions Versus Rules
Rego lets you factor out common logic in 2 different and complementary ways.
One is the _function_, which is conceptually identical to functions from most programming languages. It takes any input and returns any output. Importantly, a function can take infinitely many inputs, e.g. any string.
```live:functions:module:openable
package functions
trim_and_split(s) := result if {
t := trim(s, " ")
result := split(t, ".")
}
```
```live:functions:query
trim_and_split(" hello.world ")
```
```live:functions:output
```
The other way to factor out common logic is with a _rule_. Rules differ in that (i) they support automatic iteration and (ii) they are only defined for finitely many inputs. (Those obviously go hand-in-hand.) For example, you could define a rule that maps an application to the hostnames that app is running on:
```live:rules:module:openable
package rules
app_to_hostnames[app_name] := hostnames if {
app := apps[_]
app_name := app.name
hostnames := [hostname | name := app.servers[_]
s := sites[_].servers[_]
s.name == name
hostname := s.hostname]
}
apps := [
{
"name": "web",
"servers": ["s1", "s2"],
},
{
"name": "mysql",
"servers": ["s3"],
},
{
"name": "mongodb",
"servers": ["s4"],
},
]
sites := [
{
"servers": [
{
"name": "s1",
"hostname": "hydrogen",
},
{
"name": "s3",
"hostname": "helium",
},
{
"name": "s4",
"hostname": "nitrogen",
},
],
},
{
"servers": [
{
"name": "s2",
"hostname": "carbon",
},
],
},
]
```
And then we can iterate over all the key/value pairs of that app-to-hostname mapping (just like we could iterate over all key/value pairs of a hardcoded JSON object). You can also iterate over just the keys or just the values or you can look up the value for a key or lookup all the keys for a single value.
```live:rules/all_pairs:query
# iterate over all key/value pairs
app_to_hostnames[app]
```
```live:rules/all_pairs:output
```
```live:rules/all_values:query
# iterate over all values
app_to_hostnames[_]
```
```live:rules/all_values:output
```
```live:rules/all_keys:query
# iterate over all keys
app_to_hostnames[x] = _
```
```live:rules/all_keys:output
```
```live:rules/lookup_key:query
# lookup the value for key "web"
app_to_hostnames["web"]
```
```live:rules/lookup_key:output
```
```live:rules/lookup_value:query
# lookup keys where value includes "carbon"
app_to_hostnames[k][_] == "carbon"
```
```live:rules/lookup_value:output
```
Obviously with the `trim_and_split` function we cannot ask for all the inputs/outputs since there are infinitely many. We can't provide 1 input and ask for all the other inputs that make the function return true, again, because there could be infinitely many. The only thing we can do with a function is provide it all the inputs and ask for the output.
Functions allow you to factor out common logic that has infinitely-many input/output pairs; rules allow you to factor out common logic with finitely many input/outputs and allow you to iterate over them in the same way as native JSON objects.
To achieve automatic iteration, there is an additional syntactic requirement on a rule that is NOT present for a function: `safety`. See the FAQ entry on safety for technical details. Every rule must be `safe`, which guarantees that OPA can figure out a finite list of possible values for every variable in the body and head of a rule.
We recommend using rules where possible and using functions when rules do not work.
## Safety
The compiler will sometimes throw errors that say a rule is not `safe`. The goal of safety is to ensure that every rule has finitely many inputs/outputs. Safety ensures that every variable has finitely many possible values, so that OPA can iterate over them to find those values that make the rule true. Technically:
```
Safety: every variable appearing in the head or in a builtin or inside a negation must appear in a non-negated, non-builtin expression in the body of the rule.
```
Examples:
```live:safety:module:read_only
# Unsafe: x in head does not appear in body.
# There are infinitely many values that make p true
p[x] { some y; q[y]; r[y] }
# Safe. q and r are both rules
# Both q and r are finite; therefore p is also finite.
p[x] := y { some x, y; q[x]; r[y] }
# Unsafe: y appears inside a builtin (+) but not in the body.
# y has infinitely many possible values; so too does x.
p[x] { some y; x := y + 7 }
# Safe: the only values for y are those in q.
# Since q is a rule and finite so is p finite.
p[x] { some y; x := y + 7; q[y]}
# Unsafe: x appears inside a negation
# If q is finite, all the x's not in q are infinite.
p[x] { some x; not q[x] }
# Safe: x appears inside of r so p is no larger than r
# Since r is finite, so too is p
p[x] { some x; not q[x]; r[x] }
```
Safety has one implication about negation: you don't iterate over values NOT in a rule like `q`. Instead, you iterate over values in another rule like `r` and then use negation to CHECK whether if that value is NOT in `q`.
Embedded terms like `not p[q[_]]` sometimes produce difficult to decipher error messages. We recommend pulling the embedded terms out into the rule--the meaning is the same and often creates easier to read error messages:
```live:safety/nested:query:read_only
x := q[_]
not p[x]
```
## JSON Web Tokens (JWTs)
[JSON Web Tokens (JWTs)](https://jwt.io/) are an industry standard for exchanging information between services. Often they are used to represent information about the users logged into a system. OPA has special-purpose code for dealing with JWTs.
All JWTs with OPA come in as strings. That string is a JSON Web Token encoded with JWS Compact Serialization. JWE and JWS JSON Serialization are not supported.
You can verify tokens are properly signed.
```live:jwt_verify:query:read_only
# RS256 signature
io.jwt.verify_rs256(string, certificate)
# PS256 signature
io.jwt.verify_ps256(string, certificate)
# ES256 signature
io.jwt.verify_es256(string, certificate)
# HS256 signature
io.jwt.verify_hs256(string, certificate)
```
You can decode JWTs and use the contents of the JWT to make policy decisions.
```live:jwt_decode:module:hidden
package jwt_decode
```
```live:jwt_decode:input
{
"token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoiYWxpY2UiLCJhenAiOiJhbGljZSIsInN1Ym9yZGluYXRlcyI6W10sImhyIjpmYWxzZX0.rz3jTY033z-NrKfwrK89_dcLF7TN4gwCMj-fVBDyLoM"
}
```
```live:jwt_decode:query
io.jwt.decode(input.token)
```
```live:jwt_decode:output
```
> If nested signing was used, the header, payload and signature will represent the most deeply nested token.
You can decode **and** verify using `io.jwt.decode_verify`.
```live:jwt_decode/verify:query
io.jwt.decode_verify(input.token, {
"secret": "secret",
"alg": "hs256",
})
```
```live:jwt_decode/verify:output
```
See the [Policy Reference](./policy-reference#tokens) for additional verification constraints.
To get certificates into the policy, you can either hardcode them or provide them as environmental variables to OPA and then use the `opa.runtime` builtin to retrieve those variables.
```live:runtime:query:read_only
# all runtime information
runtime := opa.runtime()
# environment variables provided when OPA started
runtime.env
# the env variable PROD_CERTIFICATE
runtime.env.PROD_CERTIFICATE
```
## How do I Write Policies Securely?
Depending on the use case and the integration with OPA that you are using, the style of policy you choose can impact your overall security posture. Below we show three styles of authoring policy and compare them.
**Default allow**. This style of policy allows every request by default. The rules you write dictate which requests should be rejected.
```rego
package example
# entry point is 'deny'
default deny := false
deny if { ... }
deny if { ... }
```
If you assume all of the rules you write are correct, then you know that every rejection the policy produces should truly be rejected. However, there could be requests that are allowed that you may not truly want allowed, but you simply neglected to write the rule for. For operations, this is often a useful style of policy authoring because it allows you to incrementally tighten the controls for a system from wherever that system starts. For security, this style is less appropriate because it allows unknown bad actions to occur.
**Default deny**. This style of policy rejects every request by default. The rules you write dictate which requests should be allowed.
```rego
package example
# entry point is 'allow'
default allow := false
allow if { ... }
allow if { ... }
```
If you assume your rules are correct, the only requests that are accepted are known to be safe. Any statements you leave out reject requests that in actuality are safe but which you did not know were safe. For operations, these policies are less suitable for incrementally improving the policy posture of a system because the initial policy must explicitly allow all of the behaviors that are necessary for the system to operate correctly. For security, these policies ensure that any request that is allowed is known to be safe (because there is a rule saying it is safe).
**Default allow with deny override**. This style of policy rejects every request by default. You write rules that dictate which requests should be allowed, and optionally you write other rules that dictate which of those allowed requests should be rejected.
```rego
package example
# entry point is 'authz'
default authz := false
authz if {
allow
not deny
}
allow if { ... }
deny if { ... }
```
This hybrid approach to policy authoring combines the two previous styles. These policies allow relatively coarse grained parts of the request space and then carve out of each part what should actually be denied. Any deny statements that you forget lead to security problems; any allow statements you forget lead to operational problems. But since this approach allows you to implement either of the other two, it is a common pattern across use cases.
**Non-boolean policies**. The examples above focus on policies with boolean decisions. Policies that make non-boolean decisions typically have similar tradeoffs. Are you enumerating the conditions under which requests are permitted (e.g. the list of clusters to which an app SHOULD be deployed) or are you enumerating the conditions under which requests are prohibited (e.g. the list of clusters to which an app SHOULD NOT be deployed). While the details differ, the concepts are often similar.
docs/new/docs/graphql-api-authorization.md
+520
View File
@@ -0,0 +1,520 @@
---
title: "GraphQL APIs"
---
GraphQL APIs have become a popular way to query a variety of datastores and microservices, and any application or service providing a GraphQL API generally needs to control which users can run queries, mutations, and so on.
OPA makes it easy to write fine-grained, context-aware policies to implement GraphQL query authorization.
## Goals
In this tutorial, you'll use a simple GraphQL server that accepts any GraphQL request that you issue, and echoes the OPA decision back as text.
OPA will fetch policy bundles from a simple bundle server.
OPA, the bundle server, and the GraphQL server will all be run as containers.
For this tutorial, our desired policy is:
- People can see their own salaries (`query user($id: <user>) { salary }` is permitted for `<user>`)
- A manager can see their direct reports' salaries (`query user($id: <user>) { salary }` is permitted for `<user>`'s manager)
:::danger GraphQL API Authorization with OPA is currently experimental and the following tutorial is intended for demonstration purposes only. :::
## Prerequisites
This tutorial requires [Docker Compose](https://docs.docker.com/compose/install/) to run a demo web server along with OPA.
## Steps
### 1. Define our GraphQL schema.
Most modern GraphQL frameworks encourage starting with a schema, so we'll follow suit, and begin by defining the schema for this example.
**schema.gql**:
```graphql
type Employee {
id: String!
salary: Int!
}
schema {
query: Query
}
type Query {
employeeByID(id: String!): Employee
}
```
Every GraphQL service has a `query` type, and may or may not have a `mutation` type.
These types are special because they define the entry points of _every_ GraphQL query for the API covered by that schema.
For our example above, we've defined exactly one query entry point, the parameterized query `employeeByID(id: String!)`.
### 2. Create a policy bundle.
GraphQL APIs allow surprising flexibility in how queries can be constructed, which makes writing policies for them a bit more challenging than for a REST API, which usually has a more fixed structure.
To protect a particular endpoint or field, we need to see if they are referenced in the incoming GraphQL query.
By using `graphql.parse`, we can extract an [abstract syntax tree][wikipedia-ast] (AST) from the incoming query, and then walk down the tree to its leaves to see if our endpoint is the target of the query.
We can then use separate rules to enforce conditions around the `salary` field, and who is allowed to access it.
The policy below does all of the above in parts:
- Obtains the query AST (and validates it against our schema with `graphql.parse`).
- Recursive traversal with `walk()` to obtain chunks of the AST with queries of interest present.
- Selection of nodes of interest by name and structure.
- Salary field selected.
- Every query of interest found has to pass one of the `allowed_query` rules, or the entire query is rejected.
[wikipedia-ast]: https://en.wikipedia.org/wiki/Abstract_syntax_tree
- Constant/variable cases for both employees and their managers.
**example.rego**:
```live:example:module:openable
package graphqlapi.authz
subordinates := {"alice": [], "charlie": [], "bob": ["alice"], "betty": ["charlie"]}
query_ast := graphql.parse(input.query, input.schema)[0] # If validation fails, the rules depending on this will be undefined.
default allow := false
allow if {
employeeByIDQueries != {}
every query in employeeByIDQueries {
allowed_query(query)
}
}
# Allow users to see the salaries of their subordinates. (variable case)
allowed_query(q) if {
selected_salary(q)
varname := variable_arg(q, "id")
input.variables[varname] in subordinates[input.user] # Do value lookup from the 'variables' object.
}
# Allow users to see the salaries of their subordinates. (constant value case)
allowed_query(q) if {
selected_salary(q)
username := constant_string_arg(q, "id")
username in subordinates[input.user]
}
# Helper rules.
# Allow users to get their own salaries. (variable case)
allowed_query(q) if {
selected_salary(q)
varname := variable_arg(q, "id")
input.user == input.variables[varname] # Do value lookup from the 'variables' object.
}
# Allow users to get their own salaries. (constant value case)
allowed_query(q) if {
selected_salary(q)
username := constant_string_arg(q, "id")
input.user == username
}
# Helper functions.
# Build up an object with all queries of interest as values.
employeeByIDQueries contains value if {
some value
walk(query_ast, [_, value])
value.Name == "employeeByID"
count(value.SelectionSet) > 0 # Ensure we latch onto an employeeByID query.
}
# Extract the string value of a constant value argument.
constant_string_arg(value, argname) := arg.Value.Raw if {
some arg in value.Arguments
arg.Name == argname
arg.Value.Kind == 3
}
# Extract the variable name for a variable argument.
variable_arg(value, argname) := arg.Value.Raw if {
some arg in value.Arguments
arg.Name == argname
arg.Value.Kind == 0
}
# Ensure we're dealing with a selection set that includes the "salary" field.
selected_salary(value) := value.SelectionSet[_].Name == "salary"
```
Then, build a bundle.
```shell
mkdir bundles
opa build example.rego
mv bundle.tar.gz ./bundles
```
You should now see a policy bundle (`bundle.tar.gz`) in your working directory.
### 3. Bootstrap the tutorial environment using Docker Compose.
Next, create a `docker-compose.yml` file that runs OPA, a bundle server and the demo GraphQL server.
**docker-compose.yml**:
```yaml
services:
opa:
image: openpolicyagent/opa:{{< current_docker_version >}}
ports:
- "8181:8181"
command:
- "run"
- "--server"
- "--log-format=json-pretty"
- "--set=decision_logs.console=true"
- "--set=services.nginx.url=http://bundle_server"
- "--set=bundles.nginx.service=nginx"
- "--set=bundles.nginx.resource=bundle.tar.gz"
- "--set=bundles.nginx.polling.min_delay_seconds=10"
- "--set=bundles.nginx.polling.max_delay_seconds=30"
depends_on:
- bundle_server
api_server:
image: openpolicyagent/demo-graphql-api:0.1
ports:
- "6000:5000"
environment:
- OPA_ADDR=http://opa:8181
- POLICY_PATH=/v1/data/graphqlapi/authz
depends_on:
- opa
bundle_server:
image: nginx:1.20.0-alpine
ports:
- 8888:80
volumes:
- ./bundles/:/usr/share/nginx/html/
```
Then run `docker-compose` to pull and run the containers.
:::info
If running "Docker Desktop" (Mac or Windows) you may instead use the `docker compose` command.
:::
```shell
docker-compose -f docker-compose.yml up
```
Every time the demo GraphQL server receives an HTTP request, it asks OPA to decide whether an GraphQL query is authorized or not using a single RESTful API call.
An example codebase is [here][graphql-example-repo], but the crux of the (JavaScript, Apollo framework) code is shown below.
[graphql-example-repo]: https://github.com/StyraInc/graphql-apollo-example
```javascript
// we assume user is passed in as part of the request context.
var user = req.user;
// we feed in the query and schema strings, as well as the variables object.
var input = {
input: {
schema: schema, // GraphQL schema text.
query: query, // GraphQL query text.
user: user,
variables: variables, // GraphQL variable bindings.
},
};
await axios
// ask OPA for a policy decision.
// (in reality OPA URL would be constructed from environment)
.post("http://127.0.0.1:8181/v1/data/graphqlapi/authz", input)
.then(res => {
// GraphQL query allowed.
})
.catch(error => {
// GraphQL query denied.
});
```
### 4. Check that `alice` can see her own salary.
We'll define a quick shell function to make the following examples cleaner on the command line:
```shell
gql-query() {
curl --user "$1" -H "Content-Type: application/json" "$2" --data-ascii "$3"
}
```
The following command will succeed.
```shell
gql-query alice:password "localhost:6000/" '{"query":"query { employeeByID(id: \"alice\") { salary }}"}'
```
The GraphQL server queries OPA to authorize the request.
In the query, the server includes JSON data describing the incoming request.
```live:example:input
{
"schema": "type Employee {\n id: ...",
"query": "query { employeeByID(id: \"alice\") { salary }}",
"user": "alice",
"variables": {}
}
```
When the GraphQL server queries OPA it asks for a specific policy decision.
In this case, the integration is hardcoded to ask for `/v1/data/graphqlapi/authz`.
OPA translates this URL path into a query:
```live:example:query
data.graphqlapi.authz
```
The answer returned by OPA for the input above is:
```live:example:output
```
### 4. Check that `bob` can see `alice`'s salary (because `bob` is `alice`'s manager.)
```shell
gql-query bob:password "localhost:6000/" '{"query":"query { employeeByID(id: \"alice\") { salary }}"}'
```
### 5. Check that `bob` CANNOT see `charlie`'s salary.
`bob` is not `charlie`'s manager, so the following command will fail.
```shell
gql-query bob:password "localhost:6000/" '{"query":"query { employeeByID(id: \"charlie\") { salary }}"}'
```
### 6. Change the policy.
Suppose the organization now includes an HR department.
The organization wants members of HR to be able to see any salary.
Let's extend the policy to handle this.
**example-hr.rego**:
```live:hr_example:module:read_only,openable
package graphqlapi.authz
# Allow HR members to get anyone's salary.
allowed_query(q) if {
selected_salary(q)
input.user == hr[_]
}
# David is the only member of HR.
hr := ["david"]
```
Build a new bundle with the new policy included.
```shell
opa build example.rego example-hr.rego
mv bundle.tar.gz ./bundles
```
The updated bundle will automatically be served by the bundle server, but note that it might take up to the configured `max_delay_seconds` for the new bundle to be downloaded by OPA.
If you plan to make frequent policy changes you might want to adjust this value in `docker-compose.yml` accordingly.
For the sake of the tutorial we included `manager_of` and `hr` data directly inside the policies.
In real-world scenarios that information would be imported from external data sources.
### 7. Check that the new policy works.
Check that `david` can see anyone's salary.
```shell
gql-query david:password "localhost:6000/" '{"query":"query { employeeByID(id: \"alice\") { salary }}"}'
gql-query david:password "localhost:6000/" '{"query":"query { employeeByID(id: \"bob\") { salary }}"}'
gql-query david:password "localhost:6000/" '{"query":"query { employeeByID(id: \"charlie\") { salary }}"}'
gql-query david:password "localhost:6000/" '{"query":"query { employeeByID(id: \"david\") { salary }}"}'
```
### 8. (Optional) Use JSON Web Tokens to communicate policy data.
OPA supports the parsing of JSON Web Tokens via the builtin function `io.jwt.decode`.
To get a sense of one way the subordinate and HR data might be communicated in the real world, let's try a similar exercise utilizing the JWT utilities of OPA.
**example-jwt.rego**:
```live:jwt_example:module:hidden
package graphqlapi.authz
query_ast := graphql.parse(input.query, input.schema)[0] # If validation fails, the rules depending on this will be undefined.
# Helper rules.
# Allow users to see the salaries of their subordinates. (variable case)
allowed_query(q) if {
selected_salary(q)
varname := variable_arg(q, "id")
input.variables[varname] in token.payload.subordinates # Do value lookup from the 'variables' object.
}
# Allow users to see the salaries of their subordinates. (constant value case)
allowed_query(q) if {
selected_salary(q)
username := constant_string_arg(q, "id")
username in token.payload.subordinates
}
# Allow users to get their own salaries. (variable case)
allowed_query(q) if {
selected_salary(q)
varname := variable_arg(q, "id")
token.payload.user == input.variables[varname] # Do value lookup from the 'variables' object.
}
# Allow users to get their own salaries. (constant value case)
allowed_query(q) if {
selected_salary(q)
username := constant_string_arg(q, "id")
token.payload.user == username
}
# Allow HR members to get anyone's salary.
allowed_query(q) if {
selected_salary(q)
token.payload.hr == true
}
# Helper functions.
# Build up a set with all queries of interest as values.
employeeByIDQueries contains value if {
some value
walk(query_ast, [_, value])
value.Name == "employeeByID"
count(value.SelectionSet) > 0 # Ensure we latch onto an employeeByID query.
}
# Extract the string value of a constant value argument.
constant_string_arg(value, argname) := arg.Value.Raw if {
some arg in value.Arguments
arg.Name == argname
arg.Value.Kind == 3
}
# Extract the variable name for a variable argument.
variable_arg(value, argname) := arg.Value.Raw if {
some arg in value.Arguments
arg.Name == argname
arg.Value.Kind == 0
}
# Ensure we're dealing with a selection set that includes the "salary" field.
selected_salary(value) := value.SelectionSet[_].Name == "salary"
```
```live:jwt_example/new_rules:module:openable
default allow := false
allow if {
employeeByIDQueries != {}
user_owns_token # Ensure we validate the JWT token.
every query in employeeByIDQueries {
allowed_query(query)
}
}
# Helper rules ... (Same as example.rego)
# Helper functions ... (Same as example.rego)
# -------------------------------------------------------------
# JWT Token Support
# Ensure that the token was issued to the user supplying it.
user_owns_token if input.user == token.payload.azp
# Helper to get the token payload.
token := {"payload": payload} if {
[_, payload, _] := io.jwt.decode(input.token)
}
```
```live:jwt_example:input:hidden
{
"token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoiYWxpY2UiLCJhenAiOiJhbGljZSIsInN1Ym9yZGluYXRlcyI6W10sImhyIjpmYWxzZX0.rz3jTY033z-NrKfwrK89_dcLF7TN4gwCMj-fVBDyLoM",
"schema": "type Employee {\n id: ...",
"query": "query { employeeByID(id: \"alice\") { salary }}",
"user": "alice",
"variables": {}
}
```
Build a new bundle for the new policy.
```shell
opa build example-jwt.rego example-hr.rego
mv bundle.tar.gz ./bundles
```
For convenience, we'll want to store user tokens in environment variables (they're really long).
```shell
export ALICE_TOKEN="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoiYWxpY2UiLCJhenAiOiJhbGljZSIsInN1Ym9yZGluYXRlcyI6W10sImhyIjpmYWxzZX0.rz3jTY033z-NrKfwrK89_dcLF7TN4gwCMj-fVBDyLoM"
export BOB_TOKEN="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoiYm9iIiwiYXpwIjoiYm9iIiwic3Vib3JkaW5hdGVzIjpbImFsaWNlIl0sImhyIjpmYWxzZX0.n_lXN4H8UXGA_fXTbgWRx8b40GXpAGQHWluiYVI9qf0"
export CHARLIE_TOKEN="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoiY2hhcmxpZSIsImF6cCI6ImNoYXJsaWUiLCJzdWJvcmRpbmF0ZXMiOltdLCJociI6ZmFsc2V9.EZd_y_RHUnrCRMuauY7y5a1yiwdUHKRjm9xhVtjNALo"
export BETTY_TOKEN="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoiYmV0dHkiLCJhenAiOiJiZXR0eSIsInN1Ym9yZGluYXRlcyI6WyJjaGFybGllIl0sImhyIjpmYWxzZX0.TGCS6pTzjrs3nmALSOS7yiLO9Bh9fxzDXEDiq1LIYtE"
export DAVID_TOKEN="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoiZGF2aWQiLCJhenAiOiJkYXZpZCIsInN1Ym9yZGluYXRlcyI6W10sImhyIjp0cnVlfQ.Q6EiWzU1wx1g6sdWQ1r4bxT1JgSHUpVXpINMqMaUDMU"
```
These tokens encode the same information as the policies we did before (`bob` is `alice`'s manager, `betty` is `charlie`'s, `david` is the only HR member, etc).
If you want to inspect their contents, start up the OPA REPL and execute `io.jwt.decode(<token here>, [header, payload, signature])` or open the example above in the Playground.
Let's try a few queries (note: you may need to escape the `?` characters in the queries for your shell):
Check that `charlie` can't see `bob`'s salary.
```shell
gql-query charlie:password "localhost:5000/?token=$CHARLIE_TOKEN" '{"query":"query { employeeByID(id: \"bob\") { salary }}"}'
```
Check that `charlie` can't pretend to be `bob` to see `alice`'s salary.
```shell
gql-query charlie:password "localhost:5000/?token=$BOB_TOKEN" '{"query":"query { employeeByID(id: \"alice\") { salary }}"}'
```
Check that `david` can see `betty`'s salary.
```shell
gql-query david:password "localhost:5000/?token=$DAVID_TOKEN" '{"query":"query { employeeByID(id: \"betty\") { salary }}"}'
```
Check that `bob` can see `alice`'s salary.
```shell
gql-query bob:password "localhost:5000/?token=$BOB_TOKEN" '{"query":"query { employeeByID(id: \"alice\") { salary }}"}'
```
Check that `alice` can see her own salary.
```shell
gql-query alice:password "localhost:5000/?token=$ALICE_TOKEN" '{"query":"query { employeeByID(id: \"alice\") { salary }}"}'
```
## Wrap Up
Congratulations for finishing the tutorial!
You learned a number of things about API authorization with OPA:
- OPA gives you fine-grained policy control over GraphQL APIs once you set up the server to ask OPA for authorization.
- You write allow/deny policies to control which endpoints and fields can be accessed by whom.
- You can import external data into OPA and write policies that depend on that data.
- You can use OPA data structures to define abstractions over your data.
- You can use a remote bundle server for distributing policy and data.
The code for this tutorial can be found in the
[StyraInc/graphql-apollo-example][graphql-example-repo]
repository.
docs/new/docs/http-api-authorization.md
+375
View File
@@ -0,0 +1,375 @@
---
title: "HTTP APIs"
---
Anything that exposes an HTTP API (whether an individual microservice or an application as a whole) needs to control who can run those APIs and when. OPA makes it easy to write fine-grained, context-aware policies to implement API authorization.
## Goals
In this tutorial, you'll use a simple HTTP web server that accepts any HTTP GET
request that you issue and echoes the OPA decision back as text. OPA will fetch
policy bundles from a simple bundle server. Both OPA, the bundle server and the
web server will be run as containers.
For this tutorial, our desired policy is:
- People can see their own salaries (`GET /finance/salary/{user}` is permitted for `{user}`)
- A manager can see their direct reports' salaries (`GET /finance/salary/{user}` is permitted for `{user}`'s manager)
## Prerequisites
This tutorial requires [Docker Compose](https://docs.docker.com/compose/install/) to run a demo web server along with OPA.
## Steps
### 1. Create a policy bundle.
Create a policy that allows users to request their own salary as well as the salary of their direct subordinates.
**First** create a directory named `bundles` and cd into it.
```sh
mkdir bundles
cd bundles
```
**example.rego**:
```live:example:module:openable
package httpapi.authz
# bob is alice's manager, and betty is charlie's.
subordinates := {"alice": [], "charlie": [], "bob": ["alice"], "betty": ["charlie"]}
default allow := false
# Allow users to get their own salaries.
allow if {
input.method == "GET"
input.path == ["finance", "salary", input.user]
}
# Allow managers to get their subordinates' salaries.
allow if {
some username
input.method == "GET"
input.path = ["finance", "salary", username]
subordinates[input.user][_] == username
}
```
**Then**, build a bundle.
```shell
opa build example.rego
cd ..
```
You should now see a policy bundle (`bundle.tar.gz`) in your working directory (`./bundles/bundle.tar.gz`).
### 2. Bootstrap the tutorial environment using Docker Compose.
Next, create a `docker-compose.yml` file that runs OPA, a bundle server and the demo web server.
**docker-compose.yml**:
```yaml
version: "2"
services:
opa:
image: openpolicyagent/opa:{{< current_docker_version >}}
ports:
- 8181:8181
# WARNING: OPA is NOT running with an authorization policy configured. This
# means that clients can read and write policies in OPA. If you are
# deploying OPA in an insecure environment, be sure to configure
# authentication and authorization on the daemon. See the Security page for
# details: https://www.openpolicyagent.org/docs/security.html.
command:
- "run"
- "--server"
- "--addr=0.0.0.0:8181"
- "--log-format=json-pretty"
- "--set=decision_logs.console=true"
- "--set=services.nginx.url=http://bundle_server"
- "--set=bundles.nginx.service=nginx"
- "--set=bundles.nginx.resource=bundles/bundle.tar.gz"
depends_on:
- bundle_server
api_server:
image: openpolicyagent/demo-restful-api:0.3
ports:
- 5000:5000
environment:
- OPA_ADDR=http://opa:8181
- POLICY_PATH=/v1/data/httpapi/authz
depends_on:
- opa
bundle_server:
image: nginx:1.20.0-alpine
ports:
- 8888:80
volumes:
- ./bundles:/usr/share/nginx/html/bundles
```
Then run `docker-compose` to pull and run the containers.
**NOTE:** if running "Docker Desktop" (Mac or Windows) you may instead use the `docker compose` command.
```shell
docker-compose -f docker-compose.yml up
```
:::info
This example shows conceptually a 'manual' REST API integration with OPA.
You might find it easier to build your OPA integration using one of the
[language SDKs](/ecosystem/#languages) than working with the REST API directly.
:::
Every time the demo web server receives an HTTP request, it
asks OPA to decide whether an HTTP API is authorized or not
using a single RESTful API call. An example code is [here](https://github.com/open-policy-agent/contrib/blob/main/api_authz/docker/echo_server.py),
but the crux of the (Python) code is shown below.
```python
# Grab basic information. We assume user is passed on a form.
http_api_user = request.form['user']
# Get the path as a list (removing leading and trailing /)
# Example: "/finance/salary/" will become ["finance", "salary"]
http_api_path_list = request.path.strip("/").split("/")
input_dict = { # create input to hand to OPA
"input": {
"user": http_api_user,
"path": http_api_path_list, # Ex: ["finance", "salary", "alice"]
"method": request.method # HTTP verb, e.g. GET, POST, PUT, ...
}
}
# ask OPA for a policy decision
# (in reality OPA URL would be constructed from environment)
rsp = requests.post("http://127.0.0.1:8181/v1/data/httpapi/authz", json=input_dict)
if rsp.json()["allow"]:
# HTTP API allowed
else:
# HTTP API denied
```
### 3. Check that `alice` can see her own salary.
The following command will succeed.
```shell
curl --user alice:password localhost:5000/finance/salary/alice
```
The webserver queries OPA to authorize the request. In the query, the webserver
includes JSON data describing the incoming request.
```live:example:input
{
"method": "GET",
"path": ["finance", "salary", "alice"],
"user": "alice"
}
```
When the webserver queries OPA it asks for a specific policy decision. In this
case, the integration is hardcoded to ask for `/v1/data/httpapi/authz`. OPA
translates this URL path into a query:
```live:example:query
data.httpapi.authz
```
The answer returned by OPA for the input above is:
```live:example:output
```
### 4. Check that `bob` can see `alice`'s salary (because `bob` is `alice`'s manager.)
```shell
curl --user bob:password localhost:5000/finance/salary/alice
```
### 5. Check that `bob` CANNOT see `charlie`'s salary.
`bob` is not `charlie`'s manager, so the following command will fail.
```shell
curl --user bob:password localhost:5000/finance/salary/charlie
```
### 6. Change the policy.
Suppose the organization now includes an HR department. The organization wants
members of HR to be able to see any salary. Let's extend the policy to handle
this.
**example-hr.rego**:
```live:hr_example:module:read_only,openable
package httpapi.authz
# Allow HR members to get anyone's salary.
allow if {
input.method == "GET"
input.path = ["finance", "salary", _]
input.user == hr[_]
}
# David is the only member of HR.
hr := ["david"]
```
Build a new bundle with the new policy included.
```shell
opa build example.rego example-hr.rego
```
The updated bundle will automatically be served by the bundle server, but note that it might take up to the
configured `max_delay_seconds` for the new bundle to be downloaded by OPA. If you plan to make frequent policy
changes you might want to adjust this value in `docker-compose.yml` accordingly.
For the sake of the tutorial we included `manager_of` and `hr` data directly
inside the policies. In real-world scenarios that information would be imported
from external data sources.
### 7. Check that the new policy works.
Check that `david` can see anyone's salary.
```shell
curl --user david:password localhost:5000/finance/salary/alice
curl --user david:password localhost:5000/finance/salary/bob
curl --user david:password localhost:5000/finance/salary/charlie
curl --user david:password localhost:5000/finance/salary/david
```
### 8. (Optional) Use JSON Web Tokens to communicate policy data.
OPA supports the parsing of JSON Web Tokens via the builtin function `io.jwt.decode`.
To get a sense of one way the subordinate and HR data might be communicated in the
real world, let's try a similar exercise utilizing the JWT utilities of OPA.
**example-jwt.rego**:
```live:jwt_example:module:openable
package httpapi.authz
default allow := false
# Allow users to get their own salaries.
allow if {
some username
input.method == "GET"
input.path = ["finance", "salary", username]
token.payload.user == username
user_owns_token
}
# Allow managers to get their subordinate' salaries.
allow if {
some username
input.method == "GET"
input.path = ["finance", "salary", username]
token.payload.subordinates[_] == username
user_owns_token
}
# Allow HR members to get anyone's salary.
allow if {
input.method == "GET"
input.path = ["finance", "salary", _]
token.payload.hr == true
user_owns_token
}
# Ensure that the token was issued to the user supplying it.
user_owns_token if input.user == token.payload.azp
# Helper to get the token payload.
token := {"payload": payload} if {
[header, payload, signature] := io.jwt.decode(input.token)
}
```
```live:jwt_example:input:hidden
{
"token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoiYWxpY2UiLCJhenAiOiJhbGljZSIsInN1Ym9yZGluYXRlcyI6W10sImhyIjpmYWxzZX0.rz3jTY033z-NrKfwrK89_dcLF7TN4gwCMj-fVBDyLoM",
"method": "GET",
"path": ["finance", "salary", "alice"],
"user": "alice"
```
Build a new bundle for the new policy.
```shell
opa build example-jwt.rego
```
For convenience, we'll want to store user tokens in environment variables (they're really long).
```shell
export ALICE_TOKEN="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoiYWxpY2UiLCJhenAiOiJhbGljZSIsInN1Ym9yZGluYXRlcyI6W10sImhyIjpmYWxzZX0.rz3jTY033z-NrKfwrK89_dcLF7TN4gwCMj-fVBDyLoM"
export BOB_TOKEN="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoiYm9iIiwiYXpwIjoiYm9iIiwic3Vib3JkaW5hdGVzIjpbImFsaWNlIl0sImhyIjpmYWxzZX0.n_lXN4H8UXGA_fXTbgWRx8b40GXpAGQHWluiYVI9qf0"
export CHARLIE_TOKEN="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoiY2hhcmxpZSIsImF6cCI6ImNoYXJsaWUiLCJzdWJvcmRpbmF0ZXMiOltdLCJociI6ZmFsc2V9.EZd_y_RHUnrCRMuauY7y5a1yiwdUHKRjm9xhVtjNALo"
export BETTY_TOKEN="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoiYmV0dHkiLCJhenAiOiJiZXR0eSIsInN1Ym9yZGluYXRlcyI6WyJjaGFybGllIl0sImhyIjpmYWxzZX0.TGCS6pTzjrs3nmALSOS7yiLO9Bh9fxzDXEDiq1LIYtE"
export DAVID_TOKEN="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoiZGF2aWQiLCJhenAiOiJkYXZpZCIsInN1Ym9yZGluYXRlcyI6W10sImhyIjp0cnVlfQ.Q6EiWzU1wx1g6sdWQ1r4bxT1JgSHUpVXpINMqMaUDMU"
```
These tokens encode the same information as the policies we did before (`bob` is `alice`'s manager, `betty` is `charlie`'s, `david` is the only HR member, etc).
If you want to inspect their contents, start up the OPA REPL and execute `io.jwt.decode(<token here>, [header, payload, signature])` or open the example above in the Playground.
Let's try a few queries (note: you may need to escape the `?` characters in the queries for your shell):
Check that `charlie` can't see `bob`'s salary.
```shell
curl --user charlie:password "localhost:5000/finance/salary/bob?token=$CHARLIE_TOKEN"
```
Check that `charlie` can't pretend to be `bob` to see `alice`'s salary.
```shell
curl --user charlie:password "localhost:5000/finance/salary/alice?token=$BOB_TOKEN"
```
Check that `david` can see `betty`'s salary.
```shell
curl --user david:password "localhost:5000/finance/salary/betty?token=$DAVID_TOKEN"
```
Check that `bob` can see `alice`'s salary.
```shell
curl --user bob:password "localhost:5000/finance/salary/alice?token=$BOB_TOKEN"
```
Check that `alice` can see her own salary.
```shell
curl --user alice:password "localhost:5000/finance/salary/alice?token=$ALICE_TOKEN"
```
## Wrap Up
Congratulations for finishing the tutorial!
You learned a number of things about API authorization with OPA:
- OPA gives you fine-grained policy control over APIs once you set up the
server to ask OPA for authorization.
- You write allow/deny policies to control which APIs can be executed by whom.
- You can import external data into OPA and write policies that depend on
that data.
- You can use OPA data structures to define abstractions over your data.
- You can use a remote bundle server for distributing policy and data.
The code for this tutorial can be found in the
[open-policy-agent/contrib](https://github.com/open-policy-agent/contrib)
repository.
docs/new/docs/images/admission_review_flow.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 31 KiB

docs/new/docs/images/hist-envoy-grpc-100.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 207 KiB

docs/new/docs/images/hist-envoy-grpc-1000.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 222 KiB

docs/new/docs/images/hist-google-grpc-100.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 429 KiB

docs/new/docs/images/hist-google-grpc-1000.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 212 KiB

docs/new/docs/images/hist-google-vs-envoy-grpc-100.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 220 KiB

docs/new/docs/images/hist-google-vs-envoy-grpc-1000.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 237 KiB

docs/new/docs/images/thumbprint.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 123 KiB

docs/new/docs/index.md
+1247
View File
File diff suppressed because it is too large Load Diff
docs/new/docs/integration.md
+484
View File
@@ -0,0 +1,484 @@
---
title: Integrating OPA
sidebar_position: 7
---
OPA exposes domain-agnostic APIs that your service can call to manage and
enforce policies. Read this page if you want to integrate an application,
service, or tool with OPA.
When integrating with OPA there are two interfaces to consider:
- **Evaluation**: OPA's interface for asking for policy decisions. Integrating OPA is primarily focused on integrating an application, service, or tool with OPA's policy evaluation interface. This integration results in policy decisions being decoupled from that application, service, or tool.
- **Management**: OPA's interface for deploying policies, understanding status, uploading logs, and so on. This integration is typically the same across all OPA instances, regardless what software the evaluation interface is integrated with. Distributing policy, retrieving status, and storing logs in the same way across all OPAs provides a unified management plane for policy across many different software systems.
This page focuses predominantly on different ways to integrate with OPA's policy evaluation interface and how they compare. For more information about the management interface:
- See the [Bundle API](./management-bundles) for distributing policy and data to OPA.
- See the [Status API](./management-status) for collecting status reports on bundle activation and agent health.
- See the [Decision Log API](./management-decision-logs) for collecting a log of policy decisions made by agents.
- See the [Health API](./rest-api#health-api) for checking agent deployment readiness and health.
- See the [Prometheus API endpoint](./monitoring/#prometheus) to obtain insight into performance and errors.
## Evaluating Policies
OPA supports different ways to evaluate policies.
- The [REST API](./rest-api) returns decisions as JSON over HTTP.
- Also see the [Language SDKs](/ecosystem/#languages) for working with the REST API in different languages.
- The [Go API (GoDoc)](https://pkg.go.dev/github.com/open-policy-agent/opa/v1/rego) returns
decisions as simple Go types (`bool`, `string`, `map[string]interface{}`,
etc.)
- [WebAssembly](./wasm) compiles Rego policies into Wasm instructions so they can be embedded and evaluated by any WebAssembly runtime
- Custom compilers and evaluators may be written to parse evaluation plans in the low-level
[Intermediate Representation](./ir) format, which can be emitted by the `opa build` command
- The [SDK](https://pkg.go.dev/github.com/open-policy-agent/opa/v1/sdk) provides high-level APIs for obtaining the output
of query evaluation as simple Go types (`bool`, `string`, `map[string]interface{}`, etc.)
### Integrating with the REST API
To integrate with OPA outside of Go, we recommend you deploy OPA as a host-level
daemon or sidecar container. Running OPA locally on the same host as your
application or service helps ensure policy decisions are fast and highly-available.
When your application or service needs to make policy decisions it can query OPA
locally via HTTP. While it's possible to call OPA's [REST API](./rest-api) directly,
you can also find a number of [native language REST SDKs](/ecosystem/#languages)
which make the integration easier.
#### Named Policy Decisions
Use the [Data API](./rest-api#data-api) to query OPA for _named_ policy decisions:
```http
POST /v1/data/<path>
Content-Type: application/json
```
```json
{
"input": <the input document>
}
```
The `<path>` in the HTTP request identifies the policy decision to ask for. In
OPA, every rule generates a policy decision. In the example below there are two
decisions: `example/authz/allow` and `example/authz/is_admin`.
```live:authz:module:openable,read_only
package example.authz
default allow := false
allow if {
input.method == "GET"
input.path == ["salary", input.subject.user]
}
allow if is_admin
is_admin if "admin" in input.subject.groups
```
You can request specific decisions by querying for `<package path>/<rule name>`.
For example to request the `allow` decision execute the following HTTP request:
```http
POST /v1/data/example/authz/allow
Content-Type: application/json
```
```json
{
"input": <the input document>
}
```
The body of the request specifies the value of the `input` document to use
during policy evaluation. For example:
```http
POST /v1/data/example/authz/allow
Content-Type: application/json
```
```json
{
"input": {
"method": "GET",
"path": ["salary", "bob"],
"subject": {
"user": "bob"
}
}
}
```
OPA returns an HTTP 200 response code if the policy was evaluated successfully.
Non-HTTP 200 response codes indicate configuration or runtime errors. The policy
decision is contained in the `"result"` key of the response message body. For
example, the above request returns the following response:
```http
200 OK
Content-Type: application/json
```
```json
{
"result": true
}
```
If the requested policy decision is _undefined_ OPA returns an HTTP 200 response
without the `"result"` key. For example, the following request for `is_admin` is
undefined because there is no default value for `is_admin` and the input does
not satisfy the `is_admin` rule body:
```http
POST /v1/data/example/authz/is_admin
Content-Type: application/json
```
```json
{
"input": {
"subject": {
"user": "bob",
"groups": ["sales", "marketing"]
}
}
}
```
The response:
```http
200 OK
Content-Type: application/json
```
```json
{}
```
For another example of how to integrate with OPA via HTTP see the [HTTP API Authorization](./http-api-authorization) tutorial.
The [reference documentation](./rest-api) is also a good place to start.
#### Ecosystem Projects
The REST API is a common way to build integrations with OPA in distributed systems.
Browse the OPA Ecosystem for <EcosystemFeatureLink feature="rest-api-integration">
examples on REST API integrations</EcosystemFeatureLink> for inspiration.
### Integrating with the Go SDK
:::info
This section documents the v1 SDK package.
Please see [v0 Backwards Compatibility](./v0-compatibility) for notes on using
the v0 SDK package.
:::
The [SDK](https://pkg.go.dev/github.com/open-policy-agent/opa/sdk) package contains high-level APIs for embedding OPA
inside of Go programs and obtaining the output of query evaluation. To get started, import the `sdk` package:
```go
import "github.com/open-policy-agent/opa/v1/sdk"
```
A typical workflow when using the `sdk` package would involve first creating a new `sdk.OPA` object by calling
`sdk.New` and then invoking its `Decision` method to fetch the policy decision. The `sdk.New` call takes the
`sdk.Options` object as an input which allows specifying the OPA configuration, console logger, plugins, etc.
Here is an example that shows this process:
```go
package main
import (
"bytes"
"context"
"fmt"
"github.com/open-policy-agent/opa/v1/sdk"
sdktest "github.com/open-policy-agent/opa/v1/sdk/test"
)
func main() {
ctx := context.Background()
// create a mock HTTP bundle server
server, err := sdktest.NewServer(sdktest.MockBundle("/bundles/bundle.tar.gz", map[string]string{
"example.rego": `
package authz
default allow := false
allow if input.open == "sesame"
`,
}))
if err != nil {
// handle error.
}
defer server.Stop()
// provide the OPA configuration which specifies
// fetching policy bundles from the mock server
// and logging decisions locally to the console
config := []byte(fmt.Sprintf(`{
"services": {
"test": {
"url": %q
}
},
"bundles": {
"test": {
"resource": "/bundles/bundle.tar.gz"
}
},
"decision_logs": {
"console": true
}
}`, server.URL()))
// create an instance of the OPA object
opa, err := sdk.New(ctx, sdk.Options{
ID: "opa-test-1",
Config: bytes.NewReader(config),
})
if err != nil {
// handle error.
}
defer opa.Stop(ctx)
// get the named policy decision for the specified input
if result, err := opa.Decision(ctx, sdk.DecisionOptions{Path: "/authz/allow", Input: map[string]interface{}{"open": "sesame"}}); err != nil {
// handle error.
} else if decision, ok := result.Result.(bool); !ok || !decision {
// handle error.
}
}
```
If you executed this code, the output (i.e. [Decision Log](https://www.openpolicyagent.org/docs/latest/management-decision-logs/) event)
would be logged to the console by default.
Setting an `ID` in `sdk.Options` is optional, but recommended. If you do not set an `ID`, a random one will be created
for the system. While this is fine for testing, it makes it difficult to monitor the system over time, as a new ID will
be created each time the SDK is initialized, such as when the process is restarted.
#### Manually Triggering Bundle Reloads
Users of the SDK can
[manually trigger](./configuration/#bundles)
the SDK's Bundle plugin to load new bundles immediately based on external
events. When doing so, it's recommended to set `bundles[_].trigger` to `manual`
if you want to disable periodic bundle polling.
In this short example, the `bundle` plugin is loaded from the SDK instance and
triggered to check for new bundles. Do this sparingly, it is not intended to be
used as a replacement for periodic bundle polling. For best performance, only
trigger the bundle plugin when you know that new bundles are available.
```go
options := sdk.Options{
Config: bytes.NewReader(config),
Logger: logger,
Ready: make(chan struct{}), // <-- needed or else sdk.New will block
}
opa, err := sdk.New(ctx, options)
if err != nil {
log.Fatal(err)
}
defer opa.Stop(ctx)
bundle, ok := opa.Plugin("bundle").(*bundle.Plugin)
if !ok {
log.Fatal("bundle plugin not found")
}
err = bundle.Trigger(ctx)
if err != nil {
log.Fatal(err)
}
```
### Integrating with the Go API
Use the low-level
[github.com/open-policy-agent/opa/v1/rego](https://pkg.go.dev/github.com/open-policy-agent/opa/v1/rego)
package to embed OPA as a library inside services written in Go, when only policy **evaluation** — and
no other capabilities of OPA, like the management features — are desired. If you're unsure which one to
use, the SDK is probably the better option.
To get started import the `rego` package:
```go
import "github.com/open-policy-agent/opa/v1/rego"
```
The `rego` package exposes different options for customizing how policies are
evaluated. Through the `rego` package you can supply policies and data, enable
metrics and tracing, toggle optimizations, etc. In most cases you will:
1. Use the `rego` package to construct a prepared query.
2. Execute the prepared query to produce policy decisions.
3. Interpret and enforce the policy decisions.
Preparing queries in advance avoids parsing and compiling the policies on each
query and improves performance considerably. Prepared queries are safe to share
across multiple Go routines.
To prepare a query create a new `rego.Rego` object by calling `rego.New(...)`
and then invoke `rego.Rego#PrepareForEval`. The `rego.New(...)` call can be
parameterized with different options like the query, policy module(s), data
store, etc.
```go
module := `
package example.authz
default allow := false
allow if {
input.method == "GET"
input.path == ["salary", input.subject.user]
}
allow if is_admin
is_admin if "admin" in input.subject.groups
`
ctx := context.TODO()
query, err := rego.New(
rego.Query("x = data.example.authz.allow"),
rego.Module("example.rego", module),
).PrepareForEval(ctx)
if err != nil {
// Handle error.
}
```
Using the `query` returned by `rego.Rego#PrepareForEval` call the `Eval`
function to evaluate the policy:
```go
input := map[string]interface{}{
"method": "GET",
"path": []interface{}{"salary", "bob"},
"subject": map[string]interface{}{
"user": "bob",
"groups": []interface{}{"sales", "marketing"},
},
}
results, err := query.Eval(ctx, rego.EvalInput(input))
```
The `rego.PreparedEvalQuery#Eval` function returns a _result set_ that contains
the query results. If the result set is empty it indicates the query could not
be satisfied. Each element in the result set contains a set of _variable
bindings_ and a set of expression values. The query from above includes a single
variable `x` so we can lookup the value and interpret it to enforce the policy
decision.
```go
if err != nil {
// Handle evaluation error.
} else if len(results) == 0 {
// Handle undefined result.
} else if result, ok := results[0].Bindings["x"].(bool); !ok {
// Handle unexpected result type.
} else {
// Handle result/decision.
// fmt.Printf("%+v", results) => [{Expressions:[true] Bindings:map[x:true]}]
}
```
For the common case of policies evaluating to a single boolean value, there's
a helper method: With `results.Allowed()`, the previous snippet can be shortened
to
```go
results, err := query.Eval(ctx, rego.EvalInput(input))
if err != nil {
// handle error
}
if !results.Allowed() {
// handle result
}
```
For more examples of embedding OPA as a library see the
[`rego`](https://pkg.go.dev/github.com/open-policy-agent/opa/v1/rego#pkg-examples)
package in the Go documentation.
:::info
This section documents the v1 Rego package.
Please see [v0 Backwards Compatibility](./v0-compatibility) for notes on using
the v0 Rego package.
:::
#### Ecosystem Projects
The Go API is made available to allow other projects to build policy functionality into their
applications. Browse <EcosystemFeatureLink feature="go-integration">Go
integrations</EcosystemFeatureLink> in the OPA Ecosystem for inspiration.
### WebAssembly (Wasm)
Policies can be evaluated as compiled Wasm binaries. See [OPA Wasm docs](./wasm) for more details.
There are a number of projects already built on OPA's Wasm support. Take a look
in <EcosystemFeatureLink feature="wasm-integration">Wasm
integrations</EcosystemFeatureLink> in the OPA Ecosystem for more details.
### Intermediate Representation (IR)
Policies may be compiled into evaluation plans using an intermediate representation format, suitable for custom
compilers and evaluators.
See [OPA IR docs](./ir) for more details.
## Comparison
A comparison of the different integration choices are summarized below.
| Dimension | REST API | Go Lib | Wasm |
| ---------- | --------------- | -------------------------- | -------------------------- |
| Evaluation | Fast | Faster | Fastest |
| Language | Any | Only Go | Any with Wasm |
| Operations | Update just OPA | Update entire service | Update service rarely |
| Security | Must secure API | Enable only what is needed | Enable only what is needed |
Integrating OPA via the REST API is the most common, at the time of writing. OPA
is most often deployed either as a sidecar or less commonly as an external
service. Operationally this makes it easy to upgrade OPA and to configure it to
use its management services (bundles, status, decision logs, etc.). Because it
is a separate process it requires monitoring and logging (though this happens
automatically for any sidecar-aware environment like Kubernetes). OPA's
configuration and APIs must be secured according to the [security guide](./security).
Integrating OPA via the Go API only works for Go software. Updates to OPA
require re-vendoring and re-deploying the software. Evaluation has less overhead
than the REST API because all the communication happens in the same
operating-system process. All of the management functionality (bundles, decision
logs, etc.) must be either enabled or implemented. Security concerns are limited
to those management features that are enabled or implemented.
Wasm policies are embeddable in any programming language that has a Wasm
runtime. Evaluation has less overhead than the REST API (because it is evaluated
in the same operating-system process) and should outperform the Go API (because
the policies have been compiled to a lower-level instruction set). Each
programming language will need its own SDKs that implement the management
functionality and the evaluation interface. Typically new OPA language features
will not require updating the service since neither the Wasm runtime nor the
SDKs will be impacted. Updating the SDKs will require re-deploying the service.
Security is analogous to the Go API integration: it is mainly the management
functionality that presents security risks.
docs/new/docs/ir.md
+461
View File
@@ -0,0 +1,461 @@
---
title: Intermediate Representation (IR)
sidebar_position: 12
---
# Overview
OPA can compile policy queries into planned evaluation paths suitable for
further compilation or interpretation. This document explains the structure and
semantics of the intermediate representation (IR) used to represent these
planned evaluation paths. Read this document if you want to write a compiler or
interpreter for Rego.
# Structure
This section explains the structure of policies compiled into the IR.
## Policy
The root object emitted by the compiler is a `Policy` and contains the following
top-level keys:
- `static` is an object containing static data used by the compiled plans and
functions.
- `plans` is an object containing entrypoints to compiled evaluation paths.
- `funcs` is an object containing functions supporting the compiled evaluation
paths.
## Static
The `Static` object contains static data required by the plans and functions.
The static object also contains metadata that does not affect the semantics of
the policy. The static object contains the following top-level keys:
- `strings` is an array of string constants referenced by compiled statements in
the plans and functions.
- `builtin_funcs` is an array of function declarations representing built-in
functions required by the compiled statements.
- `files` is used for debugging purposes only. It is an array of filenames that
were used during compilation.
### Strings
The `Strings` array is a collection of string objects referenced by compiled
statements in the policy. Strings are referenced by their index in the
collection. Each string object contains the following fields:
- `value` is the string constant value. The string may be any valid JSON string.
### Built-in Functions
The `Built-in Functions` array is a collection of built-in function
declarations. Each declaration represents a function that must be provided by
the environment where the policy is eventually executed. Each built-in function
contains the following fields:
- `name` is the name of the function that must be provided.
- `decl` is the type definition of the function.
### Files
The `Files` array is a collection of static strings representing names of source
files used during compilation. Filenames are referred to by their index in the
files array.
## Plans
The `Plans` object contains a collection of planned evaluation paths
representing entrypoints to the policy. When users compile policies they supply
the queries to expose as entrypoints. Each plan contains the following fields:
- `name` is the entrypoint identifier, typically set to the path of the policy
decision (e.g., `authz/allow`).
- `blocks` is a collection of [`Block`](#blocks) objects representing the
compiled statements that define the entrypoint.
## Functions
The `Functions` object contains a collection of function definitions that
represent functions supporting the plans. Functions can be invoked by name
inside of plans and other functions. Each function contains the following
fields:
- `name` is the function identifier referenced by call statements.
- `path` is the function identifier referenced by dynamic call statements.
- `params` is an ordered list of local variable identifiers representing
function parameters. The parameters can be referenced inside of the blocks
that define the function.
- `return` is the local variable containing the return value of the function.
- `blocks` is collection of [`Block`](#blocks) objects representing the compiled
statements that define the function.
## Blocks
The `Block` object contains a sequence of [Statements](#statements) that must be
executed in order until a statement terminating block execution is encountered
or the end of the block is reached. Each block contains the following fields:
- `stmts` is an array of `Statement` objects.
## Statements
The `Statement` object represents an operation performed by the policy (e.g.,
function invocation, lookup, iteration, comparison, etc.) The structure is
specific to each statement type but every statement contains the following
fields:
- `type` is a string value that identifies the type of the statement.
- `stmt` is an object containing statement-specific fields.
- `file` is the index of source filename where this statement originated.
- `row` is the row in the source file where this statement originated.
- `col` is the column in the source file where this statement originated.
See the [Statement Definitions](#statement-definitions) section for an
explanation of the supported statement types.
# Execution
This section explains the execution model for compiled policies.
## Plan Execution
Compiled policies consist of one or more plans. Any plan can be invoked by name.
If no name is supplied, the first plan in the policy should be executed. Plans
consist of one or more [Blocks](#blocks) that are executed in-order. Statements
inside the blocks of a plan have implicit access to two local variables
representing the `input` and `data` documents (`0` and `1` respectively.) The
final statement in every block inside of a plan is a `ResultSetAddStmt`
statement that adds an object to an implicit result set. The object contains the
key-value bindings representing the values of variables in the original query.
If no `ResultSetAddStmt` statements are executed, the implicit result set is
empty.
## Function Execution
Compiled policies may contain zero or more functions. Any function can be
invoked by name via the `CallStmt` statement or dynamically via the
`CallDynamicStmt` statement. All functions are defined with two or more
positional arguments. The first positional argument is a local variable
representing the `input` document. The second positional argument is a local
variable representing the `data` document. Function execution terminates when a
`ReturnLocalStmt` statement is encountered. All functions include a final block
that includes a `ReturnLocalStmt`.
## Block Execution
Blocks are sequences of statements that are executed in order. Statements can be
executed if all of the input parameters are defined. If any input parameter is
undefined then the statement is undefined. The [Statement Definitions](#statement-definitions) section below indicates when a statement
may be undefined. When a statement is undefined execution breaks to the end of
the current block and resumes execution at the statement immediately following
the block (which may be the beginning of another block.) When a statement is
defined, all output parameters are defined. Execution halts if a statement
raises an exception.
# Statement Definitions
This section defines the statements that can be contained in plans and functions
and explains the input and output parameters that each statement accepts. The
set of valid parameter types are:
- `local` is a 32-bit integer representing a local variable.
- `int32` is a 32-bit integer.
- `int64` is a 64-bit integer.
- `uint32` is a 32-bit unsigned integer.
- `string` is an arbitrary-length unicode string.
- `array[...]` represents a sequence of `...` values.
In addition, parameters may be of type `operand`. The `operand` type represents
a tagged union that can refer to a local variable, boolean constant, or string
constant index:
```
{
"type": "local" | "bool" | "string_index"
"value": number | boolean | number
}
```
Local variables refer to values. The value types are any JSON value (i.e., `null`,
`true`, `false`, `number`, `string`, `array`, and `object`) as well as sets
(which are unordered value collections.)
## `ArrayAppendStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | --------- | --------------------------------- |
| `array` | `input` | `local` | The array to append a value to. |
| `value` | `input` | `operand` | The value to append to the array. |
## `AssignIntStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | ------- | -------------------------------------------- |
| `value` | `input` | `int64` | The integer value to assign to the target. |
| `target` | `output` | `local` | The local variable to assign the integer to. |
## `AssignVarOnceStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | --------- | -------------------------------------------- |
| `source` | `input` | `operand` | The value to assign to the target. |
| `target` | `output` | `local` | The local variable to assign the operand to. |
:::danger
This statement raises an exception if the `target` operand is already assigned.
:::
## `AssignVarStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | --------- | -------------------------------------------- |
| `source` | `input` | `operand` | The value to assign to the target. |
| `target` | `output` | `local` | The local variable to assign the operand to. |
## `BlockStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | ----------------- | ----------------------------- |
| `blocks` | `input` | [Blocks](#blocks) | The nested blocks to execute. |
## `BreakStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| `index` | `input` | `uint32` | The index of the block to jump out of starting with zero representing the current block and incrementing by one for each outer block. |
## `CallDynamicStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | ---------------- | ---------------------------------------------------------- |
| `path` | `input` | `array[operand]` | The path of the function to invoke. |
| `args` | `input` | `array[local]` | The positional arguments to pass to the function. |
| `result` | `output` | `local` | The local variable to assign the function return value to. |
## `CallStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | -------------- | ---------------------------------------------------------- |
| `func` | `input` | `string` | The name of the function to invoke. |
| `args` | `input` | `array[local]` | The positional arguments to pass to the function. |
| `result` | `output` | `local` | The local variable to assign the function return value to. |
## `DotStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | --------- | ------------------------------------------- |
| `source` | `input` | `operand` | The value to perform a lookup operation on. |
| `key` | `input` | `operand` | The key to lookup in the source. |
| `target` | `output` | `local` | The local variable to assign the result to. |
This statement is **undefined** if the `key` does not exist in the `source` value.
## `EqualStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | --------- | ---------------------------- |
| `a` | `input` | `operand` | The first value to compare. |
| `b` | `input` | `operand` | The second value to compare. |
This statement is **undefined** if `a` does not equal `b`.
## `IsArrayStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | --------- | ------------------- |
| `source` | `input` | `operand` | The value to check. |
This statement is **undefined** if `source` is not an array.
## `IsDefinedStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | --------- | ------------------- |
| `source` | `input` | `operand` | The value to check. |
This statement is **undefined** if `source` is undefined.
## `IsObjectStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | --------- | ------------------- |
| `source` | `input` | `operand` | The value to check. |
This statement is **undefined** if `source` is not an object.
## `IsSetStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | --------- | ------------------- |
| `source` | `input` | `operand` | The value to check. |
This statement is **undefined** if `source` is not a set.
## `IsUndefinedStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | --------- | ------------------- |
| `source` | `input` | `operand` | The value to check. |
This statement is **undefined** if `source` is not undefined.
## `LenStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | --------- | ------------------------------------------- |
| `source` | `input` | `operand` | The value to compute the length for. |
| `target` | `output` | `local` | The local variable to assign the length to. |
## `MakeArrayStmt`
| Parameter | Input/Output | Type | Description |
| ---------- | ------------ | ------- | ------------------------------------------------ |
| `capacity` | `input` | `int32` | The initial size of the array to pre-allocate. |
| `target` | `output` | `local` | The local variable to assign the array value to. |
## `MakeNullStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | ------- | ----------------------------------------------- |
| `target` | `output` | `local` | The local variable to assign the null value to. |
## `MakeNumberIntStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | ------- | ------------------------------------------------ |
| `value` | `input` | `int64` | The integer value to initialize the target with. |
| `target` | `output` | `local` | The local variable to assign the number to. |
## `MakeNumberRefStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | ------- | -------------------------------------------------------------- |
| `index` | `input` | `int32` | The index of the string constant to construct the number with. |
| `target` | `output` | `local` | The local variable to assign the number to. |
## `MakeObjectStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | ------- | ------------------------------------------- |
| `target` | `output` | `local` | The local variable to assign the object to. |
## `MakeSetStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | ------- | ---------------------------------------- |
| `target` | `output` | `local` | The local variable to assign the set to. |
## `NopStmt`
This statement is only used for debugging purposes.
## `NotEqualStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | --------- | ---------------------------- |
| `a` | `input` | `operand` | The first value to compare. |
| `b` | `input` | `operand` | The second value to compare. |
This statement is **undefined** if `a` is equal to `b`.
## `NotStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | ---------------- | --------------------------------- |
| `block` | `input` | [Block](#blocks) | The negated statement to execute. |
This statement is **undefined** if the contained block is not undefined.
## `ObjectInsertOnceStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | --------- | --------------------------------------------- |
| `key` | `input` | `operand` | The key to insert into the object. |
| `value` | `input` | `operand` | The value to insert into the object. |
| `object` | `input` | `local` | The object to insert the key-value pair into. |
:::danger
This statement raises an exception if the `object` contains an existing `key` with a different `value`.
:::
## `ObjectInsertStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | --------- | --------------------------------------------- |
| `key` | `input` | `operand` | The key to insert into the object. |
| `value` | `input` | `operand` | The value to insert into the object. |
| `object` | `input` | `local` | The object to insert the key-value pair into. |
## `ObjectMergeStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | ------- | -------------------------------------------------- |
| `a` | `input` | `local` | The object to merge into. |
| `b` | `input` | `local` | The object to merge from. |
| `target` | `output` | `local` | The local variable to assign the merged object to. |
## `ResetLocalStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | ------- | ---------------------------- |
| `target` | `output` | `local` | The local variable to reset. |
## `ResultSetAddStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | ------- | ----------------------------------- |
| `value` | `input` | `local` | The value to add to the result set. |
## `ReturnLocalStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | ------- | -------------------------------------- |
| `source` | `input` | `local` | The value to return from the function. |
## `ScanStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | ---------------- | -------------------------------------------------------------------------- |
| `source` | `input` | `local` | The value to scan. |
| `key` | `output` | `local` | The local variable to assign keys to before executing the nested block. |
| `value` | `output` | `local` | The local variable to assign values to before executing the nested block. |
| `block` | `input` | [Block](#blocks) | The nested block to execute repeatedly for each element in the collection. |
This statement is **undefined** if `source` is a scalar value or empty collection.
## `SetAddStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | --------- | --------------------------------- |
| `value` | `input` | `operand` | The value to insert into the set. |
| `set` | `input` | `local` | The set to insert the value into. |
## `WithStmt`
| Parameter | Input/Output | Type | Description |
| --------- | ------------ | ---------------- | --------------------------------------------------------------------------------------------------------------- |
| `local` | `input` | `local` | The value to mutate in the context of the nested block. |
| `path` | `input` | `array[int32]` | The path of the nested document to replace with the `value` represented as an array of string constant indices. |
| `value` | `input` | `operand` | The value to upsert. |
| `block` | `input` | [Block](#blocks) | The nested block to execute in the context of the mutation. |
# Test Suite
The OPA repository contains a [test suite](https://github.com/open-policy-agent/opa/tree/main/v1/test/cases/testdata/v1)
that is used internally to validate both the Go interpreter and the Wasm
compiler. If you are implementing your own compiler or interpreter we highly
recommend integrating the test suite into your own development environment so
that your implementation can be verified to conform with OPA's.
The test suite consists of a set of YAML files that each contain a set of test
cases. Each test cases specifies a query, set of modules, data values, and
expected outputs or expected error conditions.
To get started with the test suite, see the [Hello World](https://github.com/open-policy-agent/opa/blob/main/v1/test/cases/testdata/v0/helloworld/test-helloworld-1.yaml)
example.
The following examples show how the test suite is used internally:
- [`github.com/open-policy-agent/opa/topdown#TestRego`](https://github.com/open-policy-agent/opa/blob/main/v1/topdown/exported_test.go)
- [`github.com/open-policy-agent/opa/internal/wasm/sdk/test/e2e/external_test`](https://github.com/open-policy-agent/opa/blob/main/internal/wasm/sdk/test/e2e/external_test.go)
docs/new/docs/kafka-authorization.md
+529
View File
@@ -0,0 +1,529 @@
---
title: Kafka
---
[Apache Kafka](https://kafka.apache.org/) is a high-performance distributed
streaming platform deployed by thousands of companies. In many deployments,
administrators require fine-grained access control over Kafka topics to
enforce important requirements around confidentiality and integrity.
## Goals
This tutorial shows how to enforce fine-grained access control over Kafka
topics. In this tutorial you will use OPA to define and enforce an
authorization policy stating:
- Consumers of topics containing Personally Identifiable Information (PII) must be on allow list.
- Producers to topics with _high fanout_ must be on allow list.
In addition, this tutorial shows how to break up a policy with small helper
rules to reuse logic and improve overall readability.
## Prerequisites
This tutorial requires [Docker Compose](https://docs.docker.com/compose/install/) to run Kafka, ZooKeeper, and OPA.
Additionally, we'll use Nginx for serving policy and data bundles to OPA. This component is however easily replaceable
by any other bundle server [implementation](./management-bundles/#implementations).
## Steps
### 1. Bootstrap the tutorial environment using Docker Compose.
First, let's create some directories. We'll create one for our policy files, a second one for built bundles, and a third
one or the OPA authorizer plugin.
```bash
mkdir policies bundles plugin
```
Next, create an OPA policy that allows all requests. You will update this policy later in the tutorial.
**policies/tutorial.rego**:
```live:start:module:read_only
package kafka.authz
allow := true
```
With the policy in place, build a bundle from the contents of the `policies` directory and place it in the `bundles`
directory. The `bundles` directory will later be mounted into the Nginx container in order to distribute policy updates
to OPA.
```shell
opa build --bundle policies/ --output bundles/bundle.tar.gz
```
#### Kafka Authorizer JAR File
Next, download the latest version of the [Open Policy Agent plugin for Kafka authorization](https://github.com/StyraInc/opa-kafka-plugin)
plugin from the projects [release pages](https://github.com/StyraInc/opa-kafka-plugin/releases).
Store the plugin in the `plugin` directory (replace `${version}` with the version number of the plugin just downloaded):
```bash
mv opa-authorizer-${version}-all.jar plugin/
```
For more information on how to configure the OPA plugin for Kafka, see the plugin [repository](https://github.com/StyraInc/opa-kafka-plugin).
Next, create a `docker-compose.yaml` file that runs OPA, Nginx, ZooKeeper, and Kafka.
**docker-compose.yaml**:
```yaml
services:
nginx:
image: nginx:1.21.4
volumes:
- "./bundles:/usr/share/nginx/html"
ports:
- "80:80"
opa:
image: openpolicyagent/opa:{{< current_docker_version >}}
ports:
- "8181:8181"
command:
- "run"
- "--server"
- "--set=decision_logs.console=true"
- "--set=services.authz.url=http://nginx"
- "--set=bundles.authz.service=authz"
- "--set=bundles.authz.resource=bundle.tar.gz"
depends_on:
- nginx
zookeeper:
image: confluentinc/cp-zookeeper:6.2.1
ports:
- "2181:2181"
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
- ZOOKEEPER_CLIENT_PORT=2181
broker:
image: confluentinc/cp-kafka:6.2.1
ports:
- "9093:9093"
environment:
# Set cache expiry to low value for development in order to see decisions
KAFKA_OPA_AUTHORIZER_CACHE_EXPIRE_AFTER_SECONDS: 10
KAFKA_OPA_AUTHORIZER_URL: http://opa:8181/v1/data/kafka/authz/allow
KAFKA_AUTHORIZER_CLASS_NAME: org.openpolicyagent.kafka.OpaAuthorizer
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
KAFKA_ADVERTISED_LISTENERS: SSL://localhost:9093
KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SSL
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_SSL_KEYSTORE_FILENAME: server.keystore
KAFKA_SSL_KEYSTORE_CREDENTIALS: credentials.txt
KAFKA_SSL_KEY_CREDENTIALS: credentials.txt
KAFKA_SSL_TRUSTSTORE_FILENAME: server.truststore
KAFKA_SSL_TRUSTSTORE_CREDENTIALS: credentials.txt
KAFKA_SSL_CLIENT_AUTH: required
CLASSPATH: "/plugin/*"
volumes:
- "./plugin:/plugin"
- "./cert/server:/etc/kafka/secrets"
depends_on:
- opa
- zookeeper
```
#### Authentication
The Docker Compose file defined above requires **SSL client authentication**
for clients that connect to the broker. Enabling SSL client authentication
allows for service identities to be provided as input to your policy. The
example below shows the input structure.
```json
{
"action": {
"logIfAllowed": true,
"logIfDenied": true,
"operation": "WRITE",
"resourcePattern": {
"name": "credit-scores",
"patternType": "LITERAL",
"resourceType": "TOPIC",
"unknown": false
},
"resourceReferenceCount": 1
},
"requestContext": {
"clientAddress": "/172.22.0.1",
"clientInformation": {
"softwareName": "apache-kafka-java",
"softwareVersion": "2.8.1"
},
"connectionId": "172.22.0.5:9093-172.22.0.1:62744-2",
"header": {
"headerVersion": 2,
"name": {
"clientId": "consumer-console-producer-63933-1",
"correlationId": 10,
"requestApiKey": 1,
"requestApiVersion": 12
}
},
"listenerName": "SSL",
"principal": {
"name": "CN=anon_producer,OU=Developers",
"principalType": "User"
},
"securityProtocol": "SSL"
}
}
```
The client identity is extracted from the SSL certificates that clients
present when they connect to the broker. The user identity information is
encoded in the `input.requestContext.principal.name` field.
This field can be used inside the policy.
A detailed rundown of generating SSL certificates and JKS files required
for SSL client authentication is outside the scope of this tutorial, but the plugin
repository provides an [example script](https://github.com/StyraInc/opa-kafka-plugin/tree/main/example/opa_tutorial/create_cert.sh)
that demonstrates the creation of client certificates for the four different
users used in this tutorial:
- `anon_producer`
- `anon_consumer`
- `pii_consumer`
- `fanout_producer`
Lets' download the script and run it:
```shell
curl -O https://raw.githubusercontent.com/StyraInc/opa-kafka-plugin/main/example/opa_tutorial/create_cert.sh
chmod +x create_cert.sh
./create_cert.sh
```
We should now find a new `cert` directory created by the script,
containing the server and client certificates we'll need for TLS
authentication.
Note: Do not rely on these SSL certificates in real-world scenarios.
They are only provided for convenience/test purposes.
If you'd rather set up these users by other means, like one
of the available SASL mechanisms Kafka provides, that should work
just as well. Just make sure to update the Docker compose file
accordingly.
Once you have created the files needed for authentication,
you may launch the containers for this tutorial.
```bash
docker-compose --project-name opa-kafka-tutorial up
```
Now that the tutorial environment is running, we can define an authorization policy using OPA and test it.
### 2. Define a policy to restrict consumer access to topics containing Personally Identifiable Information (PII).
Update the `policies/tutorial.rego` with the following content.
```live:example:module:openable
#-----------------------------------------------------------------------------
# High level policy for controlling access to Kafka.
#
# * Deny operations by default.
# * Allow operations if no explicit denial.
#
# The kafka-authorizer-opa plugin will query OPA for decisions at
# /kafka/authz/allow. If the policy decision is _true_ the request is allowed.
# If the policy decision is _false_ the request is denied.
#-----------------------------------------------------------------------------
package kafka.authz
default allow := false
allow if {
not deny
}
deny if {
is_read_operation
topic_contains_pii
not consumer_is_allowlisted_for_pii
}
#-----------------------------------------------------------------------------
# Data structures for controlling access to topics. In real-world deployments,
# these data structures could be loaded into OPA as raw JSON data. The JSON
# data could be pulled from external sources like AD, Git, etc.
#-----------------------------------------------------------------------------
consumer_allowlist := {"pii": {"pii_consumer"}}
topic_metadata := {"credit-scores": {"tags": ["pii"]}}
#-----------------------------------
# Helpers for checking topic access.
#-----------------------------------
topic_contains_pii if {
"pii" in topic_metadata[topic_name].tags
}
consumer_is_allowlisted_for_pii if {
principal.name in consumer_allowlist.pii
}
#-----------------------------------------------------------------------------
# Helpers for processing Kafka operation input. This logic could be split out
# into a separate file and shared. For conciseness, we have kept it all in one
# place.
#-----------------------------------------------------------------------------
is_write_operation if {
input.action.operation == "WRITE"
}
is_read_operation if {
input.action.operation == "READ"
}
is_topic_resource if {
input.action.resourcePattern.resourceType == "TOPIC"
}
topic_name := input.action.resourcePattern.name if {
is_topic_resource
}
principal := {"fqn": parsed.CN, "name": cn_parts[0]} if {
parsed := parse_user(input.requestContext.principal.name)
cn_parts := split(parsed.CN, ".")
}
# If client certificates aren't used for authentication
else := {"fqn": "", "name": input.requestContext.principal.name}
parse_user(user) := {key: value |
parts := split(user, ",")
[key, value] := split(parts[_], "=")
}
```
The Kafka authorization plugin is configured to query for the
`data.kafka.authz.allow` decision. If the response is `true` the operation is
allowed, otherwise the operation is denied. When the integration queries OPA it
supplies a JSON representation of the operation, resource, client, and principal.
```live:example:query:hidden
data.kafka.authz.allow
```
```live:example:input
{
"action": {
"logIfAllowed": true,
"logIfDenied": true,
"operation": "READ",
"resourcePattern": {
"name": "credit-scores",
"patternType": "LITERAL",
"resourceType": "TOPIC",
"unknown": false
},
"resourceReferenceCount": 1
},
"requestContext": {
"clientAddress": "/172.22.0.1",
"clientInformation": {
"softwareName": "apache-kafka-java",
"softwareVersion": "2.8.1"
},
"connectionId": "172.22.0.5:9093-172.22.0.1:62744-2",
"header": {
"headerVersion": 2,
"name": {
"clientId": "consumer-console-producer-63933-1",
"correlationId": 10,
"requestApiKey": 1,
"requestApiVersion": 12
}
},
"listenerName": "SSL",
"principal": {
"name": "CN=pii_consumer,OU=developers",
"principalType": "User"
},
"securityProtocol": "SSL"
}
}
```
With the input value above, the answer is:
```live:example:output
```
The `./bundles` directory is mounted into the Docker container running Nginx.
When the bundle under this directory change, OPA is notified via the bundle API,
and the policies are automatically reloaded.
You can update the bundle at any time by rebuilding it.
```shell
opa build --bundle policies/ --output bundles/bundle.tar.gz
```
At this point, you can exercise the policy.
### 3. Exercise the policy that restricts consumer access to topics containing PII.
This step shows how you can grant fine-grained access to services using
Kafka. In this scenario, some services are allowed to read PII data while
others are not.
First, run `kafka-console-producer` to generate some data on the
`credit-scores` topic.
> This tutorial uses the `kafka-console-producer` and `kafka-console-consumer` scripts provided by Kafka to generate and display Kafka messages. These scripts read from STDIN and write to STDOUT and are frequently used to send and receive data via Kafka over the command line. If you are not familiar with these scripts you can learn more in Kafka's [Quick Start](https://kafka.apache.org/documentation/#quickstart) documentation.
```bash
docker run -v $(pwd)/cert/client:/tmp/client --rm --network opa-kafka-tutorial_default \
confluentinc/cp-kafka:6.2.1 \
bash -c 'for i in {1..10}; do echo "{\"user\": \"bob\", \"score\": $i}"; done | kafka-console-producer --topic credit-scores --broker-list broker:9093 -producer.config /tmp/client/anon_producer.properties'
```
This command will send 10 messages to the `credit-scores` topic. Bob's credit
score seems to be improving.
Next, run `kafka-console-consumer` and try to read data off the topic. Use
the `pii_consumer` credentials to simulate a service that is allowed to read
PII data.
```bash
docker run -v $(pwd)/cert/client:/tmp/client --rm --network opa-kafka-tutorial_default \
confluentinc/cp-kafka:6.2.1 \
kafka-console-consumer --bootstrap-server broker:9093 --topic credit-scores --from-beginning --consumer.config /tmp/client/pii_consumer.properties
```
This command will output the 10 messages sent to the topic in the first part
of this step. Once the 10 messages have been printed, exit out of the script
(^C).
Finally, run `kafka-console-consumer` again but this time try to use the
`anon_consumer` credentials. The `anon_consumer` credentials simulate a
service that has **not** been explicitly granted access to PII data.
```bash
docker run -v $(pwd)/cert/client:/tmp/client --rm --network opa-kafka-tutorial_default \
confluentinc/cp-kafka:6.2.1 \
kafka-console-consumer --bootstrap-server broker:9093 --topic credit-scores --from-beginning --consumer.config /tmp/client/anon_consumer.properties
```
Because the `anon_consumer` is not allowed to read PII data, the request will
be denied and the consumer will output an error message.
```
Not authorized to read from topic credit-scores.
...
Processed a total of 0 messages
```
### 4. Extend the policy to prevent services from accidentally writing to topics with large fanout.
First, add the following content to the policy file (`./policies/tutorial.rego`):
```live:example/deny:module:openable
deny if {
is_write_operation
topic_has_large_fanout
not producer_is_allowlisted_for_large_fanout
}
producer_allowlist := {
"large-fanout": {
"fanout_producer",
}
}
topic_has_large_fanout if {
topic_metadata[topic_name].tags[_] == "large-fanout"
}
producer_is_allowlisted_for_large_fanout if {
producer_allowlist["large-fanout"][_] == principal.name
}
```
Next, update the `topic_metadata` data structure in the same file to indicate
that the `click-stream` topic has a high fanout.
```live:updated_metadata:module:read_only
topic_metadata := {
"click-stream": {
"tags": ["large-fanout"],
},
"credit-scores": {
"tags": ["pii"],
}
}
```
Last, build a bundle from the updated policy.
```shell
opa build --bundle policies/ --output bundles/bundle.tar.gz
```
### 5. Exercise the policy that restricts producer access to topics with high fanout.
First, run `kafka-console-producer` and simulate a service with access to the
`click-stream` topic.
```bash
docker run -v $(pwd)/cert/client:/tmp/client --rm --network opa-kafka-tutorial_default \
confluentinc/cp-kafka:6.2.1 \
bash -c 'for i in {1..10}; do echo "{\"user\": \"alice\", \"button\": $i}"; done | kafka-console-producer --topic click-stream --broker-list broker:9093 -producer.config /tmp/client/fanout_producer.properties'
```
Next, run the `kafka-console-consumer` to confirm that the messages were published.
```bash
docker run -v $(pwd)/cert/client:/tmp/client --rm --network opa-kafka-tutorial_default \
confluentinc/cp-kafka:6.2.1 \
kafka-console-consumer --bootstrap-server broker:9093 --topic click-stream --from-beginning --consumer.config /tmp/client/anon_consumer.properties
```
Once you see the 10 messages produced by the first part of this step, exit the console consumer (^C).
Lastly, run `kafka-console-producer` to simulate a service that should **not**
have access to _high fanout_ topics.
```bash
docker run -v $(pwd)/cert/client:/tmp/client --rm --network opa-kafka-tutorial_default \
confluentinc/cp-kafka:6.2.1 \
bash -c 'echo "{\"user\": \"alice\", \"button\": \"bogus\"}" | kafka-console-producer --topic click-stream --broker-list broker:9093 -producer.config /tmp/client/anon_producer.properties'
```
Because `anon_producer` is not authorized to write to high fanout topics, the
request will be denied and the producer will output an error message.
```
Not authorized to access topics: [click-stream]
```
## Wrap Up
Congratulations on finishing the tutorial!
At this point you have learned how to enforce fine-grained access control
over Kafka topics. In addition, you have seen how to break down policies into
smaller rules that can be reused and improve the overall readability over the
policy.
If you want to use the Kafka Authorizer plugin that integrates Kafka with
OPA, see the build and install instructions in the
[opa-kafka-plugin](https://github.com/StyraInc/opa-kafka-plugin)
repository.
docs/new/docs/kubernetes/_category_.yaml
+4
View File
@@ -0,0 +1,4 @@
position: 15
label: "Kubernetes"
collapsible: true
collapsed: true
docs/new/docs/kubernetes/assets/kubernetes-admission-flow.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 31 KiB

docs/new/docs/kubernetes/assets/kubernetes-watchers.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 21 KiB

docs/new/docs/kubernetes/debugging.md
+166
View File
@@ -0,0 +1,166 @@
---
title: Debugging Tips
---
If you run into problems getting OPA to enforce admission control policies in
Kubernetes there are a few things you can check to make sure everything is
configured correctly. If none of these tips work, feel free to join
[our slack](https://inviter.co/opa) and ask for help.
The tips below cover the OPA-Kubernetes integration that uses kube-mgmt.
The [OPA Gatekeeper version](https://open-policy-agent.github.io/gatekeeper)
has its own docs.
### Check for the `openpolicyagent.org/kube-mgmt-status` annotation on ConfigMaps containing policies
If you are loading policies into OPA via
[kube-mgmt](https://github.com/open-policy-agent/kube-mgmt) you can check the
`openpolicyagent.org/kube-mgmt-status` annotation on ConfigMaps that contain your
policies. The annotation should be set to `{"status":"ok"}` if the policy was loaded
successfully. If errors occurred during loading (e.g., because the policy
contained a syntax error) the cause will be reported here.
If the annotation is
missing entirely, check the `kube-mgmt` container logs for connection errors
between the container and the Kubernetes API server.
### Check the `kube-mgmt` container logs for error messages
When `kube-mgmt` is healthy, the container logs will be quiet/empty. If you are
trying to enforce policies based on Kubernetes context (e.g., to check for
ingress conflicts) then you need to make sure that `kube-mgmt` can replicate
Kubernetes objects into OPA. If `kube-mgmt` is unable to list/watch resources in
the Kubernetes API server, they will not be replicated into OPA and the policy
will not get enforced.
### Check the `opa` container logs for TLS errors
Communication between the Kubernetes API server and OPA is secured with TLS. If
the CA bundle specified in the webhook configuration is out-of-sync with the
server certificate that OPA is configured with, OPA will log errors indicating a
TLS issue. Verify that the CA bundle specified in the validating or mutating
webhook configurations matches the server certificate you configured OPA to use.
### Check for POST requests in the `opa` container logs
When the Kubernetes API server queries OPA for admission control decisions, it
sends HTTP `POST` requests. If there are no `POST` requests contained in the
`opa` container logs, it indicates that the webhook configuration is wrong or
there is a network connectivity problem between the Kubernetes API server and
OPA.
- If you have access to the Kubernetes API server logs, review them to see if
they indicate the cause.
- If you are running on AWS EKS make sure your security group settings allow
traffic from Kubernetes "master" nodes to the node(s) where OPA is running.
### Ensure the webhook is configured for the proper namespaces
When you create the webhook according to the installation instructions,
it includes a namespaceSelector so that you
can decide which namespaces to ignore.
```
namespaceSelector:
matchExpressions:
- key: openpolicyagent.org/webhook
operator: NotIn
values:
- ignore
```
If OPA seems to not be making the decisions you expect, check if the namespace
is using the label `openpolicyagent.org/webhook: ignore`.
If OPA is making decision on namespaces (like `kube-system`) that you would
prefer OPA would ignore, assign the namespace the label
`openpolicyagent.org/webhook: ignore`.
### Ensure mutating policies construct JSON Patches correctly
If you are using OPA to enforce mutating admission policies you must ensure the
JSON Patch objects you generate escape "/" characters in the JSON Pointer. For
example, if you are generating a JSON Patch that sets annotations like
`acmecorp.com/myannotation` you need to escape the "/" character in the
annotation name using `~1` (per [RFC 6901](https://tools.ietf.org/html/rfc6901#section-3)).
**Correct**:
```json
{
"op": "add",
"path": "/metadata/annotations/acmecorp.com~1myannotation",
"value": "somevalue"
}
```
**Incorrect**:
```json
{
"op": "add",
"path": "/metadata/annotations/acmecorp.com/myannotation",
"value": "somevalue"
}
```
In addition, when your policy generates the response for the Kubernetes API
server, you must use the `base64.encode` built-in function to encode the JSON
Patch objects. DO NOT use the `base64url.encode` function because the Kubernetes
API server will not process it:
```live:patch:module:read_only,hidden
package system
```
**Correct**:
```live:patch/good:module:read_only,openable
main := {
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response": response,
}
response := {
"allowed": true,
"patchType": "JSONPatch",
"patch": base64.encode(json.marshal(patches)) # <-- GOOD: uses base64.encode
}
patches := [
{
"op": "add",
"path": "/metadata/annotations/acmecorp.com~1myannotation",
"value": "somevalue"
}
]
```
**Incorrect**:
```live:patch/bad:module:read_only
main := {
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response": response,
}
response := {
"allowed": true,
"patchType": "JSONPatch",
"patch": base64url.encode(json.marshal(patches)) # <-- BAD: uses base64url.encode
}
patches := [
{
"op": "add",
"path": "/metadata/annotations/acmecorp.com~1myannotation",
"value": "somevalue"
}
]
```
Also, for more examples of how to construct mutating policies and integrating
them with validating policies, see [these examples](https://github.com/open-policy-agent/library/tree/master/kubernetes/mutating-admission)
in https://github.com/open-policy-agent/library.
docs/new/docs/kubernetes/index.md
+268
View File
@@ -0,0 +1,268 @@
---
title: "Overview & Architecture"
---
In Kubernetes, [Admission Controllers](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/)
enforce policies on objects during create, update, and delete operations. Admission
control is fundamental to policy enforcement in Kubernetes.
For example, by deploying OPA as an admission controller you can:
- Require specific labels on all resources.
- Require container images come from the corporate image registry.
- Require all Pods specify resource requests and limits.
- Prevent conflicting Ingress objects from being created.
Admission controllers can also mutate incoming objects. By deploying OPA as a
mutating admission controller you can:
- Inject sidecar containers into Pods.
- Set specific annotations on all resources.
- Rewrite container images to point at the corporate image registry.
- Include node and pod (anti-)affinity selectors on Deployments.
These are just examples of policies you can enforce with admission controllers
and OPA. There are dozens of other policies you will want to enforce in your
Kubernetes clusters for security, cost, and availability reasons.
## What is OPA Gatekeeper?
[OPA Gatekeeper](https://open-policy-agent.github.io/gatekeeper) is a specialized
project providing first-class integration between OPA and Kubernetes. For
background information see this [blog post](https://kubernetes.io/blog/2019/08/06/opa-gatekeeper-policy-and-governance-for-kubernetes)
on kubernetes.io.
OPA Gatekeeper adds the following on top of plain OPA:
- An extensible, parameterized policy library.
- Native Kubernetes CRDs for instantiating the policy library (aka "constraints").
- Native Kubernetes CRDs for extending the policy library (aka "constraint templates").
- Audit functionality.
If you want to kick the tires:
- See the [Installation Instructions](https://open-policy-agent.github.io/gatekeeper/website/docs/install/)
in the README.
- See the
[demo/basic](https://github.com/open-policy-agent/gatekeeper/tree/master/demo/basic)
and
[demo/agilebank](https://github.com/open-policy-agent/gatekeeper/tree/master/demo/agilebank)
directories for examples policies and setup scripts.
**Recommendation**: OPA Gatekeeper is **the go-to project** for using OPA for
Kubernetes admission control. Plain OPA and Kube-mgmt (see below) are alternatives
that can be reached for if you want to use the management features of OPA, such as
status logs, decision logs, and bundles.
## How Does It Work With Plain OPA and Kube-mgmt?
The Kubernetes API Server is configured to query OPA for admission control
decisions when objects (e.g., Pods, Services, etc.) are created, updated, or
deleted.
![Kubernetes Admission Control Flow](./assets/kubernetes-admission-flow.png)
The API Server sends the entire Kubernetes object in the webhook request to OPA.
OPA evaluates the policies it has loaded using the admission review as `input`.
For example, the following policy denies objects that include container images
referring to illegal registries:
```live:container_image:module:openable
package kubernetes.admission
deny contains reason if {
some container
input_containers[container]
not startswith(container.image, "hooli.com/")
reason := "container image refers to illegal registry (must be hooli.com)"
}
input_containers contains container if {
container := input.request.object.spec.containers[_]
}
input_containers contains container if {
container := input.request.object.spec.template.spec.containers[_]
}
```
When `deny` is evaluated with the input defined below the answer is:
```live:container_image:query:hidden
deny
```
```live:container_image:output
```
The `input` document contains the following fields:
- `input.request.kind` specifies the type of the object (e.g., `Pod`, `Service`,
etc.)
- `input.request.operation` specifies the type of the operation, i.e., `CREATE`,
`UPDATE`, `DELETE`, `CONNECT`.
- `input.request.userInfo` specifies the identity of the caller.
- `input.request.object` contains the entire Kubernetes object.
- `input.request.oldObject` specifies the previous version of the Kubernetes
object on `UPDATE` and `DELETE`.
Here is an example of a Pod being created:
```live:container_image:input
{
"kind": "AdmissionReview",
"apiVersion": "admission.k8s.io/v1",
"request": {
"kind": {
"group": "",
"version": "v1",
"kind": "Pod"
},
"resource": {
"group": "",
"version": "v1",
"resource": "pods"
},
"namespace": "opa-test",
"operation": "CREATE",
"userInfo": {
"username": "system:serviceaccount:kube-system:replicaset-controller",
"uid": "439dea65-3e4e-4fa8-b5f8-8fdc4bc7cf53",
"groups": [
"system:serviceaccounts",
"system:serviceaccounts:kube-system",
"system:authenticated"
]
},
"object": {
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"creationTimestamp": "2019-08-13T16:01:54Z",
"generateName": "nginx-7bb7cd8db5-",
"labels": {
"pod-template-hash": "7bb7cd8db5",
"run": "nginx"
},
"name": "nginx-7bb7cd8db5-dbplk",
"namespace": "opa-test",
"ownerReferences": [
{
"apiVersion": "apps/v1",
"blockOwnerDeletion": true,
"controller": true,
"kind": "ReplicaSet",
"name": "nginx-7bb7cd8db5",
"uid": "7b6a307f-d9b4-4b65-a916-5d0b96305e87"
}
],
"uid": "266d2c8b-e43e-42d9-a19c-690bb6103900"
},
"spec": {
"containers": [
{
"image": "nginx",
"imagePullPolicy": "Always",
"name": "nginx",
"resources": {},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"volumeMounts": [
{
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
"name": "default-token-6h4dn",
"readOnly": true
}
]
}
],
"dnsPolicy": "ClusterFirst",
"enableServiceLinks": true,
"priority": 0,
"restartPolicy": "Always",
"schedulerName": "default-scheduler",
"securityContext": {},
"serviceAccount": "default",
"serviceAccountName": "default",
"terminationGracePeriodSeconds": 30,
"tolerations": [
{
"effect": "NoExecute",
"key": "node.kubernetes.io/not-ready",
"operator": "Exists",
"tolerationSeconds": 300
},
{
"effect": "NoExecute",
"key": "node.kubernetes.io/unreachable",
"operator": "Exists",
"tolerationSeconds": 300
}
],
"volumes": [
{
"name": "default-token-6h4dn",
"secret": {
"secretName": "default-token-6h4dn"
}
}
]
},
"status": {
"phase": "Pending",
"qosClass": "BestEffort"
}
},
"oldObject": null
}
}
```
The policies you give to OPA ultimately generate an admission review response
that is sent back to the API Server. Here is an example of the policy decision
sent back to the API Server.
```json
{
"kind": "AdmissionReview",
"apiVersion": "admission.k8s.io/v1",
"response": {
"allowed": false,
"status": {
"message": "container image refers to illegal registry (must be hooli.com)"
}
}
}
```
> The API Server implements a "deny overrides" conflict resolution strategy. If
> any admission controller denies the request, the request is denied (even if
> one of the later admission controllers were to allow the request.)
Policies can be loaded into OPA dynamically via ConfigMap objects using the
[kube-mgmt](https://github.com/open-policy-agent/kube-mgmt) sidecar container.
The kube-mgmt sidecar container can also load any other Kubernetes object into
OPA as JSON under `data`. This lets you enforce policies that rely on an
eventually consistent snapshot of the Kubernetes cluster as context.
![Kubernetes Watchers](./assets/kubernetes-watchers.png)
See the [Policy Authoring](./kubernetes/primer) and [Tutorial: Ingress Validation](./kubernetes/tutorial) pages for more details.
## Additional Resources
See the following pages on [kubernetes.io](https://kubernetes.io) for more
information on admission control:
- [A Guide to Kubernetes Admission
Controllers](https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/)
for a quick primer on admission controllers.
- [Dynamic Admission Control](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/)
for details on configuring external admission controllers.
## Ecosystem Projects
<EcosystemEmbed feature="kubernetes">
Kubernetes is a popular integration with OPA and there are number of projects already in this space
which might work for your use case.
</EcosystemEmbed>
docs/new/docs/kubernetes/primer.md
+544
View File
@@ -0,0 +1,544 @@
---
title: Policy Primer via Examples
---
Read this page if you are new to Kubernetes admission control with OPA and want
to learn how to write policies for Kubernetes. It covers the version
that uses kube-mgmt. The [OPA Gatekeeper version](https://open-policy-agent.github.io/gatekeeper)
has its own docs.
## Writing Policies
To get started, let's look at a common policy: ensure all images come from a
trusted registry.
```live:container_images:module:openable
package kubernetes.admission # line 1
deny contains msg if { # line 2
input.request.kind.kind == "Pod" # line 3
image := input.request.object.spec.containers[_].image # line 4
not startswith(image, "hooli.com/") # line 5
msg := sprintf("image '%v' comes from untrusted registry", [image]) # line 6
}
```
### Packages
In line 1 the `package kubernetes.admission` declaration gives the (hierarchical) name `kubernetes.admission` to the rules in the remainder of the policy. The default installation of OPA as an admission controller assumes your rules are in the package `kubernetes.admission`.
### Deny Rules
For admission control, you write `deny` statements. Order does not matter. (OPA is far more flexible than this, but we recommend writing just `deny` statements to start.) In line 2, the _head_ of the rule `deny contains msg if` says that the admission control request should be rejected and the user handed the error message `msg` if the conditions in the _body_ (the statements between the `{}`) are true.
`deny` is the _set_ of error messages that should be returned to the user. Each rule you write adds to that set of error messages.
For example, suppose you tried to create the Pod below with nginx and mysql images.
```yaml
kind: Pod
apiVersion: v1
metadata:
name: myapp
spec:
containers:
- image: nginx
name: nginx-frontend
- image: mysql
name: mysql-backend
```
The admission review request to be sent to OPA would look like this:
```live:container_images:input
{
"kind": "AdmissionReview",
"request": {
"kind": {
"kind": "Pod",
"version": "v1"
},
"object": {
"metadata": {
"name": "myapp"
},
"spec": {
"containers": [
{
"image": "nginx",
"name": "nginx-frontend"
},
{
"image": "mysql",
"name": "mysql-backend"
}
]
}
}
}
}
```
When the `deny` rule is evaluated with the input above, the answer is:
```live:container_images:query:hidden
```
```live:container_images:output
```
### Input Document
In OPA, `input` is a reserved, global variable whose value is the Kubernetes AdmissionReview object that the API server hands to any admission control webhook.
AdmissionReview objects have many fields. The rule above uses `input.request.kind`, which includes the usual group/version/kind information. The rule also uses `input.request.object`, which is the YAML that the user provided to `kubectl` (augmented with defaults, timestamps, etc.). The full `input` object is 50+ lines of YAML, so below we show just the relevant parts.
```yaml
apiVersion: admission.k8s.io/v1
kind: AdmissionReview
request:
kind:
group:
kind: Pod
version: v1
object:
metadata:
name: myapp
spec:
containers:
- image: nginx
name: nginx-frontend
- image: mysql
name: mysql-backend
```
### Dot Notation
In line 3 `input.request.kind.kind == "Pod"`, the expression `input.request.kind.kind` does the obvious thing: it descends through the YAML hierarchy. The dot (.) operator never throws any errors; if the path does not exist the value of the expression is `undefined`.
```live:container_images/kind:query:merge_down
input.request.kind
```
```live:container_images/kind:output:merge_down
```
```live:container_images/kind/kind:query:merge_down
input.request.kind.kind
```
```live:container_images/kind/kind:output:merge_down
```
```live:container_images/spec:query:merge_down
input.request.object.spec.containers
```
```live:container_images/spec:output
```
### Equality
Lines 3, 4, 6 all use a form of equality. There are 3 forms of equality in OPA.
- `x := 7` declares a local variable `x` and assigns it a value of 7. The compiler throws an error if `x` already has a value.
- `x == 7` returns true if `x` has a value of 7. The compiler throws an error if `x` has no value.
- `x = 7` either assigns the value 7 to `x` if `x` has no value or compares `x`'s value to 7 if it has a value. The compiler never throws an error.
The recommendation for rule-writing is to use `:=` and `==` wherever possible. Rules written with `:=` and `==` are easier to write and to read. `=` is invaluable in more advanced use cases, and outside of rules is the only supported form of equality.
### Arrays
Lines 4-5 find images in the Pod that don't come from the trusted registry. To do that, they use the `[]` operator, which does what you expect: index into the array.
Continuing the example from earlier:
```live:container_images/arrays:query:merge_down
input.request.object.spec.containers[0]
```
```live:container_images/arrays:output:merge_down
```
```live:container_images/arrays/image:query:merge_down
input.request.object.spec.containers[0].image
```
```live:container_images/arrays/image:output
```
The `[]` operators let you use variables to index into the array as well.
```live:container_images/arrays/vars:query:merge_down
i := 0; input.request.object.spec.containers[i]
```
```live:container_images/arrays/vars:output
```
### Iteration
The containers array has an unknown number of elements, so to implement an image registry check you need to iterate over them. Iteration in OPA requires no new syntax. In fact, OPA is always iterating--it's always searching for all variable assignments that make the conditions in the rule true. It's just that sometimes the search is so easy people don't think of it as iteration/search.
To iterate over the indexes in the `input.request.object.spec.containers` array, you just put a variable that has no value in for the index. OPA will do what it always does: find values for that variable that make the conditions true.
OPA detects when there will be multiple answers and displays all the results in a table.
```live:container_images/iteration:query:merge_down
some j; input.request.object.spec.containers[j]
```
```live:container_images/iteration:output
```
Often you don't want to invent new variable names for iteration. OPA provides the special anonymous variable `_` for exactly that reason. So in line (4) `image := input.request.object.spec.containers[_].image` finds all the images in the containers array and assigns each to the `image` variable one at a time.
### Builtins
On line 5 the _builtin_ `startswith` checks if one string is a prefix of the other. The builtin `sprintf` on line 6 formats a string with arguments. OPA has 150+ builtins detailed in [the Policy Reference](../policy-reference/#built-in-functions).
Builtins let you analyze and manipulate:
- Numbers, Strings, Regexs, Networks
- Aggregates, Arrays, Sets
- Types
- Encodings (base64, YAML, JSON, URL, JWT)
- Time
## Testing Policies
When you write policies, you should use the OPA unit-test framework _before_ sending the policies out into the OPA that is running on your cluster. The debugging process will be much quicker and effective. Here's an example test for the policy from the last section.
```live:container_images/test:module:read_only,openable
package kubernetes.test_admission # line 1
import data.kubernetes.admission # line 2
test_image_safety if { # line 3
unsafe_image := { # line 4
"request": {
"kind": {"kind": "Pod"},
"object": {
"spec": {
"containers": [
{"image": "hooli.com/nginx"},
{"image": "busybox"}
]
}
}
}
}
expected := "image 'busybox' comes from untrusted registry"
admission.deny[expected] with input as unsafe_image # line 5
}
```
**Different Package**. On line 1 the `package` directive puts these tests in a different package than admission control policy itself. This is the recommended best practice.
**Import**. On line 2 `import data.kubernetes.admission` allows us to reference the admission control policy using the name `admission` everywhere in the test package. `import` is not strictly necessary--it simply sets up an alias; you could instead reference `data.kubernetes.admission` inside the rules.
**Unit Test**. On line 3 `test_image_safety` defines a unittest. If the rule evaluates to true the test passes; otherwise it fails. When you use the OPA test runner, anything in any package starting with `test` is treated as a test.
**Assignment**. On line 4 `unsafe_image` is the input we want to use for the test. Ideally this would be a real AdmissionReview object, though those are so long that in this example we hand-rolled a partial input.
**Dot for packages**. On line 5 we use the Dot operator on a package. `admission.deny[expected]` runs the `deny` rule(s) in package `admission` and checks if the message is contained in the set defined by `deny`.
**Test Input**. Also on line 5 the stanza `with input as unsafe_image` sets the value of `input` to be `unsafe_image` while evaluating `admission.deny[expected]`.
**Running Tests**. If you've created the files _image-safety.rego_ and _test-image-safety.rego_ in the current directory then you run the tests by naming the files explicitly as shown below or by handing the `opa test` command the directory (and subdirectories) of files to load: `opa test .`
```
$ opa test image-safety.rego test-image-safety.rego
PASS: 1/1
```
## Using Context in Policies
The image-repository example shows an example where you can make a policy decision using just the one JSON/YAML file describing the resource in question. But sometimes you need to know what other resources exist in the cluster to make an allow/deny decision.
For example, it’s possible to accidentally configure two Kubernetes ingresses so that one steals traffic from the other. The policy that prevents conflicting ingresses needs to compare the ingress that’s being created/updated with all of the existing ingresses. Just knowing the new/updated ingress isn't enough information to make an allow/deny decision.
Below is a partial example of the input OPA sees when someone creates an ingress. To avoid conflicts, we want to prevent two ingresses from having the same `request.object.spec.rules.host`. If OPA has only this one ingress configuration it doesn't have enough information to make an allow/deny decision; it also needs the configurations for all of the existing ingresses.
```yaml
apiVersion: admission.k8s.io/v1
kind: AdmissionReview
request:
kind:
group: networking.k8s.io
kind: Ingress
version: v1
object:
metadata:
name: prod
spec:
rules:
- host: initech.com
http:
paths:
- path: /finance
pathType: Prefix
backend:
service:
name: banking
port:
number: 443
```
To avoid conflicting ingresses, you write a policy like the one that follows.
```live:ingress_conflicts:module:read_only
package kubernetes.admission
deny contains msg if {
some namespace, name
input.request.kind.kind == "Ingress" # line 1
newhost := input.request.object.spec.rules[_].host # line 2
oldhost := data.kubernetes.ingresses[namespace][name].spec.rules[_].host # line 3
newhost == oldhost # line 4
input.request.object.metadata.namespace != namespace # line 5
input.request.object.metadata.name != name # line 6
msg := sprintf("ingress host conflicts with ingress %v/%v", [namespace, name]) # line 7
}
```
The first part of the rule you already understand:
- Line (1) checks if the `input` is an Ingress
- Line (2) iterates over all the rules in the `input` ingress and looks up the `host` field for each of its rules.
**Existing K8s Resources** Line (3) iterates over ingresses that already exist in Kubernetes. `data` is a global variable where (among other things) OPA has a record of the current resources inside Kubernetes. The line `oldhost := data.kubernetes.ingresses[namespace][name].spec.rules[_].host` finds all ingresses in all namespaces, iterates over all the `rules` inside each of those and assigns the `host` field to the variable `oldhost`. Whenever `newhost == oldhost`, there's a conflict, and the OPA rule includes an appropriate error message into the `deny` set.
In this case the rule uses explicit variable names `namespace` and `name` for iteration so that it can use those variables again when constructing the error message in line (7).
**Schema Differences**. Both `input` and `data.kubernetes.ingresses[namespace][name]` represent ingresses, but they do it differently.
- `input` is a Kubernetes AdmissionReview object. It includes several fields in addition to the Kubernetes Ingress object itself.
- `data.kubernetes.ingresses[namespace][name]` is a native Kubernetes Ingress object as returned by the API.
Here are two examples.
<SideBySideContainer>
<SideBySideColumn>
```yaml title="data.kubernetes.ingresses[namespace][name]"
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prod
spec:
rules:
- host: initech.com
http:
paths:
- path: /finance
pathType: Prefix
backend:
service:
name: banking
port:
number: 443
```
</SideBySideColumn>
<SideBySideColumn>
```yaml title="admission_review.yaml"
apiVersion: admission.k8s.io/v1
kind: AdmissionReview
request:
kind:
group: networking.k8s.io
kind: Ingress
version: v1
operation: CREATE
userInfo:
groups:
username: alice
object:
metadata:
name: prod
spec:
rules:
- host: initech.com
http:
paths:
- path: /finance
pathType: Prefix
backend:
service:
name: banking
port:
number: 443
```
</SideBySideColumn>
</SideBySideContainer>
## Detailed Admission Control Flow
This section provides a detailed explanation of the admission control flow
introduced in the [Introduction](..) page.
It starts with someone (or something) running `kubectl` (or sending a request to
the API server.) For example, a user might run `kubectl create -f pod.yaml`:
```yaml title="pod.yaml"
kind: Pod
apiVersion: v1
metadata:
name: nginx
labels:
app: nginx
spec:
containers:
- image: nginx
name: nginx
```
When the request reaches the API server it's authenticated and authorized and
processed by the admission controllers. When the API server's Webhook admission
controller executes, the API server sends a webhook request to OPA containing an
**AdmissionReview** object.
**AdmissionReview**:
```yaml
apiVersion: admission.k8s.io/v1
kind: AdmissionReview
request:
kind:
group: ""
kind: Pod
version: v1
namespace: opa
object:
metadata:
creationTimestamp: "2018-10-27T02:12:20Z"
labels:
app: nginx
name: nginx
namespace: opa
uid: bbfee96d-d98d-11e8-b280-080027868e77
spec:
containers:
- image: nginx
imagePullPolicy: Always
name: nginx
resources: {}
terminationMessagePath: "/dev/termination-log"
terminationMessagePolicy: File
volumeMounts:
- mountPath: "/var/run/secrets/kubernetes.io/serviceaccount"
name: default-token-tm9v8
readOnly: true
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: default-token-tm9v8
secret:
secretName: default-token-tm9v8
status:
phase: Pending
qosClass: BestEffort
oldObject:
operation: CREATE
resource:
group: ""
resource: pods
version: v1
uid: 8d836dfd-e0c0-4490-93ba-85ed4a04261e
userInfo:
groups:
- system:masters
- system:authenticated
username: minikube-user
```
Typically the API server is configured (via `ValidatingWebhookConfiguration` or
`MutatingWebhookConfiguration` objects) to query OPA without providing the name
of a decision. For example:
```http
POST / HTTP/1.1
Content-Type: application/json
```
```json
{
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"request": ...
}
```
When OPA receives the webhook request, it binds the payload to the `input`
document and generates the default decision: `system.main`. The `system.main`
decision is defined by a rule that evaluates all of the admission control
policies that have been loaded into OPA.
As the administrator responsible for deploying OPA, you have full control over
the `system.main` decision (i.e., it is just another Rego policy.) A basic
implementation of the `system.main` policy simply evaluates all deny rules that
have been loaded into OPA and unions the results:
```rego
package system
import data.kubernetes.admission
main := {
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response": response,
}
default uid := ""
uid := input.request.uid
response := {
"allowed": false,
"uid": uid,
"status": {"message": reason},
} if {
reason := concat(", ", admission.deny)
reason != ""
}
else := {"allowed": true, "uid": uid}
```
The `system.main` policy MUST generate an **AdmissionReview** object containing
a response that the API server can interpret. If the request should be allowed,
the `response.allowed` field should be true. Otherwise, the `response.allowed`
field should be set to `false` and the `response.status.message` field should be
set to include an error message that indicates why the request is being
rejected. The error message will be returned to the API server caller (e.g., the
user running `kubectl`). Often the error message is the concatenation of all the
messages in the `deny` set defined above.
For example, with the input and Image Registry Safety examples above, the
response from OPA would be:
```yaml
apiVersion: admission.k8s.io/v1
kind: AdmissionReview
response:
uid: 8d836dfd-e0c0-4490-93ba-85ed4a04261e
allowed: false
status:
message: "image fails to come from trusted registry: nginx"
```
For more detail on how Kubernetes Admission Control works, see [this blog post](https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/)
on kubernetes.io.
docs/new/docs/kubernetes/tutorial.md
+559
View File
@@ -0,0 +1,559 @@
---
title: "Tutorial: Ingress Validation"
---
This tutorial shows how to deploy OPA as an admission controller from scratch.
It covers the OPA-kubernetes version that uses kube-mgmt.
The [OPA Gatekeeper version](https://open-policy-agent.github.io/gatekeeper) has its own docs.
For the purpose of the tutorial we will deploy two policies that ensure:
- Ingress hostnames must be on allowlist on the Namespace containing the Ingress.
- Two ingresses in different namespaces must not have the same hostname.
> 💡 Kubernetes does not guarantee consistency across resources. If two
> ingresses are created in parallel, there is no guarantee that OPA (or any
> other admission controller) will observe the creation of one ingress before
> the other. This means that it's not possible to enforce these policies during
> admission control 100% of the time. There will be a small window of time
> (usually on the order of milliseconds) when the eventually consistent cache
> inside of OPA (or any other admission controller) is out-of-date. To catch
> these violations we recommend you periodically audit the state of the cluster
> against your policies. Offline auditing is one of the features provided by the
> [OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper) project.
## Prerequisites
This tutorial requires Kubernetes 1.20 or later. To run the tutorial locally ensure you start a cluster with Kubernetes
version 1.20+, we recommend using [minikube](https://kubernetes.io/docs/getting-started-guides/minikube) or
[KIND](https://kind.sigs.k8s.io/).
## Steps
### 1. Enable recommended Kubernetes Admission Controllers
To implement admission control rules that validate Kubernetes resources during create, update, and delete operations,
you must enable the [ValidatingAdmissionWebhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook)
when the Kubernetes API server is started. The ValidatingAdmissionWebhook admission controller is included in
the [recommended set of admission controllers to enable](https://kubernetes.io/docs/admin/admission-controllers/#is-there-a-recommended-set-of-admission-controllers-to-use)
Start minikube:
```bash
minikube start
```
Make sure that the minikube ingress addon is enabled:
```bash
minikube addons enable ingress
```
### 2. Create a new Namespace to deploy OPA into
```bash
kubectl create namespace opa
```
Configure `kubectl` to use this namespace:
```bash
kubectl config set-context opa-tutorial --user minikube --cluster minikube --namespace opa
kubectl config use-context opa-tutorial
```
### 3. Create TLS credentials for OPA
Communication between Kubernetes and OPA must be secured using TLS. To configure TLS, use `openssl` to create a
certificate authority (CA) and certificate/key pair for OPA:
```bash
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -sha256 -key ca.key -days 100000 -out ca.crt -subj "/CN=admission_ca"
```
Generate the TLS key and certificate for OPA:
```bash
cat >server.conf <<EOF
[ req ]
prompt = no
req_extensions = v3_ext
distinguished_name = dn
[ dn ]
CN = opa.opa.svc
[ v3_ext ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = DNS:opa.opa.svc,DNS:opa.opa.svc.cluster,DNS:opa.opa.svc.cluster.local
EOF
```
```bash
openssl genrsa -out server.key 2048
openssl req -new -key server.key -sha256 -out server.csr -extensions v3_ext -config server.conf
openssl x509 -req -in server.csr -sha256 -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 100000 -extensions v3_ext -extfile server.conf
```
> Note: the Common Name value and Subject Alternative Name you give to openssl MUST match the name of the OPA service created below.
Create a Secret to store the TLS credentials for OPA:
```bash
kubectl create secret tls opa-server --cert=server.crt --key=server.key --namespace opa
```
### 4. Define OPA policy
Let's define a couple of policies to test admission control. First create a new folder to store our policies:
```bash
mkdir policies && cd policies
```
#### Policy 1: Restrict Hostnames
Create a policy that restricts the hostnames that an ingress can use. Only hostnames matching the specified regular
expressions will be allowed.
```rego title="ingress-allowlist.rego"
package kubernetes.admission
import data.kubernetes.namespaces
operations := {"CREATE", "UPDATE"}
deny contains msg if {
input.request.kind.kind == "Ingress"
operations[input.request.operation]
host := input.request.object.spec.rules[_].host
not fqdn_matches_any(host, valid_ingress_hosts)
msg := sprintf("invalid ingress host %q", [host])
}
valid_ingress_hosts := {host |
allowlist := namespaces[input.request.namespace].metadata.annotations["ingress-allowlist"]
hosts := split(allowlist, ",")
host := hosts[_]
}
fqdn_matches_any(str, patterns) if {
fqdn_matches(str, patterns[_])
}
fqdn_matches(str, pattern) if {
pattern_parts := split(pattern, ".")
pattern_parts[0] == "*"
suffix := trim(pattern, "*.")
endswith(str, suffix)
}
fqdn_matches(str, pattern) if {
not contains(pattern, "*")
str == pattern
}
```
#### Policy 2: Prohibit Hostname Conflicts
Now let's define another policy to test admission control. The following policy prevents Ingress objects in different
namespaces from sharing the same hostname.
**ingress-conflicts.rego**:
```rego title="ingress-conflicts.rego"
package kubernetes.admission
import data.kubernetes.ingresses
deny contains msg if {
some other_ns, other_ingress
input.request.kind.kind == "Ingress"
input.request.operation == "CREATE"
host := input.request.object.spec.rules[_].host
ingress := ingresses[other_ns][other_ingress]
other_ns != input.request.namespace
ingress.spec.rules[_].host == host
msg := sprintf("invalid ingress host %q (conflicts with %v/%v)", [host, other_ns, other_ingress])
}
```
#### Combine Policies
Let's define a main policy that imports the [Restrict Hostnames](#policy-1-restrict-hostnames) and
[Prohibit Hostname Conflicts](#policy-2-prohibit-hostname-conflicts) policies and provides an overall policy decision.
```rego title="main.rego"
package system
import data.kubernetes.admission
main := {
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response": response,
}
default uid := ""
uid := input.request.uid
response := {
"allowed": false,
"uid": uid,
"status": {"message": reason},
} if {
reason = concat(", ", admission.deny)
reason != ""
}
else := {"allowed": true, "uid": uid}
```
> ⚠️ When OPA receives a request, it executes a query against the document defined `data.system.main` by default.
### 5. Build and Publish OPA Bundle
Build an OPA bundle containing policies defined in the previous step. In our setup, OPA will download policies from the
bundle service and the `kube-mgmt` container will load Kubernetes resources into OPA. Since we load policy and data into
OPA from multiple sources, we need to scope the bundle to a subset of OPA’s policy and data cache by defining a manifest.
More information about this can be found [here](../management-bundles#multiple-sources-of-policy-and-data). Run the
following commands in the `policies` folder created in the previous step.
```bash
cat > .manifest <<EOF
{
"roots": ["kubernetes/admission", "system"]
}
EOF
```
```bash
opa build -b .
```
We will now serve the OPA bundle using Nginx.
```bash
docker run --rm --name bundle-server -d -p 8888:80 -v ${PWD}:/usr/share/nginx/html:ro nginx:latest
```
### 6. Deploy OPA as an Admission Controller
Next, use the file below to deploy OPA as an admission controller.
```yaml title="admission-controller.yaml"
# Grant OPA/kube-mgmt read-only access to resources. This lets kube-mgmt
# replicate resources into OPA so they can be used in policies.
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: opa-viewer
roleRef:
kind: ClusterRole
name: view
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
name: system:serviceaccounts:opa
apiGroup: rbac.authorization.k8s.io
---
# Define role for OPA/kube-mgmt to update configmaps with policy status.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: opa
name: configmap-modifier
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["update", "patch"]
---
# Grant OPA/kube-mgmt role defined above.
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: opa
name: opa-configmap-modifier
roleRef:
kind: Role
name: configmap-modifier
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
name: system:serviceaccounts:opa
apiGroup: rbac.authorization.k8s.io
---
kind: Service
apiVersion: v1
metadata:
name: opa
namespace: opa
spec:
selector:
app: opa
ports:
- name: https
protocol: TCP
port: 443
targetPort: 8443
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: opa
namespace: opa
name: opa
spec:
replicas: 1
selector:
matchLabels:
app: opa
template:
metadata:
labels:
app: opa
name: opa
spec:
containers:
# WARNING: OPA is NOT running with an authorization policy configured. This
# means that clients can read and write policies in OPA. If you are
# deploying OPA in an insecure environment, be sure to configure
# authentication and authorization on the daemon. See the Security page for
# details: https://www.openpolicyagent.org/docs/security.html.
- name: opa
image: openpolicyagent/opa:{{< current_docker_version >}}
args:
- "run"
- "--server"
- "--tls-cert-file=/certs/tls.crt"
- "--tls-private-key-file=/certs/tls.key"
- "--addr=0.0.0.0:8443"
- "--addr=http://127.0.0.1:8181"
- "--set=services.default.url=http://host.minikube.internal:8888"
- "--set=bundles.default.resource=bundle.tar.gz"
- "--log-format=json-pretty"
- "--set=status.console=true"
- "--set=decision_logs.console=true"
volumeMounts:
- readOnly: true
mountPath: /certs
name: opa-server
readinessProbe:
httpGet:
path: /health?plugins&bundle
scheme: HTTPS
port: 8443
initialDelaySeconds: 3
periodSeconds: 5
livenessProbe:
httpGet:
path: /health
scheme: HTTPS
port: 8443
initialDelaySeconds: 3
periodSeconds: 5
- name: kube-mgmt
image: openpolicyagent/kube-mgmt:2.0.1
args:
- "--replicate-cluster=v1/namespaces"
- "--replicate=networking.k8s.io/v1/ingresses"
volumes:
- name: opa-server
secret:
secretName: opa-server
```
> ⚠️ If using `kind` to run a local Kubernetes cluster, the bundle service URL should be `http://host.docker.internal:8888`.
```bash
kubectl apply -f admission-controller.yaml
```
When OPA starts, the `kube-mgmt` container will load Kubernetes Namespace and Ingress objects into OPA. You can
configure the sidecar to load any kind of Kubernetes object into OPA. The sidecar establishes watches on the
Kubernetes API server so that OPA has access to an eventually consistent cache of Kubernetes objects.
Next, generate the manifest that will be used to register OPA as an admission controller. This webhook will ignore
any namespace with the label `openpolicyagent.org/webhook=ignore`.
```bash
cat > webhook-configuration.yaml <<EOF
kind: ValidatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1
metadata:
name: opa-validating-webhook
webhooks:
- name: validating-webhook.openpolicyagent.org
namespaceSelector:
matchExpressions:
- key: openpolicyagent.org/webhook
operator: NotIn
values:
- ignore
rules:
- operations: ["CREATE", "UPDATE"]
apiGroups: ["*"]
apiVersions: ["*"]
resources: ["*"]
clientConfig:
caBundle: $(cat ca.crt | base64 | tr -d '\n')
service:
namespace: opa
name: opa
admissionReviewVersions: ["v1"]
sideEffects: None
EOF
```
The generated configuration file includes a base64 encoded representation of the CA certificate created in [Step 3](#3-create-tls-credentials-for-opa)
so that TLS connections can be established between the Kubernetes API server and OPA.
Next label `kube-system` and the `opa` namespace so that OPA does not control the resources in those namespaces.
```bash
kubectl label ns kube-system openpolicyagent.org/webhook=ignore
kubectl label ns opa openpolicyagent.org/webhook=ignore
```
Finally, register OPA as an admission controller:
```bash
kubectl apply -f webhook-configuration.yaml
```
You can follow the OPA logs to see the webhook requests being issued by the Kubernetes API server:
```
# ctrl-c to exit
kubectl logs -l app=opa -c opa -f
```
### 7. Exercise Restrict Hostnames policy
Now let's exercise the [Restrict Hostnames](#policy-1-restrict-hostnames) policy by creating two new namespaces.
```yaml title="qa-namespace.yaml"
apiVersion: v1
kind: Namespace
metadata:
annotations:
ingress-allowlist: "*.qa.acmecorp.com,*.internal.acmecorp.com"
name: qa
```
```yaml title="production-namespace.yaml"
apiVersion: v1
kind: Namespace
metadata:
annotations:
ingress-allowlist: "*.acmecorp.com"
name: production
```
```bash
kubectl create -f qa-namespace.yaml
kubectl create -f production-namespace.yaml
```
Next, define two Ingress objects. One of the Ingress objects will be permitted
and the other will be rejected.
```yaml title="ingress-ok.yaml"
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-ok
spec:
rules:
- host: signin.acmecorp.com
http:
paths:
- pathType: ImplementationSpecific
path: /
backend:
service:
name: nginx
port:
number: 80
```
```yaml title="ingress-bad.yaml"
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-bad
spec:
rules:
- host: acmecorp.com
http:
paths:
- pathType: ImplementationSpecific
path: /
backend:
service:
name: nginx
port:
number: 80
```
Finally, try to create both Ingress objects:
```bash
kubectl create -f ingress-ok.yaml -n production
kubectl create -f ingress-bad.yaml -n qa
```
The second Ingress is rejected because its hostname does not match the allowlist in the `qa` namespace.
It will report an error as follows:
```
Error from server: error when creating "ingress-bad.yaml": admission webhook "validating-webhook.openpolicyagent.org"
denied the request: invalid ingress host "acmecorp.com"
```
### 8. Exercise Prohibit Hostname Conflicts policy
Test the [Prohibit Hostname Conflicts](#policy-2-prohibit-hostname-conflicts) policy by verifying that you cannot
create an Ingress in another namespace with the same hostname as the one created earlier.
```yaml title="staging-namespace.yaml"
apiVersion: v1
kind: Namespace
metadata:
annotations:
ingress-allowlist: "*.acmecorp.com"
name: staging
```
```bash
kubectl create -f staging-namespace.yaml
```
```bash
kubectl create -f ingress-ok.yaml -n staging
```
The above command will report an error as follows:
```
Error from server (BadRequest): error when creating "ingress-ok.yaml": admission webhook
"validate.nginx.ingress.kubernetes.io" denied the request: host "signin.acmecorp.com" and
path "/" is already defined in ingress production/ingress-ok
```
## Wrap Up
Congratulations for finishing the tutorial!
This tutorial showed how you can leverage OPA to enforce admission control
decisions in Kubernetes clusters without modifying or recompiling any
Kubernetes components. Furthermore, with OPA's [Bundle](../management-bundles) feature policies can be
periodically downloaded from remote servers to satisfy changing operational requirements.
For more information about deploying OPA on top of Kubernetes, see
[Deployments - Kubernetes](../deployments#kubernetes).
docs/new/docs/management-bundles/index.md
+1420
View File
File diff suppressed because it is too large Load Diff
docs/new/docs/management-bundles/thumbprint.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 123 KiB

docs/new/docs/management-decision-logs.md
+308
View File
@@ -0,0 +1,308 @@
---
title: "Decision Logs"
---
OPA can periodically report decision logs to remote HTTP servers, using custom
plugins, or to the console output; or any combination thereof.
The decision logs contain events that describe policy queries. Each event includes
the policy that was queried, the input to the query, bundle metadata, and other
information that enables auditing and offline debugging of policy decisions.
When decision logging is enabled the OPA server will include a `decision_id`
field in API calls that return policy decisions.
See the [Configuration Reference](./configuration) for configuration details.
### Decision Log Service API
OPA expects the service to expose an API endpoint that will receive decision logs.
```http
POST /[<decision_logs.resource>] HTTP/1.1
Content-Encoding: gzip
Content-Type: application/json
```
The resource field is an optional configuration that can be used to route logs
to a specific endpoint in the service by defining the full path. If the resource path is not configured on the agent,
updates will be sent to `/logs`.
The message body contains a gzip compressed JSON array. Each array element (event)
represents a policy decision returned by OPA.
```json
[
{
"labels": {
"app": "my-example-app",
"id": "1780d507-aea2-45cc-ae50-fa153c8e4a5a",
"version": "{{< current_version >}}"
},
"decision_id": "4ca636c1-55e4-417a-b1d8-4aceb67960d1",
"bundles": {
"authz": {
"revision": "W3sibCI6InN5cy9jYXRhbG9nIiwicyI6NDA3MX1d"
}
},
"path": "http/example/authz/allow",
"input": {
"method": "GET",
"path": "/salary/bob"
},
"result": "true",
"requested_by": "[::1]:59943",
"timestamp": "2018-01-01T00:00:00.000000Z"
}
]
```
Decision log updates contain the following fields:
| Field | Type | Description |
| ---------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `[_].labels` | `object` | Set of key-value pairs that uniquely identify the OPA instance. |
| `[_].decision_id` | `string` | Unique identifier generated for each decision for traceability. |
| `[_].trace_id` | `string` | Unique identifier of a trace generated for each incoming request for traceability. This is a hex string representation compliant with the W3C trace-context specification. See more at https://www.w3.org/TR/trace-context/#trace-id. |
| `[_].span_id` | `string` | Unique identifier of a span in a trace to assist traceability. This is a hex string representation compliant with the W3C trace-context specification. See more at https://www.w3.org/TR/trace-context/#parent-id. |
| `[_].bundles` | `object` | Set of key-value pairs describing the bundles which contained policy used to produce the decision. |
| `[_].bundles[_].revision` | `string` | Revision of the bundle at the time of evaluation. |
| `[_].path` | `string` | Hierarchical policy decision path, e.g., `/http/example/authz/allow`. Receivers should tolerate slash-prefixed paths. |
| `[_].query` | `string` | Ad-hoc Rego query received by Query API. |
| `[_].input` | `any` | Input data provided in the policy query. |
| `[_].result` | `any` | Policy decision returned to the client, e.g., `true` or `false`. |
| `[_].requested_by` | `string` | Identifier for client that executed policy query, e.g., the client address. |
| `[_].request_context.http.headers` | `object` | Set of key-value pairs describing HTTP headers and their corresponding values. The header keys in this object are specified by the user as part of the decision log configuration. The values in this object represent a list of values associated with the given header key. |
| `[_].timestamp` | `string` | RFC3999 timestamp of policy decision. |
| `[_].metrics` | `object` | Key-value pairs of [performance metrics](./rest-api#performance-metrics). |
| `[_].erased` | `array[string]` | Set of JSON Pointers specifying fields in the event that were erased. |
| `[_].masked` | `array[string]` | Set of JSON Pointers specifying fields in the event that were masked. |
| `[_].nd_builtin_cache` | `object` | Key-value pairs of non-deterministic builtin names, paired with objects specifying the input/output mappings for each unique invocation of that builtin during policy evaluation. Intended for use in debugging and decision replay. Receivers will need to decode the JSON using Rego's JSON decoders. |
| `[_].req_id` | `number` | Incremental request identifier, and unique only to the OPA instance, for the request that started the policy query. The attribute value is the same as the value present in others logs (request, response, and print) and could be used to correlate them all. This attribute will be included just when OPA runtime is initialized in server mode and the log level is equal to or greater than info. |
If the decision log was successfully uploaded to the remote service, it should respond with an HTTP 2xx status. If the
service responds with a non-2xx status, OPA will requeue the last chunk containing decision log events and upload it
during the next upload event. OPA also performs an exponential backoff to calculate the delay in uploading the next chunk
when the remote service responds with a non-2xx status.
OPA periodically uploads decision logs to the remote service. In order to conserve network and memory resources, OPA
attempts to fill up each upload chunk with as many events as possible while respecting the user-specified
`upload_size_limit_bytes` config option. OPA defines an adaptive (`soft`) limit that acts as a measure for encoding
as many decisions into each chunk as possible. It uses the below algorithm to optimize the number of log events to
include in a chunk. The algorithm features three phases namely:
`Scale Up`: If the current chunk size is within 90% of the user-configured (`hard`) limit, exponentially increase the
soft limit. The exponential function is 2^x where x has a minimum value of 1
`Scale Down`: If the current chunk size exceeds the hard limit, decrease the soft limit and re-encode the decisions in
the last chunk.
`Equilibrium`: If the chunk size is between 90% and 100% of the user-configured limit, maintain soft limit value.
When an event containing `nd_builtin_cache` cannot fit into a chunk smaller than `upload_size_limit_bytes`, OPA will
drop the `nd_builtin_cache` key from the event, and will retry encoding the chunk without the non-deterministic
builtins cache information. This best-effort approach ensures that OPA reports decision log events as much as possible,
and bounds how large decision log events can get. This size-bounding is necessary, because some non-deterministic builtins
(such as `http.send`) can increase the decision log event size by a potentially unbounded amount.
### Local Decision Logs
Local console logging of decisions can be enabled via the `console` config option.
This does not require any remote server. Example of minimal config to enable:
```yaml
decision_logs:
console: true
```
This will dump all decisions to the console. See
[Configuration Reference](./configuration) for more details.
### Masking Sensitive Data
Policy queries may contain sensitive information in the `input` document that
must be removed or modified before decision logs are uploaded to the remote API
(e.g., usernames, passwords, etc.) Similarly, parts of the policy decision itself may
be considered sensitive.
By default, OPA queries the `data.system.log.mask` path prior to encoding and
uploading decision logs or calling custom decision log plugins.
OPA provides the decision log event as input to the policy query and expects
the query to return a set of JSON Pointers that refer to fields in the decision
log event to either **erase** or **modify**.
For example, assume OPA is queried with the following `input` document:
```json
{
"resource": "user",
"name": "bob",
"password": "passw0rd"
}
```
To **remove** the `password` field from decision log events related to "user"
resources, supply the following policy to OPA:
```ruby
package system.log
mask contains "/input/password" if {
# OPA provides the entire decision log event as input to the masking policy.
# Refer to the original input document under input.input.
input.input.resource == "user"
}
# To mask certain fields unconditionally, omit the rule body.
mask contains "/input/ssn"
```
When the masking policy generates one or more JSON Pointers, they will be erased
from the decision log event. The erased paths are recorded on the event itself:
```json
{
"decision_id": "b4638167-7fcb-4bc7-9e80-31f5f87cb738",
"erased": [
"/input/password",
"/input/ssn"
],
"input": {
"name": "bob",
"resource": "user"
},
------------------------- 8< -------------------------
"path": "system/main",
"requested_by": "127.0.0.1:36412",
"result": true,
"timestamp": "2019-06-03T20:07:16.939402185Z"
}
```
There are a few restrictions on the JSON Pointers that OPA will erase:
- Pointers must be prefixed with `/input`, `/result`, or `/nd_builtin_cache`.
- Pointers may point to undefined data. For example `/input/name/first` in the
example above would be undefined. Masking operations on undefined pointers are
ignored.
- Pointers can also refer to arrays both as part of the path and as the last
element in the path. For example, both `/input/users/0/name` and
`/input/users/0` would be valid.
In order to **modify** the contents of an input field, the **mask** rule may utilize the following format.
- `"op"` -- The operation to apply when masking. All operations are done at the
path specified. Valid options include:
| op | Description |
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `"remove"` | The `"path"` specified will be removed from the resulting log message. The `"value"` mask field is ignored for `"remove"` operations. |
| `"upsert"` | The `"value"` will be set at the specified `"path"`. If the field exists it is overwritten, if it does not exist it will be added to the resulting log message. |
- `"path"` -- A JSON pointer path to the field to perform the operation on.
Optional Fields:
- `"value"` -- Only required for `"upsert"` operations.
> This is processed for every decision being logged, so be mindful of
> performance when performing complex operations in the mask body, eg. crypto
> operations
```ruby
package system.log
mask contains {"op": "upsert", "path": "/input/password", "value": "**REDACTED**"} if {
# conditionally upsert password if it existed in the original event
input.input.password
}
```
To always **upsert** a value, even if it didn't exist in the original event,
the following rule format can be used.
```ruby
package system.log
# always upsert, no conditions in rule body
mask contains {"op": "upsert", "path": "/input/password", "value": "**REDACTED**"}
```
The result of this mask operation on the decision log event produces
the following output. Notice that the **mask** event field exists
to track **remove** vs **upsert** mask operations.
```json
{
"decision_id": "b4638167-7fcb-4bc7-9e80-31f5f87cb738",
"erased": [
"/input/ssn"
],
"masked": [
"/input/password"
],
"input": {
"name": "bob",
"resource": "user",
"password": "**REDACTED**"
},
------------------------- 8< -------------------------
"path": "system/main",
"requested_by": "127.0.0.1:36412",
"result": true,
"timestamp": "2019-06-03T20:07:16.939402185Z"
}
```
### Drop Decision Logs
Drop rules filters all decisions from logging where the rule evaluates to `true`.
This rule will drop all requests to the _allow_ rule in the _kafka_ package, that returned _true_:
```live:drop_rule_example/kafka_allow_rule:module:read_only
package system.log
drop if {
input.path == "kafka/allow"
input.result == true
}
```
Log only requests for _delete_ and _alter_ operations
(Kafka with the [opa-kafka-plugin](https://github.com/StyraInc/opa-kafka-plugin)):
```live:drop_rule_example/log_only_delete_alter_operations:module:read_only
package system.log
drop if {
input.path == "kafka/allow"
not input.input.action.operation in {"DELETE", "ALTER"}
}
```
The name of the drop rules by default is `drop` in the package `system.log`. It can be changed with the configuration
property `decision_logs.drop_decision`.
```yaml
decision_logs:
drop_decision: /system/log/drop
```
### Rate Limiting Decision Logs
There are scenarios where OPA may be uploading decisions faster than what the remote service is able to consume. Although
OPA provides a user-specified buffer size limit in bytes, it may be difficult to determine the ideal buffer size that will
allow the service to consume logs without being overwhelmed. The `max_decisions_per_second` config option allows users
to set the maximum number of decision log events to buffer per second. OPA will drop events if the rate limit is exceeded.
This option provides users more control over how OPA buffers log events and is an effective mechanism to make sure the
service can successfully process incoming log events.
## Ecosystem Projects
Decision Logging is an important feature of OPA which supports, in particular, auditing and debugging. The following OPA
ecosystem projects implement functionality related to Decision Logging:
<EcosystemEmbed feature="decision-logging">
These projects implement decision logging functionality.
</EcosystemEmbed>
docs/new/docs/management-discovery.md
+314
View File
@@ -0,0 +1,314 @@
---
title: "Discovery"
---
OPA can be configured to download bundles of policy and data, report status, and
upload decision logs to remote endpoints. The discovery feature helps you
centrally manage the OPA configuration for these features. You should use the
discovery feature if you want to avoid managing OPA configuration updates in
a number of different locations.
When the discovery feature is enabled, OPA will periodically download a
_discovery bundle_. Like regular bundles, the discovery bundle may contain JSON
and Rego files. OPA will evaluate the data and policies contained in the
discovery bundle to generate the rest of the configuration. There are two main
ways to structure the discovery bundle:
1. Include static JSON configuration files that define the OPA configuration.
2. Include Rego files that can be evaluated to produce the OPA configuration.
> If you need OPA to select which policy to download dynamically (e.g., based on
> environment variables like the region where OPA is running), use the second
> option.
If discovery is enabled, other features like bundle downloading and status
reporting **can** be configured manually. In case of conflicts, the bootstrap configuration
for plugins would override the discovered configuration. **In general, the bootstrap configuration
overrides the discovered configuration.**
See the [Configuration Reference](./configuration) for configuration details.
### Discovery Service API
OPA expects the service to expose an API endpoint that serves bundles.
```http
GET /<service_url>/<discovery.resource> HTTP/1.1
```
If the bundle exists, the server should respond with an HTTP 200 OK status
followed by a gzipped tarball in the message body.
```http
HTTP/1.1 200 OK
Content-Type: application/gzip
```
You can enable discovery with an OPA configuration file similar to the example
below. In some places in the documentation, the initial configuration provided
to OPA is referred to as the "boot configuration".
```yaml
services:
acmecorp:
url: https://example.com/control-plane-api/v1
credentials:
bearer:
token: "bGFza2RqZmxha3NkamZsa2Fqc2Rsa2ZqYWtsc2RqZmtramRmYWxkc2tm"
discovery:
service: acmecorp
resource: /configuration/example/discovery.tar.gz
```
Using the boot configuration above, OPA will fetch discovery bundles from:
```
https://example.com/control-plane-api/v1/configuration/example/discovery.tar.gz
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
services[discovery.service].url discovery.resource
```
> The `discovery.resource` field defaults to `bundles/<discovery.name>`. The default
> is convenient if you want to serve discovery bundles and normal bundles from the same API
> endpoint. If only one service is defined, there is no need to set `discovery.service`.
> The optional `discovery.signing` field can be used to specify the `keyid` and `scope` that should be used
> for verifying the signature of the discovery bundle. See [this](#discovery-bundle-signature) section for details.
OPA generates it's subsequent configuration by querying the Rego and JSON files
contained inside the discovery bundle. The default query is `data` however this
can be overridden by specifying the `discovery.decision`.
```yaml
services:
- name: acmecorp
url: https://example.com/control-plane-api/v1
credentials:
bearer:
token: "bGFza2RqZmxha3NkamZsa2Fqc2Rsa2ZqYWtsc2RqZmtramRmYWxkc2tm"
discovery:
name: example
resource: /configuration/example/discovery.tar.gz
decision: example/discovery
```
OPA executes the following query:
```
data.example.discovery
```
If the discovery bundle contained the following Rego file:
```ruby
package example
discovery := {
"bundles": {
"main": {
"service": "acmecorp",
"resource": bundle_name
},
},
"default_decision": "acmecorp/httpauthz/allow"
}
bundle_name := "acmecorp/httpauthz"
```
The subsequent configuration would be:
```json
{
"bundles": {
"main": {
"service": "acmecorp",
"resource": "acmecorp/httpauthz"
}
},
"default_decision": "acmecorp/httpauthz/allow"
}
```
The discovery bundle contents above are essentially static. The same result
could be achieved by constructing the discovery bundle with a static JSON file:
```json
{
"example": {
"discovery": {
"bundles": {
"main": {
"service": "acmecorp",
"resource": "acmecorp/httpauthz"
}
},
"default_decision": "acmecorp/httpauthz/allow"
}
}
}
```
> For an example of how to configure OPA dynamically see the [Example](#example)
> section below.
The subsequent configuration does not have to specify `services` or include a
reference to a service in the `bundle`, `status,` or `decision_log` sections. If
the either the `services` or references to services are missing, OPA will
default them to the value from the boot configuration.
### Example
Let's see an example of how the discovery feature can be used to dynamically
configure an OPA to download one of two bundles based on a label in the boot
configuration. Let's say the label `region` indicates the region in which the
OPA is running and it's value will decide the bundle to download.
Below is a policy file which generates an OPA configuration.
**example.rego**
```ruby
package discovery
config := {
"bundles": {
"main": {
"service": "acmecorp",
"resource": bundle_name # line 7
}
}
}
rt := opa.runtime()
region := rt.config.labels.region
bundle_name := region_bundle[region]
# region-bundle information
region_bundle := {
"US": "example/test1/p",
"UK": "example/test2/p"
}
```
The `bundle_name` variable in `line 7` of the above policy will be dynamically
selected based on the value of the label `region`. So if an OPA was started
with `region: "US"`, then the `bundle_name` will be `example/test1/p`.
Start an OPA with a boot configuration as shown below:
**opa-config.yaml**
```yaml
services:
- name: acmecorp
url: https://example.com/control-plane-api/v1
credentials:
bearer:
token: "bGFza2RqZmxha3NkamZsa2Fqc2Rsa2ZqYWtsc2RqZmtramRmYWxkc2tm"
discovery:
resource: bundles/discovery.tar.gz
decision: discovery/config
labels:
region: "US"
```
Run OPA:
```bash
opa run -s -c opa-config.yaml
```
You should see a log like below, which shows the bundle being downloaded. In
this case, the bundle name is `example/test1/p` as `region` is `US`.
```raw
INFO Bundle downloaded and activated successfully. name=example/test1/p plugin=bundle
```
Now start another OPA with a boot configuration as shown below. Notice the
`region` is `UK`:
**opa-config.yaml**
```yaml
services:
- name: acmecorp
url: https://example.com/control-plane-api/v1
credentials:
bearer:
token: "bGFza2RqZmxha3NkamZsa2Fqc2Rsa2ZqYWtsc2RqZmtramRmYWxkc2tm"
discovery:
resource: bundles/discovery.tar.gz
decision: discovery/config
labels:
region: "UK"
```
Run OPA:
```bash
opa run -s -c opa-config.yaml
```
In this case, the bundle being downloaded is `example/test2/p` as `region` is
`UK`.
```raw
INFO Bundle downloaded and activated successfully. name=example/test2/p plugin=bundle
```
This shows how the discovery feature can help in centrally managing the bundle
to be downloaded by an OPA based on a configuration label. You can use the same
strategy to dynamically configure other plugins based on the running OPA's
configuration labels or environment variables.
### Limitations
In practice, discovery services do not change frequently. These configuration sections are treated as
immutable to avoid accidental configuration errors rendering OPA unable to discover a new configuration.
If the discovered configuration changes the `discovery` section,
those changes are ignored. If the discovered configuration changes the discovery service,
an error will be logged.
If the discovered configuration changes the `labels` section, only labels that are additional compared to the bootstrap configuration are used, all other changes are ignored. If the discovery document changes its `labels` section over time, the effective set of labels is always the bootstrap configuration plus added labels from the latest discovery document.
### Discovery Bundle Signature
Like regular bundles, if the discovery bundle contains a `.signatures.json` file, OPA will verify the discovery
bundle before activating it. The format of the `.signatures.json` file and the verification steps are same as that for
regular bundles. Since the discovered configuration ignores changes to the `discovery` section, any key used for
signature verification of a discovery bundle **CANNOT** be modified via discovery.
> 🚨 We recommend that if you are using discovery you should be signing the discovery bundles because those bundles
> include the keys used to verify the non-discovery bundles. However, OPA does not enforce that recommendation. You may use
> unsigned discovery bundles that themselves require non-discovery bundles to be signed.
### Discovery Bundle Persistence
OPA can optionally persist the activated discovery bundle to disk for recovery purposes. To enable
persistence, set the `discovery.persist` field to `true`. When bundle
persistence is enabled, OPA will attempt to read the discovery bundle from disk on startup. This
allows OPA to start with the most recently activated bundle in case OPA cannot communicate
with the bundle server. OPA will try to load and activate the persisted discovery bundle on a best-effort basis. Any errors
encountered during the process will be surfaced in the bundle's status update. When communication between OPA and
the bundle server is restored, the latest bundle is downloaded, activated, and persisted. Like regular bundles, only
the discovery bundle itself is persisted. The discovered configuration that is generated by evaluating the data and
policies contained in the discovery bundle will **NOT** be persisted.
:::info
The discovery bundle is persisted at
`<persistence_directory>/bundles/<discovery.name>/bundle.tar.gz`. By default
`persistence_directory` is `.opa` in the working directory of the OPA process.
If `persistence_directory` is changed through discovery this will not affect
where the discovery plugin will store the discovery bundles, the boot
configuration will always be used.
:::
## Ecosystem Projects
Configuring OPA using Discovery Bundles is a powerful production feature.
<EcosystemEmbed feature="wasm-integration">
Wasm is a great way to integrate OPA into applications where the Go SDK is unavailable.
</EcosystemEmbed>
docs/new/docs/management-introduction/control-plane.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 117 KiB

docs/new/docs/management-introduction/distributed.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 101 KiB

docs/new/docs/management-introduction/index.md
+42
View File
@@ -0,0 +1,42 @@
---
title: "Overview & Architecture"
---
OPA exposes a set of APIs that enable unified, logically centralized policy
management. Read this page if you are interested in how to build a control plane
around OPA that enables policy distribution and collection of important
telemetry data like decision logs.
OPA enables low-latency, highly-available policy enforcement by providing a
lightweight engine for distributed architectures. By default, all of the policy
and data that OPA uses to make decisions is kept in-memory:
<!--- source: https://docs.google.com/drawings/d/1-dwGFRjv_nFydo-8tOK-C-PbWyjvRObYhePC7XaLUFw/edit?usp=sharing --->
![Host-local Architecture](integration.svg)
OPA is designed to enable _distributed_ policy enforcement. You can run OPA next
to each and every service that needs to offload policy decision-making. By
colocating OPA with the services that require decision-making, you ensure that
policy decisions are rendered as fast as possible and in a highly-available
manner.
<!--- source: https://docs.google.com/drawings/d/1wFef9_Smy0gNvJj4l8n05WCTqhmzdadiyspyRGFvHuw/edit?usp=sharing --->
![Distributed Policy Enforcement](distributed.svg)
To control and observe a set of OPAs, each OPA can be configured to connect to
management APIs that enable:
- Policy distribution ([Bundles](./management-bundles))
- Decision telemetry ([Decision Logs](./management-decision-logs))
- Agent telemetry ([Status](./management-status))
- Dynamic agent configuration ([Discovery](./management-discovery))
By configuring and implementing these management APIs you can unify control and
visibility over OPAs in your environments. OPA does not provide a control plane
service out-of-the-box today.
<!--- source: https://docs.google.com/drawings/d/1-08mHgUN5oy2phLJ6MOr7j3e0iguxg_X__3VH321iLc/edit?usp=sharing --->
![Control Plane](control-plane.svg)
docs/new/docs/management-introduction/integration.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 108 KiB

docs/new/docs/management-status.md
+311
View File
@@ -0,0 +1,311 @@
---
title: "Status"
---
OPA can periodically report status updates to remote HTTP servers. The
updates contain status information for OPA itself as well as the
[Bundles](./management-bundles) that have been downloaded and activated.
OPA sends status reports whenever one of the following happens:
- Bundles are downloaded and activated -- If the bundle download or activation fails for any reason, the status update
will include error information describing the failure. This includes Discovery bundles.
- A plugin state has changed -- All plugin status is reported, and an update to any plugin will
trigger a Status API report which contains the latest state.
The status updates will include a set of labels that uniquely identify the
OPA instance. OPA automatically includes an `id` value in the label set that
provides a globally unique identifier or the running OPA instance and a
`version` value that provides the version of OPA.
See the [Configuration Reference](./configuration) for configuration details.
## Status Service API
OPA expects the service to expose an API endpoint that will receive status
updates.
```http
POST /status[/<partition_name>] HTTP/1.1
Content-Type: application/json
```
The partition name is an optional path segment that can be used to route
status updates to different backends. If the partition name is not configured
on the agent, updates will be sent to `/status`.
```json
{
"labels": {
"app": "my-example-app",
"id": "1780d507-aea2-45cc-ae50-fa153c8e4a5a",
"version": "{{< current_version >}}"
},
"bundles": {
"http/example/authz": {
"active_revision": "ABC",
"last_request": "2018-01-01T00:00:00.000Z",
"last_successful_request": "2018-01-01T00:00:00.000Z",
"last_successful_download": "2018-01-01T00:00:00.000Z",
"last_successful_activation": "2018-01-01T00:00:00.000Z",
"metrics": {
"timer_rego_data_parse_ns": 12345,
"timer_rego_module_compile_ns": 12345,
"timer_rego_module_parse_ns": 12345
}
"name": "http/example/authz",
"size": 1048576,
"type": "snapshot",
}
},
"decision_logs": {
"code": "decision_log_error",
"message": "Upload Failed",
"http_code": "400",
"metrics": {
"counter_decision_logs_dropped": "2",
"decision_logs_nd_builtin_cache_dropped": "1"
}
},
"plugins": {
"bundle": {
"state": "OK"
},
"discovery": {
"state": "OK"
},
"status": {
"state": "OK"
}
},
"metrics": {
"prometheus": {
"go_gc_cycles_automatic_gc_cycles_total": {
"name": "go_gc_cycles_automatic_gc_cycles_total",
"help": "Count of completed GC cycles generated by the Go runtime.",
"type": "COUNTER",
"metric": [
{
"counter": {
"value": 1
}
}
]
},
"go_gc_cycles_forced_gc_cycles_total": {
"name": "go_gc_cycles_forced_gc_cycles_total",
"help": "Count of completed GC cycles forced by the application.",
"type": "COUNTER",
"metric": [
{
"counter": {
"value": 0
}
}
]
},
"go_gc_cycles_total_gc_cycles_total": {
"name": "go_gc_cycles_total_gc_cycles_total",
"help": "Count of all completed GC cycles.",
"type": "COUNTER",
"metric": [
{
"counter": {
"value": 1
}
}
]
},
"go_gc_duration_seconds": {
"name": "go_gc_duration_seconds",
"help": "A summary of the pause duration of garbage collection cycles.",
"type": "SUMMARY",
"metric": [
{
"summary": {
"sampleCount": "1",
"sampleSum": 4.1765e-05,
"quantile": [
{
"quantile": 0,
"value": 4.1765e-05
},
{
"quantile": 0.25,
"value": 4.1765e-05
},
{
"quantile": 0.5,
"value": 4.1765e-05
},
{
"quantile": 0.75,
"value": 4.1765e-05
},
{
"quantile": 1,
"value": 4.1765e-05
}
]
}
}
]
},
------------------------------8< SNIP 8<------------------------------
"http_request_duration_seconds": {
"name": "http_request_duration_seconds",
"help": "A histogram of duration for requests.",
"type": "HISTOGRAM",
"metric": [
{
"label": [
{
"name": "code",
"value": "200"
},
{
"name": "handler",
"value": "v1/data"
},
{
"name": "method",
"value": "get"
}
],
"histogram": {
"sampleCount": "2",
"sampleSum": 0.00060022,
"bucket": [
{
"cumulativeCount": "0",
"upperBound": 1e-06
},
{
"cumulativeCount": "0",
"upperBound": 5e-06
},
{
"cumulativeCount": "0",
"upperBound": 1e-05
},
{
"cumulativeCount": "0",
"upperBound": 5e-05
},
{
"cumulativeCount": "0",
"upperBound": 0.0001
},
{
"cumulativeCount": "2",
"upperBound": 0.0005
},
{
"cumulativeCount": "2",
"upperBound": 0.001
},
{
"cumulativeCount": "2",
"upperBound": 0.01
},
{
"cumulativeCount": "2",
"upperBound": 0.1
},
{
"cumulativeCount": "2",
"upperBound": 1
}
]
}
}
]
}
}
}
}
```
Status updates contain the following fields:
| Field | Type | Description |
| --------------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `labels` | `object` | Set of key-value pairs that uniquely identify the OPA instance. |
| `bundles` | `object` | Set of objects describing the status for each bundle configured with OPA. |
| `bundles[_].name` | `string` | Name of bundle that the OPA instance is configured to download. |
| `bundles[_].active_revision` | `string` | Opaque revision identifier of the last successful activation. |
| `bundles[_].last_request` | `string` | RFC3339 timestamp of last bundle request. This timestamp should be >= to the successful request timestamp in normal operation. |
| `bundles[_].last_successful_request` | `string` | RFC3339 timestamp of last successful bundle request. This timestamp should be >= to the successful download timestamp in normal operation. |
| `bundles[_].last_successful_download` | `string` | RFC3339 timestamp of last successful bundle download. |
| `bundles[_].last_successful_activation` | `string` | RFC3339 timestamp of last successful bundle activation. |
| `bundles[_].metrics` | `object` | Metrics from the last update of the bundle. |
| `bundles[_].code` | `string` | If present, indicates error(s) occurred activating this bundle. |
| `bundles[_].message` | `string` | Human readable messages describing the error(s). |
| `bundles[_].http_code` | `number` | If present, indicates an erroneous HTTP status code that OPA received downloading this bundle. |
| `bundles[_].errors` | `array` | Collection of detailed parse or compile errors that occurred during activation of this bundle. |
| `bundles[_].size` | `number` | Bundle size, in bytes |
| `bundles[_].type` | `string` | Bundle type, either `snapshot` or `delta` |
| `discovery.name` | `string` | Name of discovery bundle that the OPA instance is configured to download. |
| `discovery.active_revision` | `string` | Opaque revision identifier of the last successful discovery activation. |
| `discovery.last_request` | `string` | RFC3339 timestamp of last discovery bundle request. This timestamp should be >= to the successful request timestamp in normal operation. |
| `discovery.last_successful_request` | `string` | RFC3339 timestamp of last successful discovery bundle request. This timestamp should be >= to the successful download timestamp in normal operation. |
| `discovery.last_successful_download` | `string` | RFC3339 timestamp of last successful discovery bundle download. |
| `discovery.last_successful_activation` | `string` | RFC3339 timestamp of last successful discovery bundle activation. |
| `decision_logs.code` | `string` | If present, indicates error(s) occurred during decision log upload event. |
| `decision_logs.message` | `string` | Human readable messages describing the error(s). |
| `decision_logs.http_code` | `number` | If present, indicates an erroneous HTTP status code that OPA received during a decision log upload event. |
| `decision_logs.metrics` | `object` | Metrics from the last decision log upload event. |
| `plugins` | `object` | A set of objects describing the state of configured plugins in OPA's runtime. |
| `plugins[_].state` | `string` | The state of each plugin. |
| `metrics.prometheus` | `object` | Global performance metrics for the OPA instance. |
If the discovery bundle download or activation failed, the status update will contain
the following additional fields.
| Field | Type | Description |
| ------------------- | -------- | ------------------------------------------------------------------------------- |
| `discovery.code` | `string` | If present, indicates error(s) occurred. |
| `discovery.message` | `string` | Human readable messages describing the error(s). |
| `discovery.errors` | `array` | Collection of detailed parse or compile errors that occurred during activation. |
Services should reply with a `2xx` HTTP status if the status update is
processed successfully.
## Local Status Logs
Local console logging of status updates can be enabled via the `console` config option.
This does not require any remote server. Example of minimal config to enable:
```yaml
status:
console: true
```
This will dump all status updates to the console. See
[Configuration Reference](./configuration) for more details.
> Warning: Status update messages are somewhat infrequent but can be very verbose! The
> `metrics.prometheus` portion of the status update in particular can create a considerable
> amount of log text at info level.
## Prometheus Status Metrics
Prometheus status metrics can be enabled via the `prometheus` config option. (see [the configuration documentation](./configuration/#status))
Example of minimal config to enable:
```yaml
status:
prometheus: true
prometheus_config:
collectors:
bundle_loading_duration_ns:
buckets: [1, 1000, 10_000, 1e8]
```
When enabled the OPA instance's Prometheus endpoint exposes the metrics described on [the monitoring documentation](./monitoring/#status-metrics).
## Ecosystem Projects
<EcosystemEmbed feature="wasm-integration">
Here are some projects that use the OPA Status API to report status updates.
</EcosystemEmbed>
docs/new/docs/monitoring.md
+97
View File
@@ -0,0 +1,97 @@
---
title: Monitoring
---
## OpenTelemetry
When run as a server and configured accordingly, OPA will emit spans to an
[OpenTelemetry](https://opentelemetry.io/) collector via gRPC.
Each [REST API](./rest-api/) request sent to the server will start a span.
If processing the request involves policy evaluation, and that in turn uses
[`http.send`](./policy-reference/#http), those HTTP clients will emit descendant spans.
Furthermore, spans exported for policy evaluation requests will contain an
attribute `opa.decision_id` of the evaluation's decision ID _if_ the server
has decision logging enabled.
See [the configuration documentation](./configuration/#distributed-tracing)
for all OpenTelemetry-related configurables.
## Prometheus
OPA exposes an HTTP endpoint that can be used to collect performance metrics
for all API calls. The Prometheus endpoint is enabled by default when you run
OPA as a server.
You can enable metric collection from OPA with the following `prometheus.yml` config:
```yaml
global:
scrape_interval: 15s
scrape_configs:
- job_name: "opa"
metrics_path: "/metrics"
static_configs:
- targets:
- "localhost:8181"
```
The Prometheus endpoint exports Go runtime metrics as well as HTTP request latency metrics for all handlers (e.g., `v1/data`).
| Metric name | Metric type | Description | Status |
| -------------------------------- | ----------- | ------------------------------------------------------------------ | ------ |
| go_gc_duration_seconds | summary | A summary of the GC invocation durations. | STABLE |
| go_goroutines | gauge | Number of goroutines that currently exist. | STABLE |
| go_info | gauge | Information about the Go environment. | STABLE |
| go_memstats_alloc_bytes | gauge | Number of bytes allocated and still in use. | STABLE |
| go_memstats_alloc_bytes_total | counter | Total number of bytes allocated, even if freed. | STABLE |
| go_memstats_buck_hash_sys_bytes | gauge | Number of bytes used by the profiling bucket hash table. | STABLE |
| go_memstats_frees_total | counter | Total number of frees. | STABLE |
| go_memstats_gc_sys_bytes | gauge | Number of bytes used for garbage collection system metadata. | STABLE |
| go_memstats_heap_alloc_bytes | gauge | Number of heap bytes allocated and still in use. | STABLE |
| go_memstats_heap_idle_bytes | gauge | Number of heap bytes waiting to be used. | STABLE |
| go_memstats_heap_inuse_bytes | gauge | Number of heap bytes that are in use. | STABLE |
| go_memstats_heap_objects | gauge | Number of allocated objects. | STABLE |
| go_memstats_heap_released_bytes | gauge | Number of heap bytes released to OS. | STABLE |
| go_memstats_heap_sys_bytes | gauge | Number of heap bytes obtained from system. | STABLE |
| go_memstats_last_gc_time_seconds | gauge | Number of seconds since 1970 of last garbage collection. | STABLE |
| go_memstats_lookups_total | counter | Total number of pointer lookups. | STABLE |
| go_memstats_mallocs_total | counter | Total number of mallocs. | STABLE |
| go_memstats_mcache_inuse_bytes | gauge | Number of bytes in use by mcache structures. | STABLE |
| go_memstats_mcache_sys_bytes | gauge | Number of bytes used for mcache structures obtained from system. | STABLE |
| go_memstats_mspan_inuse_bytes | gauge | Number of bytes in use by mspan structures. | STABLE |
| go_memstats_mspan_sys_bytes | gauge | Number of bytes used for mspan structures obtained from system. | STABLE |
| go_memstats_next_gc_bytes | gauge | Number of heap bytes when next garbage collection will take place. | STABLE |
| go_memstats_other_sys_bytes | gauge | Number of bytes used for other system allocations. | STABLE |
| go_memstats_stack_inuse_bytes | gauge | Number of bytes in use by the stack allocator. | STABLE |
| go_memstats_stack_sys_bytes | gauge | Number of bytes obtained from system for stack allocator. | STABLE |
| go_memstats_sys_bytes | gauge | Number of bytes obtained from system. | STABLE |
| go_threads | gauge | Number of OS threads created. | STABLE |
| http_request_duration_seconds | histogram | A histogram of duration for requests. | STABLE |
### Status Metrics
When Prometheus is enabled in the status plugin (see [Configuration](./configuration/#status)), the OPA instance's Prometheus endpoint also exposes these metrics:
| Metric name | Metric type | Description | Status |
| ------------------------------ | ----------- | ------------------------------------------------------ | ------ |
| opa_info | gauge | Information about the OPA environment. | STABLE |
| plugin_status_gauge | gauge | Number of plugins by name and status. | STABLE |
| bundle_loaded_counter | counter | Number of bundles loaded with success. | STABLE |
| bundle_failed_load_counter | counter | Number of bundles that failed to load. | STABLE |
| last_bundle_request | gauge | Last bundle request in UNIX nanoseconds. | STABLE |
| last_success_bundle_activation | gauge | Last successful bundle activation in UNIX nanoseconds. | STABLE |
| last_success_bundle_download | gauge | Last successful bundle download in UNIX nanoseconds. | STABLE |
| last_success_bundle_request | gauge | Last successful bundle request in UNIX nanoseconds. | STABLE |
| bundle_loading_duration_ns | histogram | A histogram of duration for bundle loading. | STABLE |
## Health Checks
OPA exposes a `/health` API endpoint that can be used to perform health checks.
See [Health API](./rest-api#health-api) for details.
## Status API
OPA provides a plugin which can push status to a remote service.
See [Status API](./management-status) for details.
docs/new/docs/oauth-oidc.md
+109
View File
@@ -0,0 +1,109 @@
---
title: OAuth2 and OIDC Samples
---
OAuth2 and OpenID Connect are both pervasive technologies in modern identity systems. While verification of JSON web tokens issued by these systems is documented in the [policy reference](https://www.openpolicyagent.org/docs/latest/policy-reference/#token-verification), the policy examples below aim to cover some other common use cases.
## Metadata discovery
Rather than storing endpoints and other metadata as part of policy data, the authorization server metadata endpoint may be queried for this data.
```live:oidc:module
package oidc
issuers := {"https://issuer1.example.com", "https://issuer2.example.com"}
metadata_discovery(issuer) := http.send({
"url": concat("", [issuers[issuer], "/.well-known/openid-configuration"]),
"method": "GET",
"force_cache": true,
"force_cache_duration_seconds": 86400 # Cache response for 24 hours
}).body
claims := jwt.decode(input.token)[1]
metadata := metadata_discovery(claims.iss)
jwks_endpoint := metadata.jwks_uri
token_endpoint := metadata.token_endpoint
```
## Token verification using JWKS endpoint
Below example uses the keys published at the JWKS endpoint of the authorization server for token verification.
```live:oidc2:module
package oidc
jwks_request(url) := http.send({
"url": url,
"method": "GET",
"force_cache": true,
"force_cache_duration_seconds": 3600 # Cache response for an hour
})
jwks := jwks_request("https://authorization-server.example.com/jwks").raw_body
verified := io.jwt.verify_rs256(input.token, jwks)
```
### Key rotation
Use the keys published at the JWKS endpoint of the authorization server for token verification, with [key rotation](https://openid.net/specs/openid-connect-core-1_0.html#RotateSigKeys) taken into account.
```live:oidc3:module
package oidc
jwks_request(url) := http.send({
"url": url,
"method": "GET",
"force_cache": true,
"force_cache_duration_seconds": 3600
})
jwt_unverified := io.jwt.decode(input.token)
jwt_header := jwt_unverified[0]
# Use the key ID (kid) from the token as a cache key - if a new kid is encountered
# we obtain a fresh JWKS object as the keys have likely been rotated.
jwks_url := concat("?", [
"https://authorization-server.example.com/jwks",
urlquery.encode_object({"kid": jwt_header.kid}),
])
jwks := jwks_request(jwks_url).raw_body
jwt_verified := jwt_unverified {
io.jwt.verify_rs256(input.token, jwks)
}
claims_verified := jwt_verified[1]
```
## Token retrieval
Programmatically obtain an OAuth2 access token following the client credentials or resource owner password credential flow.
```live:oauth:module
package oauth2
token := t {
response := http.send({
"url": "https://authorization-server.example.com/token",
"method": "POST",
"headers": {
"Content-Type": "application/x-www-form-urlencoded",
"Authorization": concat(" ", [
"Basic",
base64.encode(sprintf("%v:%v", [client_id, client_secret]))
]),
},
# To use the resource owner password credentials flow, change grant_type
# to "password" and add username and password parameters to the body
"raw_body": "grant_type=client_credentials",
"force_cache": true,
"force_cache_duration_seconds": 3595, # Given an `expires_in` value of 3600
})
response.status_code == 200
t := response.body.access_token
}
```
docs/new/docs/opa-service.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 88 KiB

docs/new/docs/philosophy/data-model.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 176 KiB

docs/new/docs/philosophy/index.md
+183
View File
@@ -0,0 +1,183 @@
---
title: Philosophy
sidebar_position: 2
---
A [**policy**](#policy) is a set of rules that governs the behavior of a
software service. That policy could describe rate-limits, names of trusted
servers, the clusters an application should be deployed to, permitted network
routes, or accounts a user can withdraw money from.
Authorization is a special kind of policy that often dictates which people or
machines can run which actions on which resources. Authorization is sometimes
confused with Authentication: how people or machines prove they are who they say
they are. Authorization and more generally policy often utilize the results of
authentication (the username, user attributes, groups, claims), but makes
decisions based on far more information than just who the user is. Generalizing
away from authorization back to policy makes the distinction even clearer
because some policy decisions have nothing to do with users, e.g. policy simply
describes invariants that must hold in a software system (e.g. all binaries must
come from a trusted source).
Today policy is often a hard-coded feature of the software service it actually
governs. Open Policy Agent lets you [**decouple policy**](#policy-decoupling)
from that software service so that the people responsible for policy can read,
write, analyze, version, distribute, and in general manage policy separate from
the service itself. OPA also gives you a unified toolset to decouple policy from
any software service you like, and to write context-aware policies using any
context that you like. In short, OPA helps you decouple any policy using any
context from any software system.
## What is Policy? {#policy}
All organizations have policies. Policies are essential to the long-term success
of organizations because they encode important knowledge about how to comply
with legal requirements, work within technical constraints, avoid repeating
mistakes, and so on.
In their simplest form, policies can be applied manually based on rules that are
written down or conventions that are unspoken but permeate an organization’s
culture. Policies may also be enforced with application logic or statically
configured at deploy time.
## What is Policy Decoupling? {#policy-decoupling}
Software services should allow policies to be specified declaratively, updated
at any time without recompiling or redeploying, and enforced automatically
(which is especially valuable when decisions need to be made faster than humanly
possible).
Decoupling policy helps you build such software services at scale, makes them
adaptable to changing business requirements, improves the ability to discover
violations and conflicts, increases the consistency of policy compliance, and
mitigates the risk of human error. The policies you write can adapt more easily
to the external environment--to factors that the developer could never have
imagined at the time the software service was designed.
For example, a cloud computing service could answer questions such as:
- Can I add compute capacity?
- In what regions can I add compute capacity?
- Which instances are currently running in the wrong region?
## What is OPA?
OPA is a lightweight general-purpose policy engine that can be co-located with
your service. You can integrate OPA as a sidecar, host-level daemon, or library.
Services offload policy decisions to OPA by executing _queries_. OPA evaluates
policies and data to produce query results (which are sent back to the client).
Policies are written in a high-level declarative language and can be loaded
dynamically into OPA remotely via APIs or through the local filesystem.
## Why use OPA?
OPA is a full-featured policy engine that offloads policy decisions from your
software. You can think of it as a concierge for your software who can answer
detailed questions on behalf of your users to meet their specific needs.
OPA provides the building blocks for enabling better control and visibility over
policy in your systems.
Without OPA, you need to implement policy management for your software from scratch.
Required components such as the policy language (syntax _and_ semantics) and the
evaluation engine need to be carefully designed, implemented, tested, documented,
and then maintained to ensure correct behaviour and a positive user experience
for your customers. On top of that you must carefully consider security, tooling,
management, and more. That's a lot of work.
## How Does OPA Work?
See the [Introduction](..) for an overview of how OPA works and how to get started.
## The OPA Document Model
OPA policies (written in Rego) make decisions based on hierarchical structured data.
Sometimes we refer to this data as a document, set of attributes, piece of context,
or even just "JSON" [1]. Importantly, OPA policies can make decisions based on _arbitrary_
structured data. OPA itself is not tied to any particular domain model. Similarly,
OPA policies can represent decisions as arbitrary structured data (e.g., booleans,
strings, maps, maps of lists of maps, etc.)
Data can be loaded into OPA from outside world using push or pull interfaces that operate
synchronously or asynchronously with respect to policy evaluation. We refer to all data
loaded into OPA from the outside world as **base documents** [2]. These base documents
almost always contribute to your policy decision-making logic. However, your policies can
also make decisions based on each other. Policies almost always consist of multiple rules
that refer to other rules (possibly authored by different groups). In OPA, we refer
to the values generated by rules (a.k.a., decisions) as **virtual documents**. The term
"virtual" in this case just means the document is _computed_ by the policy, i.e.,
it's not loaded into OPA from the outside world.
Base and virtual documents can represent the exact same kind of information, e.g., numbers,
strings, lists, maps, and so on. Moreover, with Rego, you can refer to both base and virtual
documents using the exact same dot/bracket-style reference syntax. Consistency across the
types of values that can be represented and the way those values are referenced means that
_policy authors only need to learn one way of modeling and referring to information
that drives policy decision-making_. Additionally, since there is no conceptual difference
in the types of values or the way you refer to those values in base and virtual documents,
Rego lets you refer to _both_ base and virtual documents through a global variable
called `data`. Similarly, OPA lets you query for both base and virtual documents via the
`/v1/data` HTTP API [3]. This is why queries for just `data` (or `data.foo` or `data.foo.bar`, etc.)
return the combination of base and virtual documents located under that path.
Since base documents come from outside of OPA, their location under `data` is controlled
by the software doing the loading. On the other hand, the location of virtual
documents under `data` is controlled by policies themselves using the `package` directive
in the language.
Base documents can be pushed or pulled into OPA _asynchronously_ by replicating data
into OPA when the state of the world changes. This can happen periodically or when some
event (like a database change notification) occurs. Base documents loaded asynchronously
are always accessed under the `data` global variable. On the other hand, base documents can
also be pushed or pulled into OPA _synchronously_ when your software queries OPA for policy
decisions. We refer to base documents pushed synchronously as "input". Policies can
access these inputs under the `input` global variable. To pull base documents during
policy evaluation, OPA exposes (and can be extended with custom) built-in functions like
`http.send`. Built-in function return values can be assigned to local variables and
surfaced in virtual documents. Data loaded synchronously is kept outside of `data` to
avoid naming conflicts.
The following table summarizes the different models for loading base documents into OPA,
how they can be referenced inside of policies, and the actual mechanism(s) for loading.
| Model | How to access in Rego | How to integrate with OPA |
| ----------------- | --------------------------------------------------------------- | ---------------------------------------------------------------------- |
| Asynchronous Push | The `data` global variable | Invoke OPA's API(s), e.g., `PUT /v1/data` |
| Asynchronous Pull | The `data` global variable | Configure OPA's [Bundle](./management-bundles) feature |
| Synchronous Push | The `input` global variable | Provide data in policy query, e.g., inside the body of `POST /v1/data` |
| Synchronous Pull | The [built-in functions](./policy-reference), e.g., `http.send` | N/A |
Data loaded asynchronously into OPA is cached in-memory so that it can be read efficiently
during policy evaluation. Similarly, policies are also cached in-memory to ensure
high-performance and high-availability. Data _pulled_ synchronously can also be
cached in-memory. For more information on loading external data into OPA, including tradeoffs,
see the [External Data](./external-data) page.
The following diagram illustrates the base and virtual document model described above for a
hypothetical policy that renders authorization decisions (named `data.acme.allow`) based on:
- API request information pushed synchronously located under `input`.
- Entitlements data pulled asynchronously and located under `data.entitlements`.
- Resource data pulled synchronously during policy evaluation using the `http.send` built-in function.
The entitlements and resource information is _abstracted_ by rules that generate
virtual documents named `data.iam.user_has_role` and `data.acme.user_is_assigned` respectively.
<!--- source: https://docs.google.com/drawings/d/1KerjlOGRmsZvs2tqfhLh2CGGkNRFH0GWioBsHLHAuIg/edit --->
![Hypothetical Policy Document Model](data-model.svg)
> [1] OPA has excellent support for loading JSON and YAML because they are prevalent
> in modern systems; however, OPA is not tied to any particular data format. OPA
> uses its own internal representation for structures like maps and lists (a.k.a.,
> objects and arrays in JSON.)
> [2] The term "document" comes from the document-oriented database world. Document
> is just a generic term to refer to data or information encoded in some standard
> format like JSON, YAML, XML, etc. Document-oriented data does not have to adhere
> to a strict schema like data in the relational world. Documents are often deeply
> nested, hierarchical data structures containing several levels of embedded
> maps and lists.
> [3] Internally, HTTP requests like `GET /v1/data` or `GET /v1/data/foo/bar` are turned
> into Rego queries that are almost identical to the HTTP path (e.g., `data` or `data.foo.bar`)
docs/new/docs/policy-language.md
+3662
View File
File diff suppressed because it is too large Load Diff
docs/new/docs/policy-performance.md
+976
View File
@@ -0,0 +1,976 @@
---
title: Policy Performance
sidebar_position: 5
---
## High Performance Policy Decisions
Some use cases require very low-latency policy decisions. For example, a microservice API authorization decision might
have a budget in the order of 1 millisecond. OPA is a general-purpose policy engine and supports some features and
techniques to address high-performance use cases.
### Linear fragment
For such high-performance use cases, there is a fragment of the Rego language which has been engineered to evaluate
in near constant time. Adding more rules to the policy will not significantly increase the evaluation time.
For example, the following rule has one local variable `user`, and that variable can only be assigned one value. Intuitively, evaluating this rule requires checking each of the conditions in the body, and if there were N of these rules, evaluation would only require walking over each of them as well.
```live:linear:module:read_only,openable
package linear
allow if {
some user
input.method == "GET"
input.path = ["accounts", user]
input.user == user
}
```
### Use objects over arrays
One common mistake people make is using arrays when they could use objects. For example, below is an array of ID/first-name/last-names where ID is unique, and you're looking up the first-name/last-name given the ID.
```live:prefer_objects/bad:query
# DO NOT DO THIS.
# Array of objects where each object has a unique identifier
d := [{"id": "a123", "first": "alice", "last": "smith"},
{"id": "a456", "first": "bob", "last": "jones"},
{"id": "a789", "first": "clarice", "last": "johnson"}
]
# search through all elements of the array to find the ID
d[i].id == "a789"
d[i].first ...
```
Instead, use a dictionary where the key is the ID and the value is the first-name/last-name. Given the ID, you can look up the name information directly.
```live:prefer_objects/good:query
# DO THIS INSTEAD OF THE ABOVE
# Use object whose keys are the IDs for the objects.
# Looking up an object given its ID requires NO search
d := {"a123": {"first": "alice", "last": "smith"},
"a456": {"first": "bob", "last": "jones"},
"a789": {"first": "clarice", "last": "johnson"}
}
# no search required
d["a789"].first ...
```
### Use indexed statements
The linear-time fragment ensures that the cost of evaluation is no larger than the size of the policy. OPA lets you write non-linear policies, because sometimes you need to, and because sometimes it's convenient. The blog on [partial evaluation](https://blog.openpolicyagent.org/partial-evaluation-162750eaf422) describes one mechanism for converting non-linear policies into linear policies.
But as the size of the policy grows, the cost of evaluation grows with it. Sometimes the policy can grow large enough that even the linear-fragment fails to meet the performance budget.
In the linear fragment, OPA includes special algorithms that **index rules efficiently**, sometimes making evaluation constant-time, even as the policy grows. The more effective the indexing is the fewer rules need to be evaluated.
Here is an example policy from the [rule-indexing blog](https://blog.openpolicyagent.org/optimizing-opa-rule-indexing-59f03f17caf3) giving the details for these algorithms. See the rest of this section for details on indexed statements.
```live:indexed:module:openable
package indexed
default allow := false
allow if {
some user
input.method == "GET"
input.path = ["accounts", user]
input.user == user
}
allow if {
input.method == "GET"
input.path == ["accounts", "report"]
roles[input.user][_] == "admin"
}
allow if {
input.method == "POST"
input.path == ["accounts"]
roles[input.user][_] == "admin"
}
roles := {
"bob": ["admin", "hr"],
"alice": ["procurement"],
}
```
```live:indexed:query:hidden
allow
```
```live:indexed:input
{
"user": "bob",
"path": ["accounts", "bob"],
"method": "GET"
}
```
```live:indexed:output
```
#### Equality statements
For simple equality statements (`=` and `==`) to be indexed one side must be a non-nested reference that does not contain any variables and the other side must be a variable, scalar, or array (which may contain scalars and variables). For example:
| Expression | Indexed | Reason |
| --------------------------- | ------- | ---------------------------- |
| `input.x == "foo"` | yes | n/a |
| `input.x.y == "bar"` | yes | n/a |
| `input.x == ["foo", i]` | yes | n/a |
| `input.x[i] == "foo"` | no | reference contains variables |
| `input.x[input.y] == "foo"` | no | reference is nested |
#### Glob statements
For `glob.match(pattern, delimiter, match)` statements to be indexed the pattern must be recognized by the indexer and the match be a non-nested reference that does not contain any variables. The indexer recognizes patterns containing the normal glob (`*`) operator but not the super glob (`**`) or character pattern matching operators.
| Expression | Indexed | Reason |
| -------------------------------------------- | ------- | -------------------------- |
| `glob.match("foo:*:bar", [":"], input.x)` | yes | n/a |
| `glob.match("foo:**:bar", [":"], input.x)` | no | pattern contains `**` |
| `glob.match("foo:*:bar", [":"], input.x[i])` | no | match contains variable(s) |
### Early Exit in Rule Evaluation
In general, OPA has to iterate all potential variable bindings to determine the outcome
of a query. However, there are conditions under which additional iterations cannot change
the result:
1. A set of complete document rules that only have one, ground value.
2. A set of function rules that only have one, ground value.
The most common case for this are a set of `allow` rules:
```live:ee:module:read_only
package earlyexit
allow if {
input.user == "alice"
}
allow if {
input.user == "bob"
}
allow if {
input.group == "admins"
}
```
since `allow if { ... }` is a shorthand for `allow := true if { ... }`.
Intuitively, the value can be anything that does not contain a variable:
```live:eeexamples:module:read_only
package earlyexit.examples
# p, q, r and s could be evaluated with early-exit semantics:
p if {
# ...
}
q := 123 if {
# ...
}
r := {"hello": "world"} if {
# ...
}
s(x) := 12 if {
# ...
}
# u, v, w, and y could _not_
u contains x if { # not a complete document rule, but a partial set
x := 911
}
v := x if { # x is a variable, not ground
x := true
}
w := {"foo": x} if { # a compound term containing a variable
x := "bar"
}
y(z) := r if { # variable value, not ground
r := z + 1
}
```
When "early exit" is possible for a (set of) rules, iterations inside that rule will be
**cancelled** as soon as one binding matches the rule body:
```live:eeiteration:module:read_only
package earlyexit.iteration
p if {
some p
data.projects[p] == "project-a"
}
```
Since there's no possibility that could change the outcome of `data.earlyexit.iteration.p`
once a variable binding is found that satisfies the conditions, no further iteration will
occur.
The check if "early exit" is applicable for a query happens _after_ the indexing lookup,
so in this contrived example, an evaluation with input `{"user": "alice"}` _would_ exit
early; an evaluation with `{"user": "bob", "group": "admins"}` _would not_:
```live:eeindex:module:read_only
package earlyexit
allow if {
input.user == "alice"
}
allow := false if {
input.user == "bob"
}
allow if {
input.group == "admins"
}
```
This is because the index lookup for `{"user": "bob", "group": "admins"}` returns two complete
document rules with _different values_, `true` and `false`, whereas the indexer query for
`{"user": "alice"}` only returns rules with value `true`.
### Comprehension Indexing
Rego does not support mutation. As a result, certain operations like "group by" require
use of comprehensions to aggregate values. To avoid O(n^2) runtime complexity in
queries/rules that perform group-by, OPA may compute and memoize the entire collection
produced by comprehensions at once. This ensures that runtime complexity is O(n) where
n is the size of the collection that group-by/aggregation is being performed on.
For example, suppose the policy must check if the number of ports exposed on an interface
exceeds some threshold (e.g., any interface may expose up to 100 ports.) The policy is given
the port->interface mapping as a JSON array under `input`:
```json
{
"exposed": [
{
"interface": "eth0",
"port": 8080
},
{
"interface": "eth0",
"port": 8081
},
{
"interface": "eth1",
"port": 443
},
{
"interface": "lo1",
"port": 5000
}
]
}
```
In this case, the policy must count the number of ports exposed on each interface. To do this,
the policy must first aggregate/group the ports by the interface name. Conceptually,
the policy should generate a document like this:
```json
{
"exposed_ports_by_interface": {
"eth0": [8080, 8081],
"eth1": [443],
"lo1": [5000]
}
}
```
Since multiple ports could be exposed on a single interface, the policy must use a comprehension to
aggregate the port values by the interface names. To implement this logic in Rego, we would write:
```rego
some i
intf := input.exposed[i].interface
ports := [port | some j; input.exposed[j].interface == intf; port := input.exposed[j].port]
```
Without comprehension indexing, this query would be O(n^2) where n is the size of `input.exposed`.
However, with comprehension indexing, the query remains O(n) because OPA only computes the comprehension
_once_. In this case, the comprehension is evaluated and all possible values of `ports` are computed
at once. These values are indexed by the assignments of `intf`.
To implement the policy above we could write:
```rego
package example
deny contains msg if {
some i
count(exposed_ports_by_interface[i]) > 100
msg := sprintf("interface '%v' exposes too many ports", [i])
}
exposed_ports_by_interface := {intf: ports |
some i
intf := input.exposed[i].interface
ports := [port |
some j
input.exposed[j].interface == intf
port := input.exposed[j].port
]
}
```
Indices can be built for comprehensions (nested or not) that generate collections (i.e., arrays, sets, or objects)
based on variables in an outer query. In the example above:
- `intf` is the variable in the outer query.
- `[port | some j; input.exposed[j].interface == intf; port := input.exposed[j].port]` is the comprehension.
- `ports` is the variable the collection is assigned to.
In order to be indexed, comprehensions must meet the following conditions:
1. The comprehension appears in an assignment or unification statement.
1. The expression containing the comprehension does not include a `with` statement.
1. The expression containing the comprehension is not negated.
1. The comprehension body is safe when considered independent of the outer query.
1. The comprehension body closes over at least one variable in the outer query and none of these variables appear as outputs in references or `walk()` calls or inside nested comprehensions.
The following examples shows rules that are **not** indexed:
```rego
package example
not_indexed_because_missing_assignment if {
x := input[_]
[y | some y; x == input[y]]
}
not_indexed_because_includes_with if {
x := input[_]
ys := [y | some y; x := input[y]] with input as {}
}
not_indexed_because_negated if {
x := input[_]
not data.arr = [y | some y; x := input[y]]
}
not_indexed_because_safety if {
obj := input.foo.bar
x := obj[_]
ys := [y | some y; x == obj[y]]
}
not_indexed_because_no_closure if {
ys := [y | x := input[y]]
}
not_indexed_because_reference_operand_closure if {
x := input[y].x
ys := [y | x == input[y].z[_]]
}
not_indexed_because_nested_closure if {
x := 1
y := 2
_ = [i |
x == input.foo[i]
_ = [j | y == input.bar[j]]
]
}
```
> The 4th and 5th restrictions may be relaxed in the future.
### Profiling
You can also profile your policies using `opa eval`. The profiler is useful if you need to understand
why policy evaluation is slow.
The `opa eval` command provides the following profiler options:
| Option | Detail | Default |
| ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| <span class="opa-keep-it-together"> `--profile`</span> | Enables expression profiling and outputs profiler results. | off |
| <span class="opa-keep-it-together"> `--profile-sort`</span> | Criteria to sort the expression profiling results. This options implies `--profile`. | total_time_ns => num_eval => num_redo => num_gen_expr => file => line |
| <span class="opa-keep-it-together"> `--profile-limit`</span> | Desired number of profiling results sorted on the given criteria. This options implies `--profile`. | 10 |
| <span class="opa-keep-it-together"> `--count`</span> | Desired number of evaluations that profiling metrics are to be captured for. With `--format=pretty`, the output will contain min, max, mean and the 90th and 99th percentile. All collected percentiles can be found in the JSON output. | 1 |
#### Sort criteria for the profile results
- `total_time_ns` - Results are displayed is decreasing order of _expression evaluation time_
- `num_eval` - Results are displayed is decreasing order of _number of times an expression is evaluated_
- `num_redo` - Results are displayed is decreasing order of _number of times an expression is re-evaluated(redo)_
- `num_gen_expr` - Results are displayed is decreasing order of _number of generated expressions_
- `file` - Results are sorted in reverse alphabetical order based on the _rego source filename_
- `line` - Results are displayed is decreasing order of _expression line number_ in the source file
When the sort criteria is not provided `total_time_ns` has the highest sort priority
while `line` has the lowest.
The `num_gen_expr` represents the number of expressions generated for a given statement on a particular line. For example,
let's take the following policy:
```rego
package test
p if {
a := 1
b := 2
c := 3
x = a + (b * c)
}
```
If we profile the above policy we would get something like the following output:
```
+----------+----------+----------+--------------+-------------+
| TIME | NUM EVAL | NUM REDO | NUM GEN EXPR | LOCATION |
+----------+----------+----------+--------------+-------------+
| 20.291µs | 3 | 3 | 3 | test.rego:8 |
| 1µs | 1 | 1 | 1 | test.rego:7 |
| 2.333µs | 1 | 1 | 1 | test.rego:6 |
| 6.333µs | 1 | 1 | 1 | test.rego:5 |
| 84.75µs | 1 | 1 | 1 | data |
+----------+----------+----------+--------------+-------------+
```
The first entry indicates that line `test.rego:8` has a `EVAL/REDO` count of `3`. If we look at the expression on line `test.rego:8`
ie `x = a + b * c` it's not immediately clear why this line has a `EVAL/REDO` count of `3`. But we also notice that there
are `3` generated expressions (ie. `NUM GEN EXPR`) at line `test.rego:8`. This is because the compiler rewrites the above policy to
something like below:
`p = true if { __local0__ = 1; __local1__ = 2; __local2__ = 3; mul(__local1__, __local2__, __local3__); plus(__local0__, __local3__, __local4__); x = __local4__ }`
And that line `test.rego:8` is rewritten to `mul(__local1__, __local2__, __local3__); plus(__local0__, __local3__, __local4__); x = __local4__` which
results in a `NUM GEN EXPR` count of `3`. Hence, the `NUM GEN EXPR` count can help to better understand the `EVAL/REDO` counts
for a given expression and also provide more clarity into the profile results and how policy evaluation works.
#### Example Policy
The different profiling examples shown later on this page use the below
sample policy.
```live:profile:module:read_only,openable
package rbac
# Example input request
inp := {
"subject": "bob",
"resource": "foo123",
"action": "write",
}
# Example RBAC configuration.
bindings := [
{
"user": "alice",
"roles": ["dev", "test"],
},
{
"user": "bob",
"roles": ["test"],
},
]
roles := [
{
"name": "dev",
"permissions": [
{"resource": "foo123", "action": "write"},
{"resource": "foo123", "action": "read"},
],
},
{
"name": "test",
"permissions": [{"resource": "foo123", "action": "read"}],
},
]
# Example RBAC policy implementation.
default allow := false
allow if {
some role_name
user_has_role[role_name]
role_has_permission[role_name]
}
user_has_role contains role_name if {
binding := bindings[_]
binding.user == inp.subject
role_name := binding.roles[_]
}
role_has_permission contains role_name if {
role := roles[_]
role_name := role.name
perm := role.permissions[_]
perm.resource == inp.resource
perm.action == inp.action
}
```
#### Example: Display all profile results with default ordering criteria
```bash
opa eval --data rbac.rego --profile --format=pretty 'data.rbac.allow'
```
**Sample Output**
```ruby
false
+------------------------------+---------+
| METRIC | VALUE |
+------------------------------+---------+
| timer_rego_load_files_ns | 769583 |
| timer_rego_module_compile_ns | 1652125 |
| timer_rego_module_parse_ns | 482417 |
| timer_rego_query_compile_ns | 23042 |
| timer_rego_query_eval_ns | 440542 |
| timer_rego_query_parse_ns | 36250 |
+------------------------------+---------+
+-----------+----------+----------+--------------+-----------------+
| TIME | NUM EVAL | NUM REDO | NUM GEN EXPR | LOCATION |
+-----------+----------+----------+--------------+-----------------+
| 237.126µs | 1 | 1 | 1 | data.rbac.allow |
| 25.75µs | 1 | 1 | 1 | docs.rego:13 |
| 17.5µs | 1 | 1 | 1 | docs.rego:40 |
| 6.832µs | 2 | 1 | 1 | docs.rego:50 |
| 5.042µs | 1 | 1 | 1 | docs.rego:44 |
| 4.666µs | 1 | 0 | 1 | docs.rego:45 |
| 4.209µs | 1 | 1 | 1 | docs.rego:58 |
| 3.792µs | 1 | 2 | 1 | docs.rego:49 |
| 3.666µs | 1 | 2 | 1 | docs.rego:55 |
| 3.167µs | 1 | 1 | 1 | docs.rego:24 |
+-----------+----------+----------+--------------+-----------------+
```
As seen from the above table, all results are displayed. The profile results are
sorted on the default sort criteria.
To evaluate the policy multiple times, and aggregate the profiling data over those
runs, pass `--count=NUMBER`:
```bash
opa eval --data rbac.rego --profile --format=pretty --count=10 'data.rbac.allow'
```
**Sample Output**
```ruby
false
+------------------------------+--------+---------+----------+------------------------+--------------+
| METRIC | MIN | MAX | MEAN | 90% | 99% |
+------------------------------+--------+---------+----------+------------------------+--------------+
| timer_rego_load_files_ns | 140167 | 1092875 | 387233.3 | 1.0803291e+06 | 1.092875e+06 |
| timer_rego_module_compile_ns | 447208 | 1178542 | 646295.9 | 1.1565419000000001e+06 | 1.178542e+06 |
| timer_rego_module_parse_ns | 121458 | 1041333 | 349183.2 | 1.022583e+06 | 1.041333e+06 |
| timer_rego_query_compile_ns | 17542 | 47875 | 25758.4 | 47450 | 47875 |
| timer_rego_query_eval_ns | 47666 | 136625 | 68200 | 132762.5 | 136625 |
| timer_rego_query_parse_ns | 14334 | 46917 | 26270.9 | 46842 | 46917 |
+------------------------------+--------+---------+----------+------------------------+--------------+
+---------+----------+---------+----------+----------+----------+----------+--------------+-----------------+
| MIN | MAX | MEAN | 90% | 99% | NUM EVAL | NUM REDO | NUM GEN EXPR | LOCATION |
+---------+----------+---------+----------+----------+----------+----------+--------------+-----------------+
| 5.208µs | 27µs | 9.008µs | 25.525µs | 27µs | 1 | 1 | 1 | data.rbac.allow |
| 4.126µs | 17µs | 7.196µs | 16.479µs | 17µs | 1 | 1 | 1 | docs.rego:13 |
| 3.958µs | 12.833µs | 6.116µs | 12.583µs | 12.833µs | 1 | 1 | 1 | docs.rego:40 |
| 3.459µs | 10.708µs | 5.354µs | 10.499µs | 10.708µs | 2 | 1 | 1 | docs.rego:50 |
| 3.291µs | 9.209µs | 4.912µs | 9.096µs | 9.209µs | 1 | 1 | 1 | docs.rego:44 |
| 3.209µs | 8.75µs | 4.637µs | 8.62µs | 8.75µs | 1 | 0 | 1 | docs.rego:45 |
| 3.042µs | 8.333µs | 4.491µs | 8.233µs | 8.333µs | 1 | 1 | 1 | docs.rego:51 |
| 3µs | 7.25µs | 4.1µs | 7.112µs | 7.25µs | 1 | 1 | 1 | docs.rego:58 |
| 2.667µs | 5.75µs | 3.783µs | 5.72µs | 5.75µs | 1 | 2 | 1 | docs.rego:49 |
| 2.583µs | 5.708µs | 3.479µs | 5.595µs | 5.708µs | 1 | 1 | 1 | docs.rego:24 |
+---------+----------+---------+----------+----------+----------+----------+--------------+-----------------+
```
##### Example: Display top 5 profile results
```bash
opa eval --data rbac.rego --profile-limit 5 --format=pretty 'data.rbac.allow'
```
**Sample Output**
```ruby
+----------+----------+----------+--------------+-----------------+
| TIME | NUM EVAL | NUM REDO | NUM GEN EXPR | LOCATION |
+----------+----------+----------+--------------+-----------------+
| 24.624µs | 1 | 1 | 1 | data.rbac.allow |
| 15.251µs | 1 | 1 | 1 | docs.rego:13 |
| 12.167µs | 1 | 1 | 1 | docs.rego:40 |
| 9.625µs | 2 | 1 | 1 | docs.rego:50 |
| 8.751µs | 1 | 1 | 1 | docs.rego:44 |
+----------+----------+----------+--------------+-----------------+
```
The profile results are sorted on the default sort criteria.
Also `--profile` option is implied and does not need to be provided.
##### Example: Display top 5 profile results based on the 'number of times an expression is evaluated'
```bash
opa eval --data rbac.rego --profile-limit 5 --profile-sort num_eval --format=pretty 'data.rbac.allow'
```
**Sample Profile Output**
```ruby
+----------+----------+----------+--------------+-----------------+
| TIME | NUM EVAL | NUM REDO | NUM GEN EXPR | LOCATION |
+----------+----------+----------+--------------+-----------------+
| 10.541µs | 2 | 1 | 1 | docs.rego:50 |
| 4.041µs | 2 | 1 | 1 | docs.rego:56 |
| 27.876µs | 1 | 1 | 1 | data.rbac.allow |
| 19.916µs | 1 | 1 | 1 | docs.rego:40 |
| 19.416µs | 1 | 1 | 1 | docs.rego:13 |
+----------+----------+----------+--------------+-----------------+
```
As seen from the above table, the results are arranged first in decreasing
order of number of evaluations and if two expressions have been evaluated
the same number of times, the default criteria is used since no other sort criteria is provided.
In this case, total_time_ns => num_redo => file => line.
Also `--profile` option is implied and does not need to be provided.
##### Example: Display top 5 profile results based on the 'number of times an expression is evaluated' and 'number of times an expression is re-evaluated'
```bash
opa eval --data rbac.rego --profile-limit 5 --profile-sort num_eval,num_redo --format=pretty 'data.rbac.allow'
```
**Sample Profile Output**
```ruby
+---------+----------+----------+--------------+-----------------+
| TIME | NUM EVAL | NUM REDO | NUM GEN EXPR | LOCATION |
+---------+----------+----------+--------------+-----------------+
| 9.625µs | 2 | 1 | 1 | docs.rego:50 |
| 3.458µs | 2 | 1 | 1 | docs.rego:56 |
| 5.625µs | 1 | 2 | 1 | docs.rego:49 |
| 5.292µs | 1 | 2 | 1 | docs.rego:55 |
| 18.25µs | 1 | 1 | 1 | data.rbac.allow |
+---------+----------+----------+--------------+-----------------+
```
As seen from the above table, result are first arranged based on _number of evaluations_,
then _number of re-evaluations_ and finally the default criteria is used.
In this case, total_time_ns => file => line.
The `--profile-sort` options accepts repeated or comma-separated values for the criteria.
The order of the criteria on the command line determine their priority.
Another way to get the same output as above would be the following:
```bash
opa eval --data rbac.rego --profile-limit 5 --profile-sort num_eval --profile-sort num_redo --format=pretty 'data.rbac.allow'
```
## Benchmarking Queries
OPA provides CLI options to benchmark a single query via the `opa bench` command. This will evaluate similarly to
`opa eval` but it will repeat the evaluation (in its most efficient form) a number of times and report metrics.
#### Example: Benchmark rbac allow
Using the same [policy source as shown above](#example-policy):
```bash
opa bench --data rbac.rego 'data.rbac.allow'
```
Will result in an output similar to:
```
+-------------------------------------------+------------+
| samples | 27295 |
| ns/op | 45032 |
| B/op | 20977 |
| allocs/op | 382 |
| histogram_timer_rego_query_eval_ns_stddev | 25568 |
| histogram_timer_rego_query_eval_ns_99.9% | 335906 |
| histogram_timer_rego_query_eval_ns_99.99% | 336493 |
| histogram_timer_rego_query_eval_ns_mean | 40355 |
| histogram_timer_rego_query_eval_ns_median | 35846 |
| histogram_timer_rego_query_eval_ns_99% | 133936 |
| histogram_timer_rego_query_eval_ns_90% | 44780 |
| histogram_timer_rego_query_eval_ns_95% | 50815 |
| histogram_timer_rego_query_eval_ns_min | 31284 |
| histogram_timer_rego_query_eval_ns_max | 336493 |
| histogram_timer_rego_query_eval_ns_75% | 38254 |
| histogram_timer_rego_query_eval_ns_count | 27295 |
+-------------------------------------------+------------+
```
These results capture metrics of `samples` runs, where only the query evaluation is measured. All time spent preparing
to evaluate (loading, parsing, compiling, etc.) is omitted.
> Note: all `*/op` results are an average over the number of `samples` (or `N` in the JSON format)
#### Options for `opa bench`
| Option | Detail | Default |
| ------------------------------------------------------- | ------------------------------------------------- | ------- |
| <span class="opa-keep-it-together"> `--benchmem`</span> | Report memory allocations with benchmark results. | true |
| <span class="opa-keep-it-together"> `--metrics`</span> | Report additional query performance metrics. | true |
| <span class="opa-keep-it-together"> `--count`</span> | Number of times to repeat the benchmark. | 1 |
### Benchmarking OPA Tests
There is also a `--bench` option for `opa test` which will perform benchmarking on OPA unit tests. This will evaluate
any loaded tests as benchmarks. There will be additional time for any test-specific actions are included so the timing
will typically be longer than what is seen with `opa bench`. The primary use-case is not for absolute time, but to
track relative time as policies change.
#### Options for `opa test --bench`
| Option | Detail | Default |
| ------------------------------------------------------- | ------------------------------------------------- | ------- |
| <span class="opa-keep-it-together"> `--benchmem`</span> | Report memory allocations with benchmark results. | true |
| <span class="opa-keep-it-together"> `--count`</span> | Number of times to repeat the benchmark. | 1 |
#### Example Tests
Adding a unit test file for the [policy source as shown above](#example-policy):
```rego
package rbac
test_user_has_role_dev if {
user_has_role.dev with input as {"subject": "alice"}
}
test_user_has_role_negative if {
not user_has_role["super-admin"] with input as {"subject": "alice"}
}
```
Which when run normally will output something like:
```
$ opa test -v ./rbac.rego ./rbac_test.rego
data.rbac.test_user_has_role_dev: PASS (605.076µs)
data.rbac.test_user_has_role_negative: PASS (318.047µs)
--------------------------------------------------------------------------------
PASS: 2/2
```
#### Example: Benchmark rbac unit tests
```bash
opa test -v --bench ./rbac.rego ./rbac_test.rego
```
Results in output:
```
data.rbac.test_user_has_role_dev 44749 27677 ns/op 23146 timer_rego_query_eval_ns/op 12303 B/op 229 allocs/op
data.rbac.test_user_has_role_negative 44526 26348 ns/op 22033 timer_rego_query_eval_ns/op 12470 B/op 235 allocs/op
--------------------------------------------------------------------------------
PASS: 2/2
```
#### Example: Benchmark rbac unit tests and compare with `benchstat`
The benchmark output formats default to `pretty`, but support a `gobench` format which complies with the
[Golang Benchmark Data Format](https://go.googlesource.com/proposal/+/master/design/14313-benchmark-format.md).
This allows for usage of tools like [benchstat](https://godoc.org/golang.org/x/perf/cmd/benchstat) to gain additional
insight into the benchmark results and to diff between benchmark results.
Example:
```bash
opa test -v --bench --count 10 --format gobench ./rbac.rego ./rbac_test.rego | tee ./old.txt
```
Will result in an `old.txt` and output similar to:
```
BenchmarkDataRbacTestUserHasRoleDev 45152 26323 ns/op 22026 timer_rego_query_eval_ns/op 12302 B/op 229 allocs/op
BenchmarkDataRbacTestUserHasRoleNegative 45483 26253 ns/op 21986 timer_rego_query_eval_ns/op 12470 B/op 235 allocs/op
--------------------------------------------------------------------------------
PASS: 2/2
.
.
```
Repeated 10 times (as specified by the `--count` flag).
This format can then be loaded by `benchstat`:
```bash
benchstat ./old.txt
```
Output:
```
name time/op
DataRbacTestUserHasRoleDev 29.8µs ±18%
DataRbacTestUserHasRoleNegative 32.0µs ±35%
name timer_rego_query_eval_ns/op
DataRbacTestUserHasRoleDev 25.0k ±18%
DataRbacTestUserHasRoleNegative 26.7k ±35%
name alloc/op
DataRbacTestUserHasRoleDev 12.3kB ± 0%
DataRbacTestUserHasRoleNegative 12.5kB ± 0%
name allocs/op
DataRbacTestUserHasRoleDev 229 ± 0%
DataRbacTestUserHasRoleNegative 235 ± 0%
```
If later on a change was introduced that altered the performance we can run again:
```bash
opa test -v --bench --count 10 --format gobench ./rbac.rego ./rbac_test.rego | tee ./new.txt
```
```
BenchmarkDataRbacTestUserHasRoleDev 27415 43671 ns/op 39301 timer_rego_query_eval_ns/op 17201 B/op 379 allocs/op
BenchmarkDataRbacTestUserHasRoleNegative 27583 44743 ns/op 40152 timer_rego_query_eval_ns/op 17369 B/op 385 allocs/op
--------------------------------------------------------------------------------
PASS: 2/2
.
.
```
(Repeated 10 times)
Then we can compare the results via:
```bash
benchstat ./old.txt ./new.txt
```
```
name old time/op new time/op delta
DataRbacTestUserHasRoleDev 29.8µs ±18% 47.4µs ±15% +59.06% (p=0.000 n=9+10)
DataRbacTestUserHasRoleNegative 32.0µs ±35% 47.1µs ±14% +47.48% (p=0.000 n=10+9)
name old timer_rego_query_eval_ns/op new timer_rego_query_eval_ns/op delta
DataRbacTestUserHasRoleDev 25.0k ±18% 42.6k ±15% +70.51% (p=0.000 n=9+10)
DataRbacTestUserHasRoleNegative 26.7k ±35% 42.3k ±14% +58.15% (p=0.000 n=10+9)
name old alloc/op new alloc/op delta
DataRbacTestUserHasRoleDev 12.3kB ± 0% 17.2kB ± 0% +39.81% (p=0.000 n=10+10)
DataRbacTestUserHasRoleNegative 12.5kB ± 0% 17.4kB ± 0% +39.28% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
DataRbacTestUserHasRoleDev 229 ± 0% 379 ± 0% +65.50% (p=0.000 n=10+10)
DataRbacTestUserHasRoleNegative 235 ± 0% 385 ± 0% +63.83% (p=0.000 n=10+10)
```
This gives clear feedback that the evaluations have slowed down considerably by looking at the `delta`
> Note that for [benchstat](https://godoc.org/golang.org/x/perf/cmd/benchstat) you will want to run with `--count` to
> repeat the benchmarks a number of times (5-10 is usually enough). The tool requires several data points else the `p`
> value will not show meaningful changes and the `delta` will be `~`.
## Resource Utilization
Policy evaluation is typically CPU-bound unless the policies have to pull additional
data on-the-fly using built-in functions like `http.send()` (in which case evaluation
likely becomes I/O-bound.) Policy evaluation is currently single-threaded. If you
are embedding OPA as a library, it is your responsibility to dispatch concurrent queries
to different Goroutines/threads. If you are running the OPA server, it will parallelize
concurrent requests and use as many cores as possible. You can limit the number of
cores that OPA can consume by starting OPA with the [`GOMAXPROCS`](https://golang.org/pkg/runtime)
environment variable.
Memory usage scales with the size of the policy (i.e., Rego) and data (e.g., JSON) that you
load into OPA. Raw JSON data loaded into OPA uses approximately 20x more memory compared to the
same data stored in a compact, serialized format (e.g., on disk). This increased
memory usage is due to the need to load the JSON data into Go data structures like maps,
slices, and strings so that it can be evaluated. For example, if you load 8MB worth of
JSON data representing 100,000 permission objects specifying subject/action/resource triplets,
OPA would consume approximately 160MB of RAM.
Memory usage also scales linearly with the number of rules loaded into OPA. For example,
loading 10,000 rules that implement an ACL-style authorization policy consumes approximately
130MB of RAM while 100,000 rules implementing the same policy (but with 10x more tuples to check)
consumes approximately 1.1GB of RAM.
By default, OPA stores policy and data in-memory. OPA's disk storage feature allows policy and data to be stored on disk. See [this](./storage/#disk) for more details.
## Optimization Levels
The `--optimize` (or `-O`) flag on the `opa build` command controls how bundles are optimized.
> Optimization applies partial evaluation to precompute _known_ values in the policy. The goal of
> partial evaluation is to convert non-linear-time policies into linear-time policies.
By specifying the `--optimize` flag, users can control how much time and resources are spent
attempting to optimize the bundle. Generally, higher optimization levels require more time
and resources. Currently, OPA supports three optimization levels. The exact optimizations applied
in each level may change over time.
### -O=0 (default)
By default optimizations are disabled.
### -O=1 (recommended)
Policies are partially evaluated. Rules that DO NOT depend on unknowns (directly or indirectly) are
evaluated and the virtual documents they produce are inlined into call sites. Virtual documents that
are required at evaluation time are not inlined. For example, if a base or virtual document is
targeted by a `with` statement in the policy, the document will not be inlined.
Rules that depend on unknowns (directly or indirectly) are also partially evaluated however the
virtual documents they produce ARE NOT inlined into call sites. The output policy should be structurally
similar to the input policy.
The `opa build` automatically marks the `input` document as unknown. In addition to the `input` document,
if `opa build` is invoked with the `-b`/`--bundle` flag, any `data` references NOT prefixed by the
`.manifest` roots are also marked as unknown.
### -O=2 (aggressive)
Same as `-O=1` except virtual documents produced by rules that depend on unknowns may be inlined
into call sites. In addition, more aggressive inlining is applied within rules. This includes
[copy propagation](https://en.wikipedia.org/wiki/Copy_propagation) and inlining of certain negated
statements that would otherwise generate support rules.
## Storage Optimization
### In-Memory Store Read Optimization
During normal operation, data values read from storage are converted to an AST representation that is used during policy evaluation.
This conversion can be expensive both in execution time and in memory usage, especially for large data values.
The default in-memory store can be configured to optimize for read speed by precomputing the AST representation of data values during storage write operations.
This removes the time spent converting raw data values to AST during policy evaluation, improving performance.
The memory footprint of the store will increase, as processed AST values generally take up more space in memory than the corresponding raw data values, but overall memory usage of OPA might remain more stable over time, as pre-converted data is shared across evaluations and isn't recomputed for each evaluation, which can cause spikes in memory usage.
Storage write operations will be slower due to the additional processing required to precompute the AST representation of data values. This can impact startup time and bundle loading/updates, especially for large data values.
This feature can be enabled for `opa run`, `opa eval`, and `opa bench` by setting the `--optimize-store-for-read-speed` flag.
Users are recommended to do performance testing to determine the optimal configuration for their use case.
## Key Takeaways
For high-performance use cases:
- Write your policies to minimize iteration and search.
- Use objects instead of arrays when you have a unique identifier for the elements of the array.
- Consider [partial-evaluation](https://blog.openpolicyagent.org/partial-evaluation-162750eaf422) to compile non-linear policies to linear policies.
- Write your policies with indexed statements so that [rule-indexing](https://blog.openpolicyagent.org/optimizing-opa-rule-indexing-59f03f17caf3) is effective.
- Use the profiler to help identify portions of the policy that would benefit the most from improved performance.
- Use the benchmark tools to help get real world timing data and detect policy performance changes.
docs/new/docs/policy-reference.md
+1475
View File
File diff suppressed because it is too large Load Diff
docs/new/docs/policy-testing.md
+723
View File
@@ -0,0 +1,723 @@
---
title: Policy Testing
sidebar_position: 4
---
OPA gives you a high-level declarative language
([Rego](./policy-language)) to author fine-grained policies that
codify important requirements in your system.
To help you verify the correctness of your policies, OPA also gives you a
framework that you can use to write _tests_ for your policies. By writing
tests for your policies you can speed up the development process of new rules
and reduce the amount of time it takes to modify rules as requirements evolve.
:::info
The examples in this section try to represent the best practices. As such, they
make use of keywords that are meant to become standard keywords at some point in
time, but have been introduced gradually.
[See the docs on _future keywords_](./policy-language/#future-keywords) for more information.
:::
## Getting Started
Let's use an example to get started. The file below implements a simple
policy that allows new users to be created and users to access their own
profile.
**example.rego**:
```live:example:module:read_only,openable
package authz
allow if {
input.path == ["users"]
input.method == "POST"
}
allow if {
input.path == ["users", input.user_id]
input.method == "GET"
}
```
To test this policy, we will create a separate Rego file that contains test cases.
**example_test.rego**:
```live:example/test:module:read_only
package authz_test
import data.authz
test_post_allowed if {
authz.allow with input as {"path": ["users"], "method": "POST"}
}
test_get_anonymous_denied if {
not authz.allow with input as {"path": ["users"], "method": "GET"}
}
test_get_user_allowed if {
authz.allow with input as {"path": ["users", "bob"], "method": "GET", "user_id": "bob"}
}
test_get_another_user_denied if {
not authz.allow with input as {"path": ["users", "bob"], "method": "GET", "user_id": "alice"}
}
```
Both of these files are saved in the same directory.
```console
$ ls
example.rego example_test.rego
```
To exercise the policy, run the `opa test` command in the directory containing the files.
```console
$ opa test . -v
data.authz_test.test_post_allowed: PASS (1.417µs)
data.authz_test.test_get_anonymous_denied: PASS (426ns)
data.authz_test.test_get_user_allowed: PASS (367ns)
data.authz_test.test_get_another_user_denied: PASS (320ns)
--------------------------------------------------------------------------------
PASS: 4/4
```
The `opa test` output indicates that all of the tests passed.
Try exercising the tests a bit more by removing the first rule in **example.rego**.
```console
$ opa test . -v
FAILURES
--------------------------------------------------------------------------------
data.authz_test.test_post_allowed: FAIL (277.306µs)
query:1 Enter data.authz_test.test_post_allowed = _
example_test.rego:3 | Enter data.authz_test.test_post_allowed
example_test.rego:4 | | Fail data.authz_test.allow with input as {"method": "POST", "path": ["users"]}
query:1 | Fail data.authz_test.test_post_allowed = _
SUMMARY
--------------------------------------------------------------------------------
data.authz_test.test_post_allowed: FAIL (277.306µs)
data.authz_test.test_get_anonymous_denied: PASS (124.287µs)
data.authz_test.test_get_user_allowed: PASS (242.2µs)
data.authz_test.test_get_another_user_denied: PASS (131.964µs)
--------------------------------------------------------------------------------
PASS: 3/4
FAIL: 1/4
```
## Enriched Test Report With Variable Values
Sometimes, e.g. when testing rules with complex output, it can be useful to know more about the circumstances that caused a certain expression to fail a test.
The `--var-values` flag can be used to enrich the test report with the exact expression that caused a test rule to fail, including the values of any variables or references used in the expression.
Consider the following utility module:
```live:example_vars:module:read_only,openable
package authz
allowed_actions(user) := [action |
user in data.actions[action]
]
```
with accompanying tests:
```live:example_vars/test:module:read_only
package authz_test
import data.authz
test_allowed_actions_all_can_read if {
users := ["alice", "bob", "jane"]
r := ["alice", "bob"]
w := ["jane"]
p := {"read": r, "write": w}
every user in users {
"read" in authz.allowed_actions(user) with data.actions as p
}
}
```
Exercising the tests with the `--var-values` flag:
```console
opa test . --var-values
FAILURES
--------------------------------------------------------------------------------
data.authz_test.test_allowed_actions_all_can_read: FAIL (904µs)
util_test.rego:13:
"read" in authz.allowed_actions(user) with data.actions as p
| | |
| | {"read": ["alice", "bob"], "write": ["jane"]}
| "jane"
["write"]
SUMMARY
--------------------------------------------------------------------------------
util_test.rego:
data.authz_test.test_allowed_actions_all_can_read: FAIL (904µs)
--------------------------------------------------------------------------------
FAIL: 1/1
```
The test failed because it expected users with **write** permission to implicitly also have the **read** permission, an expectation the function under test didn't meet.
By including the failing expression and its local variable assignments in the test report, we make troubleshooting easier for the developer, as it's immediately apparent what assertion and combination of parameters caused the test to fail.
## Test Format
Tests are expressed as standard Rego rules with a convention that the rule
name is prefixed with `test_`. It's a good practice for tests to be placed in a package suffixed with `_test`, but not a requirement.
```live:example_format:module:read_only
package mypackage_test
import data.mypackage
test_some_descriptive_name if {
# test logic
}
```
## Test Discovery
The `opa test` subcommand runs all of the tests (i.e., rules prefixed with
`test_`) found in Rego files passed on the command line. If directories are
passed as command line arguments, `opa test` will load their file contents
recursively.
## Specifying Tests to Run
The `opa test` subcommand supports a `--run`/`-r` regex option to further
specify which of the discovered tests should be evaluated. The option supports
[re2 syntax](https://github.com/google/re2/wiki/Syntax)
## Test Results
If the test rule is undefined or generates a non-`true` value the test result
is reported as `FAIL`. If the test encounters a runtime error (e.g., a divide
by zero condition) the test result is marked as an `ERROR`. Tests prefixed with
`todo_` will be reported as `SKIPPED`. Otherwise, the test result is marked as
`PASS`.
**pass_fail_error_test.rego**:
```live:example_results:module:read_only
package example_test
import data.example
# This test will pass.
test_ok if true
# This test will fail.
test_failure if 1 == 2
# This test will error.
test_error if 1 / 0
# This test will be skipped.
todo_test_missing_implementation if {
example.allow with data.roles as ["not", "implemented"]
}
```
By default, `opa test` reports the number of tests executed and displays all
of the tests that failed or errored.
```console
$ opa test pass_fail_error_test.rego
data.example_test.test_failure: FAIL (253ns)
data.example_test.test_error: ERROR (289ns)
pass_fail_error_test.rego:15: eval_builtin_error: div: divide by zero
--------------------------------------------------------------------------------
PASS: 1/3
FAIL: 1/3
ERROR: 1/3
```
By default, OPA prints the test results in a human-readable format. If you
need to consume the test results programmatically, use the JSON output format.
```bash
opa test --format=json pass_fail_error_test.rego
```
```json
[
{
"location": {
"file": "pass_fail_error_test.rego",
"row": 4,
"col": 1
},
"package": "data.example_test",
"name": "test_ok",
"duration": 618515
},
{
"location": {
"file": "pass_fail_error_test.rego",
"row": 9,
"col": 1
},
"package": "data.example_test",
"name": "test_failure",
"fail": true,
"duration": 322177
},
{
"location": {
"file": "pass_fail_error_test.rego",
"row": 14,
"col": 1
},
"package": "data.example_test",
"name": "test_error",
"error": {
"code": "eval_internal_error",
"message": "div: divide by zero",
"location": {
"file": "pass_fail_error_test.rego",
"row": 15,
"col": 5
}
},
"duration": 345148
}
]
```
## Parameterized Tests and Data-driven Testing
A test rule can define multiple test cases for evaluation.
Test cases are declared by adding their name(s) to the rule as variables in its head's reference, and are evaluated through regular enumeration.
**example_test.rego**:
```live:example_test_cases:module:read_only
package example_test
test_concat[note] if {
some note, tc in {
"empty + empty": {
"a": [],
"b": [],
"exp": [],
},
"empty + filled": {
"a": [],
"b": [1, 2],
"exp": [1, 2],
},
"filled + filled": {
"a": [1, 2],
"b": [3, 4],
"exp": [1, 2, 3], # Faulty expectation, this test case will fail
},
}
act := array.concat(tc.a, tc.b)
act == tc.exp
}
```
```console
$ opa test example_test.rego
example_test.rego:
data.example_test.test_concat: FAIL (263.375µs)
empty + empty: PASS
empty + filled: PASS
filled + filled: FAIL
--------------------------------------------------------------------------------
FAIL: 1/1
```
Just as in regular evaluation, test-case data doesn't need to be declared as inline Rego, but can be loaded from json and yaml data files:
**file_example_test.rego**:
```live:example_file_test_cases:module:read_only
package example_test
import data.test_cases
test_concat[note] if {
some note, tc in test_cases
act := array.concat(tc.a, tc.b)
act == tc.exp
}
```
**file_example_test.yaml**:
```yaml
test_cases:
empty + empty:
a: []
b: []
exp: []
empty + filled:
a: []
b: [1, 2]
exp: [1, 2]
filled + filled:
a: [1, 2]
b: [3, 4]
exp: [1, 2, 3] # Faulty expectation, this test case will fail
```
```console
$ opa test file_example_test.rego file_example_test.yaml
file_example_test.rego:
data.example_test.test_concat: FAIL (280µs)
empty + empty: PASS
empty + filled: PASS
filled + filled: FAIL
--------------------------------------------------------------------------------
FAIL: 1/1
```
Test cases can be nested by declaring multiple test case name variables in the head reference.
This is useful when e.g. the same set of test cases can be used for asserting the same behaviour across slightly different circumstances:
**nested_example_test.rego**:
```live:example_nested_test_cases:module:read_only
package example_test
test_sign_token[note][alg] if {
some note, tc in {
"claims": {
"claims": {"foo": "bar"},
},
"no claims": {
"claims": {},
},
}
some alg in [
"HS256",
"HS333", # unknown signing algorithm, this test case will fail
"HS512",
]
secret := "foobar"
key := base64.encode(secret)
token := io.jwt.encode_sign({
"typ": "JWT",
"alg": alg
}, tc.claims, {
"kty": "oct",
"k": key
})
[valid, _, payload] := io.jwt.decode_verify(token, {"secret": secret})
valid
payload = tc.claims
}
```
```console
$ opa test nested_example_test.rego
nested_example_test.rego:
data.example_test.test_sign_token: FAIL (1.214541ms)
claims: FAIL
HS256: PASS
HS333: FAIL
HS512: PASS
no claims: FAIL
HS256: PASS
HS333: FAIL
HS512: PASS
--------------------------------------------------------------------------------
FAIL: 1/1
```
## Data and Function Mocking
OPA's `with` keyword can be used to replace the data document or called functions with mocks.
Both base and virtual documents can be replaced.
When replacing functions, built-in or otherwise, the following constraints are in place:
1. Replacing `internal.*` functions, or `rego.metadata.*`, or `eq`; or relations (`walk`) is not allowed.
2. Replacement and replaced function need to have the same arity.
3. Replaced functions can call the functions they're replacing, and those calls
will call out to the original function, and not cause recursion.
Below is a simple policy that depends on the data document.
**authz.rego**:
```live:with_keyword:module:read_only,openable
package authz
allow if {
some x in data.policies
x.name == "test_policy"
matches_role(input.role)
}
matches_role(my_role) if input.user in data.roles[my_role]
```
Below is the Rego file to test the above policy.
**authz_test.rego**:
```live:with_keyword/tests:module:read_only
package authz_test
import data.authz
policies := [{"name": "test_policy"}]
roles := {"admin": ["alice"]}
test_allow_with_data if {
authz.allow with input as {"user": "alice", "role": "admin"}
with data.policies as policies
with data.roles as roles
}
```
To exercise the policy, run the `opa test` command.
```console
$ opa test -v authz.rego authz_test.rego
data.authz_test.test_allow_with_data: PASS (697ns)
--------------------------------------------------------------------------------
PASS: 1/1
```
Below is an example to replace a **rule without arguments**.
**authz.rego**:
```live:with_keyword_rules:module:read_only
package authz
allow1 if allow2
allow2 if 2 == 1
```
**authz_test.rego**:
```live:with_keyword_rules/tests:module:read_only
package authz_test
import data.authz
test_replace_rule if {
authz.allow1 with authz.allow2 as true
}
```
```console
$ opa test -v authz.rego authz_test.rego
data.authz_test.test_replace_rule: PASS (328ns)
--------------------------------------------------------------------------------
PASS: 1/1
```
Here is an example to replace a rule's **built-in function** with a user-defined function.
**authz.rego**:
```live:with_keyword_builtins:module:read_only
package authz
import data.jwks.cert
allow if {
[true, _, _] = io.jwt.decode_verify(input.headers["x-token"], {"cert": cert, "iss": "corp.issuer.com"})
}
```
**authz_test.rego**:
```live:with_keyword_builtins/tests:module:read_only
package authz_test
import data.authz
mock_decode_verify("my-jwt", _) := [true, {}, {}]
mock_decode_verify(x, _) := [false, {}, {}] if x != "my-jwt"
test_allow if {
authz.allow with input.headers["x-token"] as "my-jwt"
with data.jwks.cert as "mock-cert"
with io.jwt.decode_verify as mock_decode_verify
}
```
```console
$ opa test -v authz.rego authz_test.rego
data.authz_test.test_allow: PASS (458.752µs)
--------------------------------------------------------------------------------
PASS: 1/1
```
In simple cases, a function can also be replaced with a value, as in
```live:with_keyword_builtins/tests/value:module:read_only
test_allow_value if {
authz.allow
with input.headers["x-token"] as "my-jwt"
with data.jwks.cert as "mock-cert"
with io.jwt.decode_verify as [true, {}, {}]
}
```
Every invocation of the function will then return the replacement value, regardless
of the function's arguments.
Note that it's also possible to replace one built-in function by another; or a non-built-in
function by a built-in function.
**authz.rego**:
```live:with_keyword_funcs:module:read_only
package authz
replace_rule if {
replace(input.label)
}
replace(label) if {
label == "test_label"
}
```
**authz_test.rego**:
```live:with_keyword_funcs/tests:module:read_only
package authz_test
import data.authz
test_replace_rule if {
authz.replace_rule with input.label as "does-not-matter" with replace as true
}
```
```console
$ opa test -v authz.rego authz_test.rego
data.authz_test.test_replace_rule: PASS (648.314µs)
--------------------------------------------------------------------------------
PASS: 1/1
```
## Coverage
In addition to reporting pass, fail, and error results for tests, `opa test`
can also report _coverage_ for the policies under test.
The coverage report includes all of the lines evaluated and not evaluated in
the Rego files provided on the command line. When a line is not covered it
indicates one of two things:
- If the line refers to the head of a rule, the body of the rule was never true.
- If the line refers to an expression in a rule, the expression was never evaluated.
It is also possible that [rule indexing](./policy-performance/#use-indexed-statements)
has determined some path unnecessary for evaluation, thereby affecting the lines
reported as covered.
If we run the coverage report on the original **example.rego** file without
`test_get_user_allowed` from **example_test**.rego the report will indicate
that line 8 is not covered.
```bash
opa test --coverage --format=json example.rego example_test.rego
```
```json
{
"files": {
"example.rego": {
"covered": [
{
"start": {
"row": 3
},
"end": {
"row": 5
}
},
{
"start": {
"row": 9
},
"end": {
"row": 11
}
}
],
"not_covered": [
{
"start": {
"row": 8
},
"end": {
"row": 8
}
}
],
"covered_lines": 6,
"not_covered_lines": 1,
"coverage": 85.7
},
"example_test.rego": {
"covered": [
{
"start": {
"row": 3
},
"end": {
"row": 4
}
},
{
"start": {
"row": 7
},
"end": {
"row": 8
}
},
{
"start": {
"row": 11
},
"end": {
"row": 12
}
}
],
"covered_lines": 6,
"coverage": 100
},
"covered_lines": 12,
"not_covered_lines": 1,
"coverage": 92.3
}
}
```
## Ecosystem Projects
<EcosystemEmbed feature="policy-testing">
Here are some projects that can help you with policy testing:
</EcosystemEmbed>
docs/new/docs/privacy.md
+79
View File
@@ -0,0 +1,79 @@
---
title: Privacy
---
This document provides details about OPA's anonymous information reporting feature.
## Overview
OPA periodically reports its version and specific anonymous runtime statistics to a publicly hosted, external service.
The reports contain the OPA version number (e.g., v0.12.3), a randomly generated UUID and the following runtime statistics:
- heap usage in bytes
This feature is only applicable to the `opa run` and `opa version` commands.
In case of the `opa run` command, this feature is **ON by-default** and can be easily disabled by specifying
the `--disable-telemetry` flag. When OPA is started in either the server or repl mode, OPA calls the external service
on a best-effort basis and shares the version it's running and other statistics such as current memory usage.
The time taken to execute the remote call and process the subsequent response from the external service does not
delay OPA's start-up.
In case of the `opa version` command, this feature can be enabled by specifying the `--check` or `-c` flag.
## External Service
OPA uploads its information by default at [telemetry.openpolicyagent.org](https://telemetry.openpolicyagent.org).
The environment variable `OPA_TELEMETRY_SERVICE_URL` can be used to configure the external service OPA reports to.
Sample HTTP request from OPA to the external service looks like this:
```http
POST /v1/version HTTP/1.1
Host: telemetry.openpolicyagent.org
Content-Type: application/json
User-Agent: "Open Policy Agent/v0.12.3 (darwin, amd64)"
```
```json
{
"id": "08c1d850-6065-478a-b9b5-a8f9f464ad33",
"version": "v0.12.3",
"heap_usage_bytes": "596000"
}
```
The _id_ field in the request body above is a version 4 random UUID generated when OPA starts.
The external service checks the OPA version reported by a remote OPA client and responds with information about the
latest OPA release. This information includes a link to download the latest OPA version, release notes etc.
Sample response from the external service looks like this:
```json
{
"latest": {
"download": "https://openpolicyagent.org/downloads/v0.19.2/opa_darwin_amd64",
"release_notes": "https://github.com/open-policy-agent/opa/releases/tag/v0.19.2",
"latest_release": "v0.19.2"
}
}
```
The external service response contains a link to download the latest released OPA binary for client's platform, and a link
to the OPA release notes.
## Benefits
- OPA's anonymous version reporting feature provides users with up-to-date information about new OPA versions while
still executing the familiar OPA `run` and `version` commands. It helps users stay abreast of OPA's latest capabilities
and hence empowers them to make informed decisions while upgrading their OPA deployments.
- OPA maintainers and the [Cloud Native Computing Foundation](https://cncf.io) (CNCF) executive staff can use the version
reports for obtaining more information about OPA usage and engagement. For example, the information can be used in
making better decisions about OPA's deprecation cycle.
- Reporting a running OPA's memory usage can help to better understand how much memory an OPA instance is consuming and
thereby drive optimization efforts around better resource utilization. Some users have concerns around OPA's memory usage
and hence this information can help OPA maintainers quantify the number of impacted OPA deployments and also guide future
features and priorities for the project.
docs/new/docs/rest-api.md
+2216
View File
File diff suppressed because it is too large Load Diff
docs/new/docs/security.md
+654
View File
@@ -0,0 +1,654 @@
---
title: Security
---
This document provides guidelines for deploying OPA inside untrusted
environments. You should read this document if you are deploying OPA as a
service.
Securing the API involves configuring OPA to use TLS, authentication, and
authorization so that:
- Traffic between OPA and clients is encrypted.
- Clients verify the OPA API endpoint identity.
- OPA verifies client identities.
- Clients are only granted access to specific APIs or sections of [The `data` Document](./philosophy/#the-opa-document-model).
## TLS and HTTPS
HTTPS is configured by specifying TLS credentials via command line flags at
startup:
- `--tls-cert-file=<path>` specifies the path of the file containing the TLS certificate.
- `--tls-private-key-file=<path>` specifies the path of the file containing the TLS private key.
OPA will exit immediately with a non-zero status code if only one of these flags
is specified.
The server can track the certificate and key files' contents, and reload them if necessary:
- `--tls-cert-refresh-period=<duration>` specifies how often OPA should check the TLS certificate and
private key file for changes (defaults to 0s, disabling periodic refresh). This argument accepts
any duration, such as "30s", "5m" or "24h".
Note that for using TLS-based authentication, a CA cert file can be provided:
- `--tls-ca-cert-file=<path>` specifies the path of the file containing the CA cert.
If provided, it will be used to validate clients' TLS certificates when using TLS
authentication (see below).
By default, OPA ignores insecure HTTP connections when TLS is enabled. To allow
insecure HTTP connections in addition to HTTPS connections, provide another
listening address with `--addr`. For example:
```bash
opa run --server \
--log-level debug \
--tls-cert-file public.crt \
--tls-private-key-file private.key \
--addr https://0.0.0.0:8181 \
--addr http://localhost:8282
```
### 1. Generate the TLS credentials for OPA (Example)
```bash
openssl genrsa -out private.key 2048
openssl req -new -x509 -sha256 -key private.key -out public.crt -days 1
```
> We have generated a self-signed certificate for example purposes here. DO NOT
> rely on self-signed certificates outside of development without understanding
> the risks.
### 2. Start OPA with TLS enabled
```bash
opa run --server --log-level debug \
--tls-cert-file public.crt \
--tls-private-key-file private.key
```
### 3. Try to access the API with HTTP
```bash
curl http://localhost:8181/v1/data
```
### 4. Access the API with HTTPS
```bash
curl -k https://localhost:8181/v1/data
```
:::info
We have to use cURL's `-k/--insecure` flag because we are using a self-signed certificate.
:::
## Interface Binding
OPA can be configured to listen on specific interfaces using the `--addr` flag. For example:
```bash
opa run --server \
--log-level debug \
--addr 0.0.0.0:8181 \
```
By default, OPA binds to `localhost`, which prevents the OPA server from being exposed to services running outside of the same machine.
In situations where OPA is not intended to be exposed to remote services, it is recommended to bind OPA to the localhost interface, which only allows connections from the same machine. If it is necessary to expose OPA to remote services, ensure to follow the security recommendations on this page, such as requiring authentication.
## Authentication and Authorization
This section shows how to configure OPA to authenticate and authorize client
requests. Client-side authentication of the OPA API endpoint should be handled
with TLS.
Authentication and authorization allow OPA to:
- Verify client identities.
- Control client access to APIs and data.
Both are configured via command line flags:
- `--authentication=<scheme>` specifies the authentication scheme to use.
- `--authorization=<scheme>` specifies the authorization scheme to use.
By default, OPA does not perform authentication or authorization and these flags
default to `off`.
For authentication, OPA supports:
- [Bearer tokens](./rest-api#bearer-tokens): Bearer tokens are enabled by
starting OPA with `--authentication=token`. When the `token` authentication
mode is enabled, OPA will extract the Bearer token from incoming API requests
and provide to the authorization handler. When you use the `token`
authentication, you must configure an authorization policy that checks the
tokens. If the client does not supply a Bearer token, the `input.identity`
value will be undefined when the authorization policy is evaluated.
- Client TLS certificates: Client TLS authentication is enabled by starting
OPA with `--authentication=tls`. When this authentication mode is enabled,
OPA will require all clients to provide a client certificate. It is verified
against the CA certificate(s) provided via `--tls-ca-cert-file`. Upon successful
verification, the `input.identity` value is set to the TLS certificate's
subject.
Note that TLS authentication does not disable non-HTTPS listeners. To ensure
that all your communication is secured, it should be paired with an
authorization policy (see below) that at least requires the client identity
(`input.identity`) to _be set_.
For authorization, OPA relies on policy written in Rego. Authorization is
enabled by starting OPA with `--authorization=basic`.
When the `basic` authorization scheme is enabled, a minimal authorization policy
must be provided on startup. The authorization policy must be structured as follows:
```live:system_ns:module:read_only
# The "system" namespace is reserved for internal use
# by OPA. Authorization policy must be defined under
# system.authz as follows:
package system.authz
default allow := false # Reject requests by default.
allow if {
# Logic to authorize request goes here.
}
```
When OPA receives a request, it executes a query against the document defined
`data.system.authz.allow`. The implementation of the policy may span multiple
packages however it is recommended that administrators keep the policy under the
`system` namespace.
If the document produced by the `allow` rule is `true`, the request is
processed normally. If the document is undefined or **not** `true`, the
request is rejected immediately. The count of requests rejected by an OPA instance
are surfaced via the performance metrics in the [Status](./management-status) information.
OPA provides the following `input` document when executing the authorization
policy. Since the schema for the `input` document is known to OPA, it performs automatic type checking of this document
and reports any errors resulting from the schema check. The `--skip-known-schema-check` flag can be passed to `opa run`
to disable automatic type checking of this `input` document.
<!-- TODO(sr): check if "jsonc" looks alright on netlify -->
```jsonc
{
# Identity value established by authentication scheme.
# When Bearer tokens are used, the identity is
# set to the Bearer token value.
# When TLS client certificates are used, the identity
# is set to the certificate subject RDNSequence.
# E.g. "OU=opa-client-01,O=Example"
# Note: client certificate data is available in the
# 'client_certificates' key.
"identity": "",
# Client certificates provided by the client when calling OPA
# over an mTLS connection. Represented in input as a list of
# Go x509.Certificate objects marshalled as JSON.
"client_certificates": [],
# One of {"GET", "POST", "PUT", "PATCH", "DELETE"}.
"method": "",
# URL path represented as an array.
# For example: /v1/data/exempli-gratia
# is represented as ["v1", "data", "exampli-gratia"]
"path": [...],
# URL parameters represented as an object of string arrays.
# For example: metrics&explain=true is represented as
# {"metrics": [""], "explain": ["true"]}
"params": {"...": ...},
# Request headers represented as an object of string arrays.
#
# Example Request Headers:
#
# host: acmecorp.com
# x-custom: secretvalue
#
# Example input.headers Value:
#
# {"Host": ["acmecorp.com"], "X-Custom": ["mysecret"]}
#
# Example header check:
#
# input.headers["X-Custom"][_] == "mysecret"
#
# Header keys follow canonical MIME form. The first character and any
# characters following a hyphen are uppercase. The rest are lowercase.
# If the header key contains space or invalid header field bytes,
# no conversion is performed.
"headers": {"...": [...]},
# Request message body if present for applicable APIs.
#
# Example Request:
#
# POST v1/data HTTP/1.1
# Content-Type: application/json
#
# {"input": {"action": "trade", "stock": "ACME"}}
#
# Example input.body Value:
#
# {"input": {"action": "trade", "stock": "ACME"}}
#
# Example body check:
#
# input.body.input.stock == "ACME"
#
# The 'body' field is provided for the following APIs:
#
# * POST v1/data
# * POST v0/data
# * POST /
"body": ...,
}
```
At a minimum, the authorization policy should grant access to a special root
identity:
```live:system_authz_secret:module:read_only
package system.authz
default allow := false # Reject requests by default.
allow if { # Allow request if...
"secret" == input.identity # Identity is the secret root key.
}
```
When OPA is configured with this minimal authorization policy, requests without
authentication are rejected:
```http
GET /v1/policies HTTP/1.1
```
Response:
```http
HTTP/1.1 401 Unauthorized
Content-Type: application/json
```
```json
{
"code": "unauthorized",
"message": "request rejected by administrative policy"
}
```
However, if Bearer token authentication is enabled and the request includes the
secret from above, the request is allowed:
```http
GET /v1/policies HTTP/1.1
Authorization: Bearer secret
```
Response:
```http
HTTP/1.1 200 OK
Content-Type: application/json
```
Besides boolean responses, authorization policies can change the message included
in the deny response. Do do that, policy decisions must yield an object response as
follows:
```live:system_authz_object_resp:module:read_only
package system.authz
default allow := {
"allowed": false,
"reason": "unauthorized resource access",
}
allow := {"allowed": true} if { # Allow request if...
"secret" == input.identity # identity is the secret root key.
}
allow := {"allowed": false, "reason": reason} if {
not input.identity
reason := "no identity provided"
}
```
### Token-based Authentication Example
When Bearer tokens are used for authentication, the policy should at minimum
validate the identity:
```live:system_authz_bearer:module:read_only
package system.authz
# Tokens may defined in policy or pushed into OPA as data.
tokens := {
"my-secret-token-foo": {
"roles": ["admin"]
},
"my-secret-token-bar": {
"roles": ["service-1"]
},
"my-secret-token-baz": {
"roles": ["service-2", "service-3"]
}
}
default allow := false # Reject requests by default.
allow if { # Allow request if...
input.identity == "secret" # Identity is the secret root key.
}
allow if { # Allow request if...
tokens[input.identity] # Identity exists in "tokens".
}
```
To complete this example, the policy could further restrict tokens to specific
documents:
```live:system_authz_bearer_complete:module:read_only
package system.authz
# Rights may be defined in policy or pushed into OPA as data.
rights := {
"admin": {
"path": "*"
},
"service-1": {
"path": ["v1", "data", "exempli", "gratia"]
},
"service-2": {
"path": ["v1", "data", "par", "example"]
}
}
# Tokens may be defined in policy or pushed into OPA as data.
tokens := {
"my-secret-token-foo": {
"roles": ["admin"]
},
"my-secret-token-bar": {
"roles": ["service-1"]
},
"my-secret-token-baz": {
"roles": ["service-2", "service-3"]
}
}
default allow := false # Reject requests by default.
allow if { # Allow request if...
some right
identity_rights[right] # Rights for identity exist, and...
right.path == "*" # Right.path is '*'.
}
allow if { # Allow request if...
some right
identity_rights[right] # Rights for identity exist, and...
right.path == input.path # Right.path matches input.path.
}
identity_rights contains right if { # Right is in the identity_rights set if...
token := tokens[input.identity] # Token exists for identity, and...
role := token.roles[_] # Token has a role, and...
right := rights[role] # Role has rights defined.
}
```
### TLS-based Authentication Example
To set up authentication based on mutual TLS, we will need three certificates:
1. the CA cert (self-signed),
2. the server cert (signed by the CA), and
3. the client cert (signed by the CA).
We use `openssl` to create the example certificates and keys used in this demo. In production, creation of certificates
and keys should be handled by an automated process out of scope for this tutorial.
Note that we also create an extra client cert (client-2). While this certificate is signed by the same CA, it's identity
is different. We'll use this to show our authorization policy in action.
```bash
# CA
openssl ecparam -out ca-key.pem -name prime256v1 -genkey
openssl req -x509 -new -nodes -key ca-key.pem -days 30 -out ca.pem -subj "/CN=my-ca"
# client 1
cat <<EOF >req.cnf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[v3_req]
basicConstraints = CA:FALSE
subjectAltName = @alt_names
[alt_names]
URI.1 = spiffe://example.com/client-1
EOF
openssl ecparam -out client-key-1.pem -name prime256v1 -genkey
openssl req -new -key client-key-1.pem -out csr.pem -subj "/CN=client-1" -config req.cnf
openssl x509 -req -in csr.pem -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out client-cert-1.pem -days 10 -extensions v3_req -extfile req.cnf -sha256
# client 2
cat <<EOF >req.cnf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[v3_req]
basicConstraints = CA:FALSE
subjectAltName = @alt_names
[alt_names]
URI.1 = spiffe://example.com/client-2
EOF
openssl ecparam -out client-key-2.pem -name prime256v1 -genkey
openssl req -new -key client-key-2.pem -out csr.pem -subj "/CN=client-2" -config req.cnf
openssl x509 -req -in csr.pem -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out client-cert-2.pem -days 10 -extensions v3_req -extfile req.cnf -sha256
# create server cert with IP and DNS SANs
cat <<EOF >req.cnf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = opa.example.com
IP.1 = 127.0.0.1
URI.1 = spiffe://example.com/server
EOF
openssl ecparam -out server-key.pem -name prime256v1 -genkey
openssl req -new -key server-key.pem -out csr.pem -subj "/CN=server" -config req.cnf
openssl x509 -req -in csr.pem -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -days 10 -extensions v3_req -extfile req.cnf -sha256
```
We also create an example authorization policy file, called `check.rego`. This example `system.authz` policy will check
the certificate ID against a list of allowed paths as defined in a simple Access Control List.
:::danger
When choosing messages to return to unauthorized clients in `system.authz` policies, be careful not to expose sensitive
information such as which paths are allowed.
:::
```live:system_authz_x509:module:read_only
package system.authz
id_uri := input.client_certificates[0].URIs[0]
id_string := sprintf("%s://%s%s", [id_uri.Scheme, id_uri.Host, id_uri.Path])
# client_acl represents an access control list and may defined in policy or pushed into OPA as data changes.
client_acl := {
"spiffe://example.com/client-1": [["v1", "data"]],
"spiffe://example.com/client-2": [],
}
default allow := {"allowed": false, "reason": "Access denied: unknown caller"}
allow := {"allowed": true} if {
input.path in client_acl[id_string]
} else := {
"allowed": false,
"reason": sprintf("%s is not allowed to call /%s", [
id_string,
concat("/", input.path),
]),
}
```
Now, we're ready to starting the server with `-authentication=tls` and the
certificate-related parameters:
```console
$ opa run -s \
--tls-cert-file server-cert.pem \
--tls-private-key-file server-key.pem \
--tls-ca-cert-file ca.pem \
--authentication=tls \
--authorization=basic \
-a https://127.0.0.1:8181 \
check.rego
{"addrs":["https://127.0.0.1:8181"],"diagnostic-addrs":[],"level":"info","msg":"Initializing server.","time":"2023-01-04T10:31:12Z"}
```
We can use `curl` to validate our TLS-based authentication setup:
First, we use the client certificate that was signed by the CA, and has a subject
matching our authorization policy:
```console
$ curl --key client-key-1.pem \
--cert client-cert-1.pem \
--cacert ca.pem \
--resolve opa.example.com:8181:127.0.0.1 \
https://opa.example.com:8181/v1/data
{"result":{}}
```
Note that we're passing the CA cert to curl -- this is done to have curl accept
the server's certificate, which has been signed by our CA cert.
Since we've set up an IP SAN, we may also `curl https://127.0.0.1:8181/v1/data`
directly. (To keep our examples focused, we'll do that from here on.)
Using a valid certificate whose subject will be declined by our authorization
policy:
```console
$ curl --key client-key-2.pem \
--cert client-cert-2.pem \
--cacert ca.pem \
https://127.0.0.1:8181/v1/data
{
"code": "unauthorized",
"message": "spiffe://example.com/client-2 is not allowed to call /v1/data"
}
```
Finally, we'll attempt to query without a client certificate:
```console
$ curl --cacert ca.pem https://127.0.0.1:8181/v1/data
curl: (56) LibreSSL SSL_read: error:1404C412:SSL routines:ST_OK:sslv3 alert bad certificate, errno 0
```
As you can see, TLS-based authentication disallows these request before even invoking the `system.authz` policy.
## Secure Health and Monitoring
Often OPA is deployed locally to the host where the client resides (side-car or
similar model). In these deployments it is ideal to only expose the API via
`localhost` to prevent any remote clients from reaching OPA at all. The downside
to this approach is that it blocks remote monitoring systems that require access
to `/health` or `/metrics`.
The solution is to configure OPA with a separate diagnostic listener by
providing the `--diagnostic-addr` flag, for example:
```
$ opa run \
-s \
--addr localhost:8181 \
--diagnostic-addr :8282
```
The configuration above would expose only `/health` and `/metrics` API's on port
`8282` while keeping the normal REST API bound to `localhost:8181`.
> When the diagnostic listener is enabled, the `/metrics` and `/health` APIs will
> still be exposed on the normal listener.
## Hardened Configuration Example
You can run a hardened OPA deployment with minimal configuration. There are a
few things to keep in mind:
- Limit API access to host-local clients executing policy queries.
- Configure TLS (for localhost TCP) or a UNIX domain socket.
- Do not pass credentials as command-line arguments.
- Run OPA as a non-root user ideally inside it's own account.
With OPA configured to fetch policies using the [Bundles](./management-bundles) feature
you can configure OPA with a restrictive authorization policy that only grants
clients access to the default policy decision, i.e., `POST /`:
```live:hardened_example:module:read_only
package system.authz
# Deny access by default.
default allow := false
# Allow anonymous access to the default policy decision.
allow if {
input.method == "POST"
input.path == [""]
}
```
The example below shows flags that tell OPA to:
- Authorize all API requests (`--authorization=basic`)
- Listen on localhost for HTTPS (not HTTP!) connections (`--addr`, `--tls-cert-file`, `--tls-private-key-file`)
- Download bundles from a remote HTTPS endpoint (`--set` flags and `--set-file` flag)
```bash
opa run \
--server \
--authorization=basic \
--addr=https://localhost:8181 \
--tls-cert-file=/var/tmp/server.crt \
--tls-private-key-file=/var/tmp/server.key \
--set=bundles.authz.service=default \
--set=bundles.authz.resource=myapp_authz_bundle \
--set=services.default.url=https://control.acmecorp.com \
--set-file=services.default.credentials.bearer.token=/var/tmp/secret-bearer-token
```
> The `/var/tmp/secret-bearer-token` will store the credential in plaintext. You
> should make sure that file permission(s) are setup to limit access.
docs/new/docs/ssh-and-sudo-authorization.md
+438
View File
@@ -0,0 +1,438 @@
---
title: SSH and sudo
---
Host-level access controls are an important part of every organization's
security strategy. Using [Linux-PAM](http://tldp.org/HOWTO/User-Authentication-HOWTO/x115.html) and OPA
we can extend policy-based access control to SSH and sudo.
## Goals
This tutorial shows how you can use OPA and Linux-PAM to enforce fine-grained,
host-level access controls over SSH and sudo.
Linux-PAM can be configured to delegate authorization decisions to plugins
(shared libraries). In this case, we have created an OPA-based plugin that can
be configured to authorize SSH and sudo access. The OPA-based Linux-PAM plugin
used in this tutorial can be found at [open-policy-agent/contrib](https://github.com/open-policy-agent/contrib/tree/main/pam_opa).
For this tutorial, our desired policy is:
- Admins can SSH into any host and run sudo commands.
- Normal users can SSH into hosts that they have _contributed_ to and run sudo commands.
Furthermore, we'll assume we have the following set of users and hosts:
- `frontend-dev` is a developer who contributes to the app running on the `frontend` host.
- `backend-dev` is a developer who contributes to the app running on the `backend` host.
- `ops` is an administrator for the organization.
Authentication (verifying user identity) is outside the scope of OPA's
responsibility so this tutorial relies on identities being statically
defined. In real-world scenarios authentication can be delegated to SSH itself
(authorized_keys) or other identity management systems.
Let's get started.
## Prerequisites
This tutorial requires [Docker Compose](https://docs.docker.com/compose/install/) to run dummy SSH hosts along
with OPA. The dummy SSH hosts are just containers running sshd inside.
## Steps
### 1. Bootstrap the tutorial environment using Docker Compose.
First, create a `tutorial-docker-compose.yaml` file that runs OPA and the containers that
represent our backend and frontend hosts.
**tutorial-docker-compose.yaml**:
```yaml
version: "2"
services:
opa:
image: openpolicyagent/opa:{{< current_docker_version >}}
ports:
- "8181:8181"
# WARNING: OPA is NOT running with an authorization policy configured. This
# means that clients can read and write policies in OPA. If you are
# deploying OPA in an insecure environment, be sure to configure
# authentication and authorization on the daemon. See the Security page for
# details: https://www.openpolicyagent.org/docs/security.html.
command:
- "run"
- "--server"
- "--set=decision_logs.console=true"
- "--set=services.nginx.url=http://bundle_server"
- "--set=bundles.nginx.service=nginx"
- "--set=bundles.nginx.resource=bundles/bundle.tar.gz"
depends_on:
- bundle_server
frontend:
image: openpolicyagent/demo-pam
ports:
- "2222:22"
volumes:
- ./frontend_host_id.json:/etc/host_identity.json
backend:
image: openpolicyagent/demo-pam
ports:
- "2223:22"
volumes:
- ./backend_host_id.json:/etc/host_identity.json
bundle_server:
image: nginx:1.20.0-alpine
ports:
- 8888:80
volumes:
- ./bundles:/usr/share/nginx/html/bundles
```
The `tutorial-docker-compose.yaml` file requires two other local files:
`frontend_host_id.json` and `backend_host_id.json`. These files are mounted
into the containers representing our hosts. The content of the file provides
_context_ that the PAM module provides as input when executing queries
against OPA.
Create the extra files required by tutorial-docker-compose.yaml:
```shell
echo '{"host_id": "frontend"}' > frontend_host_id.json
echo '{"host_id": "backend"}' > backend_host_id.json
```
> In real-world scenarios, these files could contain arbitrary information that we want to expose to the policy.
Finally, run `docker-compose` to pull and run the containers.
```shell
docker-compose -f tutorial-docker-compose.yaml up
```
This tutorial uses a special Docker image named `openpolicyagent/demo-pam` to simulate an SSH server.
This image contains pre-created Linux accounts for our users, and the required PAM module is
pre-configured inside the `sudo` and `sshd` files in `/etc/pam.d/`.
### 2. Create a Bundle for the policies and data.
In another terminal, create the policies and data that OPA will use to control access to the hosts.
First, create folder called bundles and cd into it.
```bash
mkdir bundles
cd bundles
```
Next, create a policy that will tell the PAM module to collect context that is required for authorization.
For more details on what this policy should look like, see [this documentation](https://github.com/open-policy-agent/contrib/tree/main/pam_opa/pam#pull).
**pull.rego**:
```live:ssh_pull:module:read_only
package pull
# Which files should be loaded into the context?
files := ["/etc/host_identity.json"]
# Which environment variables should be loaded into the context?
env_vars := []
```
Create the policies that will authorize SSH and sudo requests.
The `input` which makes up the authorization context in the policy below will also
include some default values, such as the username making the request. See
[this documentation](https://github.com/open-policy-agent/contrib/tree/main/pam_opa/pam#authz)
to get a better understanding of what the `input` to the authorization policy will look like.
Unlike the _pull_ policy, we'll create separate _authz_ policies
for SSH and `sudo` for more fine-grained control.
In production, it makes more sense to have this separation for _display_ and _pull_ as well.
Create the SSH authorization policy. It should allow admins to SSH into all hosts,
and non-admins to only SSH into hosts that they contributed code to.
**sshd_authz.rego**:
```live:sshd_authz:module:read_only
package sshd.authz
import input.pull_responses
import input.sysinfo
import data.hosts
# By default, users are not authorized.
default allow := false
# Allow access to any user that has the "admin" role.
allow if {
data.roles.admin[_] == input.sysinfo.pam_username
}
# Allow access to any user who contributed to the code running on the host.
#
# This rule gets the "host_id" value from the file "/etc/host_identity.json".
# It is available in the input under "pull_responses" because we
# asked for it in our pull policy above.
#
# It then compares all the contributors for that host against the username
# that is asking for authorization.
allow if {
hosts[pull_responses.files["/etc/host_identity.json"].host_id].contributors[_] == sysinfo.pam_username
}
# If the user is not authorized, then include an error message in the response.
errors contains "Request denied by administrative policy" if {
not allow
}
```
Create the `sudo` authorization policy. It should allow only admins to use `sudo`.
**sudo_authz.rego**:
```live:sudo_authz:module:read_only
package sudo.authz
# By default, users are not authorized.
default allow := false
# Allow access to any user that has the "admin" role.
allow if {
data.roles.admin[_] == input.sysinfo.pam_username
}
# If the user is not authorized, then include an error message in the response.
errors contains "Request denied by administrative policy" if {
not allow
}
```
Now we need to create the data that represents our roles, hots, and contributors into OPA.
Create a folder called roles, and the following data file.
```shell
mkdir roles
cat <<EOF > roles/data.json
{
"admin": ["ops"]
}
EOF
```
Create a folder called hosts, and the following data file.
```shell
mkdir hosts
cat <<EOF > hosts/data.json
{
"frontend": {
"contributors": [
"frontend-dev"
]
},
"backend": {
"contributors": [
"backend-dev"
]
}
}
EOF
```
Finally create the bundle for the bundle server to use.
```bash
opa build -b .
```
Now you should have the following file structure setup.
```
.
└── tutorial-docker-compose.yaml
├── backend_host_id.json
├── frontend_host_id.json
├── bundles
│   ├── bundle.tar.gz
│   ├── pull.rego
│   ├── sshd_authz.rego
│   ├── sudo_authz.rego
│   ├── hosts
│   │   └── data.json
│   ├── roles
│   │   └── data.json
```
### 3. SSH and sudo as a user with the `admin` role.
First, let's try to access the hosts as the `ops` user. Recall, the `ops` user
has been granted the `admin` role (via the `PUT /data/roles` request above) and
users with the `admin` role can login to any host and perform sudo commands.
Login to the `frontend` host (which has SSH listening on port 2222) and run a command with sudo as the `ops` user.
```shell
ssh -p 2222 ops@localhost \
-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
sudo ls /
exit
```
You will see a lot of verbose logs from `sudo` as the PAM module goes through the motions.
This is intended so you can study how the PAM module works.
You can disable verbose logging by changing the `log_level` argument in the PAM
configuration. For more details see
[this documentation](https://github.com/open-policy-agent/contrib/tree/main/pam_opa/pam#configuration).
### 4. SSH as a user without the `admin` role.
Let's try a user without the admin role. Recall, that a non-admin user can SSH
into any host that they have _contributed to_.
The `frontend-dev` user contributed code to the `frontend` host so they should be
able to login.
```shell
ssh -p 2222 frontend-dev@localhost \
-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
```
Only admins can use `sudo`, so you shouldn't be able to run `sudo ls /`.
Since `frontend-dev` did not contribute to the code running on the
`backend` host (which has SSH listening on port 2223), they should not be able
to login.
```shell
ssh -p 2223 frontend-dev@localhost \
-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
```
### 5. Elevate a user's rights through policy.
Suppose you have a ticketing system for elevation, where you generate tickets for users
that need elevated rights, send the ticket to the user, and expire those tickets when
their rights should be removed.
Let's mock the current state of this simple ticketing system's API with some data.
```shell
mkdir elevate
cat <<EOF > elevate/data.json
{
"tickets": {
"frontend-dev": "1234"
}
}
EOF
```
This means that for now, if the `frontend-dev` user can provide ticket number `1234`,
they should be able to SSH into all servers.
Let's write policy to ensure that this happens.
First, we need to make the PAM module take input from the user.
**display.rego**:
```live:display:module:read_only
package display
# What should be prompted to the user?
display_spec := [
{
"message": "Please enter an elevation ticket if you have one:",
"style": "prompt_echo_on",
"key": "ticket"
}
]
```
Then we need to make sure that the authorization takes this input into account.
**sudo_authz_elevated.rego**:
```live:sudo_authz/elevate:module:read_only
# A package can be defined across multiple files.
package sudo.authz
import data.elevate
import input.display_responses
import input.sysinfo
# Allow this user if the elevation ticket they provided matches our mock API
# of an internal elevation system.
allow if {
elevate.tickets[sysinfo.pam_username] == display_responses.ticket
}
```
Now we need to build a new bundle for OPA to use.
```shell
opa build -b .
```
Confirm that the user `frontend-dev` can indeed use `sudo`.
```shell
ssh -p 2222 frontend-dev@localhost \
-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
sudo ls /
```
You should be prompted with the message that we defined in our _display_ policy
for both the SSH and `sudo` authorization cycles.
This happens because the _display_ policy is shared by the PAM configurations of SSH and `sudo`.
In production, it is more practical to use separate policy packages for each PAM configuration.
We have not defined the SSH _authz_ policy to work with elevation, so you can enter any value
into the prompt that comes up for for SSH.
For `sudo`, enter the ticket number `1234` to get access.
Lastly, update the mocked elevation API and confirm the user's original rights are restored.
```shell
cat <<EOF > elevate/data.json
{
"tickets": {}
}
EOF
```
Once again, build the bundle with this new data
```bash
opa build -b .
```
You will find that running `sudo ls /` as the `frontend-dev` user is disallowed again.
It is possible to configure the _display_ policy to only make the PAM module prompt for the
elevation ticket when our mock API has a non-empty `tickets` object. So when there are no
elevated users, there will be no prompt for a ticket. This can be done using the Rego
[`count` aggregate](./policy-reference/#aggregates).
## Wrap Up
Congratulations for finishing the tutorial!
You learned a number of things about SSH with OPA:
- OPA gives you fine-grained access control over SSH, `sudo`, and any other application that uses PAM.
Although this tutorial used the some of the same policies for both
SSH and sudo, you should use separate, fine-grained policies for each application that supports PAM.
- Writing allow/deny policies to control who has access to what using context from the user and host.
- Importing external data into OPA and writing policies that depend on that data.
The code for the PAM module used in this tutorial can be found in the
[open-policy-agent/contrib](https://github.com/open-policy-agent/contrib)
repository.
docs/new/docs/storage.md
+182
View File
@@ -0,0 +1,182 @@
---
title: Storage
---
## Disk
This page outlines configuration options relevant to using the disk storage
feature of OPA.
Configuration options are to be found in [the configuration docs](./configuration/#disk-storage).
:::info
The persistent disk storage enables OPA to work with data that does not fit
into the memory resources granted to the OPA server.
It is **not** supposed to be used as the primary source of truth for that data.
The on-disk storage should be considered ephemeral: you need to secure the
means to restore that data.
Backup and restore, or repair procedures for data corruption are not provided
at this time.
:::
### Partitions
Partitions determine how the JSON data is split up when stored in the
underlying key-value store.
For example, this table shows how an example document would be stored given
different configured partitions:
```json
{
"users": {
"alice": { "roles": ["admin"] },
"bob": { "roles": ["viewer"] }
}
}
```
| Partitions | Keys | Values |
| -------------- | -------------------- | ---------------------------------------------------------------- |
| (1) none | `/users` | `{"alice": {"roles": ["admin"]}, "bob": {"roles": ["viewer"]}}}` |
| --- | --- | --- |
| (2) `/users` | `/users/alice` | `{"roles": ["admin"]}` |
| | `/users/bob` | `{"roles": ["viewer"]}` |
| --- | --- | --- |
| (3) `/users/*` | `/users/alice/roles` | `["admin"]` |
| | `/users/bob/roles` | `["viewer"]` |
Partitioning has consequences on performance: in the example above, the
number of keys to retrieve from the database (and the amount of data of
its values) varies.
| Query | Partitions | Number of keys read |
| ------------------ | ---------- | ----------------------------- |
| `data.users` | (1) | 1 |
| | (2) | 2 |
| | (3) | 2 |
| --- | --- | --- |
| `data.users.alice` | (1) | 1 with `bob` data thrown away |
| | (2) | 2 |
| | (3) | 2 |
For example, retrieving the full extent of `data.users` from the disk store
will require a single key fetch with the partitions of (1).
With (2), the storage engine will fetch two keys and their values.
Retrieving a single user's data, e.g. `data.users.alice`, will require
reading a single key and all the users data with (1); but throw away most
of it: all the data not belonging to `alice`.
There is no one-size-fits-all setting for partitions: good settings depend
on the actual usage, and that comes down to the policies that are used with
OPA.
Commonly, you would optimize the partition settings for those queries that
are performance critical.
To figure out suboptimal partitioning, please have a look at the exposed
metrics.
OPA stores some internal values (such as bundle metadata) in the data store,
under `/system`. Partitions for that part of the data store are managed by
OPA, and providing any overlapping partitions in the config will raise an
error.
### Metrics
Using the [REST API](./rest-api/), you can include the `?metrics` query string
to gain insights into the disk storage access related to a certain OPA query.
```
$ curl 'http://localhost:8181/v1/data/tenants/acme1/bindings/user1?metrics' | opa eval -I 'input.metrics' -fpretty
{
"counter_disk_read_bytes": 339,
"counter_disk_read_keys": 3,
"counter_server_query_cache_hit": 1,
"timer_disk_read_ns": 40736,
"timer_rego_external_resolve_ns": 251,
"timer_rego_input_parse_ns": 656,
"timer_rego_query_eval_ns": 66616,
"timer_server_handler_ns": 117539
}
```
The `timer_disk_*_ns` timers give an indication about how much time
was spent with the different disk operations.
Available timers are
- `timer_disk_read_ns`
- `timer_disk_write_ns`
- `timer_disk_commit_ns`
Also note the `counter_disk_*` counters in the metrics:
- `counter_disk_read_keys`: number of keys retrieved
- `counter_disk_written_keys`: number of keys written
- `counter_disk_deleted_keys`: number of keys deleted
- `counter_disk_read_bytes`: bytes retrieved
Suboptimal partition settings can be spotted when the amount of
keys and bytes retrieved for a query is unproportional to the
actual data returned: the query likely had to retrieve a giant
JSON object, and most of it was thrown away.
### Debug Logging
Pass `--log-level debug` to `opa run` to see all the underlying storage
engine's logs.
When debug logging is _enabled_, the service will output some
statistics about the configured disk partitions and their key
sizes.
```
[DEBUG] partition /tenants/acme3/bindings (pattern /tenants/*/bindings): key count: 10000 (estimated size 598890 bytes)
[DEBUG] partition /tenants/acme4/bindings (pattern /tenants/*/bindings): key count: 10000 (estimated size 598890 bytes)
[DEBUG] partition /tenants/acme8/bindings (pattern /tenants/*/bindings): key count: 10000 (estimated size 598890 bytes)
[DEBUG] partition /tenants/acme9/bindings (pattern /tenants/*/bindings): key count: 10000 (estimated size 598890 bytes)
[DEBUG] partition /tenants/acme0/bindings (pattern /tenants/*/bindings): key count: 10000 (estimated size 598890 bytes)
[DEBUG] partition /tenants/acme2/bindings (pattern /tenants/*/bindings): key count: 10000 (estimated size 598890 bytes)
[DEBUG] partition /tenants/acme6/bindings (pattern /tenants/*/bindings): key count: 10000 (estimated size 598890 bytes)
```
Note that this process will iterate over all database keys.
It only happens on startup, when debug logging is enabled.
### Fine-tuning Badger settings (superflags)
While partitioning should be the first thing to look into to tune the memory usage and
performance of the on-disk storage engine, this configurable gives you the means to
change many internal aspects of how Badger uses memory and disk storage.
:::danger
To be used with care!
Any of the Badger settings used by OPA can be overridden using this feature.
There is no validation happening for configurables set using this flag.
When the embedded Badger version changes, these configurables could change,
too.
:::
The configurables correspond to Badger options that can be set on [the library's Options struct](https://pkg.go.dev/github.com/dgraph-io/badger/v3#Options).
The following configurables can _not_ be overridden:
- `dir`
- `valuedir`
- `detectconflicts`
Aside from conflict detection, Badger in OPA uses the default options [you can find here](https://github.com/dgraph-io/badger/blob/v3.2103.2/options.go#L128-L187).
Conflict detection is disabled because the locking scheme used within OPA does not allow
for having multiple concurrent writes.
#### Example
```yaml
storage:
disk:
directory: /tmp/disk
badger: nummemtables=1; numgoroutines=2; maxlevels=3
```
docs/new/docs/system.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 41 KiB

docs/new/docs/terraform.md
+914
View File
@@ -0,0 +1,914 @@
---
title: Terraform
---
Terraform lets you describe the infrastructure you want and automatically creates, deletes, and modifies
your existing infrastructure to match. OPA makes it possible to write policies that test the changes
Terraform is about to make before it makes them. Such tests help in different ways:
- tests help individual developers sanity check their Terraform changes
- tests can auto-approve run-of-the-mill infrastructure changes and reduce the burden of peer-review
- tests can help catch problems that arise when applying Terraform to production after applying it to staging
Terraform is a popular integration case for OPA and there are already a number
of popular tools for running policy on HCL and plan JSONs. Browse existing
<EcosystemFeatureLink feature="terraform"> tools using OPA and
Terraform</EcosystemFeatureLink> in the OPA Ecosystem.
## Goals
In this tutorial, you'll learn how to use OPA to implement unit tests for Terraform plans that create
and delete auto-scaling groups and servers.
## Prerequisites
This tutorial requires
- [Terraform 0.12.6](https://releases.hashicorp.com/terraform/0.12.6/)
- [OPA](https://github.com/open-policy-agent/opa/releases)
(This tutorial _should_ also work with the
[latest version of Terraform](https://www.terraform.io/downloads.html), but
it is untested. Contributions welcome!)
# Getting Started
## Steps
### 1. Create and save a Terraform plan
Create a [Terraform](https://www.terraform.io/docs/index.html) file that includes an
auto-scaling group and a server on AWS.
(You will need to modify the `shared_credentials_file` to point to your AWS credentials.)
```shell
cat >main.tf <<EOF
provider "aws" {
region = "us-west-1"
}
resource "aws_instance" "web" {
instance_type = "t2.micro"
ami = "ami-09b4b74c"
}
resource "aws_autoscaling_group" "my_asg" {
availability_zones = ["us-west-1a"]
name = "my_asg"
max_size = 5
min_size = 1
health_check_grace_period = 300
health_check_type = "ELB"
desired_capacity = 4
force_delete = true
launch_configuration = "my_web_config"
}
resource "aws_launch_configuration" "my_web_config" {
name = "my_web_config"
image_id = "ami-09b4b74c"
instance_type = "t2.micro"
}
EOF
```
Then initialize Terraform and ask it to calculate what changes it will make and store the output in `plan.binary`.
```shell
terraform init
terraform plan --out tfplan.binary
```
### 2. Convert the Terraform plan into JSON
Use the command [terraform show](https://www.terraform.io/docs/commands/show.html) to convert the Terraform plan into
JSON so that OPA can read the plan.
```shell
terraform show -json tfplan.binary > tfplan.json
```
Here is the expected contents of `tfplan.json`.
```json
{
"format_version": "0.1",
"terraform_version": "0.12.6",
"planned_values": {
"root_module": {
"resources": [
{
"address": "aws_autoscaling_group.my_asg",
"mode": "managed",
"type": "aws_autoscaling_group",
"name": "my_asg",
"provider_name": "aws",
"schema_version": 0,
"values": {
"availability_zones": [
"us-west-1a"
],
"desired_capacity": 4,
"enabled_metrics": null,
"force_delete": true,
"health_check_grace_period": 300,
"health_check_type": "ELB",
"initial_lifecycle_hook": [],
"launch_configuration": "my_web_config",
"launch_template": [],
"max_size": 5,
"metrics_granularity": "1Minute",
"min_elb_capacity": null,
"min_size": 1,
"mixed_instances_policy": [],
"name": "my_asg",
"name_prefix": null,
"placement_group": null,
"protect_from_scale_in": false,
"suspended_processes": null,
"tag": [],
"tags": null,
"termination_policies": null,
"timeouts": null,
"wait_for_capacity_timeout": "10m",
"wait_for_elb_capacity": null
}
},
{
"address": "aws_instance.web",
"mode": "managed",
"type": "aws_instance",
"name": "web",
"provider_name": "aws",
"schema_version": 1,
"values": {
"ami": "ami-09b4b74c",
"credit_specification": [],
"disable_api_termination": null,
"ebs_optimized": null,
"get_password_data": false,
"iam_instance_profile": null,
"instance_initiated_shutdown_behavior": null,
"instance_type": "t2.micro",
"monitoring": null,
"source_dest_check": true,
"tags": null,
"timeouts": null,
"user_data": null,
"user_data_base64": null
}
},
{
"address": "aws_launch_configuration.my_web_config",
"mode": "managed",
"type": "aws_launch_configuration",
"name": "my_web_config",
"provider_name": "aws",
"schema_version": 0,
"values": {
"associate_public_ip_address": false,
"enable_monitoring": true,
"ephemeral_block_device": [],
"iam_instance_profile": null,
"image_id": "ami-09b4b74c",
"instance_type": "t2.micro",
"name": "my_web_config",
"name_prefix": null,
"placement_tenancy": null,
"security_groups": null,
"spot_price": null,
"user_data": null,
"user_data_base64": null,
"vpc_classic_link_id": null,
"vpc_classic_link_security_groups": null
}
}
]
}
},
"resource_changes": [
{
"address": "aws_autoscaling_group.my_asg",
"mode": "managed",
"type": "aws_autoscaling_group",
"name": "my_asg",
"provider_name": "aws",
"change": {
"actions": [
"create"
],
"before": null,
"after": {
"availability_zones": [
"us-west-1a"
],
"desired_capacity": 4,
"enabled_metrics": null,
"force_delete": true,
"health_check_grace_period": 300,
"health_check_type": "ELB",
"initial_lifecycle_hook": [],
"launch_configuration": "my_web_config",
"launch_template": [],
"max_size": 5,
"metrics_granularity": "1Minute",
"min_elb_capacity": null,
"min_size": 1,
"mixed_instances_policy": [],
"name": "my_asg",
"name_prefix": null,
"placement_group": null,
"protect_from_scale_in": false,
"suspended_processes": null,
"tag": [],
"tags": null,
"termination_policies": null,
"timeouts": null,
"wait_for_capacity_timeout": "10m",
"wait_for_elb_capacity": null
},
"after_unknown": {
"arn": true,
"availability_zones": [
false
],
"default_cooldown": true,
"id": true,
"initial_lifecycle_hook": [],
"launch_template": [],
"load_balancers": true,
"mixed_instances_policy": [],
"service_linked_role_arn": true,
"tag": [],
"target_group_arns": true,
"vpc_zone_identifier": true
}
}
},
{
"address": "aws_instance.web",
"mode": "managed",
"type": "aws_instance",
"name": "web",
"provider_name": "aws",
"change": {
"actions": [
"create"
],
"before": null,
"after": {
"ami": "ami-09b4b74c",
"credit_specification": [],
"disable_api_termination": null,
"ebs_optimized": null,
"get_password_data": false,
"iam_instance_profile": null,
"instance_initiated_shutdown_behavior": null,
"instance_type": "t2.micro",
"monitoring": null,
"source_dest_check": true,
"tags": null,
"timeouts": null,
"user_data": null,
"user_data_base64": null
},
"after_unknown": {
"arn": true,
"associate_public_ip_address": true,
"availability_zone": true,
"cpu_core_count": true,
"cpu_threads_per_core": true,
"credit_specification": [],
"ebs_block_device": true,
"ephemeral_block_device": true,
"host_id": true,
"id": true,
"instance_state": true,
"ipv6_address_count": true,
"ipv6_addresses": true,
"key_name": true,
"network_interface": true,
"network_interface_id": true,
"password_data": true,
"placement_group": true,
"primary_network_interface_id": true,
"private_dns": true,
"private_ip": true,
"public_dns": true,
"public_ip": true,
"root_block_device": true,
"security_groups": true,
"subnet_id": true,
"tenancy": true,
"volume_tags": true,
"vpc_security_group_ids": true
}
}
},
{
"address": "aws_launch_configuration.my_web_config",
"mode": "managed",
"type": "aws_launch_configuration",
"name": "my_web_config",
"provider_name": "aws",
"change": {
"actions": [
"create"
],
"before": null,
"after": {
"associate_public_ip_address": false,
"enable_monitoring": true,
"ephemeral_block_device": [],
"iam_instance_profile": null,
"image_id": "ami-09b4b74c",
"instance_type": "t2.micro",
"name": "my_web_config",
"name_prefix": null,
"placement_tenancy": null,
"security_groups": null,
"spot_price": null,
"user_data": null,
"user_data_base64": null,
"vpc_classic_link_id": null,
"vpc_classic_link_security_groups": null
},
"after_unknown": {
"ebs_block_device": true,
"ebs_optimized": true,
"ephemeral_block_device": [],
"id": true,
"key_name": true,
"root_block_device": true
}
}
}
],
"configuration": {
"provider_config": {
"aws": {
"name": "aws",
"expressions": {
"region": {
"constant_value": "us-west-1"
}
}
}
},
"root_module": {
"resources": [
{
"address": "aws_autoscaling_group.my_asg",
"mode": "managed",
"type": "aws_autoscaling_group",
"name": "my_asg",
"provider_config_key": "aws",
"expressions": {
"availability_zones": {
"constant_value": [
"us-west-1a"
]
},
"desired_capacity": {
"constant_value": 4
},
"force_delete": {
"constant_value": true
},
"health_check_grace_period": {
"constant_value": 300
},
"health_check_type": {
"constant_value": "ELB"
},
"launch_configuration": {
"constant_value": "my_web_config"
},
"max_size": {
"constant_value": 5
},
"min_size": {
"constant_value": 1
},
"name": {
"constant_value": "my_asg"
}
},
"schema_version": 0
},
{
"address": "aws_instance.web",
"mode": "managed",
"type": "aws_instance",
"name": "web",
"provider_config_key": "aws",
"expressions": {
"ami": {
"constant_value": "ami-09b4b74c"
},
"instance_type": {
"constant_value": "t2.micro"
}
},
"schema_version": 1
},
{
"address": "aws_launch_configuration.my_web_config",
"mode": "managed",
"type": "aws_launch_configuration",
"name": "my_web_config",
"provider_config_key": "aws",
"expressions": {
"image_id": {
"constant_value": "ami-09b4b74c"
},
"instance_type": {
"constant_value": "t2.micro"
},
"name": {
"constant_value": "my_web_config"
}
},
"schema_version": 0
}
]
}
}
}
```
The json plan output produced by terraform contains a lot of information. For this tutorial, we will be interested by:
- `.resource_changes`: array containing all the actions that terraform will apply on the infrastructure.
- `.resource_changes[].type`: the type of resource (eg `aws_instance` , `aws_iam` ...)
- `.resource_changes[].change.actions`: array of actions applied on the resource (`create`, `update`, `delete`...)
For more information about the json plan representation, please check the [terraform documentation](https://www.terraform.io/docs/internals/json-format.html#plan-representation)
### 3. Write the OPA policy to check the plan
The policy computes a score for a Terraform that combines
- The number of deletions of each resource type
- The number of creations of each resource type
- The number of modifications of each resource type
The policy authorizes the plan when the score for the plan is below a threshold
and there are no changes made to any IAM resources.
(For simplicity, the threshold in this tutorial is the same for everyone, but in
practice you would vary the threshold depending on the user.)
**policy/terraform.rego**:
```rego
package terraform.analysis
import input as tfplan
########################
# Parameters for Policy
########################
# acceptable score for automated authorization
blast_radius := 30
# weights assigned for each operation on each resource-type
weights := {
"aws_autoscaling_group": {"delete": 100, "create": 10, "modify": 1},
"aws_instance": {"delete": 10, "create": 1, "modify": 1},
}
# Consider exactly these resource types in calculations
resource_types := {"aws_autoscaling_group", "aws_instance", "aws_iam", "aws_launch_configuration"}
#########
# Policy
#########
# Authorization holds if score for the plan is acceptable and no changes are made to IAM
default authz := false
authz if {
score < blast_radius
not touches_iam
}
# Compute the score for a Terraform plan as the weighted sum of deletions, creations, modifications
score := s if {
all_resources := [x |
some resource_type, crud in weights
del := crud.delete * num_deletes[resource_type]
new := crud.create * num_creates[resource_type]
mod := crud.modify * num_modifies[resource_type]
x := (del + new) + mod
]
s := sum(all_resources)
}
# Whether there is any change to IAM
touches_iam if {
all_resources := resources.aws_iam
count(all_resources) > 0
}
####################
# Terraform Library
####################
# list of all resources of a given type
resources[resource_type] := all_resources if {
some resource_type, _ in resource_types
all_resources := [name |
some name in tfplan.resource_changes
name.type == resource_type
]
}
# number of creations of resources of a given type
num_creates[resource_type] := num if {
some resource_type, _ in resource_types
all_resources := resources[resource_type]
creates := [res |
some res in all_resources
"create" in res.change.actions
]
num := count(creates)
}
# number of deletions of resources of a given type
num_deletes[resource_type] := num if {
some resource_type, _ in resource_types
all_resources := resources[resource_type]
deletions := [res |
some res in all_resources
"delete" in res.change.actions
]
num := count(deletions)
}
# number of modifications to resources of a given type
num_modifies[resource_type] := num if {
some resource_type, _ in resource_types
all_resources := resources[resource_type]
modifies := [res |
some res in all_resources
"update" in res.change.actions
]
num := count(modifies)
}
```
### 4. Evaluate the OPA policy on the Terraform plan
To evaluate the policy against that plan, you hand OPA the policy, the Terraform plan as input, and
ask it to evaluate `terraform/analysis/authz`.
```shell
opa exec --decision terraform/analysis/authz --bundle policy/ tfplan.json
```
```json
{
"result": [
{
"path": "tfplan.json",
"result": true
}
]
}
```
If you're curious, you can ask for the score that the policy used to make the authorization decision.
In our example, it is 11 (10 for the creation of the auto-scaling group and 1 for the creation of the server).
```shell
opa exec --decision terraform/analysis/score --bundle policy/ tfplan.json
```
```json
{
"result": [
{
"path": "tfplan.json",
"result": 11
}
]
}
```
If as suggested in the previous step, you want to modify your policy to make an authorization decision
based on both the user and the Terraform plan, the input you would give to OPA would take the form
`{"user": <user>, "plan": <plan>}`, and your policy would reference the user with `input.user` and
the plan with `input.plan`. You could even go so far as to provide the Terraform state file and the AWS
EC2 data to OPA and write policy using all of that context.
### 5. Create a Large Terraform plan and Evaluate it
Create a Terraform plan that creates enough resources to exceed the blast-radius permitted
by policy.
```shell
cat >main.tf <<EOF
provider "aws" {
region = "us-west-1"
}
resource "aws_instance" "web" {
instance_type = "t2.micro"
ami = "ami-09b4b74c"
}
resource "aws_autoscaling_group" "my_asg" {
availability_zones = ["us-west-1a"]
name = "my_asg"
max_size = 5
min_size = 1
health_check_grace_period = 300
health_check_type = "ELB"
desired_capacity = 4
force_delete = true
launch_configuration = "my_web_config"
}
resource "aws_launch_configuration" "my_web_config" {
name = "my_web_config"
image_id = "ami-09b4b74c"
instance_type = "t2.micro"
}
resource "aws_autoscaling_group" "my_asg2" {
availability_zones = ["us-west-2a"]
name = "my_asg2"
max_size = 6
min_size = 1
health_check_grace_period = 300
health_check_type = "ELB"
desired_capacity = 4
force_delete = true
launch_configuration = "my_web_config"
}
resource "aws_autoscaling_group" "my_asg3" {
availability_zones = ["us-west-2b"]
name = "my_asg3"
max_size = 7
min_size = 1
health_check_grace_period = 300
health_check_type = "ELB"
desired_capacity = 4
force_delete = true
launch_configuration = "my_web_config"
}
EOF
```
Generate the Terraform plan and convert it to JSON.
```shell
terraform init
terraform plan --out tfplan_large.binary
terraform show -json tfplan_large.binary > tfplan_large.json
```
Evaluate the policy to see that it fails the policy tests and check the score.
```shell
opa exec --decision terraform/analysis/authz --bundle policy/ tfplan_large.json
opa exec --decision terraform/analysis/score --bundle policy/ tfplan_large.json
```
### 6. (Optional) Run OPA using a remote policy bundle
In addition to loading policies from the local filesystem, `opa exec` can fetch policies from remote locations via [Bundles](./management-bundles). To see this in action, first build the policies into a bundle:
```shell
opa build policy/
```
Next, serve the bundle via nginx:
```bash
docker run --rm --name bundle_server -d -p 8888:80 -v ${PWD}:/usr/share/nginx/html:ro nginx:latest
```
Then run `opa exec` with bundles enabled:
```
opa exec --decision terraform/analysis/authz \
--set services.bundle_server.url=http://localhost:8888 \
--set bundles.tutorial.resource=bundle.tar.gz \
tfplan_large.json
```
## Wrap Up
Congratulations for finishing the tutorial!
You learned a number of things about Terraform Testing with OPA:
- OPA gives you fine-grained policy control over Terraform plans.
- You can use data other than the plan itself (e.g. the user) when writing authorization policies.
Keep in mind that it's up to you to decide how to use OPA's Terraform tests and authorization decision. Here are some ideas.
- Add it as part of your Terraform wrapper to implement unit tests on Terraform plans
- Use it to automatically approve run-of-the-mill Terraform changes to reduce the burden of peer-review
- Embed it into your deployment system to catch problems that arise when applying Terraform to production after applying it to staging
If you'd like to explore an additional example that uses terraform modules please continue below.
# Working with Modules
## Module Steps
### 1. Create and save Terraform module plan
Create a new Terraform file that includes a
security group and security group from a module.
(This example uses the module from
[terraform-aws-modules](https://github.com/terraform-aws-modules/terraform-aws-security-group))
```shell
cat >main.tf <<EOF
provider "aws" {
region = "us-east-1"
}
data "aws_vpc" "default" {
default = true
}
module "http_sg" {
source = "git::https://github.com/terraform-aws-modules/terraform-aws-security-group.git?ref=v3.10.0"
name = "http-sg"
description = "Security group with HTTP ports open for everybody (IPv4 CIDR), egress ports are all world open"
vpc_id = data.aws_vpc.default.id
ingress_cidr_blocks = ["0.0.0.0/0"]
}
resource "aws_security_group" "allow_tls" {
name = "allow_tls"
description = "Allow TLS inbound traffic"
vpc_id = data.aws_vpc.default.id
ingress {
description = "TLS from VPC"
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["10.0.0.0/8"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
Name = "allow_tls"
}
}
EOF
```
Then initialize Terraform and ask it to calculate what changes it will make and store the output in `tfplan.binary`.
```shell
terraform init
terraform plan --out tfplan.binary
```
### 2. Convert the new Terraform plan into JSON
Use the Terraform show command to produce the json representation of the terraform plan
```shell
terraform show -json tfplan.binary > tfplan2.json
```
### 3. Write the OPA policy to collect resources
The policy evaluates if a security group is valid based on the contents of it's description:
- Resources can be specified under the root module or in child modules
- We want to evaluate against the combined group of these resources
- This example is scoped to the planned changes section of the json representation
The policy uses the walk keyword to explore the json structure, and uses conditions to filter for the specific paths where resources would be found.
**policy/terraform_module.rego**:
```rego
package terraform.module
deny contains msg if {
some r
desc := resources[r].values.description
contains(desc, "HTTP")
msg := sprintf("No security groups should be using HTTP. Resource in violation: %v", [r.address])
}
resources contains r if {
some path, value
# Walk over the JSON tree and check if the node we are
# currently on is a module (either root or child) resources
# value.
walk(input.planned_values, [path, value])
# Look for resources in the current value based on path
some r in module_resources(path, value)
}
# Variant to match root_module resources
module_resources(path, value) := value if {
# Expect something like:
#
# {
# "root_module": {
# "resources": [...],
# ...
# }
# ...
# }
#
# Where the path is [..., "root_module", "resources"]
reverse_index(path, 1) == "resources"
reverse_index(path, 2) == "root_module"
}
# Variant to match child_modules resources
module_resources(path, value) := value if {
# Expect something like:
#
# {
# ...
# "child_modules": [
# {
# "resources": [...],
# ...
# },
# ...
# ]
# ...
# }
#
# Where the path is [..., "child_modules", 0, "resources"]
# Note that there will always be an index int between `child_modules`
# and `resources`. We know that walk will only visit each one once,
# so we shouldn't need to keep track of what the index is.
reverse_index(path, 1) == "resources"
reverse_index(path, 3) == "child_modules"
}
reverse_index(path, idx) := path[count(path) - idx]
```
### 4. Evaluate the OPA policy on the Terraform module plan
To evaluate the policy against that plan, you hand OPA the policy, the Terraform plan as input, and
ask it to evaluate `data.terraform.module.deny`.
```shell
opa exec --decision terraform/module/deny --bundle policy/ tfplan2.json
```
This should return one of the two resources. The security group created by the module uses HTTP in its description and therefore fails the evaluation.
```shell
{
"result": [
{
"path": "tfplan2.json",
"result": [
"No security groups should be using HTTP. Resource in violation: module.http_sg.aws_security_group.this_name_prefix[0]"
]
}
]
}
```
## Module Wrap Up
Congratulations on finishing the tutorial!
You learned OPA can be used to determine if a proposed configuration is authorized.
Additional use cases might include:
- Ensuring all resources have tags before they are created
- Making sure naming standards for resources are followed
- Security or operational requirements
# Ecosystem Projects
<EcosystemEmbed feature="terraform">
As further reading, you might be interested to review the Terraform integrations
from the OPA Ecosystem.
</EcosystemEmbed>
docs/new/docs/v0-compatibility.md
+188
View File
@@ -0,0 +1,188 @@
---
title: v0 Backwards Compatibility
sidebar_position: 14
---
## Running OPA in v0.x compatibility mode
The v1.0 release of OPA comes with functionality to run in a backwards
compatible, v0.x mode. This is used by running OPA with the `--v0-compatible`
flag or using the v0.x compatible options in Go integrations. When enabled, OPA
instances and Go integrations will behave as they do in pre v1.0 releases.
### When to use v0.x compatibility mode
**Use of v0.x compatibility mode is not recommended for most users**. This
mode is intended to help users with large volumes of third party Rego stay up to
date while performing a longer term migration to a OPA v1.0 compatible OPA
feature set. Examples of when this applies:
- You run a service for customers who supply their own Rego.
- You use OPA as part of a managed platform and need to run a mix of v0.x and
v1.0 OPAs based on customer demands.
Users with control over their Rego and OPA deployments are instead encouraged
to migrate their Rego to be compatible with OPA v1.0 using the below tooling options:
1. The `rego.v1` import makes OPA apply all restrictions that are enforced by default in OPA v1.0.
If a Rego module imports `rego.v1`, it means applicable `future.keywords` imports are implied. It is illegal to import both `rego.v1` and `future.keywords` in the same module.
2. The `--v0-v1` flag on the `opa fmt` command will rewrite existing modules to use the `rego.v1` import instead of `future.keywords` imports.
3. The `--v0-v1` flag on the `opa check` command will check that either the `rego.v1` import or applicable `future.keywords` imports are present if any of the `in`, `every`, `if` and `contains` keywords are used in a module.
### v0.x compatibility mode in the OPA binary
The `--v0-compatible` flag is supported on the following commands in OPA v1.0.x
releases:
- `bench`: supports Rego v0.x syntax modules, use of `import rego.v1` is optional.
- `build`: supports Rego v0.x syntax modules, use of `import rego.v1` is optional.
- `deps`: supports Rego v0.x syntax modules, use of `import rego.v1` is optional.
- `check`*: supports Rego v0.x syntax modules, use of `import rego.v1` is optional.
- `eval`: supports Rego v0.x syntax modules, use of `import rego.v1` is optional.
- `exec`: supports Rego v0.x syntax modules, use of `import rego.v1` is optional.
- `fmt`*: formats modules to be compatible with OPA v0.x syntax. See note about
`--v0-v1` flag below.
- `inspect`: supports Rego v0.x syntax modules, use of `import rego.v1` is optional.
- `parse`: supports Rego v0.x syntax modules, use of `import rego.v1` is optional.
- `run`: supports modules (including discovery bundle) using Rego v0.x syntax, use of `import rego.v1` is optional. Binds server listeners to all interfaces by default, rather than localhost.
- `test`: supports Rego v0.x syntax modules, use of `import rego.v1` is optional.
Note (*): the `check` and `fmt` commands also support the `--v0-v1` flag,
which will check/format Rego modules as if compatible with the Rego syntax of
_both_ the old 0.x OPA version and current OPA v1.0.
Note (*): Pre v1.0 versions of OPA also support a comparable `--v1-compatible`
flag which can be used to produce and consume Rego v1 bundles. See
[Upgrading to v1.0](./v0-upgrade) for more information on how to use this flag
as part of an upgrade to OPA v1.0.
### v0.x compatibility mode in Rego package
There are three ways to enable v0.x compatibility mode in the [Rego package](https://pkg.go.dev/github.com/open-policy-agent/opa/rego):
1. Set the Rego version on modules
2. Set the Rego version on bundle manifests
3. Use the SetRegoVersion Rego argument
1 & 2 are preferred as they are more granular and make it easier to run a
mix of v0.x and v1.0 compatible Rego in the same OPA instance and thus better
support a gradual upgrade path.
The `SetRegoVersion` method on [Module](https://pkg.go.dev/github.com/open-policy-agent/opa/ast#Module.SetRegoVersion?)
can be used like this:
```go
m := ast.Module{
Package: regoCode,
}
m.SetRegoVersion(ast.RegoV0)
```
Similarly, the [Bundle Manifest](https://pkg.go.dev/github.com/open-policy-agent/opa/bundle#Manifest.SetRegoVersion) Rego version
can be set like this:
```go
b := Bundle{
// ...
}
b.SetRegoVersion(ast.RegoV0)
```
If you cannot set the Rego version on modules or bundle manifests, you
can use the [`SetRegoVersion`](https://pkg.go.dev/github.com/open-policy-agent/opa/rego#SetRegoVersion) Rego argument to control the Rego version used when
evaluating policies.
Users are encouraged to use the more granular options where possible to better
allow them to upgrade Rego used in their system to Rego v1 gradually.
In the example below, `SetRegoVersion` is used as a Rego argument instructing
the supplied Rego to be handled as v0.x syntax:
```go
// Only to be used if the above are not suitable.
r := rego.New(
rego.Query("data.foo.bar"),
rego.Module("policy.rego", regoCode),
rego.SetRegoVersion(ast.RegoV1), // <---
)
```
Finally, another option is to import the `v0` package instead. The program
below imports the v0 package instead:
```go
package main
import (
"context"
"encoding/json"
"fmt"
"github.com/open-policy-agent/opa/rego"
// rather than the v1 import, which is:
// "github.com/open-policy-agent/opa/v1/rego"
)
func main() {
module := `package example
messages[msg] {
msg := "foo"
}
`
r := v0rego.New(
rego.Query("data.example.messages"),
rego.Module("example.rego", module),
)
rs, _ = rv0.Eval(context.TODO())
bs, _ = json.Marshal(rs)
fmt.Println(string(bs))
}
```
:::danger
**Note**: Using v0 packages and v1 packages in the same program is considered an
anti-pattern and is not recommended or supported. Any interoperability between
the two packages is not guaranteed and should be considered unsupported.
:::
### v0.x compatibility mode in the OPA Go SDK
In OPA 1.0, the recommended
[SDK package](https://pkg.go.dev/github.com/open-policy-agent/opa/v1/sdk)
import for most users is `github.com/open-policy-agent/opa/v1/sdk`.
Those who need to support v0 bundles should set the Rego version on bundle
manifests as outlined above wherever possible. For users unable to do this, use
of a v0 import of the SDK package is required. For example:
```go
package main
import (
"bytes"
"context"
"fmt"
"github.com/open-policy-agent/opa/sdk" // <-- import v0 sdk package
)
func main() {
opa, _ := sdk.New(ctx, sdk.Options{
ID: "opa-1",
Config: bytes.NewReader(config),
})
defer opa.Stop(ctx)
// ...
}
```
Users in this scenario should look to version bundles as soon as possible to
allow them to use a v1 SDK instead.
docs/new/docs/v0-upgrade/index.md
+534
View File
@@ -0,0 +1,534 @@
---
title: Upgrading to v1.0
sidebar_position: 14
---
All users should plan to upgrade to OPA v1.0 eventually. Some users, with more
control over the Rego loaded into the OPA instances they run will be able to do
so more quickly. Other users with less control or running third party Rego may
wish to upgrade to OPA v1.0 and use the [v0 compatibility](./v0-compatibility) functionality to
upgrade gradually.
This documentation covers the different upgrade scenarios and the best course of
action for each. The documentation makes use of the following concepts:
- **Bundle Producer**: A system based on `opa build` that produces a bundle that
is loaded by consumers.
- **Bundle Consumer**: An OPA instance that loads and evaluates policy
from a bundle in the system.
- **Authoring**: The process of writing Rego policies before bundles are
produced and consumed. In managed systems, this might be a user's only contact
point with OPA.
In some systems, where OPA is used without a bundle, there is no producer. This
simplifies the upgrade process.
Users are encouraged to upgrade to OPA v1.0 as soon as possible. Using the v0
compatible functionality until updating Rego is preferred to delaying the
upgrade. The first part of this guide refers to upgrading OPA instances used for
producing and consuming bundles. This is the first step users should take unless
their Rego is already v1.0 compatible. See [Upgrading Rego](#upgrading-rego)
below for information on how to upgrade Rego policies to be v1.0 compatible.
## General Upgrade Approach: Upgrade Producers, then Consumers
Users will need to upgrade to OPA v1.0 in their own way, depending on their
release and change management processes, use cases and risk tolerance. The
general advice is to upgrade producers first, then consumers. This is because
the updated producers would be able to set the Rego version on bundle manifests
and as a result it wouldn't be necessary to run consumers with the `--v0-compatible` flag.
Also since it's likely there are much more consumers than producers, upgrading producers
first would lead to a smoother upgrade process.
Some users may wish to migrate to OPA v1.0 all at once, with adequate testing and validation
this is possible. Not all steps are necessary for all users so a hybrid approach
is also an option depending on your context.
The rest of this documentation is designed to meet users where they find
themselves and direct them down the smoothest path to upgrade to OPA
v1.0.
## Detailed Producer & Consumer Version Scenarios
Tabulated in this section are the different versions of OPA users might be
working with in different parts of their systems. Select the scenario that best
matches your setup to find the recommended upgrade path.
If you are in doubt, [Scenario 1](#scenario-1) is the most common starting
point and we recommended you start there.
| | v0.x Consumer | Mix Consumer | v1.0 Consumer |
| ----------------- | ------------------------------------ | ------------------------- | ------------------------------------ |
| **v0.x Producer** | [Scenario 1](#scenario-1) (All v0.x) | [Scenario 4](#scenario-4) | [Scenario 7](#scenario-7) |
| **Mix Producer** | [Scenario 2](#scenario-2) | [Scenario 5](#scenario-5) | [Scenario 8](#scenario-8) |
| **v1.0 Producer** | [Scenario 3](#scenario-3) | [Scenario 6](#scenario-6) | [Scenario 9](#scenario-9) (All v1.0) |
<!-- source https://docs.google.com/drawings/d/137EObOVhMIVk9NEWOX0u_1eQOqe0MRkPQlmwKXkYWEU/edit -->
![Upgrade flows to OPA v1.0](opa-v0-upgrade.png)
## Upgrade Scenarios
Listed below are the different upgrade scenarios and the recommended migration
plans for each case.
### Scenario 1: v0.x Producer, v0.x Consumer
All OPA runtimes - both bundle consumers and producers - are v0.x. This is the
most common starting point for users upgrading from a v0.x version of OPA.
#### How to Run
- Policies are authored to be v0.x compatible.
#### Next
Start upgrading producers to v1.0 ([Scenario 2](#scenario-2)) until all producers
are v1.0 ([Scenario 3](#scenario-3)).
### Scenario 2: Mix Producer, v0.x Consumer
Some bundle producers are v1.0, while some remain on v0.x. All bundle consumers
are v0.x. This might be the case if you have bundles from different tenants or
users using different OPA versions and you cannot control the versions they use.
#### Pre-requisites
Users cannot proceed with upgrade until they have either a single version of bundle
producers or have a means to control the use of `--v0-compatible` on newer
producers.
#### How to Run
- Policies are authored to be v0.x compatible.
- v0.x producers are run as-is.
- v1.0 producer is run with `--v0-compatible`, or modules have `rego.v1` import.
- v0.x consumers are run as-is.
#### Next
Continue migrating producers to v1.0 until all producers have been upgraded
([Scenario 3](#scenario-3)).
### Scenario 3: v1.0 Producer, v0.x Consumer
OPA bundles are produced by OPA v1.0 instances, consumers are still on v0.x. This
scenario is common as users upgrade to OPA v1.0 by upgrading their producers
first.
#### Pre-requisites
Control of producers to set `--v0-compatible` or use `rego.v1` imports is
required.
#### How to Run
- Policies are authored to be v0.x compatible.
- v0.x consumers are run as-is.
- v1.0 producer is run with `--v0-compatible`, or modules have `rego.v1` import.
- Since policies will always be consumed by a v0.x OPA, all policies _must_ be v0.x compliant.
#### Next
Now that all producers are v1.0, and consumers are still not all v1.0, it's time
to get all the consumers to v1.0, ([Scenario 6](#scenario-6)).
### Scenario 4: v0.x Producer, Mix Consumer
Producers are v0.x, consumers are a mix of v0.x and v1.0. This scenario might occur
when users have partially upgraded OPA instances to v1.0, but have not yet
upgraded their consumers. This is not a recommended step if it can be avoided as
it's recommended to upgrade producers first.
#### Pre-requisites
OPA v1.0 consumers must be able to run with `--v0-compatible` to accept v0.x
bundles. This upgrade path cannot continue until this is possible.
#### How to Run
- Policies are authored to be v0.x compatible.
- v0.x producers and v0.x consumers are run as is.
### Next
Upgrade producers to v1.0 and continue the upgrade from that point. Generally, it's recommended to
upgrade producers first, however depending on your existing OPAs v1.0 consumers deployments, you may prefer to
upgrade all your producers to v1.0 rather than to downgrade consumers.
### Scenario 5: Mix Producer, Mix Consumer
Mixed versions of OPA are being used for both bundle production and consumption.
#### Pre-requisites
As users have a mix of bundle producers, they must have control over the runtime
options for the producers to set `--v0-compatible`. Users must also have control
over their v1.0 consumers to set the `--v0-compatible` flag. Both these conditions
must be met for the upgrade to proceed.
#### How to Run
- Policies are authored to be v0.x compatible.
- v1.0 consumers are run with `--v0-compatible`.
- v1.0 producers are run with `--v0-compatible`.
### Next
Please gradually upgrade producers to v1.0 until all producers are v1.0 ([Scenario 6](#scenario-6)).
### Scenario 6: v1.0 Producer, Mix Consumer
All consumers can be run without flags, as the bundle will contain
attributes to inform v1.0 OPAs to accept v0.x modules.
#### How to Run
- Policies are authored to be v0.x compatible.
- v0.x consumers are run as-is. Bundles will contain v0.x policies
- v1.0 consumers are run as-is. Bundles will contain `rego_version` attribute, so v0.x modules are accepted.
#### Pre-requisites
If users cannot set their OPA v1.0 producers to use `--v0-compatible` to be
compatible with their v0.x consumers, then this upgrade path is blocked.
#### Next
Running exclusively v1.0 producers and consumers, ([Scenario 9](#scenario-9)), is
the next and final step.
### Scenario 7: v0.x Producer, v1.0 Consumer
All consumers are v1.0, but producers are v0.x. This scenario might occur when
OPAs used for evaluation are upgraded before the policy bundling system.
#### Pre-requisites
If v1.0 consumers cannot be run with `--v0-compatible`, when loading v0.x consumer
generated bundle, the bundles cannot include `rego_version` attribute. This means
the upgrade path is blocked until either the consumers can create bundles with a
Rego version or the `--v0-compatible` flag is available for producers.
#### How to Run
- Policies are authored to be v0.x compatible.
- v1.0 consumers are run with `--v0-compatible`
#### Next
Upgrade producers to v1.0 ([Scenario 8](#scenario-8)) until all producers are v1.0
([Scenario 9](#scenario-9)).
### Scenario 8: Mix Producer, v1.0 Consumer
All consumers are v1.0, but producers are a mix of v0.x and v1.0.
#### How to Run
- Policies are authored to be v0.x compatible.
- v0.x producers are run as is.
- v1.0 consumers are run with `--v0-compatible`
- v1.0 producers are run with `--v0-compatible`
#### Pre-requisites
If using v0.x bundles, it must be possible to use `--v0-compatible` on the bundle
producers in order for them to work in the v1.0 consumers.
v1.0 consumers will accept v1.0 producer bundles, as these will have the Rego version specified in the manifest;
they won't however accept bundles from v0.x producers unless they have `--v0-compatible` set.
#### Next
Upgrade producers to v1.0 ([Scenario 9](#scenario-9)), completing the upgrade.
### Scenario 9: v1.0 Producer, v1.0 Consumer
Once you have all consumers and producers running at v1.0 then you have
completed the upgrade to OPA v1.0. If you are using `--v0-compatible`
functionality, the next task is to upgrade the Rego loaded into OPAs to Rego v1.
Regardless of whether you are now upgrading your Rego, we encourage users to
use `opa check`, `opa check --strict` and to lint their Rego projects if you
have not already done so to identify issues.
## Changes to Rego in OPA v1.0
Once you have upgraded OPA instance to v1.0, or if you are upgrading all at
once, you will need to upgrade your Rego policies to Rego v1.0. This section
outlines the changes in Rego v1.0.
### The `future.keywords` imports
The `in`, `every`, `if` and `contains` keywords have been introduced over time,
and Rego v0.x required an opt-in to prevent them from breaking policies that
existed before their introduction. The `future.keywords` imports facilitate this
opt-in mechanism. These keywords help to increase the readability of policies
and provide syntactic sugar for commonly used operations such as iteration,
membership checks, defining multi-value rules, and so on. There is growing
adoption of these keywords and their usage is prevalent in the OPA
documentation, Rego Playground, etc.
In OPA v1.0 the `in`, `every`, `if` and `contains` keywords are part of the
language by default and the `future.keywords` imports will become a no-op. A
policy that makes use of these keywords, but doesn't import `future.keywords` is
valid in OPA v1.0 but not in older versions of OPA.
### Enforce use of `if` and `contains` keywords in rule head declarations
In Rego v0.x, there is semantic ambiguity between rules like `a.b {true}` and
`a.b.c {true}`. Although syntactically similar, the former generates a set with
the entry `b` at path `data.a`, while the latter generates an object with the
attribute `"c": true` at path `data.a.b`. This inconsistency makes it difficult
for new users to understand how Rego works. The `if` keyword is more than just
syntactic sugar. When used in a rule head, that rule doesn't contribute to a
partial set unless the `contains` keyword is also used. E.g. `a.b if {true}`
will generate an object with the attribute `"b": true` at path `data.a`. To make
things simpler, OPA v1.0 requires the usage of `if` and `contains` keywords when
declaring rules. This would mean:
- All rules are single-value by default. When the value is omitted from the
head, it defaults to `true`.
- To make rules multi-value (i.e. partial set rules), use the `contains` keyword
to convert the value into a set.
The `contains` keyword is required to disambiguate rules that generate a single
value from rules that generate multiple values. The `if` keyword ensures that
the semantics of rules do not change between v0.x and v1.0 Rego. The table below
illustrates why `if` is required.
| rule | output in v0.x | output in v1.0 |
| ------------------- | -------------------------- | -------------------------- |
| `p { true }` | `{"p": true}` | `compile error` |
| `p.a { true }` | `{"p": {"a"}}` | `compile error` |
| `p.a.b { true }` | `{"p": {"a": {"b": true}}` | `compile error` |
| `p if { true }` | `{"p": true}` | `{"p": true}` |
| `p.a if { true }` | `{"p":{"a": true}}` | `{"p":{"a": true}}` |
| `p.a.b if { true }` | `{"p": {"a": {"b": true}}` | `{"p": {"a": {"b": true}}` |
| `p contains “a”` | `{"p": {"a"}}` | `{"p": {"a"}}` |
If the Rego language was changed so that all rules were single-value by default,
unless the `contains` keyword was used to make them multi-value, then the
outcome of a rule like `p.a { true }` would change between v0.x and v1.0
without generating an error. Generating errors in this case is preferable to
changing the semantics of existing rules. Therefore, use of the `if` keyword is
a requirement in OPA v1.0.
In OPA v1.0, the `if` keyword is only required for rules with a declared body.
Constants, rules that only consist of a value assignment, do not require `if`.
The following forms therefore remain valid in OPA v1.0:
| rule | output in v0.x | output in v1.0 |
| ------------ | ------------------------ | ------------------------ |
| `p := 1` | `{"p": 1}` | `{"p": 1}` |
| `p.a := 1` | `{"p": {"a": 1}}` | `{"p": {"a": 1}}` |
| `p.a.b := 1` | `{"p": {"a": {"b": 1}}}` | `{"p": {"a": {"b": 1}}}` |
Since the `if` keyword can only be used in front of a rule body, rules with no
body and no value assignment, i.e. a solitary reference, are not allowed in
the v1.0 Rego syntax:
| rule | output in v0.x | output in v1.0 |
| ------- | --------------------------- | --------------- |
| `p` | `compile error` | `compile error` |
| `p.a` | `{"p": {"a"}}` | `compile error` |
| `p.a.b` | `{"p": {"a": {"b": true}}}` | `compile error` |
The below table gives examples of v0.x valid Rego syntax which are
invalid in OPA v1.0, along with the equivalent valid syntax in OPA v1.0:
| invalid in v1.0 | v1.0 equivalent | Note |
| ---------------- | ---------------------------- | ------------------------- |
| `p { true }` | `p if { true }` | `Single-value rule` |
| `p.a` | `p contains "a"` | `Multi-value insertion` |
| `p.a { true }` | `p contains "a" if { true }` | `Multi-value rule` |
| `p.a.b` | `p.a.b := true` | `Single-value assignment` |
| `p.a.b { true }` | `p.a.b if { true }` | `Single-value rule` |
Following is an example of how to define a rule that generates a set:
```rego
package play
a contains b if { b := 1 }
```
When the above rule is evaluated the output is (sets are serialized into arrays
in JSON):
```json
{
"a": [1]
}
```
Following is an example of how to define a rule that generates an object:
```rego
package play
a[b] if { b := 1}
```
When the above rule is evaluated the output is:
```json
{
"a": {
"1": true
}
}
```
The requirement of `if` and `contains` keywords remove the ambiguity
between single-value and multi-value rule declaration. This makes Rego code
easier to author and read; thereby making it simpler for users to author their
policies.
### Prohibit duplicate imports
As part of `strict` mode in OPA 0.x, the Rego compiler prohibits duplicate imports where one import shadows another.
OPA v1.0 enforces this check by default.
An import shadowing another is most likely an authoring error and probably
unintentional. OPA checking this by default will help to avoid policy
evaluations resulting in error-prone decisions.
### `input` and `data` keywords are reserved
The Rego compiler ensures that `input` and `data` are reserved keywords and may
not be used as names for rules and variable assignments. This is part of`strict` mode in OPA 0.x
The `input` document holds the user-provided input, while the data pushed into
OPA and rule evaluation results are nested under the `data` document. Hence, if
a rule or variable shadows `input` or `data` you have the unintended consequence
of erasing information under these inside the local scope, resulting in incorrect policy decisions. In
OPA v1.0 such scenarios are avoided by default.
Note, using the [with](./policy-language/#with-keyword) keyword to insert
values into - or to fully replace - the `input` or `data` documents, as in
`my_func(x) with input as {...}` does not constitute shadowing and is therefore
allowed in OPA v1.0.
### Prohibit use of deprecated builtins
As part of `strict` mode in OPA 0.x, the Rego compiler prohibits use of deprecated built-in functions. In OPA v1.0,
these built-ins have been removed.
The following built-in functions are deprecated: `any`, `all`, `re_match`,
`net.cidr_overlap`, `set_diff`, `cast_array`, `cast_set`, `cast_string`,
`cast_boolean`, `cast_null`, `cast_object`. In some cases, new built-in
functions have been added that provide functionality at least similar to a
deprecated built-in.
### Rego-versioned bundles
A bundle built with OPA `v0.64.0` or later, contain a `rego_version` attribute
in their [manifest](./management-bundles/#bundle-file-format), which the OPA
consuming that bundle will use when processing the contained modules. A bundle's
internal rego-version takes precedence over the presence of the
`--v1-compatible` flag; therefore, prerequisite knowledge about what Rego syntax
any consumed bundle contains is not needed. The `--v1-compatible` flag (and
`--v0-compatible` in v1.0) on the `opa build` command allows the user to control
the `rego-version` of the built bundle.
See [Upgrading to v1.0](./v0-upgrade) for more information on how to use
versioned bundles as part of an upgrade to OPA v1.0.
## Compilation Constraints and Checks
Below constraints and safety checks are enforced by default in v1.0 during compilation. These checks along with the ones in [v1.0 strict mode](./policy-language/#strict-mode)
were part of the compiler `strict` mode in OPA 0.x.
| Name | Description |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Duplicate imports | Duplicate [imports](./policy-language/#imports), where one import shadows another, are prohibited. |
| `input` and `data` reserved keywords | `input` and `data` are reserved keywords, and may not be used as names for rules and variable assignment. |
| Use of deprecated built-ins | Use of deprecated functions is prohibited, and these will be removed in OPA 1.0. Deprecated built-in functions: `any`, `all`, `re_match`, `net.cidr_overlap`, `set_diff`, `cast_array`, `cast_set`, `cast_string`, `cast_boolean`, `cast_null`, `cast_object` |
## Upgrading Rego
Users with v0.x Rego projects are encouraged to follow the below process to
upgrade their Rego code to conform to best practices, and to be compatible with
OPA v1.0. These steps are largely based on the process outlined in this
[detailed blog post](https://www.styra.com/blog/renovating-rego/).
Before starting the upgrade, users are recommended to ensure they have a local
OPA binary of version 1.0 or later.
1. `opa check --v0-v1`, this will catch any parse or compilation errors.
2. `opa check --v0-v1 --strict`, this will raise a number of other issues found in code
that might make it incompatible with OPA v1.0 such as the use of deprecated
built-ins or duplicate imports.
3. Automatically reformat your code for OPA v1.0 with `opa fmt --write --v0-v1`.
4. `regal lint`, the [Regal linter](../ecosystem/entry/regal/) has many more rules to
test for issues in Rego code that can lead to errors, poor performance or
unexpected behaviour.
If you run into any issues while upgrading a Rego project, please drop a message
in the #help channel on the [OPA Slack](https://slack.openpolicyagent.org/).
## Upgrading OPA Instances
Prior to OPA 1.0, when running in server mode (`opa run --server/-s`), OPA would
bind to all interfaces by default. In OPA 1.0,
[OPA will bind to `localhost`](https://github.com/open-policy-agent/opa/issues/6286)
by default instead. Though not inherently insecure in a trusted environment,
it's good practice to bind OPA to localhost by default if OPA is not intended to
be exposed to remote services.
If you need to replicate the v0.x behaviour, you can use the `--addr` flag to
bind to all interfaces. For example:
```sh
opa run --server --addr 0.0.0.0:8181
```
:::info
When running OPA in a container, binding to all interfaces is required
when the instance needs to be accessed by the host or another container.
:::
More information can be found in the
[security documentation](../security/#interface-binding).
## Upgrading for Go Integrations
Both users of the
[v0 SDK](https://pkg.go.dev/github.com/open-policy-agent/opa/sdk)
and
[v0 Rego](https://pkg.go.dev/github.com/open-policy-agent/opa/rego) packages are
encoraged to upgrade to the new v1 packages instead. These can be found here:
- [SDK v1](https://pkg.go.dev/github.com/open-policy-agent/opa/v1/sdk)
- [Rego v1](https://pkg.go.dev/github.com/open-policy-agent/opa/v1/rego)
In order to upgrade to a v1 package, you need to make the following change:
Before:
```
import (
"github.com/open-policy-agent/opa/rego"
)
```
After:
```
import (
"github.com/open-policy-agent/opa/v1/rego"
)
```
This will be needed for all OPA packages your application depends on, not just
`rego` and `sdk`, other commonly used packages are: `ast`, `bundle`, `compile`,
`types` & `topdown`.
As of OPA 1.0, all v0 packages have been deprecated. While they will remain for
the lifetime of OPA 1.0, you are encouraged to upgrade as soon as possible.
If you need to use v0 functionality, you can still use v1 packages. Please see
the [Backwards Compatibility](./v0-compatibility/) documentation for more
details.
docs/new/docs/v0-upgrade/opa-v0-upgrade.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 28 KiB

docs/new/docs/wasm.md
+382
View File
@@ -0,0 +1,382 @@
---
title: WebAssembly
sidebar_position: 13
---
# What is WebAssembly (Wasm)?
As described on [https://webassembly.org/](https://webassembly.org/)
> WebAssembly (abbreviated Wasm) is a binary instruction format for a
> stack-based virtual machine. Wasm is designed as a portable target for
> compilation of high-level languages like C/C++/Rust, enabling deployment on
> the web for client and server applications.
## Overview
OPA is able to compile Rego policies into executable Wasm modules that can be
evaluated with different inputs and external data. This is _not_ running the OPA
server in Wasm, nor is this just cross-compiled Golang code. The compiled Wasm
module is a planned evaluation path for the source policy and query.
## Current Status
The core language is supported fully but there are a number of built-in
functions that are not, and probably won't be natively supported in Wasm (e.g.,
`http.send`). Built-in functions that are not natively supported can be
implemented in the host environment (e.g., JavaScript).
## Compiling Policies
You can compile Rego policies into Wasm modules using the `opa build` subcommand.
For example, the `opa build` command below compiles the `example.rego` file into a
Wasm module and packages it into an OPA bundle. The `wasm` target requires at least
one entrypoint rule (specified by `-e`, or a metadata `entrypoint` annotation).
```bash
opa build -t wasm -e example/allow example.rego
```
The output of a Wasm module built this way contain the `result` of evaluating the
entrypoint rule. For example:
```json
[
{
"result": <value of data.example.allow>
}
]
```
The output of policy evaluation is a set of variable assignments. The variable
assignments specify values that satisfy the expressions in the policy query
(i.e., if the variables in the query are replaced with the values from the
assignments, all of the expressions in the query would be defined and not
false.)
When policies are compiled into Wasm, the user provides the path of the policy
decision that should be exposed by the Wasm module. The policy decision is
assigned to a variable named `result`. The policy decision can be ANY JSON value
(boolean, string, object, etc.) but there will be at-most-one assignment. This
means that callers should first check if the set of variable assignments is
empty (indicating an undefined policy decision) otherwise they should select the
`"result"` key out of the variable assignment set.
> For more information on `opa build` run `opa build --help`.
### Advanced Compiling Options
You can also compile Rego policies into Wasm modules from Go using the lower-level
[rego](https://pkg.go.dev/github.com/open-policy-agent/opa/rego#Rego.Compile) API
that produces raw Wasm executables and the higher-level
[compile](https://pkg.go.dev/github.com/open-policy-agent/opa/compile#Compiler.Build)
API that produces OPA bundle files. The compile API is recommended.
## Using Compiled Policies
### JavaScript SDK
There is a JavaScript SDK available that simplifies the process of loading and
evaluating compiled policies. If you want to evaluate Rego policies inside
JavaScript we recommend you use the
[Javascript SDK](https://github.com/open-policy-agent/npm-opa-wasm).
There is also an
[example NodeJS application](https://github.com/open-policy-agent/npm-opa-wasm/tree/master/examples/nodejs-app)
provided for reference.
### Other Languages
A number of other languages have OPA Wasm support too <EcosystemFeatureLink
feature="wasm-integration"> via various community SDKs </EcosystemFeatureLink>
in the OPA Ecosystem.
### From Scratch
If you want to integrate Wasm compiled policies into a language or runtime that
does not have SDK support, read this section.
#### Instantiating the Wasm Module
Before you can evaluate Wasm compiled policies you need to instantiate the Wasm
module produced by the compilation process described earlier on this page.
To load the compiled Wasm module refer the documentation for the Wasm runtime
that you are using. At a high-level you must provide a memory buffer and a set
of import functions. The memory buffer is a contiguous, mutable byte-array that
allows you to pass data to the policy and receive output from the policy. The
import functions are dependencies of the compiled policies.
#### ABI Versions
Wasm modules built using OPA 0.27.0 onwards contain a global variable named
`opa_wasm_abi_version` that has a constant i32 value indicating the ABI version
this module requires. Described below you find ABI versions `1.x`.
There's another i32 constant exported, `opa_wasm_abi_minor_version`, used
to track backwards-compatible changes.
Using tools like `wasm-objdump` (`wasm-objdump -x policy.wasm`), the ABI
version can be found here:
```
Global[3]:
- global[0] i32 mutable=1 - init i32=121904
- global[1] i32 mutable=0 <opa_wasm_abi_version> - init i32=1
- global[2] i32 mutable=0 <opa_wasm_abi_minor_version> - init i32=0
Export[19]:
[...]
- global[1] -> "opa_wasm_abi_version"
- global[2] -> "opa_wasm_abi_minor_version"
```
Note the `i32=1` of `global[1]`, exported by the name of `opa_wasm_abi_version`.
##### Version notes
| ABI | Notes |
| --- | --------------------------------------------------------------------------------------------------------------------- |
| 1.0 | Start of ABI versioning. |
| 1.1 | Adds export `memory`. |
| 1.2 | Adds exported function `opa_eval`. |
| 1.3 | Adds exported functions `opa_value_free`, `opa_heap_blocks_stash`, `opa_heap_blocks_restore`, `opa_heap_stash_clear`. |
#### Exports
The primary exported functions for interacting with policy modules are listed below.
In the ABI column, you can find the ABI version with which the export was introduced.
| Function | Description | ABI |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- |
| `eval` | <div> `int32 eval(ctx_addr)`</div><div>Evaluates the loaded policy with the provided evaluation context. The return value is reserved for future use. </div> | 1.0 |
| `builtins` | <div> `value_addr builtins(void)`</div><div>Returns the address of a mapping of built-in function names to numeric identifiers that are required by the policy. </div> | 1.0 |
| `entrypoints` | <div> `value_addr entrypoints(void)`</div><div>Returns the address of a mapping of entrypoints to numeric identifiers that can be selected when evaluating the policy. </div> | 1.0 |
| `opa_eval_ctx_new` | <div> `ctx_addr opa_eval_ctx_new(void)`</div><div>Returns the address of a newly allocated evaluation context. </div> | 1.0 |
| `opa_eval_ctx_set_input` | <div> `void opa_eval_ctx_set_input(ctx_addr, value_addr)`</div><div>Set the input value to use during evaluation. This must be called before each `eval()` call. If the input value is not set before evaluation, references to the `input` document result produce no results (i.e., they are undefined.) </div> | 1.0 |
| `opa_eval_ctx_set_data` | <div> `void opa_eval_ctx_set_data(ctx_addr, value_addr)` </div><div>Set the data value to use during evaluation. This should be called before each `eval()` call. If the data value is not set before evaluation, references to base `data` documents produce no results (i.e., they are undefined.) </div> | 1.0 |
| `opa_eval_ctx_set_entrypoint` | <div> `void opa_eval_ctx_set_entrypoint(ctx_addr, entrypoint_id)` </div><div>Set the entrypoint to evaluate. By default, entrypoint with id `0` is evaluated. </div> | 1.0 |
| `opa_eval_ctx_get_result` | <div> `value_addr opa_eval_ctx_get_result(ctx_addr)`</div><div>Get the result set produced by the evaluation process. </div> | 1.0 |
| `opa_malloc` | <div> `addr opa_malloc(int32 size)`</div><div>Allocates size bytes in the shared memory and returns the starting address. </div> | 1.0 |
| `opa_free` | <div> `void opa_free(addr)`</div><div>Free a pointer. Calls `opa_abort` on error. </div> | 1.0 |
| `opa_json_parse` | <div> `value_addr opa_json_parse(str_addr, size)`</div><div>Parses the JSON serialized value starting at str_addr of size bytes and returns the address of the parsed value. The parsed value may refer to a null, boolean, number, string, array, or object value. </div> | 1.0 |
| `opa_value_parse` | <div> `value_addr opa_value_parse(str_addr, size)`</div><div>The same as `opa_json_parse` except Rego set literals are supported. </div> | 1.0 |
| `opa_json_dump` | <div> `str_addr opa_json_dump(value_addr)`</div><div>Dumps the value referred to by `value_addr` to a null-terminated JSON serialized string and returns the address of the start of the string. Rego sets are serialized as JSON arrays. Non-string Rego object keys are serialized as strings. </div> | 1.0 |
| `opa_value_dump` | <div> `str_addr opa_value_dump(value_addr)`</div><div>The same as `opa_json_dump` except Rego sets are serialized using the literal syntax and non-string Rego object keys are not serialized as strings. </div> | 1.0 |
| `opa_heap_ptr_set` | <div> `void opa_heap_ptr_set(addr)`</div><div>Set the heap pointer for the next evaluation. </div> | 1.0 |
| `opa_heap_ptr_get` | <div> `addr opa_heap_ptr_get(void)`</div><div>Get the current heap pointer. </div> | 1.0 |
| `opa_value_add_path` | <div> `int32 opa_value_add_path(base_value_addr, path_value_addr, value_addr)`</div><div>Add the value at the `value_addr` into the object referenced by `base_value_addr` at the given path. The `path_value_addr` must point to an array value with string keys (eg: `["a", "b", "c"]`). Existing values will be updated. On success the value at `value_addr` is no longer owned by the caller, it will be freed with the base value. The path value must be freed by the caller after use by calling `opa_value_free`. (The original path string passed to `opa_json_parse` or `opa_value_parse` to create the value must be freed by calling `opa_free`.) If an error occurs the base value will remain unchanged. Example: base object `{"a": {"b": 123}}`, path `["a", "x", "y"]`, and value `{"foo": "bar"}` will yield `{"a": {"b": 123, "x": {"y": {"foo": "bar"}}}}`. Returns an error code (see below). </div> | 1.0 |
| `opa_value_remove_path` | <div> `int32 opa_value_remove_path(base_value_addr, path_value_addr)`</div><div>Remove the value from the object referenced by `base_value_addr` at the given path. Values removed will be freed. The path value must be freed by the caller after use by calling `opa_value_free`. (The original path string parsed by `opa_json_parse` or `opa_value_parse` must be released using `opa_free`.) The `path_value_addr` must point to an array value with string keys (eg: `["a", "b", "c"]`). Returns an error code (see below). </div> | 1.0 |
| `opa_value_free` | <div> `void opa_value_free(value_addr)`</div><div>Free a value such as one generated by `opa_value_parse` or `opa_json_parse` reference at `value_addr`</div> | 1.3 |
| `opa_heap_blocks_stash` | <div> `void opa_heap_blocks_stash(void)`</div><div>Stash free heap blocks in a shadow heap to enable `eval` or `opa_eval` to allocate only blocks that it can subsequently free with a call to `opa_heap_ptr_set`. The caller should subsequently call `opa_heap_ptr_get` and store the value to save before calling `opa_heap_bloks_restore`</div> | 1.3 |
| `opa_heap_blocks_restore` | <div> `void opa_heap_blocks_restore(void)`</div><div>Restore heap blocks stored by `opa_heap_blocks_stash` to the heap. This should only be called after a `opa_heap_ptr_set` to the a heap pointer recorded by `opa_heap_ptr_get` after the previous call to `opa_heap_blocks_stash`.</div> | 1.3 |
| `opa_heap_stash_clear` | <div> `void opa_heap_stash_clear(void)`</div><div>Drop all heap blocks saved by `opa_heap_blocks_stash`. This leaks memory in the VM unless the caller subsequently invokes `opa_heap_ptr_set` to a value taken prior to calling `opa_heap_blocks_stash`. (see below)</div> | 1.3 |
| `opa_eval` | <div> `str_addr opa_eval(_ addr, entrypoint_id int32, data value_addr, input str_addr, input_len int32, heap_ptr addr, format int32)`</div><div>One-off policy evaluation method. Its arguments are everything needed to evaluate: entrypoint, address of data in memory, address and length of input JSON string in memory, heap address to use, and the output format (`0` is JSON, `1` is "value", i.e. serialized Rego values). The first argument is reserved for future use and must be `0`. Returns the address to the serialised result value. </div> | 1.2 |
The addresses passed and returned by the policy modules are 32-bit integer
offsets into the shared memory region. The `value_addr` parameters and return
values refer to OPA value data structures: `null`, `boolean`, `number`,
`string`, `array`, `object`, and `set`.
**Error codes:**
OPA Wasm Error codes are int32 values defined as:
| Value | Name | Description |
| ----- | -------------------- | ----------------------------------- |
| 0 | OPA_ERR_OK | No error. |
| 1 | OPA_ERR_INTERNAL | Unrecoverable internal error. |
| 2 | OPA_ERR_INVALID_TYPE | Invalid value type was encountered. |
| 3 | OPA_ERR_INVALID_PATH | Invalid object path reference. |
#### Imports
Policy modules require the following function imports at instantiation-time:
| Namespace | Name | Params | Result | Description |
| --------- | -------------- | ------------------------------------------------------------------------------ | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `env` | `opa_abort` | `(addr)` | `void` | Called if an internal error occurs. The `addr` refers to a null-terminated string in the shared memory buffer. |
| `env` | `opa_println` | `(addr)` | `void` | Called to emit a message from the policy evaluation. The `addr` refers to a null-terminated string in the shared memory buffer. |
| `env` | `opa_builtin0` | <span class="opa-keep-it-together"> `(builtin_id, ctx)`</span> | `addr` | Called to dispatch the built-in function identified by the `builtin_id`. The `ctx` parameter reserved for future use. The result `addr` must refer to a value in the shared-memory buffer. The function accepts 0 arguments. |
| `env` | `opa_builtin1` | <span class="opa-keep-it-together"> `(builtin_id, ctx, _1)`</span> | `addr` | Same as previous except the function accepts 1 argument. |
| `env` | `opa_builtin2` | <span class="opa-keep-it-together"> `(builtin_id, ctx, _1, _2)`</span> | `addr` | Same as previous except the function accepts 2 arguments. |
| `env` | `opa_builtin3` | <span class="opa-keep-it-together"> `(builtin_id, ctx, _1, _2, _3)`</span> | `addr` | Same as previous except the function accepts 3 arguments. |
| `env` | `opa_builtin4` | <span class="opa-keep-it-together"> `(builtin_id, ctx, _1, _2, _3, _4)`</span> | `addr` | Same as previous except the function accepts 4 arguments. |
The policy module also requires a shared memory buffer named `env.memory`.
#### Memory Buffer
A shared memory buffer must be provided as an import for the policy module with
the name `env.memory`. The buffer must be large enough to accommodate the input,
provided data, and result of evaluation.
#### Built-in Functions
After instantiating the policy module, call the exported `builtins` function to
receive a mapping of built-in functions required during evaluation. The result
maps required built-in function names to the identifiers supplied to the
built-in function callbacks (e.g., `opa_builtin0`, `opa_builtin1`, etc.)
For example:
```javascript
const memory = new WebAssembly.Memory({ initial: 5 });
const policy_module = await WebAssembly.instantiate(
byte_buffer,
/* import object */
);
const addr = policy_module.instance.exports.builtins();
const str_addr = policy_module.instance.exports.opa_json_dump(addr);
const builtin_map = deserialize_null_terminated_JSON_string(memory, str_addr);
```
The built-in function mapping will contain all of the built-in functions that
may be required during evaluation. For example, the following query refers to
the `http.send` built-in function which is not included in the policy module:
```live:builtin:module:read_only
result := http.send({"method": "get", "url": "https://example.com/api/lookup/12345"})
```
If this query was compiled to Wasm the built-in map would contain a single
element:
```json
{
"http.send": 0
}
```
When the evaluation runs, the `opa_builtin1` callback would invoked with
`builtin_id` set to `0`.
#### Evaluation
Once instantiated, the policy module is ready to be evaluated. Use the
`opa_eval_ctx_new` exported function to create an evaluation context. Use the
`opa_eval_ctx_set_input` and `opa_eval_ctx_set_data` exported functions to specify
the values of the `input` and base `data` documents to use during evaluation.
To evaluate, call to the exported `eval` function with the eval context address
as the only parameter.
#### Input
The (optional) `input` document for a policy can be provided by loading a JSON
string into the shared memory buffer. Use the `opa_malloc` exported function to
allocate a buffer the size of the JSON string and copy the contents in at the
returned address. After the raw string is loaded into memory you will need to
call the `opa_json_parse` exported method to get an address to the parsed input
document for use in evaluations. Set the address via the
`opa_eval_ctx_set_input` exported function supplying the evaluation context
address and parsed input document address.
#### External Data
External data can be loaded for use in evaluation. Similar to the `input` this
is done by loading a JSON string into the shared memory buffer. Use `opa_malloc`
and `opa_json_parse` followed by `opa_eval_ctx_set_data` to set the address on
the evaluation context.
Data can be updated by using the `opa_value_add_path` and `opa_value_remove_path`
and providing the same value address as the base. Similarly, use `opa_malloc` and
`opa_json_parse` for the updated value and creating the path.
After loading the external data use the `opa_heap_ptr_get` exported method to save
the current point in the heap before evaluation. After evaluation this should be
reset by calling `opa_heap_ptr_set` to ensure that evaluation restarts back at the
saved data and re-uses heap space. This is particularly important if re-evaluating many
times with the same data.
If you want to continue to update data in between query evaluations then the calling
convention is a little more sophisticated due to the way that `eval` and `opa_eval`
release temporary memory between queries. The functions `opa_heap_blocks_stash` and
`opa_heap_blocks_restore` provide a safe way to stash free heap memory during queries
and then restore it for use when adding or removing further external data. Without
using these, the `eval` and `opa_eval` calls will leak all heap blocks below the
heap pointer. The calling convention is as follows:
- It's always prudent at VM initialization to call `opa_malloc` with a size of 0
and then `opa_heap_ptr_get` to obtain the initial value of the heap pointer for the
VM. Call this the "initial heap pointer".
- If you never load external data, you can use the "initial heap pointer" as your
"data heap pointer" for calls to `eval` or `opa_eval`.
- On the first load of external data, after calling `opa_heap_ptr_get` also call
`opa_heap_blocks_stash`. This will save free heap memory for reuse later and prevent
calls to `eval` or `opa_eval` from leaking that memory. Call the saved heap pointer
after the initial data document load the "data heap pointer". This "data heap
pointer" is the value that should be used in `eval` or `opa_eval` calls.
- On subsequent calls to modify the data document using `opa_value_add_path` or
`opa_value_remove_path`, do the following:
1. Call `opa_heap_ptr_set` passing the "data heap pointer" to reset the heap
and clear any memory left from `eval` or `opa_eval` calls.
2. Call `opa_heap_blocks_restore` to reinstate the heap stashed heap memory.
3. Call `opa_malloc`/`opa_json_parse`/`opa_free` to create the "path" and
"value" arguments (in WASM value form) as usual.
4. Call `opa_value_add_path` or `opa_value_remove_path` as usual.
5. Call `opa_value_free` on the "path" argument to release it as usual.
6. Call `opa_value_blocks_stash` to stash any free heap blocks to
protect them during `eval` or `opa_eval` calls.
7. Call `opa_heap_ptr_get` to get a new "data heap pointer". It may
be larger or smaller than the previous value depending upon which
internal memory the calls allocated or released.
- The calling convention for `eval` and `opa_eval` don't change at all.
- If, at any point, you wish to reset the VM to an initial state with regard
to the policy data then do the following:
1. Call `opa_heap_stash_clear` to drop all stashed heap blocks (if any).
2. Call `opa_heap_ptr_set` with the "initial heap pointer" to reset the
heap to its initial state.
3. Use the "initial heap pointer" as your new "data heap pointer" until
the next time you add external data in the VM.
It might seem counter-intuitive to hide available heap memory from calls
to `eval` or `opa_eval`. But that memory was never truly available for
queries in the first place. The very first call to `opa_heap_ptr_set`
(either before `eval` or which `opa_eval` calls internally) resets the heap
and leaks any free blocks on the heap. In versions of the ABI prior to
1.3 this memory was simply lost. Note, however, that multiple queries
would not continue to leak memory since they would always reset the heap
pointer to the same value. The WASM engine would only leak further
memory if there were subsequent calls to `opa_value_add_path` or
`opa_value_remove_path` followed by more queries. ABI 1.3 introduced the
calling convention using `opa_heap_blocks_stash` and
`opa_heap_blocks_restore` to allow for interleaving query evaluations
with incremental data document modifications.
#### Entrypoints
The compiled policy may have one or more entrypoints. If no entrypoint is set
on the evaluation context the default entrypoint (`0`) will be evaluated. SDKs
can call `entrypoints()` after instantiating the module to retrieve the
entrypoint name to entrypoint identifier mapping. SDKs can set the entrypoint to
evaluate by calling `opa_eval_ctx_set_entrypoint` on the evaluation context. If
an invalid entrypoint identifier is passed, the `eval` function will invoke `opa_abort`.
#### Results
After evaluation results can be retrieved via the exported
`opa_eval_ctx_get_result` function. Pass in the evaluation context address. The
return value is an address in the shared memory buffer to the structured result.
To access the JSON result use the `opa_json_dump` exported function to retrieve
a pointer in shared memory to a null terminated JSON string.
The result of evaluation is the set variable bindings that satisfy the
expressions in the query. For example, the query `x = 1; y = 2; y > x` would
produce the following result set:
```json
[
{
"x": 1,
"y": 2
}
]
```
Sets are represented as JSON arrays.
## Ecosystem Projects
<EcosystemEmbed feature="wasm-integration">
Wasm is a great way to integrate OPA into applications where the Go SDK is unavailable.
</EcosystemEmbed>
docs/new/docusaurus.config.js
+439
View File
@@ -0,0 +1,439 @@
const { themes } = require("prism-react-renderer");
const lightCodeTheme = themes.github;
const darkCodeTheme = themes.dracula;
const semver = require("semver");
import fs from "fs/promises";
const path = require("path");
const { loadPages } = require("./src/lib/ecosystem/loadPages");
// TODO: update this to "/" when this is the main site.
const baseUrl = "/new/";
// With JSDoc @type annotations, IDEs can provide config autocompletion
/** @type {import("@docusaurus/types").DocusaurusConfig} */
(
module.exports = {
title: "Open Policy Agent",
tagline: "Policy-based control for cloud native environments",
url: "https://openpolicyagent.org",
baseUrl: baseUrl,
// Build-time options
onBrokenLinks: "throw",
onBrokenMarkdownLinks: "throw",
trailingSlash: false,
presets: [
[
"@docusaurus/preset-classic",
/** @type {import("@docusaurus/preset-classic").Options} */
{
docs: {
path: "docs",
routeBasePath: "/docs/",
breadcrumbs: false,
sidebarPath: require.resolve("./src/lib/sidebars.js"),
},
blog: false,
theme: {
customCss: require.resolve("./src/css/custom.css"),
},
},
],
],
themeConfig: {
colorMode: {
disableSwitch: true,
// TODO: our graphics need an update to support this
respectPrefersColorScheme: false,
},
metadata: [
{ name: "msapplication-TileColor", content: "#2b5797" },
{ name: "theme-color", content: "#ffffff" },
],
headTags: [
{
tagName: "link",
attributes: {
rel: "icon",
href: "/favicon.ico",
},
},
{
tagName: "link",
attributes: {
rel: "apple-touch-icon",
sizes: "180x180",
href: "/apple-touch-icon.png",
},
},
{
tagName: "link",
attributes: {
rel: "icon",
type: "image/png",
sizes: "32x32",
href: "/favicon-32x32.png",
},
},
{
tagName: "link",
attributes: {
rel: "icon",
type: "image/png",
sizes: "16x16",
href: "/favicon-16x16.png",
},
},
{
tagName: "link",
attributes: {
rel: "manifest",
href: "/site.webmanifest",
},
},
{
tagName: "link",
attributes: {
rel: "mask-icon",
href: "/safari-pinned-tab.svg",
color: "#5bbad5",
},
},
],
navbar: {
title: "Open Policy Agent",
logo: {
alt: "OPA Logo",
src: "img/nav/logo.png",
},
items: [
{ to: "/docs/", label: "Docs", position: "right" },
{ to: "/ecosystem/", label: "Ecosystem", position: "right" },
{ to: "/security", label: "Security", position: "right" },
{ to: "/support", label: "Support", position: "right" },
{ to: "/community", label: "Community", position: "right" },
{ href: "https://play.openpolicyagent.org/", label: "Play", position: "right" },
{ href: "https://blog.openpolicyagent.org/", label: "Blog", position: "right" },
{
type: "html",
position: "right",
value: `
<a href="https://github.com/open-policy-agent/opa"
target="_blank"
rel="noopener noreferrer"
aria-label="GitHub repository">
<img src="${baseUrl}img/nav/github.png" alt="GitHub" style="width: 24px; height: auto; margin-left: 8px;" />
</a>
`,
},
{
type: "html",
position: "right",
value: `
<a href="https://slack.openpolicyagent.org/"
target="_blank"
rel="noopener noreferrer"
aria-label="Slack community">
<img src="${baseUrl}img/nav/slack.png" alt="Slack" style="width: 24px; height: auto; margin-left: 8px;" />
</a>
`,
},
],
},
footer: {
style: "light",
links: [],
copyright:
`Open Policy Agent is a <a href="https://www.cncf.io/">Cloud Native Computing Foundation</a> Graduated project.
<img src="${baseUrl}img/footer/cncf.svg" alt="CNCF Logo" style="max-width: 10rem; vertical-align: middle; margin: 0 10px;"><br />
© ${new Date().getFullYear()}
Open Policy Agent contributors.
<a href="https://github.com/open-policy-agent/opa/blob/main/LICENSE">Licensed under the Apache License, Version 2.0</a>.
See the <a href="${baseUrl}/docs/contributing">contributing documentation</a> for information about contributing.
The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page.`,
},
prism: {
theme: lightCodeTheme,
darkTheme: darkCodeTheme,
additionalLanguages: [
"rego",
"hcl",
"json",
"java",
"scala",
"gradle",
"javadoc",
"sql",
"http",
"diff",
"typescript",
"ini",
"cypher",
"csharp",
"shell-session",
"go-module",
"docker",
"javastacktrace",
"properties",
"log",
],
magicComments: [
{
className: "code-block-terminal-command",
line: "terminal-command",
},
{
className: "code-block-terminal-command",
line: "cmd",
},
{
className: "code-block-diff-add-line",
line: "diff-add",
block: { start: "diff-add-start", end: "diff-add-end" },
},
{
className: "code-block-diff-remove-line",
line: "diff-remove",
block: { start: "diff-remove-start", end: "diff-remove-end" },
},
{
className: "theme-code-block-highlighted-line",
line: "highlight-next-line",
block: { start: "highlight-start", end: "highlight-end" },
},
{
className: "code-block-error-line",
line: "error-next-line",
block: { start: "error-start", end: "error-end" },
},
],
},
mermaid: {
theme: { light: "base", dark: "dark" },
options: {
themeVariables: { // https://mermaid.js.org/config/theming.html#theme-variables
fontFamily: "sans-serif",
primaryColor: "#76d3ed",
secondaryColor: "#fff",
tertiaryColor: "#fff",
},
},
},
},
plugins: [
[
require.resolve("@easyops-cn/docusaurus-search-local"),
{
indexPages: true,
},
],
() => ({
name: "raw-loader",
configureWebpack() {
return {
module: {
rules: [
{ test: /\.rego$/, use: "raw-loader" },
{ test: /\.txt$/, use: "raw-loader" },
],
},
};
},
}),
async function ecosystemLanguagePageGen(context, options) {
return {
name: "ecosystem-language-gen",
async loadContent() {
const languages = await loadPages(path.join(context.siteDir, "src/data/ecosystem/languages/*.md"));
return { languages };
},
async contentLoaded({ content, actions }) {
const { pagesByLanguage, languages } = content;
await Promise.all(
Object.keys(languages).map(async (language) => {
const routePath = path.join(baseUrl, `/ecosystem/by-language/${language}`);
return actions.addRoute({
path: routePath,
component: require.resolve("./src/EcosystemLanguage.js"),
exact: true,
modules: {},
customData: { language },
});
}),
);
},
};
},
async function ecosystemFeaturePageGen(context, options) {
return {
name: "ecosystem-feature-gen",
async loadContent() {
const features = await loadPages(path.join(context.siteDir, "src/data/ecosystem/features/*.md"));
return { features };
},
async contentLoaded({ content, actions }) {
const { features } = content;
await Promise.all(
Object.keys(features).map(async (feature) => {
const routePath = path.join(baseUrl, `/ecosystem/by-feature/${feature}`);
return actions.addRoute({
path: routePath,
component: require.resolve("./src/EcosystemFeature.js"),
exact: true,
modules: {},
customData: { feature },
});
}),
);
},
};
},
async function ecosystemData(context, options) {
return {
name: "ecosystem-data",
async loadContent() {
const entries = await loadPages(path.join(context.siteDir, "src/data/ecosystem/entries/*.md"));
const languages = await loadPages(path.join(context.siteDir, "src/data/ecosystem/languages/*.md"));
const features = await loadPages(path.join(context.siteDir, "src/data/ecosystem/features/*.md"));
const featureCategories = await loadPages(
path.join(context.siteDir, "src/data/ecosystem/feature-categories/*.md"),
);
return {
entries,
languages,
features,
featureCategories,
};
},
async contentLoaded({ content, actions }) {
const { createData } = actions;
const { entries, languages, features, featureCategories } = content;
await createData("entries.json", JSON.stringify(entries, null, 2));
await createData("languages.json", JSON.stringify(languages, null, 2));
await createData("features.json", JSON.stringify(features, null, 2));
await createData("feature-categories.json", JSON.stringify(featureCategories, null, 2));
},
};
},
async function builtinData(context, options) {
return {
name: "builtin-data",
async loadContent() {
const filePath = "../../builtin_metadata.json";
const fileContent = await fs.readFile(filePath, "utf-8");
const builtins = JSON.parse(fileContent);
return { builtins };
},
async contentLoaded({ content, actions }) {
const { createData } = actions;
const { builtins } = content;
await createData("builtins.json", JSON.stringify(builtins, null, 2));
},
};
},
async function ecosystemPagesGen(context, options) {
return {
name: "ecosystem-entries-pages-gen",
async loadContent() {
const entries = await loadPages(path.join(context.siteDir, "src/data/ecosystem/entries/*.md"));
return { entries };
},
async contentLoaded({ content, actions }) {
const { entries } = content;
await Promise.all(
Object.values(entries).map(async (entry) => {
const routePath = path.join(baseUrl, `/ecosystem/entry/${entry.id}`);
return actions.addRoute({
path: routePath,
component: require.resolve("./src/EcosystemEntry.js"),
exact: true,
modules: {},
customData: { id: entry.id },
});
}),
);
},
};
},
async function versionsData(context, options) {
return {
name: "versions-data",
async loadContent() {
const capabilitiesDir = path.resolve(__dirname, "../../capabilities");
let sortedVersions = [];
const dirents = await fs.readdir(capabilitiesDir, { withFileTypes: true });
const versionStrings = dirents
.filter(dirent => dirent.isFile() && dirent.name.endsWith(".json"))
.map(dirent => dirent.name.replace(".json", ""));
const validVersions = versionStrings.filter(v => semver.valid(v));
sortedVersions = semver.sort(validVersions);
return { versions: sortedVersions };
},
async contentLoaded({ content, actions }) {
const { createData } = actions;
const { versions } = content;
await createData("versions.json", JSON.stringify(versions, null, 2));
},
};
},
async function versionsPageGen(context, options) {
return {
name: "version-page-gen",
async contentLoaded({ content, actions }) {
return actions.addRoute({
path: path.join(baseUrl, `/docs/archive`),
component: require.resolve("./src/Archive.js"),
exact: true,
modules: {},
});
},
};
},
],
clientModules: [
require.resolve("./src/lib/playground.js"),
],
stylesheets: [
{
href: "https://unpkg.com/@antonz/codapi@0.19.8/dist/snippet.css",
},
],
scripts: [
{
src: "https://unpkg.com/@antonz/codapi@0.19.8/dist/snippet.js",
defer: true,
},
],
}
);
docs/new/package-lock.json
Generated
+19467
View File
File diff suppressed because it is too large Load Diff
docs/new/package.json
+25
View File
@@ -0,0 +1,25 @@
{
"name": "new",
"version": "1.0.0",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [],
"author": "",
"license": "ISC",
"description": "",
"dependencies": {
"@docusaurus/core": "^3.7.0",
"@docusaurus/preset-classic": "^3.7.0",
"@easyops-cn/docusaurus-search-local": "^0.49.2",
"docusaurus-lunr-search": "^3.6.0",
"md-front-matter": "^1.0.4",
"raw-loader": "^4.0.2",
"react-markdown": "^10.1.0",
"styled-components": "^6.1.18"
},
"engines": {
"node": ">=22.0.0"
}
}
docs/new/src/Archive.js
+123
View File
@@ -0,0 +1,123 @@
import Heading from "@theme/Heading";
import Layout from "@theme/Layout";
import React from "react";
const semver = require("semver");
import versions from "@generated/versions-data/default/versions.json";
const Archive = (props) => {
const title = "OPA Documentation Archive";
const lastOldDocsVersion = "v1.4.2";
const oldDocsVersions = [
"v0.11.0",
"v0.12.2",
"v0.13.5",
"v0.14.2",
"v0.15.1",
"v0.16.2",
"v0.17.3",
"v0.18.0",
"v0.19.2",
"v0.20.5",
"v0.21.1",
"v0.22.0",
"v0.23.2",
"v0.24.0",
"v0.25.2",
"v0.26.0",
"v0.27.1",
"v0.28.0",
"v0.29.4",
"v0.30.2",
"v0.31.0",
"v0.32.1",
"v0.33.1",
"v0.34.2",
"v0.35.0",
"v0.36.1",
"v0.37.2",
"v0.38.1",
"v0.39.0",
"v0.40.0",
"v0.41.0",
"v0.42.2",
"v0.43.1",
"v0.44.0",
"v0.45.0",
"v0.46.3",
"v0.47.4",
"v0.48.0",
"v0.49.2",
"v0.50.2",
"v0.51.0",
"v0.52.0",
"v0.53.1",
"v0.54.0",
"v0.55.0",
"v0.56.0",
"v0.57.1",
"v0.58.0",
"v0.59.0",
"v0.60.0",
"v0.61.0",
"v0.62.1",
"v0.63.0",
"v0.64.1",
"v0.65.0",
"v0.66.0",
"v0.67.1",
"v0.68.0",
"v0.69.0",
"v0.70.0",
"v1.0.1",
"v1.1.0",
"v1.2.0",
"v1.3.0",
"v1.4.2",
];
const firstDocsVersion = semver.valid("0.17.2");
const descVersions = versions.slice().reverse()
.filter(version => {
return semver.gt(version, lastOldDocsVersion) || oldDocsVersions.includes(version);
});
const getArchiveUrl = (version) => {
const urlVersionPart = version.replaceAll(".", "-");
return `https://${urlVersionPart}--opa-docs.netlify.app/`;
};
return (
<Layout title={title}>
<div className="container margin-vert--lg">
<Heading as="h1">{title}</Heading>
<p>Please find a list of archived OPA docs versions here:</p>
<div style={{ marginTop: "1rem" }}>
<ul
style={{
listStyle: "none",
paddingLeft: 0,
}}
>
{descVersions.map((version) => (
<li key={version}>
<a
href={getArchiveUrl(version)}
target="_blank"
rel="noopener noreferrer"
>
{version}
</a>
</li>
))}
</ul>
</div>
</div>
</Layout>
);
};
export default Archive;
docs/new/src/EcosystemEntry.js
+167
View File
@@ -0,0 +1,167 @@
import Heading from "@theme/Heading";
import Layout from "@theme/Layout";
import React from "react";
import ReactMarkdown from "react-markdown";
import entries from "@generated/ecosystem-data/default/entries.json";
import getLogoAsset from "./lib/ecosystem/getLogoAsset.js";
const EcosystemEntry = (props) => {
const { id } = props.route.customData;
const page = entries[id];
const {
title,
subtitle,
labels,
inventors,
blogs,
code,
videos,
tutorials,
content,
} = page;
return (
<Layout title={title}>
<div className="container margin-vert--lg">
<div style={{ display: "flex", alignItems: "center", marginBottom: "1rem" }}>
<img
src={getLogoAsset(id)}
alt={`${title} Logo`}
style={{ maxWidth: "150px", height: "auto", marginRight: "1rem" }}
/>
<Heading as="h1" style={{ margin: 0 }}>
{title}
</Heading>
</div>
{subtitle && <p style={{ fontSize: "1.2rem", color: "#555" }}>{subtitle}</p>}
{/* Content (Markdown) */}
{content && (
<div style={{ marginTop: "2rem" }}>
<div style={{ marginTop: "0.5rem" }}>
<ReactMarkdown>
{content}
</ReactMarkdown>
</div>
</div>
)}
{/* Inventors */}
{/* Disabled until we have inventor pages */}
{false && inventors?.length > 0 && (
<div style={{ marginBottom: "1rem" }}>
<strong>Inventors:</strong> {inventors.join(", ")}
</div>
)}
{/* Blogs */}
{blogs?.length > 0 && (
<div style={{ marginBottom: "1rem" }}>
<strong>Blogs:</strong>
<ul>
{blogs.map((url, idx) => (
<li key={`blog-${idx}`}>
<a href={url} target="_blank" rel="noopener noreferrer">
{url}
</a>
</li>
))}
</ul>
</div>
)}
{/* Code */}
{code?.length > 0 && (
<div style={{ marginBottom: "1rem" }}>
<strong>Code:</strong>
<ul>
{code.map((url, idx) => (
<li key={`code-${idx}`}>
<a href={url} target="_blank" rel="noopener noreferrer">
{url}
</a>
</li>
))}
</ul>
</div>
)}
{/* Videos */}
{videos?.length > 0 && (
<div>
<strong>Videos:</strong>
<ul>
{videos.map((video, idx) => {
// If the video is a simple string URL
if (typeof video === "string") {
return (
<li key={`video-${idx}`}>
<a href={video} target="_blank" rel="noopener noreferrer">
{video}
</a>
</li>
);
}
// Structured video object
return (
<li key={`video-${idx}`}>
<a href={video.link} target="_blank" rel="noopener noreferrer">
{video.title}
{video.venue && ` - ${video.venue}`}
</a>
{Array.isArray(video.speakers) && (
<ul>
{video.speakers.map((speaker, sIdx) => {
if (typeof speaker === "string") {
return <li key={`speaker-${sIdx}`}>{speaker}</li>;
}
return (
<li key={`speaker-${sIdx}`}>
{speaker.name} - {speaker.organization}
</li>
);
})}
</ul>
)}
</li>
);
})}
</ul>
</div>
)}
{/* Tutorials */}
{tutorials?.length > 0 && (
<div style={{ marginBottom: "1rem" }}>
<strong>Tutorials:</strong>
<ul>
{tutorials.map((url, idx) => (
<li key={`tutorial-${idx}`}>
<a href={url} target="_blank" rel="noopener noreferrer">
{url}
</a>
</li>
))}
</ul>
</div>
)}
{/* Labels */}
{labels && (
<div style={{ margin: "1rem 0" }}>
<strong>Category:</strong> {labels.category} <br />
<strong>Layer:</strong> {labels.layer}
</div>
)}
</div>
</Layout>
);
};
export default EcosystemEntry;
docs/new/src/EcosystemFeature.js
+88
View File
@@ -0,0 +1,88 @@
import Heading from "@theme/Heading";
import Layout from "@theme/Layout";
import React from "react";
import ReactMarkdown from "react-markdown";
import Card from "./components/Card";
import getLogoAsset from "./lib/ecosystem/getLogoAsset.js";
import sortPagesByRank from "./lib/ecosystem/sortPagesByRank.js";
import entries from "@generated/ecosystem-data/default/entries.json";
import featureCategories from "@generated/ecosystem-data/default/feature-categories.json";
import features from "@generated/ecosystem-data/default/features.json";
import languages from "@generated/ecosystem-data/default/languages.json";
const EcosystemFeature = (props) => {
const { feature } = props.route.customData;
const pagesByFeature = {};
for (const pageId in entries) {
const page = entries[pageId];
const features = page.docs_features || {};
for (const featureKey of Object.keys(features)) {
if (!pagesByFeature[featureKey]) {
pagesByFeature[featureKey] = [];
}
pagesByFeature[featureKey].push(page);
}
}
const pages = pagesByFeature[feature] || [];
const sortedPages = sortPagesByRank(pages);
const title = features[feature].title;
const content = features[feature].content;
return (
<Layout title={title}>
<div className="container margin-vert--lg">
<Heading as="h1" style={{ margin: 0 }}>
{title}
</Heading>
{content && (
<div style={{ marginTop: "1rem" }}>
<div style={{ marginTop: "0.5rem" }}>
<ReactMarkdown>
{content}
</ReactMarkdown>
</div>
</div>
)}
<div
style={{
marginTop: "2rem",
display: "flex",
flexWrap: "wrap",
justifyContent: "center",
gap: 20,
}}
>
{sortedPages.map((id) => {
const page = pages[id];
const cardData = {
title: page.title,
note: page.docs_features[feature].note,
icon: getLogoAsset(page.id),
link: `/ecosystem/entry/${page.id}`,
link_text: "View Details",
};
return (
<div key={id} style={{ flex: "1 1 30%", minWidth: "250px", maxWidth: "400px" }}>
<Card item={cardData} />
</div>
);
})}
</div>
</div>
</Layout>
);
};
export default EcosystemFeature;
docs/new/src/EcosystemLanguage.js
+85
View File
@@ -0,0 +1,85 @@
import Heading from "@theme/Heading";
import Layout from "@theme/Layout";
import React from "react";
import ReactMarkdown from "react-markdown";
import Card from "./components/Card";
import getLogoAsset from "./lib/ecosystem/getLogoAsset.js";
import sortPagesByRank from "./lib/ecosystem/sortPagesByRank.js";
import entries from "@generated/ecosystem-data/default/entries.json";
import featureCategories from "@generated/ecosystem-data/default/feature-categories.json";
import features from "@generated/ecosystem-data/default/features.json";
import languages from "@generated/ecosystem-data/default/languages.json";
const EcosystemFeature = (props) => {
const { language } = props.route.customData;
const pagesByLanguage = {};
for (const pageId in entries) {
const page = entries[pageId];
const lang = page.for_language;
if (!lang) continue;
if (!pagesByLanguage[lang]) {
pagesByLanguage[lang] = [];
}
pagesByLanguage[lang].push(page);
}
const pages = pagesByLanguage[language] || [];
const sortedPages = sortPagesByRank(pages);
const title = languages[language].title;
const content = languages[language].content;
return (
<Layout title={title}>
<div className="container margin-vert--lg">
<Heading as="h1" style={{ margin: 0 }}>
{title}
</Heading>
{content && (
<div style={{ marginTop: "1rem" }}>
<div style={{ marginTop: "0.5rem" }}>
<ReactMarkdown>
{content}
</ReactMarkdown>
</div>
</div>
)}
<div
style={{
marginTop: "2rem",
display: "flex",
flexWrap: "wrap",
justifyContent: "center",
gap: 20,
}}
>
{sortedPages.map((id) => {
const page = pages[id];
const cardData = {
title: page.title,
note: page.subtitle,
icon: getLogoAsset(page.id),
link: `/ecosystem/entry/${page.id}`,
link_text: "View Details",
};
return (
<div key={id} style={{ flex: "1 1 30%", minWidth: "250px", maxWidth: "400px" }}>
<Card item={cardData} />
</div>
);
})}
</div>
</div>
</Layout>
);
};
export default EcosystemFeature;
docs/new/src/components/BuiltinTable.js
+128
View File
@@ -0,0 +1,128 @@
import builtins from "@generated/builtin-data/default/builtins.json";
import React from "react";
import ReactMarkdown from "react-markdown";
function capitalize(str) {
return str.charAt(0).toUpperCase() + str.slice(1);
}
export default function BuiltinTable({
category,
id,
title,
children,
}) {
const categoryFns = builtins._categories[category];
if (!categoryFns) return <p>No built-ins found for category "{category}".</p>;
const htmlID = id || category;
const htmlTitle = title || capitalize(category);
return (
<div>
<h2 className="anchor" id={htmlID}>
{htmlTitle}
<a
href={`#${htmlID}`}
className="hash-link"
aria-label={`Direct link to ${htmlTitle}`}
title={`Direct link to ${htmlTitle}`}
>
</a>
</h2>
<table style={{ width: "100%", tableLayout: "fixed" }}>
<colgroup>
<col />
<col style={{ width: "100%" }} />
<col />
</colgroup>
<thead>
<tr>
<th>Function</th>
<th>Description</th>
<th>Meta</th>
</tr>
</thead>
<tbody>
{categoryFns.map((name) => {
const fn = builtins[name];
if (!fn) return null;
const anchor = `builtin-${category}-${name}`;
const isInfix = !!fn.infix;
const isRelation = !!fn.relation;
const args = fn.args || [];
const result = fn.result || {};
const signature = isInfix
? `${args[0]?.name || "x"} ${fn.infix} ${args[1]?.name || "y"}`
: isRelation
? `${name}(${args.map((a) => a.name).join(", ")}, ${result.name})`
: `${result.name || "result"} := ${name}(${args.map((a) => a.name).join(", ")})`;
return (
<tr key={anchor} id={anchor}>
<td>
<a href={`#${anchor}`}>
<code>{isInfix ? signature : name}</code>
</a>
</td>
<td>
<p>
<code>{signature}</code>
</p>
{fn.description && <ReactMarkdown>{fn.description}</ReactMarkdown>}
{args.length > 0 && (
<div>
<strong>Arguments:</strong>
{args.map((arg, i) => (
<div key={i} style={{ marginBottom: "0.5rem" }}>
<div>
<code>{arg.name}</code> <span>({arg.type})</span>
</div>
<div>
<ReactMarkdown>{arg.description}</ReactMarkdown>
</div>
</div>
))}
</div>
)}
<strong>Returns:</strong>
<div>
<code>{result.name}</code> <span>({result.type})</span>
<div>
<ReactMarkdown>{result.description}</ReactMarkdown>
</div>
</div>
</td>
<td>
<div>
{fn.introduced && fn.introduced !== "edge" && fn.introduced !== "v0.17.0" && (
<a
href={`https://github.com/open-policy-agent/opa/releases/${fn.introduced}`}
target="_blank"
rel="noopener noreferrer"
>
<span>{fn.introduced}</span>
</a>
)}
{fn.introduced === "edge" && <span>edge</span>}
{fn.wasm
? <span>Wasm</span>
: <span>SDK-dependent</span>}
</div>
</td>
</tr>
);
})}
</tbody>
</table>
{children}
</div>
);
}
docs/new/src/components/Card.js
+45
View File
@@ -0,0 +1,45 @@
import Link from "@docusaurus/Link";
import Heading from "@theme/Heading";
import ReactMarkdown from "react-markdown";
export default function Card({ item }) {
return (
<div
style={{
border: "1px solid #ddd",
borderRadius: 8,
padding: 16,
marginBottom: 16,
maxWidth: 400,
}}
>
{item.icon && (
<img
src={item.icon}
alt={item.title}
style={{ maxWidth: 40, marginBottom: 10 }}
/>
)}
<Heading as="h4">{item.title}</Heading>
<ReactMarkdown>
{item.note}
</ReactMarkdown>
{item.links && (
<ul>
{item.links.map((link, idx) => (
<li key={idx}>
<a href={link.url} target="_blank" rel="noopener noreferrer">
{link.text}
</a>
</li>
))}
</ul>
)}
{item.link && (
<Link className="button button--primary button--sm" to={item.link}>
{item.link_text}
</Link>
)}
</div>
);
}
docs/new/src/components/EcosystemEmbed.js
+68
View File
@@ -0,0 +1,68 @@
import useBaseUrl from "@docusaurus/useBaseUrl";
import React from "react";
import ReactMarkdown from "react-markdown";
import Card from "./Card";
import getLogoAsset from "../lib/ecosystem/getLogoAsset.js";
import sortPagesByRank from "../lib/ecosystem/sortPagesByRank.js";
import entries from "@generated/ecosystem-data/default/entries.json";
export default function EcosystemEmbed({ feature, children }) {
const allPages = entries;
const featurePages = [];
for (const pageId in allPages) {
const page = allPages[pageId];
if (page.docs_features && page.docs_features[feature]) {
featurePages.push(page);
}
}
// if there are too many, then we just provide a link to the page.
if (featurePages.length > 5) {
return (
<div className="margin-vert--lg">
Browse {featurePages.length} projects related to "{feature}" in the{" "}
<a href={useBaseUrl(`/ecosystem/by-feature/${feature}`)}>OPA Ecosystem</a>.
</div>
);
}
const sortedPages = sortPagesByRank(featurePages);
return (
<div className="margin-vert--lg">
{children}
<div
style={{
marginTop: "2rem",
display: "flex",
flexWrap: "wrap",
justifyContent: "center",
gap: 20,
}}
>
{sortedPages.map((id) => {
const page = featurePages[id];
if (!page) return null;
const cardData = {
title: page.title,
note: page.docs_features[feature]?.note ?? "No note available",
icon: getLogoAsset(page.id),
link: useBaseUrl(`/ecosystem/entry/${page.id}`),
link_text: "View Details",
};
return (
<div key={id} style={{ flex: "1 1 30%", minWidth: "250px", maxWidth: "400px" }}>
<Card item={cardData} />
</div>
);
})}
</div>
</div>
);
}
docs/new/src/components/EcosystemFeatureLink.js
+25
View File
@@ -0,0 +1,25 @@
import useBaseUrl from "@docusaurus/useBaseUrl";
import React from "react";
import entries from "@generated/ecosystem-data/default/entries.json";
import sortPagesByRank from "../lib/ecosystem/sortPagesByRank.js";
export default function EcosystemFeatureLink({ feature, children }) {
const allPages = entries;
const featurePages = [];
for (const pageId in allPages) {
const page = allPages[pageId];
if (page.docs_features && page.docs_features[feature]) {
featurePages.push(page);
}
}
let message = "1 project";
if (featurePages.length > 1) {
message = `${featurePages.length} projects`;
}
return <a href={useBaseUrl(`/ecosystem/by-feature/${feature}`)}>{children} ({message})</a>;
}
docs/new/src/components/ImageCard.js
+45
View File
@@ -0,0 +1,45 @@
import Link from "@docusaurus/Link";
import Heading from "@theme/Heading";
export default function ImageCard({ item }) {
return (
<div
style={{
border: "1px solid #ddd",
borderRadius: 8,
padding: 16,
marginBottom: 16,
maxWidth: 400,
textAlign: "center",
}}
>
{item.image && (
<img
src={item.image}
alt={item.title}
style={{ width: "100%", height: "auto", borderRadius: 8, marginBottom: 10 }}
/>
)}
<Heading as="h4" style={{ marginTop: 10 }}>
{item.title}
</Heading>
<p>{item.note}</p>
{item.links && (
<ul style={{ listStyleType: "none", padding: 0 }}>
{item.links.map((link, idx) => (
<li key={idx} style={{ marginBottom: 5 }}>
<a href={link.url} target="_blank" rel="noopener noreferrer">
{link.text}
</a>
</li>
))}
</ul>
)}
{item.link && (
<Link className="button button--primary button--sm" to={item.link} style={{ marginTop: 10 }}>
{item.link_text}
</Link>
)}
</div>
);
}
docs/new/src/components/PlaygroundExample.js
+178
View File
@@ -0,0 +1,178 @@
import Link from "@docusaurus/Link";
import { MDXProvider } from "@mdx-js/react";
import CodeBlock from "@theme/CodeBlock";
import * as components from "@theme/MDXComponents"; // Import default MDX components from Docusaurus theme
import React from "react";
import RunSnippet from "./runSnippet";
import SideBySideColumn from "./SideBySide/Column";
import SideBySideContainer from "./SideBySide/Container";
export default function PlaygroundExample({
dir,
}) {
let files = dir.keys().reduce((acc, key) => {
let fileName = key.replace(`./`, "");
if (!fileName.includes(".")) {
return acc;
}
if (!fileName.endsWith(".json")) {
acc[fileName] = dir(key).default;
} else {
acc[fileName] = dir(key);
}
return acc;
}, {});
const config = files["config.json"];
const input = files["input.json"];
const data = files["data.json"];
const policy = files["policy.rego"];
const title = files["title.txt"];
const intro = files["intro.md"];
const outro = files["outro.md"];
const output = files["output.json"];
const showInput = config?.showInput ?? true;
const showData = config?.showData ?? true;
const showTitles = config?.showTitles ?? true;
const command = config?.command ?? "data.play";
const state = encodeToBase64(JSON.stringify({
i: JSON.stringify(input, null, 2),
d: JSON.stringify(data, null, 2),
p: policy,
}));
const url = `https://play.openpolicyagent.org/?state=${state}`;
const showNotes = output && output.some(rule => rule.note);
let dataString = JSON.stringify(data, null, 2);
if (config && config.showData && config.dataLineLimit) {
dataString = dataString.split("\n").slice(0, config.dataLineLimit).join("\n") + "\n...";
}
// id is used to stop contents from other examples on the same page being used
const id = getId(state);
const snippetFiles = `#${id}-input.json:input.json #${id}-data.json:data.json`;
return (
<div>
{title && <h2>{title}</h2>}
{intro && intro()}
{showInput && (
<SideBySideContainer>
<SideBySideColumn>
<MDXProvider components={components}>
<CodeBlock language={"rego"} title="policy.rego">
{policy}
</CodeBlock>
<RunSnippet command="data.play" id={`${id}-policy.rego`} files={snippetFiles} />
</MDXProvider>
</SideBySideColumn>
<SideBySideColumn>
<MDXProvider components={components}>
<CodeBlock language={"json"} title="input.json">
{JSON.stringify(input, null, 2)}
</CodeBlock>
<RunSnippet id={`${id}-input.json`} />
</MDXProvider>
{showData && (
<MDXProvider components={components}>
<CodeBlock language={"json"} title="data.json">
{dataString}
</CodeBlock>
<RunSnippet id={`${id}-data.json`} />
</MDXProvider>
)}
</SideBySideColumn>
</SideBySideContainer>
)}
{!showData && (
<div className="dn">
{/* this is needed to include the contents of data.json, but hidden when the config turned it off */}
<CodeBlock language={"json"} title="data.json">
{JSON.stringify(data, null, 2)}
</CodeBlock>
<RunSnippet id={`${id}-data.json`} />
</div>
)}
{!showInput && (
<MDXProvider components={components}>
{/* this is needed to include the contents of input.json, but hidden when the config turned it off */}
<div className="dn">
<CodeBlock language={"json"} title="input.json">
{JSON.stringify(input, null, 2)}
</CodeBlock>
<RunSnippet id={`${id}-input.json`} />
</div>
<CodeBlock language={"rego"} title={showTitles ? "policy.rego" : ""}>
{policy}
</CodeBlock>
<RunSnippet command={command} id={`${id}-policy.rego`} files={snippetFiles} playgroundLink={url} />
</MDXProvider>
)}
{output && (
<p>
<Link to={url}>Open in OPA Playground</Link>
</p>
)}
{output && (
<table>
<thead>
<tr>
<th>Rule</th>
<th>Output Value</th>
{showNotes && <th>Notes</th>}
</tr>
</thead>
<tbody>
{output.map((rule, index) => {
return (
<tr key={index}>
<td>{rule.ref}</td>
<td>
{rule.value !== undefined
&& rule.value !== "undefined"
&& <code>{JSON.stringify(rule.value)}</code>}
{rule.value === "undefined" && <code>undefined</code>}
</td>
{showNotes && <td>{rule.note}</td>}
</tr>
);
})}
</tbody>
</table>
)}
{outro && outro()}
</div>
);
}
function encodeToBase64(str) {
const utf8Bytes = new TextEncoder().encode(str);
const base64String = btoa(String.fromCharCode.apply(null, utf8Bytes));
return base64String;
}
// djb2 http://www.cse.yorku.ca/~oz/hash.html
function getId(str) {
let hash = 5381;
for (let i = 0; i < str.length; i++) {
hash = (hash * 33) ^ str.charCodeAt(i);
}
return (hash >>> 0).toString(36).slice(0, 6);
}
docs/new/src/components/SideBySide/Column.js
+9
View File
@@ -0,0 +1,9 @@
import React from "react";
export default function SideBySideColumn({ children }) {
return (
<div>
{children}
</div>
);
}
docs/new/src/components/SideBySide/Container.js
+38
View File
@@ -0,0 +1,38 @@
import React from "react";
import styled from "styled-components";
const Container = styled.div`
display: flex;
flex-wrap: wrap;
justify-content: space-between;
@media (max-width: 1200px) {
flex-direction: column;
}
`;
const Child = styled.div`
margin-top: 1rem;
width: 50%;
&:first-child {
padding-right: 0.25rem;
}
&:last-child {
padding-left: 0.25rem;
}
@media (max-width: 1200px) {
width: 100%;
padding: 0;
}
`;
export default function SideBySideContainer({ children }) {
return (
<Container>
{React.Children.map(children, (child, index) => <Child>{child}</Child>)}
</Container>
);
}

Some files were not shown because too many files have changed in this diff Show More