ewilliams28/releases
1
0
Fork 0
mirror of https://github.com/open-policy-agent/opa.git synced 2026-08-12 19:32:48 -06:00
Code Issues Packages Projects Releases Wiki Activity

Add initial version of the language reference

Browse Source 
This covers language introduction, grammar, basics, and details on various
aspects of Opalog:

- Scalar values
- Composite values
- Variables
- References
- Rules
- Generating sets
- Generating objects
- Incremental definitions
- Negation
- Modules
- Packages
- Imports
- Comments
- Example data
This commit is contained in:
Torin Sandall
2016-03-09 21:28:50 -08:00
parent e2e4c1cab8
commit 6f1f0de27c
2 changed files with 775 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/CONCEPTS.md
+2 -7
View File
@@ -283,12 +283,7 @@ Content-Type: application/json
]
```
<!--
## What's Next
// Once at least one of these exist we can link to them. For now this section is hidden.
For more information on how to write policy definitions and queries, see [Opalog: OPA's Query Language](./LANGUAGE.md).
For more information on the architecture of OPA, see [OPA's Architecture](./ARCHITECTURE.md).
-->
For more information on how to write policy definitions and queries, see [Opalog: OPA's Query Language](./LANGUAGE.md).
docs/LANGUAGE.md
+773
View File
@@ -0,0 +1,773 @@
# Opalog: OPA's Query Language
OPA includes a policy engine that is purpose built for reasoning about information represented in structured documents such as JSON. Data stored in in the policy engine can be queried using OPA's native query language: Opalog.
## What is Opalog?
Opalog was inspired by [Datalog](https://en.wikipedia.org/wiki/Datalog), which is a well understood, decades old query language. Opalog extends Datalog to support structured document models such as JSON.
Opalog queries are assertions on data stored in OPA. These queries can be used to define policies that enumerate instances of data that violate the expected state of the system.
## Why use Opalog?
Use Opalog for defining policy that is easy to read and write.
Opalog focuses on providing powerful support for referencing nested documents and ensuring that queries are correct and unambiguous.
Opalog is declarative so policy authors can focus on what queries should return rather than how queries should be executed. These queries are simpler and more concise than the equivalent in an imperative language. Like other applications which support declarative query languages, OPA is able to optimize queries to improve performance, e.g., indexing, concurrent evaluation, reordering, etc.
## The Basics
This section introduces the main aspsects of Opalog.
The simplest rule contains a single expression and is defined in terms of a [Scalar Value](#scalar-values):
```opalog
pi :- 3.14159
```
Rules define the content of documents. We can query for the content of the "pi" document generated by the rule above:
```
pi
# 3.14159
```
Rules can also be defined in terms of [Composite Values](#composite-values):
```opalog
rect :- {"width": 2, "height": 4}
```
The result:
```
rect
# {"width": 2, "height": 4}
```
Many expressions are defined in terms of [Equality](#equality). These expressions can be thought of as assertions. The simplest type of assertion takes two scalar values:
```opalog
v :- 42 = "the meaning of life"
```
If we query for the contents of "v" we see the expression has been evaluated:
```
v
# false
```
The order of operands in an equality expression does not matter:
```opalog
u :- "the meaning of life" = 42
```
The result is the same:
```
u
# false
```
We can define rules in terms of [Variables](#variables) as well:
```opalog
t :- x = 42, y = 41, x > y
```
Multiple expressions are separated by the comma (",") character. In order for the rule to be true, all of the expressions in the rule must true for some set of variable bindings. There may be multiple sets of variable bindings that make the rule true. The body of a rule can be understood intuitively as "\<expression-1> AND \<expression-2> AND ... AND \<expression-N>".
When we query for the contents of "t" we see the obvious result:
```
t
# true
```
The order of expressions in a rule does not affect the document's content:
```opalog
s :- x > y, y = 41, x = 42
```
The query result is the same:
```
s
# true
```
Opalog supports [References](#references) to nested documents. For example:
```opalog
sites :- [{"name": "prod"}, {"name": "smoke1"}, {"name": "dev"}]
r :- sites[i].name = "prod"
```
The rule "r" above asserts that there exists one document within the "sites"
document which has the name "prod".
The result:
```
r
# true
```
We can generalize the example above with a rule that defines a set document instead of a boolean document:
```opalog
sites :- [{"name": "prod"}, {"name": "smoke1"}, {"name": "dev"}]
q[] = name :- sites[].name = name
```
When we query for "q" we obtain a set of names:
```
q
# ["prod", "smoke1", "dev"]
```
We can re-write the rule "r" from above to make use of "q". We will call the new rule "p":
```opalog
p :- "prod" = q[]
```
The result will be the same:
```
p
# true
```
Rules which have arguments can be queried with input values:
```
q[] = "smoke2"
# false
q[] = "dev"
# true
```
If you made it this far, congratulations. This section introduced the main
aspsects of Opalog. The rest of this document provides more detail on various
aspects of Opalog. Opalog's syntax is defined at the end of this document in
the [Opalog Grammar](#grammar) section.
## <a name="scalar-values"></a> Scalar Values
Scalar values are the simplest type of term in Opalog. Scalar values can be
strings, numbers, booleans, or null.
Documents can be defined solely in terms of scalar values. This is useful for
defining constants that are referenced in multiple places. For example:
```opalog
greeting :- "Hello"
max_height :- 42
pi :- 3.14159
allowed :- true
sentinel :- null
```
These documents can be queried like any other:
```
greeting
# "foo"
max_height
# 42
pi
# 3.14159
allowed
# true
sentinel
# null
```
## <a name="composite-values"></a> Composite Values
Composite values define collections. In simple cases, composite values
can be treated as constants like [Scalar Values](#scalar-values):
```opalog
cube :- {"width": 3, "height": 4, "depth": 5}
```
The result:
```
cube.width
# 3
```
Composite values can also be defined in terms of [Variables](#variables) or
[References](#references). For example:
```opalog
p[] = x :-
foo = 42,
bar = false,
baz = null,
x = {"foo": foo, "bar": [bar, baz]}
```
The result:
```
p
# [{"foo": 42, "bar": [false, null]}]
```
By defining composite values in terms of variables and references, rules can
define abstractions over raw data and other rules.
## <a name="variables"></a> Variables
Variables are another kind of term in Opalog. They can appear in both the
head and body of rules.
Variables appearing in the head of a rule can be thought of as input and
output of the rule. Unlike many programming languages, where a variable is either an input or an output, in Opalog a variable is simultaneously an input and an output. If a query supplies a value for a variable, that variable is an input, and if the query does not supply a value for a variable, that variable is an output.
For example:
```opalog
sites :- [{"name": "prod"}, {"name": "smoke1"}, {"name": "dev"}]
q[] = name :- sites[].name = name
```
In this case, if we evaluate "q" without providing an input value for "name" we obtain the names of all of the sites:
```
q
# ["prod", "smoke1", "dev"]
```
On the other hand, if we evaluate "q" with an input value for "name" we can determine whether "name" exists in the document defined by "q":
```
q[] = "smoke2"
# false
q[] = "dev"
# true
```
Variables appearing in the head of a rule must also appear in a non-negated
equality expression within the same rule. This property ensures that if the
rule is evaluated and all of the expressions evaluate to true for some set of
variable bindings, the variable in the head of the rule will be defined.
## <a name="references"></a> References
Referenced are used to access nested documents.
The examples in this section use the data defined in the [Examples](#examples)
section.
The simplest reference contains no variables. For example, the following
reference returns the hostname of the second server in the first site document
from our example data:
```
sites[0].servers[1].hostname
# "helium"
```
References are typically written using the "dot-access" style. The canonical form does away with "." and closely resembles dictionary lookup in a language such as Python:
```
sites[0]["servers"][1]["hostname"]
# "helium"
```
Both forms are valid, however, the "dot-access" style is typically more readable. Note, there are two cases where brackets need to be used:
1. String keys containing characters other than [a-z], [A-Z], [0-9], or "_" (underscore).
1. Non-string keys such as numbers, booleans, and null.
1. Variable keys which are described later.
References are always prefixed with a variable that identifes the root
document. In the example above this is "p". The root document may be:
- a local variable inside a rule.
- a rule inside the same package.
- a document stored in OPA.
- a documented temporarily provided to OPA as part of a transaction.
References can include variables as keys. References written this way are used to select a value from every element in a collection.
The following reference will select the hostnames of all the servers in our
example data:
```
sites[i].servers[j].hostname
# ["hydrogen", "helium", "lithium", "berylium", "boron"]
```
Conceptually, this is the same as the following imperative code (Python):
```python
def hostnames(sites):
result = []
for site in sites:
for server in site.servers:
result.append(server.hostname)
return result
```
Rules are often written in terms of multiple expressions that contain references to documents. In the following example, the rule defines a document containing pairs of app names and hostnames of the server where the app is deployed:
```opalog
apps_and_hostnames[] = pair :-
apps[i].name = name,
apps[i].servers[] = server,
sites[j].servers[k].name = server,
sites[j].servers[k].hostname = hostname,
pair = [name, hostname]
```
The result:
```
apps_and_hostnames
# [
# ["web", "hydrogen"],
# ["web", "helium"],
# ["web", "berylium"],
# ["web", "boron"],
# ["web", "nitrogen"],
# ["mysql", "lithium"],
# ["mysql", "carbon"],
# ["mongodb", "oxygen"]
# ]
```
Don't worry about understanding everything in this example right now. There are just two important points:
1. Several variables appear more than once in the body. When a variable is used in multiple locations, OPA will only produce documents for the rule with the variable bound to the same value in all expressions.
2. The rule is joining the "apps" and "sites" documents implicitly. In Opalog (and other languages based on Datalog) joins are implicit.
Using a different key on the same array or object provides the equivalent of self-join in SQL. For example, the following rule defines a document containing apps deployed
on the same site as "mysql":
```opalog
same_site[] = name :-
apps[i].name = "mysql",
apps[i].servers[] = server,
sites[j].servers[].name = server,
sites[j].servers[].name = other_server,
server != other_server,
apps[j].servers[] = other_server,
apps[j].name = name
```
The result:
```
same_site
# ["web", "web", "web", "web"]
```
## Rules
Rules define the content of [Virtual Documents](./CONTENT.md#data-model) in
OPA. When OPA evaluates a rule, we say OPA *generates* the content of the
document that is defined by the rule.
The examples in this section make use of the data defined
in the [Examples](#examples) section.
### <a name="set-documents"></a> Generating Sets
The following rule documents a set containing the hostnames of all servers:
```opalog
hostnames[] = name :- sites[].servers[].hostname = name
```
When we query for the content of "hostnames" we see the same data as we would if we queried using the `sites[].servers[].hostname` reference directly:
```
hostnames
# [
# "hydrogen",
# "helium",
# "lithium",
# "berylium",
# "boron",
# "carbon",
# "nitrogen",
# "oxygen"
# ]
```
This example introduces a few important aspects of Opalog.
First, the rule defines a set document where the contents are defined by the
variable "name". We know this rule defines a set document because the head
does not include a key, i.e., all rules follow this syntax:
```
rule ::= head ":-" body
head ::= name "[" [key] "]" [value]
name ::= var
key ::= var
value ::= var
```
> Set documents are collections of values without keys. OPA represents set documents as arrays when serializing to JSON or other formats which do not support a set data type. The important distinction between sets and arrays or objects is that sets are un-keyed while arrays and objects are keyed, i.e., you cannot refer to the index of an element within a set. While you cannot refer to sets as arrays or objects, you can refer to the values of arrays or objects as sets.
Second, the `sites[].servers[].hostname` fragment selects the "hostname" attribute from all of the objects in the "servers" collection. From reading the fragment in isolation we cannot tell whether the fragment refers to sets, arrays, or objects. We only know that it refers to a collections of values.
> Under the hood, OPA implicitly adds a variable to the reference for the purpose of evaluation, e.g., `sites[].servers[].hostname` becomes `sites[$a].servers[$b].hostname`. This translation happens for arrays, objects, AND sets. In the case of sets, OPA can determine when a reference points to a set and does not attempt to bind a value to it (i.e., an element index).
Third, the `sites[].servers[].hostname = name` expression binds the value of the "hostname" attribute to the variable "name", which is also declared in the head of the rule.
### <a name="object-documents"></a> Generating Objects
Rules that define objects are very similar to rules that define sets.
```opalog
apps_by_hostname[hostname] = app :-
sites[].servers[] = server,
server.hostname = hostname,
apps[i].servers[] = server.name,
apps[i].name = app
```
The rule above defines an object that maps hostnames to app names. The main
difference between this rule and one which defines a set is the rule head: in
addition to declaring a value, the rule head also declares a key for the
document.
The result:
```
apps_by_hostname["helium"]
# "web"
```
### <a name="incremental-definitions"></a> Incremental Definitions
A rule may be defined multiple times with the same name. When a rule is
defined this way, we refer to the rule definition as *incremental* because
each definition is additive. The document content for an incrementally defined
rule is the union of the document content for each of the individual rules.
For example, we can write a rule that abstracts over our "servers" and
"containers" data as "instances":
```opalog
instances[] = instance :-
sites[].servers[] = server,
instance = {"address": server.hostname, "name": server.name}
instances[] = instance :-
containers[] = container,
instance = {"address": container.ipaddress, "name": container.name}
```
An incrementally defined rule can be intuitively understood as "\<rule-1> OR
\<rule-2> OR ... OR \<rule-N>".
## <a name="negation"></a> Negation
To generate the content of a [Virtual Document](./CONCEPTS.md#data-model), OPA attempts to bind variables in the body of the rule such that all expressions in the rule evaluate to True.
This generates the correct result when the expressions represent assertions about what states should exist in the data stored in OPA. In some cases, you want to express that certain states *should not* exist in the data stored in OPA. In these cases, negation must be used.
For safety, a variable appearing in a negated expression must also appear in another non-negated equality expression in the rule.
> OPA will reorder expressions to ensure that negated expressions are evaluated after other non-negated expressions with the same variables. OPA will reject rules containing negated expressions that do not meet the safety criteria described above.
The simplest use of negation involves only scalar values or variables and is equivalent to complementing the operator:
```opalog
t :- 42 = x, not x = "the meaning of life"
```
The result:
```
t
# true
```
Negation is required to check whether some value *does not* exist in a
collection. I.e., complementing the operator in an expression such as `p[] =
"foo"` yields `p[] != "foo"`, however, this is not equivalent to `not p[] =
"foo"`.
For example, we can write a rule that defines a document containing names of
apps not deployed on the "prod" site:
```opalog
not_in_production[] = name :-
apps[i].name = name,
apps[i].servers[] = server,
not prod_server_names[] = server
prod_server_names[] = name :-
sites[i].name = "prod",
sites[i].servers[].name = name
```
The result:
```
not_in_production
# ["mongodb"]
```
## Modules
In Opalog, policies are defined inside *modules*. Modules consist of:
- Exactly one [Package](#packages) declaration.
- Zero or more [Import](#imports) statements.
- Zero or more [Rule](#rules) definitions.
Modules are typically represented in Unicode text and encoded in UTF-8.
### <a name="comments"></a> Comments
Comments begin with the `#` character and continue until the end of the line.
### <a name="packages"></a> Packages
Packages group the rules defined in one or more modules into a particular
namespace. Because rules are namespaced they can be safely shared across
projects.
Modules contributing to the same package do not have to be located in the same
directory.
The rules defined in a module are automatically exported. I.e., they can be
queried under OPA's [Data API](CONCEPTS.md#data-api) provided the appropriate
package is given, e.g., given the following module:
```opalog
package opa.examples
pi :- 3.14159
```
The "pi" document can be queried via the Data API:
```
GET /v1/data/opa/examples/pi
```
### <a name="imports"></a> Imports
Import statements declare dependencies that modules have on documents
defined outside the package. By importing a document, the identifiers
exported by that document can be referenced within the current module.
All modules contain an implicit statement which imports the "data" document.
Modules use the same syntax to declare dependencies on [Base
Documents](./CONCEPTS.md#data-model) and [Virtual
Documents](./CONCEPTS.md#data-model).
```opalog
package opa.examples
import data.servers
http_servers[] = server :-
server = servers[]
server.protocols[] = "http"
```
Imports can include an optional `alias` statement to handle namespacing
issues:
```opalog
package opa.examples
import data.servers as srvs
http_servers[] = server :-
server = srvs[]
server.protocols[] = "http"
```
## <a name="operators"></a> Operators
### <a name="equality"></a> Equality
The equality operator (`=`) is used to define expressions that assert that
two values are the same. If the expression is defined in terms of one or more
variables then the expression will evaluate to true if one of the variables is
unbound. If the neither operand is an unbound variable, the expression is
evaluated by comparing the *values* referenced by the operands.
OPA attempts to *bind* variables to values when it encounters unbound variables
in equality expressions. Binding a variable affects subsequent evaluation of
expressions such that the variable will be treated as a constant (with the
bound value) instead of a variable.
### <a name="inequality"></a> Inequality
The following inequality operators are supported:
| Symbol | Example | Description |
| --- | --- | --- |
| `!=` | `"foo" != x.y` | Returns true if the left hand side does not equal the right hand side, false otherwise. |
| `<` | `"foo" < x.y` | Returns true if the left hand side is less than the right hand side, false otherwise. |
| `>` | `"foo" > x.y` | Returns true if the left hand side is greater than the right hand side, false otherwise. |
| `>=` | `"foo" >= x.y` | Returns true if the left hand side is less than or equal to the right hand side, false otherwise. |
| `<=` | `"foo" <= x.y` | Returns true if the left hand side is greater than or equal to the right hand side, false otherwise. |
If either operand is a variable, the variable must appear in a non-negated
equality expression within the same rule.
## <a name="examples"></a> Examples
The rules below define the content of two documents describing a simplistic
deployment environment. These documents are referenced in other sections
above. If you are experimenting with OPA, you can copy-paste the content below
into an OPA CLI session and reference it when following along above.
```opalog
sites :- [
{
"region": "east",
"name": "prod",
"servers": [
{
"name": "web-0",
"hostname": "hydrogen"
},
{
"name": "web-1",
"hostname": "helium"
},
{
"name": "db-0",
"hostname": "lithium"
}
]
},
{
"region": "west",
"name": "smoke",
"servers": [
{
"name": "web-1000",
"hostname": "berylium"
},
{
"name": "web-1001",
"hostname": "boron"
},
{
"name": "db-1000",
"hostname": "carbon"
}
]
},
{
"region": "west",
"name": "dev",
"servers": [
{
"name": "web-dev",
"hostname": "nitrogen"
},
{
"name": "db-dev",
"hostname": "oxygen"
}
]
}
]
apps :- [
{
"name": "web",
"servers": ["web-0", "web-1", "web-1000", "web-1001", "web-dev"]
},
{
"name": "mysql",
"servers": ["db-0", "db-1000"]
},
{
"name": "mongodb",
"servers": ["db-dev"]
}
]
containers :- [
{
"image": "redis",
"ipaddress": "10.0.0.1",
"name": "big_stallman"
},
{
"image": "nginx",
"ipaddress": "10.0.0.2",
"name": "cranky_euclid"
}
]
```
## <a name="grammar"></a> Opalog Grammar
Opalog's syntax is defined by the following grammar:
```
module ::= package { import } policy
package ::= "package" ref
import ::= "import" package [ "as" var ]
policy ::= { rule }
rule ::= var [ rule-args ] :- rule-body
rule-args ::= "[" [ var ] "]" [ "=" var ]
rule-body ::= [ literal { "," literal } ]
literal ::= expr | "not" expr
expr ::= term
| expr-builtin
| expr-infix
expr-builtin ::= var "(" [ term { , term } ] ")"
expr-infix ::= term bool-operator term
term ::= ref | var | scalar | array | object
bool-operator ::= "=" | "!=" | "<" | ">" | ">=" | "<="
ref ::= var { ref-arg }
ref-arg ::= ref-arg-dot | ref-arg-brack
ref-arg-brack ::= "[" ( scalar | var ) "]"
ref-arg-dot ::= "." var
var ::= ALPHA { ALPHA | DIGIT | "_" }
scalar ::= STRING | NUMBER | TRUE | FALSE | NULL
array ::= "[" term { "," term } "]"
object ::= "{" object-item { "," object-item } "}"
object-item ::= ( scalar | ref | var ) ":" term
```
The grammar defined above makes use of the following syntax. See [the Wikipedia page on EBNF](https://en.wikipedia.org/wiki/Extended_Backus–Naur_Form) for more details:
```
[] optional (zero or one instances)
{} repetition (zero or more instances)
| alteration (one of the instances)
() grouping (order of expansion)
STRING JSON string
NUMBER JSON number
TRUE JSON true
FALSE JSON false
NULL JSON null
ALPHA ASCII characters A-Z and a-z
DIGIT ASCII characters 0-9
```