Commit Graph

73 Commits

Author SHA1 Message Date
Patrick East 346aa964e8 Add support for multiple bundles
This change brings in support for multiple bundles to be downloaded
and activated OPA.

This is enabled by using the new config option `bundles` to define
the bundles, and deprecates the older `bundle` option.

The new `bundles` keyword and structure is propagated through to the
decision logs, status API, provenance, stored manifests, etc. Check
out the doc changes for all the updated structures.

That being said any existing configuration using `bundle` will *not*
see the new structure, everything is intended to be backwards
compatible (almost to a fault).

Fixes: #721

Signed-off-by: Patrick East <east.patrick@gmail.com>
2019-07-31 03:43:38 -04:00
Torin Sandall 221974f28a docs: Add example of merging two objects
This is answers a common question that was recently raised in #1588.

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2019-07-30 08:54:53 -07:00
repenno 15725be32a Add jws encoding built-in functions
fixes #1174

Signed-off-by: repenno <rapenno@gmail.com>
2019-07-16 09:04:36 -04:00
Jason Hoch 26e0b325cb docs: Tweaks to improve consistency
The preceding example uses `x>y`, without which the variable `y`
is unused.

Signed-off-by: Jason Hoch <jrshoch@gmail.com>
2019-07-15 12:34:43 -07:00
Patrick East 41c0c97aee docs: Add details for .'s in --set keys
It is not very discoverable as to how the `--set` options work when
keys have `.` characters in them. This adds in a section to the config
docs on how to do it.

Signed-off-by: Patrick East <east.patrick@gmail.com>
2019-07-09 17:26:51 -07:00
Jessica de Leeuw 543b3a677f content: Fixed typo in language-reference.md
Signed-off-by: Jessica de Leeuw <jessicadl@google.com>
2019-07-03 06:48:09 +08:00
Torin Sandall 7d15f462ec docs: Add section on how to express "FOR ALL" in Rego
This is a common question that comes up. Until we have a keyword that
lets users express "FOR ALL" we should have docs we can point to.

Fixes #1307

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2019-06-29 01:20:40 +08:00
Xin Jin 00db47315a docs: fix minor typos and add principal in AWS example.
Signed-off-by: Xin Jin <xin@styra.com>
2019-06-29 00:46:29 +08:00
Torin Sandall 6aa521515c docs: Remove cast_xyz functions from docs
Fixes #1405

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2019-06-28 03:30:30 +08:00
Patrick East 1b6ee8de21 docs: Clarify config for console decision logger
Clarify the requirements/caveats for using the local decision logger
and include a blurb on the decision log.

Signed-off-by: Patrick East <east.patrick@gmail.com>
2019-06-26 06:37:32 -07:00
Torin Sandall 096670bda4 ast: Update rule index to support glob.match
These changes update the rule index to support glob.match calls. The
changes update the build step to transform glob.match(pattern,
delimiter, match) calls into ref/value pairs like equality
expressions. In addition, the build step creates a mapper function
that transforms resolved values into lookup values during traversal.

As part of these changes, the build operation has been refactored so
that most of the process is encapsulated by a new refindices struct
that's separate from the actual rule index/trie. Hopefully this makes
the code easier to follow.

The index build step for glob.match only recognizes statements where
the match operand was given as a reference (and then was rewritten by
the earlier step in the compiler.) If this is too brittle we could
revisit the build step and close over all intermediate assignments.

Fixes #1496

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2019-06-26 08:57:25 +08:00
Julien Garcia Gonzalez 758f1bbcc9 Fix link in the kubernetes-admission-control docs
This PR is fixing docs link in the kubernetes-admission-control-validation docs

Signed-off-by: Julien Garcia Gonzalez <julien@giantswarm.io>
2019-06-24 12:11:01 -07:00
Ken Fukuyama 4412102707 Add example of JSON pointers using array indexes.
Add specific example using array indexes to show what expression is
allowed and what is not explicitly.

Signed-off-by: Ken Fukuyama <kenfdev@gmail.com>
2019-06-19 15:57:16 -07:00
Patrick East a1abbbf4c9 logs: console decision logging option
Add option to log decision logs locally. They'll get logged via
Logrus at info level.

To enable configure OPA with something like:

```
decision_logs:
    console: true
```

This will work alongside remote services and plugins. It will also
log the masked events in the case a masking policy is set.

Fixes: #1334
Signed-off-by: Patrick East <east.patrick@gmail.com>
2019-06-13 13:27:51 -07:00
Torin Sandall 0b326b0a3e docs: Remove mention of persistent storage
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2019-06-13 09:21:52 -07:00
omerlh 1088d4fe6f allow user to define the query used for discovery
Signed-off-by: omerlh <omerl@soluto.com>
2019-06-13 09:19:39 -07:00
Prince Rachit Sinha c5d64e86d1 minor corrections
Signed-off-by: Prince Rachit Sinha <atheos.commune@gmail.com>
2019-06-13 07:46:48 -07:00
Patrick East a1171b27f1 Revert "docs(): updating query api input example to match expected"
This reverts commit b0e5ce55d3.

Signed-off-by: Patrick East <east.patrick@gmail.com>
2019-06-11 18:26:26 -07:00
Patrick East 5f19dd7117 Add docs for empty config options
Signed-off-by: Patrick East <east.patrick@gmail.com>
2019-06-11 18:25:52 -07:00
Torin Sandall 113e168528 bundle: Add support for loading YAML files from bundles
These changes simply update the bundle reader to accept data.yaml
files in addition to data.json files. This allows callers to supply
YAML files to OPA without having to convert them into JSON.

Fixes #1471

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2019-06-11 17:57:13 -07:00
Torin Sandall bd5f28e75e docs: Fix mention of reference head var
This actually goes back to the very first draft of the language
reference.

Fixes #1477

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2019-06-11 17:52:51 -07:00
Art Begolli b0e5ce55d3 docs(): updating query api input example to match expected
Signed-off-by: Art Begolli <ab14764@bristol.ac.uk>
2019-06-11 12:57:09 -07:00
Craig Hooper bee2c6d2aa Add timezone functionality to builtin time/date functions
Signed-off-by: Craig Hooper <craig.hooper@gmail.com>
2019-06-11 09:27:50 -07:00
Omer Levi Hevroni 575f6df8e2 Improve bundle docs
Signed-off-by: omerlh <omerl@soluto.com>
2019-06-05 09:42:11 -07:00
Torin Sandall c0cc24676d logs: Update decision logger to support masking
These changes enhance the in-built decision logger to support masking
of input and result fields for cases where sensitive information is
passed to OPA (or generated by the policy.)

Also, fix rego package to avoid panic-ing on programmatically created
queriest that lack Location information.

Fixes #779

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2019-06-04 16:14:46 -07:00
Abdennebi Mohamed 22f62ce61c Correct some typos
Signed-off-by: Patrick East <east.patrick@gmail.com>
2019-06-03 23:56:49 +02:00
Nick Williams a360905c5d Support AWS signatures for bundle HTTP operations
ref. Issue #1340

Signed-off-by: Nick Williams <nhw@me.com>
2019-06-03 21:52:27 +02:00
Torin Sandall d821d80db8 server: Add request headers to authorization input
These changes update the server to include request headers in the
authorization input document.

Fixes #1456

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2019-06-03 07:11:59 -07:00
Omer Levi Hevroni ed559e4a4b remove 404 status code
Signed-off-by: omerlh <omerl@soluto.com>
2019-05-29 16:39:38 +02:00
Torin Sandall 3263f54a74 ast: Rename 'var' to 'some'
This commit renames the 'var' keyword to 'some'. 'some' is more
descriptive than 'var' and will better complement an 'every' or
'forall' keyword representing for universal quantifiers.

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2019-05-18 21:27:32 -07:00
Torin Sandall 92a0e2487c docs: Update docs to include var keyword
Also, fix a couple examples to use := instead of the old datalog syntax.

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2019-05-18 21:27:32 -07:00
Tim Hinrichs b290000daf docs: Add namespace selector to k8s tutorial
Avoid OPA policies stopping changes to OPA's own configuration.
Do the same thing for kube-system.

Signed-off-by: Tim Hinrichs <tim@styra.com>
2019-05-17 12:55:10 -07:00
Patrick East 61f6f4a2b3 Switch the ceph example to use the correct image tag shortcode
Signed-off-by: Patrick East <east.patrick@gmail.com>
2019-05-16 13:21:10 -07:00
Patrick East 0ed56fef03 Change to using custom text log formatter
This is less sophisticated than the default logrus one, but it is
much easier to read the server output. Primarily with JSON fields
and multi-line strings. They are decoded and prettified.

Signed-off-by: Patrick East <east.patrick@gmail.com>
2019-05-10 12:04:56 -07:00
Patrick East 9be5e7b945 Make json logs the default and add json-pretty
The `json-pretty` log format flips on the “pretty” flag for logrus
which gives multi-line formatted json log output.

This also changes to make the `json` format to be the default as it
is likely to be what deployers want when running with `—server`.

Fixes: #1202
Signed-off-by: Patrick East <east.patrick@gmail.com>
2019-05-10 12:04:56 -07:00
Patrick East d07b6b3723 Add additional compiler metrics for each stage
Each stage of the compiler now has its own timer when instrumentation
is enabled.

To keep things consistent the naming follows the lowercase and
underscore style that the existing ones have. Each stage now needs
to define not only its name but its metric name too.

This does make a change to the ast compiler API by requiring the
additional naming information when adding extra stages.

And example of the new metrics:

```
+----------------------------------------------------------+--------------------+
|                          METRIC                          |       VALUE        |
+----------------------------------------------------------+--------------------+
| counter_eval_op_virtual_cache_miss                       | 1                  |
| histogram_eval_op_plug_75%                               | 881                |

<snip>

| histogram_eval_op_rule_index_min                         | 12995              |
| histogram_eval_op_rule_index_stddev                      | 0                  |
| timer_compile_stage_check_recursion_ns                   | 3080               |
| timer_compile_stage_check_rule_conflicts_ns              | 3141               |
| timer_compile_stage_check_safety_rule_bodies_ns          | 26366              |
| timer_compile_stage_check_safety_rule_heads_ns           | 10365              |
| timer_compile_stage_check_types_ns                       | 13269              |
| timer_compile_stage_rebuild_indices_ns                   | 13742              |
| timer_compile_stage_resolve_refs_ns                      | 20219              |
| timer_compile_stage_rewrite_assignments_ns               | 27630              |
| timer_compile_stage_rewrite_comprehension_terms_ns       | 10804              |
| timer_compile_stage_rewrite_dynamic_terms_ns             | 12036              |
| timer_compile_stage_rewrite_equals_ns                    | 7569               |
| timer_compile_stage_rewrite_expr_terms_ns                | 14067              |
| timer_compile_stage_rewrite_refs_in_head_ns              | 25290              |
| timer_compile_stage_rewrite_with_values_ns               | 9931               |
| timer_compile_stage_set_graph_ns                         | 11567              |
| timer_compile_stage_set_module_tree_ns                   | 3739               |
| timer_compile_stage_set_rule_tree_ns                     | 3093               |
| timer_eval_op_plug_ns                                    | 3203               |
| timer_eval_op_rule_index_ns                              | 12995              |
| timer_query_compile_stage_check_safety_ns                | 60071              |
| timer_query_compile_stage_check_types_ns                 | 55424              |
| timer_query_compile_stage_resolve_refs_ns                | 20801              |
| timer_query_compile_stage_rewrite_assignments_ns         | 20198              |
| timer_query_compile_stage_rewrite_comprehension_terms_ns | 10297              |
| timer_query_compile_stage_rewrite_dynamic_terms_ns       | 10086              |
| timer_query_compile_stage_rewrite_expr_terms_ns          | 9161               |
| timer_query_compile_stage_rewrite_to_capture_value_ns    | 21270              |
| timer_query_compile_stage_rewrite_with_values_ns         | 7114               |
| timer_rego_input_parse_ns                                | 575                |
| timer_rego_module_compile_ns                             | 423224             |
| timer_rego_module_parse_ns                               | 481                |
| timer_rego_query_compile_ns                              | 237144             |
| timer_rego_query_eval_ns                                 | 149115             |
| timer_rego_query_parse_ns                                | 344023             |
+----------------------------------------------------------+--------------------+
```

Fixes: #1059

Signed-off-by: Patrick East <east.patrick@gmail.com>
2019-05-08 09:41:38 -07:00
Torin Sandall 459a1b3699 docs: Add note about HTTP_PROXY and friends
Fixes #1410

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2019-05-07 10:08:45 -07:00
David Katz 0a4744f96a Checks for Ingress on UPDATE
Adds operation support for both CREATE and UPDATE.
This allows OPA to prevent updates to an Ingress when
it conflicts with another namespace.

Only enforcing on CREATE could allow an actor to first
apply a valid ingress and then update to a conflicting one.

Signed-off-by: David Katz <david.katz@mongodb.com>
2019-05-04 09:15:43 -06:00
Torin Sandall 2c14d2f5b2 docs: Update security docs with query parameter example
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2019-05-03 13:26:26 -06:00
Patrick East 589105aff5 Add param to include bundle activation in /health response
There is a new parameter for /health REST API which will include the
Configured bundle activation in the response. Example:

GET /health?bundle=true HTTP/1.1

Without the parameter the behavior stays the same, with it the server
will respond with 500’s until the status has been updated with an
activation time.

The docs for kubernetes ready probe has been updated to show this as
it makes for a better ready check than the original behavior when
remote bundles are being used.

Fixes: #1153
Signed-off-by: Patrick East <east.patrick@gmail.com>
2019-05-02 15:28:12 -06:00
Peter W. Morreale 9cf6281b31 Add provenance query param documentation
Describe the effects (and provide example output) of
the new provenance query parameter.

Signed-off-by: Peter W. Morreale <pmorreale@statestreet.com>
2019-05-01 20:26:37 -07:00
Patrick East 00e4b0025f Raise error when loading empty module in bundle
When we parse modules we won’t get an error back if the module contents
is empty, but we do get back a `nil` module value.

In the bundle loader we need to catch this before going further with
trying to load the module as it is not actually a valid module and
will break assumptions made about it.

According to the docs Modules must, at a minimum, have a package
declaration. With that in mind this seems like the right behavior to
enforce on the bundled rego files since we treat them as modules.

Fixes: #1393
Signed-off-by: Patrick East <east.patrick@gmail.com>
2019-04-30 15:05:49 -06:00
Torin Sandall d75271818b docs: Reformat the cheatsheet and move next to reference
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2019-04-30 13:34:54 -07:00
Patrick East f5953ce902 Fix extra ‘v’ in older versions of the docs
The older markdown still had `v{{< version >}}` shortcodes being used
which broke when the version changed to include the “v” in it.

This adds in some backwards compatibility logic for the shortcode and
adds newer, more specific, ones to replace it going forward. Part of
The issue is that typically the documentation will only want to
Reference the version that it is documenting and not always the
latest.

Fixes: #1382
Signed-off-by: Patrick East <east.patrick@gmail.com>
2019-04-27 20:11:35 -07:00
Patrick East c8ea625d65 Separate static content specific to doc content
We mixed together doc content and website content. We don’t need to
version the website content (logos, main site stuff, etc) but the doc
static content (mostly images) should be tied to the markdown that
uses it.

Signed-off-by: Patrick East <east.patrick@gmail.com>
2019-04-26 17:12:15 -07:00
Patrick East 230fae190d Move code snippets into docs content and version them
Previously the code content was only kept in master and was
Subsequently referenced by the versioned doc contents. This causes
A few problems.. so we will version it along with the doc content.

To get this to work we make a handful of changes:

* Move the docs up into just opa/docs/content/*, we’re going to remove
  The difference between local/dev and production builds soon.
* Move the “code” directory into the content dir, it is content tied
  To the markdown files already there.
* Change the Hugo config to ignore the code directories
* Change the “code” shortcode to load the versioned code snippet
* Change load-docs to copy the code directory in addition to the other
  markdown content.

Signed-off-by: Patrick East <east.patrick@gmail.com>
2019-04-26 17:12:15 -07:00
Patrick East f3aa3cde94 Use full tag name in version field for docs
It’s kinda weird that we have the “v” prefix all over when we started
out with the full tag name anyway. This change just keeps it all set
to the tag version.

Signed-off-by: Patrick East <east.patrick@gmail.com>
2019-04-26 17:12:15 -07:00
Sedat Gökcen 33fba2597e Correct the doc of array.slice function
The minor mistake in the documentation of array.slice function is corrected.

Signed-off-by: Sedat Gökcen <sedat@sedooe.com>
2019-04-23 08:01:36 -07:00
Torin Sandall ac27b855cc Update bundle docs with caching info
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2019-04-22 10:15:45 -07:00
Torin Sandall faa573f6a6 Add hardened configuration example to security page
Fixes #1172

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2019-04-18 22:24:25 -07:00