Have done this some time in the past, but there was a few
new issues this would highlight now that we're on Go 1.24.
Mostly:
- Use `b.Loop()` in benchmarks
- Use `strings.SplitSeq` where possible
- Remove `omitempty` tag for types that can't be empty
Signed-off-by: Anders Eknert <anders@eknert.com>
Go 1.23 is no longer supported as per Go release policy.
Changes:
- Use Go v1.24.6 as the project SDK requirement
- Apply lint fixes for Go 1.24
- Fix "non-constant format string in call" issues as seen in CI.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
A tiny first step to have more tooling correctly report
virtual and base document conflicts, as detailed in #7694.
This PR fixes the `opa check` command to report conflicts
of this type when the `-b`/`--bundle` flag is provided. The
bundle flag is required as without that, `opa check` should
only verify policies and not load data at all.
While I was in the `cmd` directory, I got annoyed with how
many of these commands store the same constants for their
`--format` flag, so I decided to fix that too, even if it
wasn't related to what I originally planned to do. I hope
it's not too distracting.
Signed-off-by: Anders Eknert <anders@styra.com>
Following up on #7566, and now applying the more exciting
modernizations. fmt.Appendf was new to me! But especially
the contains checks are so much better IMHO. I have reviewed
all changes myself and did a few manual changes where it
became obvious that things could be improved a little further.
(the modernize analyzer still has some issues running against
OPA, and I have manually worked around those for the time being)
Signed-off-by: Anders Eknert <anders@styra.com>
to allow for using Rego v1 bundles in `opa build`/`check`/`eval`/`test`.
Before this change, a bundle with `1` as `rego_version`/`file_rego_versions` would be rejected when evaluated with the `--v0-compatible` flag with the error:
```
rego_parse_error: illegal capabilities: rego_v1 feature required for parsing v1 Rego
```
This is fixed by adding the `rego_v1` feature to the `v0` default capabilities applied when using the `--v0-compatible` flag. Note: this allows OPA to accept Rego `v1` modules inside bundles, but modules without a specified Rego version, such as freestanding non-bundle modules or modules inside bundles with no specified Rego version, are parsed as `v0`.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
And update code to conform to the rule.
- Replace unnecessary fmt.Sprintf with string concatenation
- Replace fmt.Sprint with more efficient strconv.Itoa
- Replace static fmt.Errorf calls with more efficient errors.New
Thanks @srenatus for pushing me down this rabbit hole!
Signed-off-by: Anders Eknert <anders@styra.com>
And a few other small fixes in tests. This i not so much
about performance but about choosing the best tool for a
given task :) But that the alternatives are also faster
doesn't hurt either.
Signed-off-by: Anders Eknert <anders@styra.com>
Fixing issue in the `opa eval` command where `--v0-compatible` was ignored for PE support modules when
used in combination with the `pretty` output format; producing v1 support modules instead of v0.
Fixing: #7248
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.
Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.
Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Previously, this would panic deep win the call stack when trying to do the actual
partial eval. Now, we're parsing the arguments and decline bad input with an error
message.
Fixes#7127.
Signed-off-by: kunal.das <kd.kunaldas92@gmail.com>
The previous version has been failing without any good reason for me,
so let's try this.
About the version pick: It's not the latest version (v1.62.0 at the
moment), because that would introduce a new revive rule,
redeclares-builtin-id, and that flags every variable called `min` or
`max` in the code base. I had started addressing these, but they were
just too many.
The new issues related to this version are mostly that it complains
whenever it finds a non-static string that makes its way into a printf-
like function. However, that's a common pattern in some place here, so
I've sprinkled some nolint:govet on it.
Signed-off-by: Stephan Renatus <stephan@styra.com>
A new optimized read mode has been added to the default in-memory store, where data written to the store is eagerly converted to AST values (the data format used during evaluation). This pre-converted data is faster to read, and won’t cause memory spikes during load; but comes with slower data writes (affects startup and bundle load/update time) and a larger lowest overall memory footprint for OPA. Can be enabled for `opa run`, `opa eval`, and `opa bench` by setting the `—optimize-store-for-read-speed`. See http://localhost:8888/docs/edge/policy-performance/#storage-optimization.
Implements: #4147
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Co-authored-by: Ashutosh Narkar <anarkar4387@gmail.com>
This is the last few tests to be refactored before all tests are compatible with the v1-by-default switch coming in OPA 1.0.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
To make OPA behave as v0.x post v1.0 release.
If used simultaneously with `--v1-compatible` flag, the `--v0-compatible` flag takes precedence.
Also, future-proofing `cmd` package tests for 1.0.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Prioritizing generating v0 Rego with `rego.v1` import when producing support modules for non-`--v1-compatible` optimized builds.
Affects `opa build` when the `-O` flag is used for optimization, and `opa eval` for partial evaluation with the `-p` flag.
Fixes: #6450
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
When `rego.v1` is in the list of imports directly applied on the `rego.Rego` SDK struct, this import, and it's effects, is applied to the query when parsed.
This change affects the `eval` and `bench` commands when the `--imports` flag is used.
Fixes: #6701
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Adding a global `rego_version` attribute to bundle manifest, to inform OPA runtime about what rego-version (v0/v1) to use to parse/compile contained Rego files.
The rego-version of individual Rego files can be overridden through the `file_rego_versions` manifest attribute.
Implements: #6578
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* ast+cmd+rego: Adding `--rego-v1` flag to `opa eval`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--rego-v1` flag to `opa build`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Formatting PE support modules to comply with rego-v1 when required
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Removing rego.v1 import when formatting for rego-v1 (not rego-v0-compat-v1)
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* touch up
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Fixing linting issues
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Consolidating `Bundle.FormatModules()` and `Bundle.FormatModulesForRegoVersion()`
Suggested by @ashutosh-narkar
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding descriptions to `RegoVersion`
Requested by @ashutosh-narkar
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Using `--v1-compatible` flag instead of `--rego-v1`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Updating docs
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Reintroducing `ParserOptions.RegoV1Compatible` to avoid breaking change
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* cmd & tester
Adding `--v1-compatible` flag to `opa test`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--v1-compatible` flag to `opa fmt`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Adding `--v1-compatible` flag to `opa check`
Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Making linter happy
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
* removing `Println` in test
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
* Review modifications suggested by @ashutosh-narkar
* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
* Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
* removing `Println` in test
* Updating docs with per-command behavioural descriptions for `--v1-compatible`.
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
---------
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This commit changes the string representation of the ident token from
'ident' to 'identifier' as this shows up in error messages and should be
a bit friendlier for users.
Signed-off-by: Torin Sandall <torin@styra.com>
* test: Add --schema flag to opa test
* runner: added annotation/metadata reading for opa test
Fixes: #5923
Signed-off-by: Renato Cordeiro <renato@renatocordeiro.com>
The number of EVAL/REDO counts in the profile result
are sometimes difficult to understand. This is mainly due to the
fact that the compiler rewrites expressions and assigns the
same location to each generated expression and the profiler
keys the counters by the location. So users have no idea
that multiple expressions may be contributing to the profile
result for a given line in the policy.
This change attempts to provide more clarity to the profile
output by including the number of generated expressions for
each given expression thereby helping to better understand
the result and also how the evaluation works.
Fixes: #2552
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
* ast: Enforce inlined schemas without --schemas flag
Enabled for commands:
* eval
* check
* test
Always parse annotations.
Always use `schemas` annotations with inlined schemas for type checking.
Ignore `schemas` annotations with schema refs if no `--schema` flag was provided.
Fixes: #5506
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Fixing an issue where if a `rego.metadata.*` function is called in a rule, any present `schemas` annotation will be used to inform type checking, even if no `--schema` flag was used for the command.
Only commands that takes the `--schema` flag can now use the `schemas` annotation to inform type checking; i.e. `eval` and `check`.
Fixes: 5430
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
This commit adds type coercion guards around the AST annotation
set building logic, to avoid panics when nil values appear.
Nil node values can happen here after copy-propagation, because
the AST nodes can be ripped out from under the annotation system
when they are pruned away or inlined into other rules.
Fixes: #5368
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This commit adds support to the `compile` package for extracting
entrypoints from Rego `entrypoint` metadata annotations. The new
entrypoint annotations can be applied to any valid entrypoint target,
e.g. rules and packages.
An example policy using the new entrypoint annotations:
package test
# METADATA
# entrypoint: true
allow {
input.x
}
The `build` and `eval` CLI commands have been updated so that they do
not require an entrypoint to be explicitly provided with `-e`, so long
as at least one valid rule/package is marked with an entrypoint
annotation.
The new feature is additive; entrypoints can still be explicitly
provided to these commands by `-e`, and those will be used alongside the
entrypoints discovered from the metadata annotations.
Fixes: #3459
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
* format: allow ignoreing source locations
* cmd/eval: format disregarding source locations for partial result
Before, we'd see this output:
```
$ opa eval -p -fsource 'time.clock(input.x)==time.clock(input.y)'
# Query 1
time.clock(time.clock(input.x), input.y)
```
Now, we get the proper answer: `time.clock(input.y, time.clock(input.x))`.
Note that it's a _display_ issue; the JSON output of PE has not been affected.
Fixes#4609.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
If the following conditions hold for a set of rules returned by the indexer,
it will set EarlyExit: true, and change how the complete virtual doc or
function is evaluated:
- all rule head values are ground
- all rule head values match
This implies that some cases where early exit would be possible will not be
covered:
p = x {
x := true
input.foo == "bar"
}
p = x {
x := true
input.baz == "quz"
}
To indicate that "early exit" is possible, the indexer result message is
amended. Also, the "Exit" trace event will have a message of "early" when
"early exit" actually happens in eval:
$ echo '{"x":"x", "y":"y"}' | opa eval -I -fpretty --explain=full -d r.rego data.r.r
query:1 Enter data.r.r = _
query:1 | Eval data.r.r = _
query:1 | Index data.r.r (matched 2 rules, early exit)
r.rego:11 | Enter data.r.r
r.rego:12 | | Eval input.y = "y"
r.rego:11 | | Exit data.r.r
query:1 | Exit data.r.r = _
query:1 Redo data.r.r = _
query:1 | Redo data.r.r = _
r.rego:11 | Redo data.r.r
r.rego:12 | | Redo input.y = "y"
r.rego:11 | Exit data.r.r early
With `r.rego` as
package r
r {
input.x = "x"
}
r = 2 {
input.z = "z"
}
r {
input.y = "y"
}
This is done in in a way such that early-exit will abort array/set/object
iterations on data:
r {
data.i[_] = "one"
data.j[_] = "four"
}
f(x, y) {
data.i[_] = x
data.j[_] = y
}
Complete rules (r) and functions (f) that iterate over sets, arrays, and
objects from either data (evalTree) or a term that's returned by some
other rule etc (evalTerm).
The CLI and golang packages expose ways to disable 'early-exit':
This is in line with how indexing can be disabled. It's supposed to be
used as a debugging measure, so it's only exposed as a CLI flag to
`opa eval`.
Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This adds a new top-level key to the capabilities structure, `allow_net`.
It currently is only used for restricting the typechecker's ability to fetch
remote refs in JSON schemas, but could be used more widely in the future.
It works like this:
- If it's not present, any host can be contacted
- If it's present, the items will be the hosts or IP addresses that may be
contacted; anything not in the list is prohibited.
- As a consequence, If it's present and empty (`[]`), no host can be contacted
Introducing a package-level var to gojsonschema isn't the prettiest solution,
but since we want this in an all-or-nothing way right now anyways, it does
the trick. And it's more ergonomic than adding extra parameters all over the
place.
Fixes#3746.
Also:
* move some profiling-related default params into newEvalCommandParams
* replace some errors.Wrap by fmt.Errorf in loader pkg
* remove some != nil handling where it didn't make a difference when
working on the schema set
* reduces indentation in code examples in `opa eval -h` and `opa check -h`
by replacing tabs by four spaces.
* ast: allow testing with remote refs without networking
It would be nice to ensure that the remote refs feature actually works,
without introducing a network dependency into our tests.
This commit adds the kube 1.14 definitions into ast/testdata, and uses
that from a httptest.Server instance in the unit tests.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This was introduced in #3624: turns out the eval command's code only
passed parsedInput into the rego package, and that only looked at
rawInput when feeding it into the wasm engine.
The approach taken here is to make the rego package more robust: It
should do the right thing if parsedInput is provided for an eval
under the wasm target: the parsing work was already done, let's not
dischard it.
Fixes#3666.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).
This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.
Breaking API Changes:
- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr
Signed-off-by: Will Beason <willbeason@google.com>
This commit does not change any functionality it just refactors the
schema implementation a bit:
* There is no reason to expose the ByPath collection for now. This
change simplifies things for the caller because they can just
perform get/put operations on the SchemaSet.
* Rename setTypesWithSchema to loadSchema.
* Move schema code into a separate file with it's own test cases
separate from the compiler.
Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Sebastian Spaink