Commit Graph

79 Commits

Author SHA1 Message Date
Ville Vesilehto f77322b3fb build: bump Go version requirement to 1.24 (#7839)
Go 1.23 is no longer supported as per Go release policy.

Changes:

- Use Go v1.24.6 as the project SDK requirement
- Apply lint fixes for Go 1.24
- Fix "non-constant format string in call" issues as seen in CI.

Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
2025-08-24 09:02:09 +02:00
Anders Eknert 8ba08ac80c Apply modernize linter fixes (#7599)
Following up on #7566, and now applying the more exciting
modernizations. fmt.Appendf was new to me! But especially
the contains checks are so much better IMHO. I have reviewed
all changes myself and did a few manual changes where it
became obvious that things could be improved a little further.

(the modernize analyzer still has some issues running against
OPA, and I have manually worked around those for the time being)

Signed-off-by: Anders Eknert <anders@styra.com>
2025-05-20 23:12:13 +02:00
Anders Eknert e43ef0a979 Use any in place of interface{} (#7566)
Earlier this evening I tried to run the Go
[modernize](https://pkg.go.dev/golang.org/x/tools/gopls/internal/analysis/modernize)
analyzer on OPA. That didn't go as planned:

- https://github.com/golang/go/issues/73661
- https://github.com/golang/go/issues/73663

While we wait for that to be fixed, I figured an old-fashioned
search-and-replace across the repo may work for at least the
`interface{}` to `any` conversion. That should help make it easier
to see the other fixes as applied by the modernize tool once it has
had those issues resolved.

Signed-off-by: Anders Eknert <anders@styra.com>
2025-05-12 13:57:48 +02:00
Anders Eknert afb30d3f9d Add gocritic linter, fix a bunch of stuff (#7377)
Brace yourselves! For there are many touched files here. No changes
in semantics however.

Spent a long time trying out the various optional rules gocritic
provides, and settled for a few of them. There are more I really
like, but that would take many hours to address across the codebase.

Perhaps others find gocritic too pedantic? If so, we can merge the
fixes without enabling the rule.

Signed-off-by: Anders Eknert <anders@styra.com>
2025-02-24 16:28:41 +01:00
Anders Eknert b0100a66cd testing: replace reflect.DeepEqual where possible (#7286)
And a few other small fixes in tests. This i not so much
about performance but about choosing the best tool for a
given task :) But that the alternatives are also faster
doesn't hurt either.

Signed-off-by: Anders Eknert <anders@styra.com>
2025-01-21 10:34:53 +01:00
Johan Fylling a179a24c48 v1 API
All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.

Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.

Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.

Signed-off-by: Johan Fylling <johan.dev@fylling.se>
2024-12-12 15:27:34 +01:00
Johan Fylling 7bb6dbe36b Preparing for v1 API
Moving (most) source to v1 root package to prepare for v0/v1 API separation.

Signed-off-by: Johan Fylling <johan.dev@fylling.se>
2024-12-12 15:09:03 +01:00
Anders Eknert af8f915846 Add mirror linter
This flags unnecessary conversions of byte->string and vice versa

Signed-off-by: Anders Eknert <anders@eknert.com>
2024-09-26 13:45:37 -07:00
Johan Fylling 4ba95d0cc4 format: Bracketing keyword ref elements in formatter output (#7010)
Also future-proofing format pkg tests to be 1.0 compatible.

Signed-off-by: Johan Fylling <johan.dev@fylling.se>
2024-09-26 10:24:19 +02:00
Johan Fylling 546eeac58e rego-v1: Future-proofing ast package tests to be 1.0 compatible (#6992)
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
2024-09-18 10:22:15 +02:00
Johan Fylling 7cd3fecae7 ast: Make Module.String() include if/contains for v1 modules (#7000)
Fixes: #6973
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
2024-09-06 15:09:02 +02:00
Johan Fylling 91874a1561 rego-v1: Future-proofing compiler pkg tests (#6985)
* Updating `compile` package tests
* Respect optimizer rego-version for optimized support modules
* Setting parser options to use module rego-version when `bundle.FormatModulesForRegoVersion()` should preserve parsed module rego-version
* Enforcing requested rego-version for partial-eval support modules

Signed-off-by: Johan Fylling <johan.dev@fylling.se>
2024-09-05 10:50:28 +02:00
Anders Eknert f10cc1f4c8 Change required scope of entrypoint from rule to document (#6963)
And automatically change implied `scope` from `rule` to `document` when
no `scope` is provided (on rule metadata).

Fixes #6798

Signed-off-by: Anders Eknert <anders@styra.com>
2024-08-28 13:22:30 +02:00
Tyler Schade f37b5ccbc2 cmd+bundle: Add --follow-symlinks flag to include symlinked files when building bundles (#6800)
Signed-off-by: Tyler Schade <tyler.schade@solo.io>
2024-07-02 15:10:18 +02:00
Johan Fylling b58e87f89b ast: Importing rego.v1 in v0 support modules when applicable (#6698)
Prioritizing generating v0 Rego with `rego.v1` import when producing support modules for non-`--v1-compatible` optimized builds.

Affects `opa build` when the `-O` flag is used for optimization, and `opa eval` for partial evaluation with the `-p` flag.

Fixes: #6450
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
2024-04-24 19:02:30 +02:00
Johan Fylling e23d771711 Add rego_version attribute to bundle manifest (#6579)
Adding a global `rego_version` attribute to bundle manifest, to inform OPA runtime about what rego-version (v0/v1) to use to parse/compile contained Rego files.
The rego-version of individual Rego files can be overridden through the `file_rego_versions` manifest attribute.

Implements: #6578

Signed-off-by: Johan Fylling <johan.dev@fylling.se>
2024-04-10 19:06:07 +02:00
Philip Conrad 88eaaa93ad compile/compile: Fix panic from CLI + metadata entrypoint overlaps.
This commit fixes a panic that could occur when `opa build` was provided
an entrypoint from both a CLI flag, and via entrypoint metadata
annotation.

The fix is simple: deduplicate the slice of entrypoint refs that the
compiler uses, before compiling WASM or Plan targets.

Fixes: #6661

Co-authored-by: Daniel Herzig <danielherzig96@gmail.com>
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
2024-04-03 16:13:17 -07:00
Johan Fylling 38c2f0c5e0 Adding --v1-compatible flag to build, opa eval (#6478)
* ast+cmd+rego: Adding `--rego-v1` flag to `opa eval`

Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>

* Adding `--rego-v1` flag to `opa build`

Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>

* Formatting PE support modules to comply with rego-v1 when required

Signed-off-by: Johan Fylling <johan.dev@fylling.se>

* Removing rego.v1 import when formatting for rego-v1 (not rego-v0-compat-v1)

Signed-off-by: Johan Fylling <johan.dev@fylling.se>

* touch up

Signed-off-by: Johan Fylling <johan.dev@fylling.se>

* Fixing linting issues

Signed-off-by: Johan Fylling <johan.dev@fylling.se>

* Consolidating `Bundle.FormatModules()` and `Bundle.FormatModulesForRegoVersion()`

Suggested by @ashutosh-narkar

Signed-off-by: Johan Fylling <johan.dev@fylling.se>

* Adding descriptions to `RegoVersion`

Requested by @ashutosh-narkar

Signed-off-by: Johan Fylling <johan.dev@fylling.se>

* Using `--v1-compatible` flag instead of `--rego-v1`

Signed-off-by: Johan Fylling <johan.dev@fylling.se>

* Updating docs

Signed-off-by: Johan Fylling <johan.dev@fylling.se>

* Reintroducing `ParserOptions.RegoV1Compatible` to avoid breaking change

Signed-off-by: Johan Fylling <johan.dev@fylling.se>

* cmd & tester

Adding `--v1-compatible` flag to `opa test`

Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>

* Adding `--v1-compatible` flag to `opa fmt`

Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>

* Adding `--v1-compatible` flag to `opa check`

Fixes: #6463
Signed-off-by: Johan Fylling <johan.dev@fylling.se>

* Making linter happy

Signed-off-by: Johan Fylling <johan.dev@fylling.se>

* Review modifications suggested by @ashutosh-narkar

* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description

Signed-off-by: Johan Fylling <johan.dev@fylling.se>

* Review modifications suggested by @ashutosh-narkar

* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
  * Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`

Signed-off-by: Johan Fylling <johan.dev@fylling.se>

* Review modifications suggested by @ashutosh-narkar

* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
  * Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`

Signed-off-by: Johan Fylling <johan.dev@fylling.se>

* Review modifications suggested by @ashutosh-narkar

* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
  * Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
* removing `Println` in test

Signed-off-by: Johan Fylling <johan.dev@fylling.se>

* Review modifications suggested by @ashutosh-narkar

* Changing `ParserOptions.RegoV1Compatible` take precedence over `ParserOptions.RegoVersion`
* Fixing comment in test
* Updating `fmt --rego-v1` CLI description
* Adding back `Opts.RegoV1` and deprecating.
  * Making `Opts.RegoV1` take precedence over `Opts.RegoVersion`
* `TestPartialWitRegoV1` -> `TestPartialWithRegoV1`
* removing `Println` in test
* Updating docs with per-command behavioural descriptions for `--v1-compatible`.

Signed-off-by: Johan Fylling <johan.dev@fylling.se>

---------

Signed-off-by: Johan Fylling <johan.dev@fylling.se>
2023-12-20 10:59:14 +01:00
Torin Sandall 16fccc9a76 ast: extend capabilities to return min compatible version
This commit adds the ability to determine the minimum compatible OPA
version for a set of capabilities. This can be coupled with the
capabilities generated by the compiler to determine the min. compatible
version of a policy/bundle. The build has been extended to generate the
version index that lets us quickly check the required version for each
builtin/feature/keyword in the capabilities.

Signed-off-by: Torin Sandall <torin@styra.com>
2023-10-30 09:36:35 -07:00
Johan Fylling d7e9d13ea8 topdown: Fixing issues with optimizing rules with refs in the head
Issues fixed:

* production of support module with forbidden characters in first var of rule ref (#6338)
* panic when policy contains rules with a general ref in the head (#6339)

Fixes: #6338
Fixes: #6339

Signed-off-by: Johan Fylling <johan.dev@fylling.se>
2023-10-26 10:33:28 -07:00
Philip Conrad 434d324840 types: New algorithm for (Any).Union + new benchmarks (#6228)
This commit swaps out the old algorithm in the `(Any).Union` method that
would explode on some inputs to one that uses dramatically less memory,
and runs in nearly linear time and memory usage.

This commit also includes improvements to the
`BenchmarkCompileDynamicPolicy` benchmark, as well as new benchmarks for
the `Any` type's `Merge` and `Union` methods.

Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
2023-09-20 12:27:30 -04:00
Jakub Kulich 322cdd543e compile: bundle roots can be provided by caller
Previously, bundle roots couldn't be provided and neither .manifest file couldn't be provided, because it was replaced by the .manifest generated by the compiler.

Fixes: #6085

Signed-off-by: Jakub Kulich <jakubkulich@gmail.com>
2023-07-14 11:39:00 -07:00
Ashutosh Narkar 7ea3ee034b Add ability to specify namespace for optimized files
Currently the namespace for partially evaluated files
in an optimized bundle cannot be modified. As a result
if more than one optimized bundle is loaded in OPA, a root
conflict error would occur as the optimized bundles have a
root called "partial" automatially added to their
manifest. This change allows the namespace to be configured via
the build command.

Fixes: #5933

Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
2023-06-25 09:42:39 -07:00
Kieran Othen b65c68e340 Add ability to load bundles from an arbitrary filesystem
Support OPA Client SDK programs loading bundles from an arbitraty filesystem, such as an in-memory filesystem, which unlocks additional uses that include compiling a bundle to an intermediate representation from a client program rather than the OPA command line.

Fixes #5833

bundle: Add filesystem support
Soften constraint in `Equal` method to support bundle comparison for rootless filesystems, eg treat "/file" and "file" as equal for both URLs and Paths
Add `WithPathFormat` for `DirectoryLoader` builders to centralise logic for how paths are returned during file traversal, ie in `NextFile`
Add support for specifiying the root directory for `dirLoaderFS`

compile: Add filesystem support
Add `WithFS` builder helper to pass into `initload.LoadPaths` to load bundles from a filesystem

internal/runtime/init: Add filesystem support
Pass newly supplied `fsys fs.FS` parameter in `LoadPaths` into file loader builder

loader: Add filesystem support
Add new `GetBundleDirectLoaderFS` which can load bundles from the supplied filesystem

runtime: Add filesystem support
Pass-through nil parameter as `fsys fs.FS` parameter into `initLoad.LoadPaths` (OPA servers/repls are not in scope for loading from filesystem)

util/test: Add in-memory filesystem support
Add new `WithTestFS` helper to allow tests that currently use `WithTempFS` to choose between a disk-based or memory-based filesystem - now used throughout `compile_test`

Signed-off-by: Kieran Othen <kieran.othen@mac.com>
2023-04-26 12:28:17 -07:00
Philip Conrad eb3cf9096d ast/compile: Guard recursive module equality check. (#5757)
This commit moves a recursive AST module equality check in
checkRuleConflicts() behind a guard condition, so that it is evaluated
dramatically less often. This fixes a performance regression for
compiling large bundles.

Fixes: #5756

Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
2023-03-14 11:58:52 -04:00
Johan Fylling f93d0f8fea bundle: Retain metadata annotations for Wasm entrypoints during inspection (#5603)
* Pruning METADATA blocks associated with Wasm compiled entrypoints from Rego source in bundle
* Adding metadata annotations to wasm entrypoint declarations in bundle .manifest file
* Reading metadata annotations from both Rego source and .manifest file in bundle during `inspect`

Fixes: #5588
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
2023-02-03 12:15:21 +01:00
Johan Fylling 9f7e5e6be6 compile: Allow object generating rules to be annotated as entrypoint (#5578)
Using ground ref of rule when generating entrypoint from annotation
as variable in head suffix cannot be included in generated entrypoint path.

Fixes: #5577
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
2023-01-23 21:47:24 +01:00
shm12 30356a7fc2 build: Fix wrong windows bundle tar files path separator (#5478)
Fix #5473.
 
Signed-off-by: shm12 <shm.bia@gmail.com>
2022-12-19 10:59:25 +01:00
Stephan Renatus 0e6cb8808c planner: fix ref heads processing (#5418)
With the introduction of ref heads in #4660, the planned IR
still mostly worked, but it was bypassing the CallDynamic
optimization when it shouldn't have.

This commit re-works some of the rule planning to more robustly
handle ref heads.

Also adds a few test cases to get a grip on what should and
should not happen.

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2022-11-28 20:31:05 +01:00
Stephan Renatus d238828776 cmd/{build,check}: respect capabilities for parsing (#5326)
Before, the capabilities were plumbled through in most places:

1. checking which builtins exist
2. passed along to the optimizer
3. passed along to the planner

But they hadn't been passed along to the file loader. As such, it could not
pass the caps along to the parser either. This is now done, but adding a new
method to the FileLoader interface.

Fixes #5323.

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2022-11-01 18:50:03 +01:00
Philip Conrad 3e640489bd compile+types: Speed up typechecker when working with Refs (#5307)
* compile/compile_bench_test: Add basic compilation benchmark.
* types/types: Replace linear copy loops with copy() Go builtin.
* types/types: Switch linear scan -> binary search.

This commit switches out the linear scan in the `Any` type's `Contains`
method for a more efficient binary search. This results in around a 3-5x
speedup for policy compilation as a whole, according to the benchmark.

* ast/visit: Remove for loop item copies.

This commit refactors the visitor functions in the `ast/visit` package
to avoid extra copying of items on each loop iteration. This shaves off
around 10-15% memory usage during type-checking during compilation, and
provides speed benefits due to reduced GC as a result.

Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
2022-10-27 09:48:39 +02:00
Philip Conrad e2eabe88c1 ast+compile+cmd: Support entrypoint annotations. (#5246)
This commit adds support to the `compile` package for extracting
entrypoints from Rego `entrypoint` metadata annotations. The new
entrypoint annotations can be applied to any valid entrypoint target,
e.g. rules and packages.

An example policy using the new entrypoint annotations:

    package test

    # METADATA
    # entrypoint: true
    allow {
        input.x
    }

The `build` and `eval` CLI commands have been updated so that they do
not require an entrypoint to be explicitly provided with `-e`, so long
as at least one valid rule/package is marked with an entrypoint
annotation.

The new feature is additive; entrypoints can still be explicitly
provided to these commands by `-e`, and those will be used alongside the
entrypoints discovered from the metadata annotations.

Fixes: #3459

Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
2022-10-24 14:35:17 -04:00
Stephan Renatus 965301f90e ast: support dotted heads (#4660)
This change allows rules to have string prefixes in their heads -- we've
come to call them "ref heads".

String prefixes means that where before, you had

    package a.b.c
    allow = true

you can now have

    package a
    b.c.allow = true

This allows for more concise policies, and different ways to structure
larger rule corpuses.

Backwards-compatibility:

- There are code paths that accept ast.Module structs that don't necessarily
  come from the parser -- so we're backfilling the rule's Head.Reference
  field from the Name when it's not present.
  This is exposed through (Head).Ref() which always returns a Ref.

  This also affects the `opa parse` "pretty" output:

  With x.rego as

    package x
    import future.keywords
    a.b.c.d if true
    e[x] if true

  we get

    $ opa parse x rego
    module
     package
      ref
       data
       "x"
     import
      ref
       future
       "keywords"

     rule
      head
       ref
        a
        "b"
        "c"
        "d"
       true
      body
       expr index=0
        true
     rule
      head
       ref
        e
        x
       true
      body
       expr index=0
        true

  Note that

    Name: e
    Key: x

  becomes

    Reference: e[x]

  in the output above (since that's how we're parsing it, back-compat edge cases aside)

- One special case for backcompat is `p[x] { ... }`:

    rule                    | ref   | key | value | name
    ------------------------+-------+-----+-------+-----
    p[x] { ... }            | p     | x   | nil   | "p"
    p contains x if { ... } | p     | x   | nil   | "p"
    p[x] if { ... }         | p[x]  | nil | true  | ""

  For interpreting a rule, we now have the following procedure:

  1. if it has a Key, it's a multi-value rule; and its Ref defines the set:

     Head{Key: x, Ref: p} ~> p is a set
     ^-- we'd get this from `p contains x if true`
         or `p[x] { true }` (back compat)

  2. if it has a Value, it's a single-value rule; its Ref may contain vars:

     Head{Ref: p.q.r[s], Value: 12} ~> body determines s, `p.q.r.[s]` is 12
     ^-- we'd get this from `p.q.r[s] = 12 { s := "whatever" }`

     Head{Key: x, Ref: p[x], Value: 3} ~> `p[x]` has value 3, `x` is determined
                                          by the rule body
     ^-- we'd get this from `p[x] = 3 if x := 2`
         or `p[x] = 3 { x := 2 }` (back compat)

     Here, the Key isn't used, it's present for backwards compatibility: for ref-
     less rule heads, `p[x] = 3` used to be a partial object: key x, value 3,
     name "p"

- The destinction between complete rules and partial object rules disappears.
  They're both single-value rules now.

- We're now outputting the refs of the rules completely in error messages, as
  it's hard to make sense of "rule r" when there's rule r in package a.b.c and
  rule b.c.r in package a.

Restrictions/next steps:

- Support for ref head rules in the REPL is pretty poor so far. Anything that
  works does so rather accidentally. You should be able to work with policies
  that contain ref heads, but you cannot interactively define them.
  
  This is because before, we'd looked at REPL input like

      p.foo.bar = true

  and noticed that it cannot be a rule, so it's got to be a query. This is no
  longer the case with ref heads.

- Currently vars in Refs are only allowed in the last position. This is expected
 to change in the future.

- Also, for multi-value rules, we can not have a var at all -- so the following
  isn't supported yet:

      p.q.r[s] contains t if { ... }

-----

Most of the work happens when the RuleTree is derived from the ModuleTree -- in
the RuleTree, it doesn't matter if a rule was `p` in `package a.b.c` or `b.c.p`
in `package a`.

As such, the planner and wasm compiler hasn't seen that many adaptations:

- We're putting rules into the ruletree _including_ the var parts, so

  p.q.a = 1
  p.q.[x] = 2 { x := "b" }

  end up in two different leaves:

  p
  `-> q
       `-> a = 1
       `-> [x] = 2`

- When planing a ref, we're checking if a rule tree node's children have
  var keys, and plan "one level higher" accordingly:

  Both sets of rules, p.q.a and p.q[x] will be planned into one function
  (same as before); and accordingly return an object {"a": 1, "b": 2}

- When we don't have vars in the last ref part, we'll end up planning
  the rules separately. This will have an effect on the IR.

  p.q = 1
  p.r = 2

  Before, these would have been one function; now, it's two. As a result,
  in Wasm, some "object insertion" conflicts can become "var assignment
  conflicts", but that's in line with the now-new view of "multi-value"
  and "single-value" rules, not partial {set/obj} vs complete.
* planner: only check ref.GroundPrefix() for optimizations

In a previous commit, we've only mapped

    p.q.r[7]

as p.q.r;  and as such, also need to lookup the ref

    p.q.r[__local0__]

via p.q.r

(I think. Full disclosure: there might be edge cases here that are unaccounted
for, but right now, I'm aiming for making the existing tests green...)


New compiler stage:

In the compiler, we're having a new early rewriting step to ensure that the
RuleTree's keys are comparible. They're ast.Value, but some of them cause us
grief:

- ast.Object cannot be compared structurally; so

      _, ok := map[ast.Value]bool{ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")}): true}[ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")})]

  `ok` will never be true here.

- ast.Ref is a slice type, not hashable, so adding that to the RuleTree would
  cause a runtime panic:

      p[y.z] { y := input }

  is now rewritten to

    p[__local0__] { y := input; __local0__ := y.z }

This required moving the InitLocalVarGen stage up the chain, but as it's still
below ResolveRefs, we should be OK.

As a consequence, we've had to adapt `oracle` to cope with that rewriting:

1. The compiler rewrites rule head refs early because the rule tree expects
   only simple vars, no refs, in rule head refs. So `p[x.y]` becomes
   `p[local] { local = x.y }`
2. The oracle circles in on the node it's finding the definition for based
   on source location, and the logic for doing that depends on unaltered
   modules.

So here, (2.) is relaxed: the logic for building the lookup node stack can
now cope with generated statements that have been appended to the rule bodies.


There is a peculiarity about ref rules and extents:

See the added tests: having a ref rule implies that we get an empty object
in the full extent:

    package p
    foo.bar if false

makes the extent of data.p: {"foo": {}}

This is somewhat odd, but also follows from the behaviour we have right now
with empty modules:

    package p.foo
    bar if false

this also gives data.p the extent {"foo": {}}.

This could be worked around by recording, in the rule tree, when a node was
added because it's an intermediary with no values, but only children.

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2022-10-14 10:15:54 +02:00
Stephan Renatus 283b1e11f8 ir: make golang code public (#5141)
This has been semi-public anyways: people depend on the JSON structure to be
kept as-is.

So we might as well make the structs public, and make working with this easier
from golang. No need to copy the struct definitions manually.

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2022-09-16 07:56:13 +02:00
Philip Conrad b2d92a33c1 Add prealloc linter check + linter fixes (#5139)
This commit adds the `prealloc` linter to the list of linters for OPA, and fixes up the miscellaneous locations in the code that the linter found where we could easily preallocate slices.

Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
2022-09-15 15:09:54 -04:00
Anders Eknert b5cbbb5d8e opa build -t wasm|plan: fail on unmatched entrypoints
Fixes #3957

Signed-off-by: Anders Eknert <anders@eknert.com>
2022-09-07 14:52:15 +02:00
Stephan Renatus 3fbe069f9d cmd/build+compile: allow opt-out of dependents gathering (#5038)
* compile: allow opt-out of dependents gathering

With `.WithPruneUnused(true)`, the compiler (of the compile package) no longer
collects dependents of its entrypoints.

The resulting bundle, if used with the wasm target, will no longer be
semantically equivalent to the bundle built with the rego target.

Since we're unable to have entrypoints for functions, this allows building
modules that we couldn't build before.

Fixes #5035.

* cmd/build: expose new configurable via --prune-unused

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2022-08-23 10:31:29 +02:00
Stephan Renatus 137d7b6f72 storage/inmem: Allow disabling util.Roundtrip on Write (#5015)
Add option to inmem.store which allows disabling the round-tripping
through JSON when adding data to the store.

This option is intended for callers who can guarantee the objects they
pass to Write are JSON objects, and have properly ensured the object
will be only be accessed by store once added.

Fixes #4708.

This is continuance of https://github.com/open-policy-agent/opa/pull/4709,
adding these bits:

* storage/inmem: backwards-compat nitpicks, test adaptations

  I might have overshot here, but adding variable-length function parameters
  is not a backwards-compatible move. Concretely, if you had been using code like

      var x func() storage.Store = inmem.New

  going from New() to New(...Opts) would break it.

* storage/inmem: use it where possible without roundtrip

* storage/inmem: deal with nil map

  It looks like this is something the roundtrip had guarded us from.
  Now, we'll explicitly check this.

  This came up when running the bundle tests with roundtripping disabled.

* loader: add StoreWithOpts convenience method

Co-authored-by: Will Beason <willbeason@google.com>
Co-authored-by: Philip Conrad <conradp@chariot-chaser.net>
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2022-08-18 08:38:21 +02:00
Kevin St. Pierre 646c8418ae add compiler.Compiler and compiler.optimizer WithEnablePrintStatements option and pass through to ast.Compiler
When using the compiler.Compiler; rego "print" statements get removed and there is no way on the API to enable them.
So the PrintHook is broken.

Signed-off-by: Kevin St. Pierre <kevin@styra.com>
2022-07-22 13:22:14 -07:00
Jason Hall 4dd7fb1c0d Remove use of github.com/pkg/errors (#4696)
This package is deprecated, archived, and in maintenance mode, since Go
errors support wrapping natively.

For #2152.

Signed-off-by: Jason Hall <jason@chainguard.dev>
2022-05-18 11:29:35 +02:00
Maren-Sofie Stubø a18f53d187 compile: adds metadata field to .manifest (#4306)
Fixes: #4289

Signed-off-by: marensws <msws@live.no>
2022-01-31 14:26:46 +01:00
Torin Sandall 91027d1e26 cmd+compile: add new plan target
This commit adds a new "plan" target to the compile package and build
command. This targets lets users compile out query plans into
bundles. This way the compiled query plan can be consumed by
transpilers/interpreters outside of OPA.

Fixes #4133

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2022-01-29 06:25:30 -08:00
davidmarne-wf 3a56a7114d compile: fix file renaming check in optimization (#4154)
This was resulting in a max of 2 optimized entrypoints in a single file.

Signed-off-by: davidmarne-wf <david.marne@workiva.com>
2021-12-17 07:45:26 +01:00
Torin Sandall a1358d9a0f compile: Add debug message if optimized entrypoint is discarded (#4030)
Previously, `opa build -O=1` would silently discard the support
module for the entrypoint if the entrypoint could not be PE-ed. This
could be somewhat confusing if users expect an optimized/ directory to
show up in the bundle. This change just adds a debug message to the
compiler so that we can see when this happens.

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2021-11-18 08:22:34 +01:00
Stephan Renatus 7b8b6d8faa ast/term: sort set's keys slice on insert (#4028)
Before, we'd sort a set's `keys` slice in-place, and only when it
was actually compared to another set. This side-effect of a comparison
can be unexpected.

Now, we'll keep the `keys` slice sorted by inserting every new element
into its sorted position.

Analogous to #3823, where we did this with objects' `keys` slices.

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2021-11-17 19:54:34 +01:00
Stephan Renatus c0c3cd18a6 wasm: introduce one-off eval function, use it instead (#3627)
* wasm/sdk: check version, call old eval path for ABI 1.1

  Fixes #3146.

* docs/wasm: document addition as ABI 1.2

* wasm-sdk: overwrite previous inputs, don't accumulate them

  There is a little room for optimization here, should the input
  ever grow so large that it eats up too much precious heap space,
  we could look into changing this so that the memory used for it
  can be reclaimed.

* internal/compiler/wasm: commit generated wasm

  I've noticed that since the CI build running on macos-latest doesn't
  have docker installed, it cannot update these files itself at build
  time. We thus end up with macos binaries that have the wasm binary
  data from the main branch, not the PR.

  This can be observed from the test failure:

      Run make ci-binary-smoke-test-wasm BINARY=opa_darwin_amd64
      chmod +x "_release/0.31.0-dev/opa_darwin_amd64"
      "_release/0.31.0-dev/opa_darwin_amd64" eval -t "wasm" 'time.now_ns()'
      make: *** [ci-binary-smoke-test-wasm] Error 2
      {
        "errors": [
          {
            "message": "caller not found: opa_eval (opa_eval)"
          }
        ]
      }
      Error: Process completed with exit code 2.

  Since I had previously commit the CSV data that drives the dead
  code elimination process, that optimization had failed to find a
  function it expected to have.

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2021-07-15 09:41:40 +02:00
Torin Sandall d8287db653 ast, compile: Update opa build --debug output
This commit tweaks the --debug output format to be slightly more
readable. The comprehension indexing messages now have a prefix to
make them more discernable and the optimizer output is now aggregated
so that it's more concise and clear what parameters are being passed
to partial evaluation.

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
2021-05-24 12:21:08 -04:00
Will Beason 3be1d08b87 Change check-lint to use golangci-lint (#3465)
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).

This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.

Breaking API Changes:

- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr

Signed-off-by: Will Beason <willbeason@google.com>
2021-05-19 07:52:02 +02:00
Andre Håland 4b5f53b548 Deterministic order of rules for optimized bundles
Before this fix, an optimized bundle would get
non-deterministic order of the support rules.

One example use case where it is necessary for the optimized
bundle to be deterministic is when a hash of the bundle is
used for version tag.

Fixes: #3453

Signed-off-by: Andre Håland <andre.haland@gmail.com>
2021-05-14 12:59:49 -04:00
Stephan Renatus f7d7e0f692 build: dump policy IR if passing --debug (#3367)
Issue #2260 shows me that we did that last summer. Let's keep doing it.

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2021-04-13 14:15:58 +02:00