16 Commits

Author SHA1 Message Date
Johan Fylling a179a24c48 v1 API
All packages, except for `cmd` and `internal`, have been moved into a new `v1` root package.

Old packages are kept for backwards-compatibility reasons. All contained code is replaced with simple type aliases and proxy functions to `v1` implementations.

Old packages default to the Rego v0 syntax, new `v1` packages default to the Rego v1 syntax.

Signed-off-by: Johan Fylling <johan.dev@fylling.se>
2024-12-12 15:27:34 +01:00
Johan Fylling 7bb6dbe36b Preparing for v1 API
Moving (most) source to v1 root package to prepare for v0/v1 API separation.

Signed-off-by: Johan Fylling <johan.dev@fylling.se>
2024-12-12 15:09:03 +01:00
Johan Fylling 06942c6c29 rego-v1: Future-proofing ir, loader, profiler, and refactor pkg tests to be 1.0 compatible (#7021)
Updating tests in packages:

* ir
* loader
* profiler
* refactor

Signed-off-by: Johan Fylling <johan.dev@fylling.se>
2024-09-18 13:10:46 +02:00
Ashutosh Narkar 7f2895ddc8 profiler: Add number of generated expr to profile o/p
The number of EVAL/REDO counts in the profile result
are sometimes difficult to understand. This is mainly due to the
fact that the compiler rewrites expressions and assigns the
same location to each generated expression and the profiler
keys the counters by the location. So users have no idea
that multiple expressions may be contributing to the profile
result for a given line in the policy.

This change attempts to provide more clarity to the profile
output by including the number of generated expressions for
each given expression thereby helping to better understand
the result and also how the evaluation works.

Fixes: #2552

Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
2023-04-18 23:54:19 -07:00
Anders Eknert 95708108f3 linters: add unconvert (#5318)
Got a few warnings from my IDE about redundant type conversions,
so I decided to look into it. Added the unconvert linter to our
checks, and fixed the violations. Added two ignore comments as I
wasn't sure about whether they'd change the semantics of the code.

Signed-off-by: Anders Eknert <anders@eknert.com>
2022-10-27 13:35:39 +02:00
Philip Conrad 2d62aa9df3 topdown/builtins: Refactor registration functions and signatures (#5225)
* topdown/aggregates: Refactor to newer style.
* topdown/arithmetic: Refactor to newer style.
* topdown/array: Refactor to newer style.
* topdown/binary: Refactor to newer style.
* topdown/crypto: Refactor to newer style.
* topdown/casts: Refactor to newer style.
* topdown/comparison: Refactor to newer style.
* topdown/regex: Refactor to newer style.
* topdown/strings: Refactor to newer style.
* topdown/time: Refactor to newer style.
* topdown/type: Refactor to newer style.
* topdown/sets: Refactor to newer style.
* topdown/encoding: Refactor to newer style.
* topdown/type_name: Refactor to newer style.
* topdown/glob: Refactor to newer style.
* topdown/parse: Refactor to newer style.
* topdown/cidr: Refactor to newer style.

   This particular file was trickier to refactor than the others so far
   because the builtins were reused in 1-2 spots, which required
   renaming/wrapping tricks to get everything working again.

Also includes:
* builtins: Add function for extracting (Term, error) tuples.
* tests: Switch `test.sleep` examples to newer style.
* topdown: Use 'operands' over 'args' everywhere.
* topdown/builtins: Deprecate functional-style builtins.

Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
2022-10-11 09:20:49 +02:00
Philip Conrad b2d92a33c1 Add prealloc linter check + linter fixes (#5139)
This commit adds the `prealloc` linter to the list of linters for OPA, and fixes up the miscellaneous locations in the code that the linter found where we could easily preallocate slices.

Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
2022-09-15 15:09:54 -04:00
Stephan Renatus ef716a3142 opa eval: add '--count=#' flag (#3655)
`opa eval --count=# --profile` will now evaluate the query # times,
and display aggregated results for metrics and the expression-time
profile.

Before, `opa eval --profile` would yield output like this (JSON):

    "metrics": {
      "timer_rego_external_resolve_ns": 233,
      "timer_rego_load_files_ns": 138866,
      "timer_rego_module_compile_ns": 432654,
      "timer_rego_module_parse_ns": 81454,
      "timer_rego_query_compile_ns": 54892,
      "timer_rego_query_eval_ns": 135624,
      "timer_rego_query_parse_ns": 3886
    },
    "profile": [
      {
        "total_time_ns": 75705,
        "num_eval": 4,
        "num_redo": 2,
        "location": {
          "file": "t.rego",
          "row": 8,
          "col": 2
        }
      },

Now, both of these are replaced by "aggragated_" variants that
include some statistics:

    "aggregated_metrics": {
      "timer_rego_external_resolve_ns": {
        "75%": 1618.5,
        "90%": 1933.8000000000002,
        "95%": 1954,
        "99%": 1954,
        "99.9%": 1954,
        "99.99%": 1954,
        "count": 10,
        "max": 1954,
        "mean": 1137.1,
        "median": 1140,
        "min": 311,
        "stddev": 514.9278493148337
      },

and

    "aggregated_profile": [
      {
        "total_time_ns_stats": {
          "75%": 63369.75,
          "90%": 69523.5,
          "95%": 69691,
          "99%": 69691,
          "99.9%": 69691,
          "99.99%": 69691,
          "count": 10,
          "max": 69691,
          "mean": 39030.9,
          "median": 53758.5,
          "min": 3390,
          "stddev": 27635.790954666016
        },
        "num_eval": 1,
        "num_redo": 1,
        "location": {
          "file": "t.rego",
          "row": 9,
          "col": 2
        }
      }

The table format has been adjusted as well, and only displays a subset of
the statistics gathered:
Especially the 99.9 and 99.99 percentiles hardly matter for the sample
sizes I'd expect to be used here: they will almost certainly match "max".

The output is less unwieldy now:

    +------------------------------+---------+----------+---------------+----------------+---------------+
    |            METRIC            |   MIN   |   MAX    |     MEAN      |      90%       |      99%      |
    +------------------------------+---------+----------+---------------+----------------+---------------+
    | timer_rego_load_files_ns     | 349969  | 2549399  | 1.4760619e+06 | 2.5312689e+06  | 2.549399e+06  |
    | timer_rego_module_compile_ns | 1087507 | 24537496 | 1.120074e+07  | 2.41699473e+07 | 2.4537496e+07 |
    | timer_rego_module_parse_ns   | 275531  | 1915263  | 1.126406e+06  | 1.9016968e+06  | 1.915263e+06  |
    | timer_rego_query_compile_ns  | 61663   | 64395    | 63062.5       | 64374.1        | 64395         |
    | timer_rego_query_eval_ns     | 161812  | 1198092  | 637754        | 1.1846622e+06  | 1.198092e+06  |
    | timer_rego_query_parse_ns    | 6078    | 6078     | 6078          | 6078           | 6078          |
    +------------------------------+---------+----------+---------------+----------------+---------------+
    +----------+-------------+-------------+-------------+-------------+----------+----------+-----------------+
    |   MIN    |     MAX     |    MEAN     |     90%     |     99%     | NUM EVAL | NUM REDO |    LOCATION     |
    +----------+-------------+-------------+-------------+-------------+----------+----------+-----------------+
    | 43.875µs | 26.135469ms | 11.494512ms | 25.746215ms | 26.135469ms | 1        | 1        | data.rbac.allow |
    | 21.478µs | 211.461µs   | 98.102µs    | 205.72µs    | 211.461µs   | 1        | 1        | rbac.rego:13    |
    | 19.652µs | 123.537µs   | 73.161µs    | 122.75µs    | 123.537µs   | 1        | 1        | rbac.rego:40    |
    | 12.303µs | 117.277µs   | 61.59µs     | 116.733µs   | 117.277µs   | 2        | 1        | rbac.rego:50    |
    | 12.224µs | 93.214µs    | 51.289µs    | 92.217µs    | 93.214µs    | 1        | 1        | rbac.rego:44    |
    | 5.561µs  | 84.121µs    | 43.002µs    | 83.469µs    | 84.121µs    | 1        | 1        | rbac.rego:51    |
    | 5.56µs   | 71.712µs    | 36.545µs    | 71.158µs    | 71.712µs    | 1        | 0        | rbac.rego:45    |
    | 4.958µs  | 66.04µs     | 33.161µs    | 65.636µs    | 66.04µs     | 1        | 2        | rbac.rego:49    |
    | 4.326µs  | 65.836µs    | 30.461µs    | 65.083µs    | 65.836µs    | 1        | 1        | rbac.rego:6     |
    | 3.948µs  | 43.399µs    | 24.167µs    | 43.055µs    | 43.399µs    | 1        | 2        | rbac.rego:55    |
    +----------+-------------+-------------+-------------+-------------+----------+----------+-----------------+

Fixes #3651.

Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
2021-07-21 21:32:01 +02:00
Will Beason 3be1d08b87 Change check-lint to use golangci-lint (#3465)
golint is deprecated. The author of the code no longer supports the
codebase. golangci-lint is faster than golint, and is in use by other
opa repositories (e.g. Gatekeeper).

This commit changes tools.go to reference golangci (so it ends up in
vendor) and modifies check-lint to use golangci instead.

Breaking API Changes:

- plugins/rest/rest.go: Fix typo "AllowInsureTLS" -> "AllowInsecureTLS"
- storage/errors.go: Removed unused IndexingNotSupportedErr

Signed-off-by: Will Beason <willbeason@google.com>
2021-05-19 07:52:02 +02:00
Patrick East b45ab28375 rego: Plumb the newer QueryTracer through Rego API's
This will deprecate the older API's that used the `topdown.Tracer` in
favor of the newer `topdown.QueryTracer` interface. Usages of the old
API have been swapped, although some of the testing is left with them
to ensure we still support them (until we remove the deprecated API).

Signed-off-by: Patrick East <east.patrick@gmail.com>
2020-07-15 12:01:12 -07:00
Patrick East a233a316da profiler: Update Profiler tracer to be a QueryTracer
Using the newer `topdown.QueryTracer` interface yields a significant
performance improvement for the tracer (up to ~10% for relevant
 benchmarks).

Signed-off-by: Patrick East <east.patrick@gmail.com>
2020-06-08 10:04:34 -04:00
Patrick East 837011c5bc profiler: Disable local var plugging on trace
This adds support for the `topdown.CustomTracer` to the `Profiler` so
that we can signal via config to not plug local vars on trace events.

This improves the performance substantially in cases where there are
many/large local variables.

Fixes: #2245
Signed-off-by: Patrick East <east.patrick@gmail.com>
2020-05-21 15:11:11 -07:00
Ashutosh Narkar 4e5f791673 profiler: Group expressions with missing locations
Previously the profiler would panic if it encountered an expression with no location info. This change groups such expressions by giving them a fake location so that their evaluation results can be captured.

Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
2020-04-08 11:30:08 -07:00
Ashutosh Narkar eb5e5b243f Add profiler to OPA eval command
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
2018-07-20 16:00:05 -07:00
Ashutosh Narkar 45f589b2a7 Sort profiler result based on criteria to return top N results
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
2018-07-03 10:34:04 -07:00
Ashutosh Narkar e85546d1b6 Add Rego profiler
Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
2018-07-03 10:34:04 -07:00