This commit adds a new GraphQL builtin for validating GraphQL schemas,
applying stronger validation rules than what the current GraphQL parsing
builtins apply by default.
Fixes: #5125
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
Got a few warnings from my IDE about redundant type conversions,
so I decided to look into it. Added the unconvert linter to our
checks, and fixed the violations. Added two ignore comments as I
wasn't sure about whether they'd change the semantics of the code.
Signed-off-by: Anders Eknert <anders@eknert.com>
Before, this was OK:
test_a {
. mock_f := true
allow with f as mock_f
}
but this had panicked:
mock_f := true
test_a {
allow with f as mock_f
}
Which, from a user perspective, is quite incomprehensible. Technically,
the first snippet was a (supported) replacement-by-value, and the second
was an unsupported replacement by a rule that was not a function.
Furthermore, the second case wasn't properly caught in the 'with' validations.
Now, we'll capture the situation, and start supporting it. Both snippets will
now work the same, as one would expect from the language surface.
Fixes#5299.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit adds the `net.cidr_is_valid` builtin, which makes
validating network CIDR strings much easier in policies.
Example policy:
allow {
net.cidr_is_valid("192.168.0.0/24")
}
This builtin works for both IPv4 and IPv6 CIDR strings.
Signed-off-by: Ricardo Maraschini <ricardo.maraschini@gmail.com>
When force_cache is true, do not use the Date header set
by the server as the value to use for TTL initialization,
but rather create it from the current instant.
* Rename all current interQuery* tests to intraQuery*
* Add e2e test cases for forced http.send interQueryCaching
Fixes#4960
Signed-off-by: Anders Eknert <anders@eknert.com>
We had been taking care of this case for equal(), but not for glob.match.
Now, we'll get the correct results for this policy, with and without indexing:
package play
p = x {
x := glob.match("/a/*/c", ["/"], input.path)
}
Fixes#5283.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
Previously, the planner didn't account for the variable to become known in the
process of planning term b.
In the case here, foo became known when planning the ref `input.foos[foo]`, the
rhs of the `foo = input.foos[foo]` unification.
Fixes#5271.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit wraps up the mutex-requiring code within the `glob.match`
builtin in helper function, so that the deferred mutex unlock will
always occur before the call at the end of the function to `iter()`.
This helps prevent deadlocks during evaluation.
Fixes#5273.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
The previous code in evalVirtualComplete had the intention of skipping
functions, but failed to cature the `default f(x) := 1` case.
It's somewhat peculiar, since the syntax is accepted, but there is no eval
logic supporting it in topdown. (IR/Wasm support default functions.)
Either way, a function definition alone should not anything to the extent of
a package.
Fixes#5202.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
The compiler ensures that all the keys we see there are scalars. For strings,
nothing changes -- they're handled just like before -- but this now also allows
numbers and booleans.
An example policy that exploded with a panic before is
package p
a[0] = true
when querying the full extent of `data.p` or `data.p.a`.
Fixes#5252.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This change allows rules to have string prefixes in their heads -- we've
come to call them "ref heads".
String prefixes means that where before, you had
package a.b.c
allow = true
you can now have
package a
b.c.allow = true
This allows for more concise policies, and different ways to structure
larger rule corpuses.
Backwards-compatibility:
- There are code paths that accept ast.Module structs that don't necessarily
come from the parser -- so we're backfilling the rule's Head.Reference
field from the Name when it's not present.
This is exposed through (Head).Ref() which always returns a Ref.
This also affects the `opa parse` "pretty" output:
With x.rego as
package x
import future.keywords
a.b.c.d if true
e[x] if true
we get
$ opa parse x rego
module
package
ref
data
"x"
import
ref
future
"keywords"
rule
head
ref
a
"b"
"c"
"d"
true
body
expr index=0
true
rule
head
ref
e
x
true
body
expr index=0
true
Note that
Name: e
Key: x
becomes
Reference: e[x]
in the output above (since that's how we're parsing it, back-compat edge cases aside)
- One special case for backcompat is `p[x] { ... }`:
rule | ref | key | value | name
------------------------+-------+-----+-------+-----
p[x] { ... } | p | x | nil | "p"
p contains x if { ... } | p | x | nil | "p"
p[x] if { ... } | p[x] | nil | true | ""
For interpreting a rule, we now have the following procedure:
1. if it has a Key, it's a multi-value rule; and its Ref defines the set:
Head{Key: x, Ref: p} ~> p is a set
^-- we'd get this from `p contains x if true`
or `p[x] { true }` (back compat)
2. if it has a Value, it's a single-value rule; its Ref may contain vars:
Head{Ref: p.q.r[s], Value: 12} ~> body determines s, `p.q.r.[s]` is 12
^-- we'd get this from `p.q.r[s] = 12 { s := "whatever" }`
Head{Key: x, Ref: p[x], Value: 3} ~> `p[x]` has value 3, `x` is determined
by the rule body
^-- we'd get this from `p[x] = 3 if x := 2`
or `p[x] = 3 { x := 2 }` (back compat)
Here, the Key isn't used, it's present for backwards compatibility: for ref-
less rule heads, `p[x] = 3` used to be a partial object: key x, value 3,
name "p"
- The destinction between complete rules and partial object rules disappears.
They're both single-value rules now.
- We're now outputting the refs of the rules completely in error messages, as
it's hard to make sense of "rule r" when there's rule r in package a.b.c and
rule b.c.r in package a.
Restrictions/next steps:
- Support for ref head rules in the REPL is pretty poor so far. Anything that
works does so rather accidentally. You should be able to work with policies
that contain ref heads, but you cannot interactively define them.
This is because before, we'd looked at REPL input like
p.foo.bar = true
and noticed that it cannot be a rule, so it's got to be a query. This is no
longer the case with ref heads.
- Currently vars in Refs are only allowed in the last position. This is expected
to change in the future.
- Also, for multi-value rules, we can not have a var at all -- so the following
isn't supported yet:
p.q.r[s] contains t if { ... }
-----
Most of the work happens when the RuleTree is derived from the ModuleTree -- in
the RuleTree, it doesn't matter if a rule was `p` in `package a.b.c` or `b.c.p`
in `package a`.
As such, the planner and wasm compiler hasn't seen that many adaptations:
- We're putting rules into the ruletree _including_ the var parts, so
p.q.a = 1
p.q.[x] = 2 { x := "b" }
end up in two different leaves:
p
`-> q
`-> a = 1
`-> [x] = 2`
- When planing a ref, we're checking if a rule tree node's children have
var keys, and plan "one level higher" accordingly:
Both sets of rules, p.q.a and p.q[x] will be planned into one function
(same as before); and accordingly return an object {"a": 1, "b": 2}
- When we don't have vars in the last ref part, we'll end up planning
the rules separately. This will have an effect on the IR.
p.q = 1
p.r = 2
Before, these would have been one function; now, it's two. As a result,
in Wasm, some "object insertion" conflicts can become "var assignment
conflicts", but that's in line with the now-new view of "multi-value"
and "single-value" rules, not partial {set/obj} vs complete.
* planner: only check ref.GroundPrefix() for optimizations
In a previous commit, we've only mapped
p.q.r[7]
as p.q.r; and as such, also need to lookup the ref
p.q.r[__local0__]
via p.q.r
(I think. Full disclosure: there might be edge cases here that are unaccounted
for, but right now, I'm aiming for making the existing tests green...)
New compiler stage:
In the compiler, we're having a new early rewriting step to ensure that the
RuleTree's keys are comparible. They're ast.Value, but some of them cause us
grief:
- ast.Object cannot be compared structurally; so
_, ok := map[ast.Value]bool{ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")}): true}[ast.NewObject([2]*ast.Term{ast.StringTerm("foo"), ast.StringTerm("bar")})]
`ok` will never be true here.
- ast.Ref is a slice type, not hashable, so adding that to the RuleTree would
cause a runtime panic:
p[y.z] { y := input }
is now rewritten to
p[__local0__] { y := input; __local0__ := y.z }
This required moving the InitLocalVarGen stage up the chain, but as it's still
below ResolveRefs, we should be OK.
As a consequence, we've had to adapt `oracle` to cope with that rewriting:
1. The compiler rewrites rule head refs early because the rule tree expects
only simple vars, no refs, in rule head refs. So `p[x.y]` becomes
`p[local] { local = x.y }`
2. The oracle circles in on the node it's finding the definition for based
on source location, and the logic for doing that depends on unaltered
modules.
So here, (2.) is relaxed: the logic for building the lookup node stack can
now cope with generated statements that have been appended to the rule bodies.
There is a peculiarity about ref rules and extents:
See the added tests: having a ref rule implies that we get an empty object
in the full extent:
package p
foo.bar if false
makes the extent of data.p: {"foo": {}}
This is somewhat odd, but also follows from the behaviour we have right now
with empty modules:
package p.foo
bar if false
this also gives data.p the extent {"foo": {}}.
This could be worked around by recording, in the rule tree, when a node was
added because it's an intermediary with no values, but only children.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
The incoming relative path is now converted as follows:
1. filepath.Abs: foo/bar -> c:\a\b\c\foo\bar
2. filepath.ToShasl: c:\a\b\c\foo\bar -> c:/a/b/c/foo/bar
3. prepend `file://`
That's something the runtime can make sense of, and the smoke test passes.
Fixes#5134.
Includes:
* ci: add smoke test with a bundle
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
It looks like we didn't have any before. While it's inherently random, we still
know that for the same inputs, within a single evaluation run, we should only
get one output. So let's assert that.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This commit adds support for AST objects to be usable in place of
strings for several of the GraphQL built-in functions, to improve
the composability of the GraphQL set of built-ins, and to dramatically
reduce the amount of redundant parsing when writing GraphQL policies.
Fixes: #4742
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
JWT specification allows exp and nbf to be missing, but if they are present, they must be of type "number." Before this change, JWTs that violated this would result in OPA panic. Added assertion on expected type.
Fixes#5165
Signed-off-by: Charlie Flowers <cflowers@gmail.com>
We're now also properly rewriting cases like
rule {
x := 1
allow with input as [x]
}
Follow-up to #5148.
Signed-off-by: liu-du <duliujimmy@hotmail.com>
Before, a test like this:
test_allow {
base_test_case := {"x": 1}
allow with input as object.union(base_test_case, {"x": 2})
}
would fail to compile. Now, it works as expected.
Fixes#5148.
Signed-off-by: liu-du <duliujimmy@hotmail.com>
This commit adds the `prealloc` linter to the list of linters for OPA, and fixes up the miscellaneous locations in the code that the linter found where we could easily preallocate slices.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
The `object.union_n` builtin could panic from being fed an array with a wrong-typed element because the error value from a type assertion was being ignored. This could only occur under special circumstances, such as when the malformed array came in through a ref, such as `input`.
This commit fixes the `object.union_n` builtin by no longer ignoring the type assertion's error value. This allows the builtin to fail gracefully, instead of carrying on blithely and hitting a panic.
Fixes#5073
Signed-off-by: Charles Daniels <charles@styra.com>
This commit addresses issues around vendoring the 3rd party GraphQL
parser library, `vektah/gqlparser`, which is used by our GraphQL builtins.
By directly depending on the library, we accidentally forced all of our
library users to have to match `gqlparser` versions, which could cause
problems if they wanted to use downstream GraphQL libraries.
To fix this version clash problem, this PR internalizes `vektah/gqlparser`
v2.4.8 into the `internal/gqlparser` package (we can update to v2.5.0
later). Scripts are included to automate some of the internalizing
process if we wish to update the library in the future.
Fixes: #5065
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
Adding `error: ` to each message is redundant, and breaks tooling
that expects a uniform format of `<builtin-name>: message` in error
messages from OPA (i.e. Jarl 😄).
Signed-off-by: Anders Eknert <anders@eknert.com>
This commit eliminates the potential for a slow disk write to cause a test failure in the certificate rotation tests by adding a file `Sync()` call in the function used to copy certificate files around.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This commit cleans up the output of `units.parse` for integer values. Previously, all values from `units.parse` had their precision set to 10 places past the decimal point, regardless of whether that much precision was actually needed. This commit removes the extra decimal places for integers, giving cleaner outputs.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
* testcases: Fix missing `modules` in units tests.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
* test/cases/units: Fix YAML formatting in units regression test.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
Hacking on IR implementations for the builtins revealed
some functions missing tests, which are added here:
* rego.metadata.rule
* rego.metadata.chain
* crypto.x509.parse_rsa_private_key
Also brought back the comment providing instructions for
running the tests in the helloworld directory, as this was
mistakenly removed by the massive cleanup PR some month back.
Signed-off-by: Anders Eknert <anders@eknert.com>
This commit ensures that the `graph.reachable` and `graph.reachable_paths` builtins check the types of both of their operands, and return type errors instead of default values if a wrong type is provided.
Fixes#4951
Using null for delimiters disables delimiters in glob matching. Preferable over regex on some cases for performance reasons.
Fixes#4923.
Signed-off-by: vinhph0906 <vinhph0906@gmail.com>
Co-authored-by: Stephan Renatus <stephan.renatus@gmail.com>
Add support for checking if `super` array contains
every element of `sub` set to object.subset.
object.subset allows `super` to be array and `sub` to be set.
Fixes: #4858
Signed-off-by: x-color <36035885+x-color@users.noreply.github.com>
Previously, we used big.Float for units parsing, but this resulted in
occasional rounding issues, due to values like 0.001 not being perfectly
representable in floating-point format.
This issue was fixed by switching units parsing to use big.Rat, since
Rationals can generally handle such quantities with perfect precision.
Fixes#4856.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
This commit updates the `*.is_valid` functions to no longer produce
errors when providing wrong-typed arguments. Instead, they will now
return true/false for all inputs. Tests and WASM versions of these
builtins have been updated to match the new behavior.
Fixes#4760.
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
`contains` provides an alternative way to declare partial sets:
p contains x {
x := { "foo": "bar"
}
which is the same as
p[x] {
x := { "foo": "bar"
}
The keyword is enabled by importing `future.keywords.contains`, and
when it _is enabled_, the format will be used for all partial sets in
that file for pretty-printing.
`if` is a new keyword allowing for more readable rule definitions:
The syntax is
NAME [if] { EXPR [EXPR...] }
and the is a shorthand allows dropping the braces around the expression
if there is only one:
NAME if EXPR
For example, this allows expressions like
allow if not deny
f(xs) if every x in xs { x != "foo" }
The one exception here are partial sets: they cannot use `if` UNLESS
they use `contains`:
p[x] { x := "foo" } # valid
p contains x { x := "bar" } # valid
p contains x if { x := "bar" } # valid
p[x] if { x := "foo" } # invalid
This is because we want to interpret that differently (as an object
rule defining `p.foo = true`) in the near future.
The formatter works in the same way: if `future.keywords.if` is imported, it
will be used where it can be used.
We don't want to be too eager when it comes to introducing syntactic sugar.
So this will be rewritten, because head and body expression are on the same
line:
p := 5 if { time.day_of_week() == "Monday" }
# => p := 5 if time.day_of_week() == "Monday"
but this won't:
p := 5 if {
time.day_of_week() == "Monday"
}
The rationale here is that if the policy author decided that they want this on
an extra line, we won't mess with it.
This also sidesteps the need to check if both the head and the single body
expression have a comment.
This change includes various docs updates. Notable exceptions are the GK docs,
since it will take a while for these keywords to be come available there; and
the frontpage: merging a PR would update the frontpage immediately, and we
don't want to show something there that isn't available in the latest release.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This reverts commit 2fbc99c324.
See #4771 for details.
We'll get this back in, but we're backing out of it for now.
So, the previous (annoying, buggy) behavior is going to come back, but a
proper, future fix of the sprintf behaviour will incorporate this work.
Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>
This implements the new object.subset() builtin.
Based on my benchmarking, this offers a 2.77x speedup compared to
implementing the same thing in pure Rego, and is also easier to read.
Fixes#4358
Signed-off-by: Charles Daniels <charles@styra.com>
Stephan Renatus