Mostly automated fixes from running:
```
go run golang.org/x/tools/go/analysis/passes/modernize/cmd/modernize@latest --fix ./...
```
But carefully reviewed, and several fixes reverted as they looked like
they potentially could be less performant, and in a few cases due to
bugs in the analyzer that changed semantics of the code. Will report
these upstream.
Mostly good fixes though!
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
The `json.patch` built-in is quite versatile, and compared to
patching via e.g. `object.union` et. al. often communicates
intent better, IMO. But while it uses some fairly advanced
logic for complex patch operations, it doesn't perform all that
great on simple ones. This is a first and pretty basic attempt
to improve that somewhat by picking the most low-hangig performance
fruits, like avoiding repeated allocations of temporary term pointers.
The main allocation source is the creation of EditTree's, and this
remains a problem. I have created a sync pool but only managed to
get the outermost edit tree to recycle, as I found it really hard
to track where it's safe to release those created in the deeply
nested calls. Additionally, I managed to trigger stack overflows
trying to recycle child trees, so there seems to be some circular
refs? Or I just did something wrong.
If someone wants to look into this and pick up where
I left, that'd be great!
- Add InternedIntRange for testing, primarily
- Intern keys used in json.patch patches
- Clean up json.X built-in benchmarks
- Reduce allocations in edit tree function
- Avoid using intermediate data structures
for JSON patches
- Some unrelated interning fixes to reduce noise
in tests and benchmarks (e.g. do less stuff in
var inits)
Selected benchmark that I used while working on this:
**Before**
```
BenchmarkJSONPatchAddShallowScalar/object-10-16 147853 8008 ns/op 9667 B/op 206 allocs/op
BenchmarkJSONPatchAddShallowScalar/array-10-16 201704 5889 ns/op 7256 B/op 173 allocs/op
BenchmarkJSONPatchAddShallowScalar/set-10-16 182566 6733 ns/op 8103 B/op 156 allocs/op
```
**After**
```
BenchmarkJSONPatchAddShallowScalar/object-10-16 197414 6066 ns/op 7256 B/op 133 allocs/op
BenchmarkJSONPatchAddShallowScalar/array-10-16 278121 4427 ns/op 5285 B/op 100 allocs/op
BenchmarkJSONPatchAddShallowScalar/set-10-16 233884 4839 ns/op 6243 B/op 113 allocs/op
```
Signed-off-by: Anders Eknert <anders.eknert@apple.com>
* Ensure HTTP transport is setup when SNI is used
An operation specific http.Transport is not always setup in http.send.
In these cases, the ServerName populated in the TLS configuration is not
used by the client. Existing tests did not exercise this path because
they would always configure additional TLS options that resulted in an
operation specific transport.
This change moves the Host header and tls_server_name processing closer
to the rest of the TLS logic and sets isTLS to true to ensure a custom
transport is configured.
Signed-off-by: Matthew Sykes <matthew.sykes@fmr.com>
* Simplify http.Transport construction for http.send
When unix domain sockets are used, the URL scheme is forced to "http".
This change effectively disables TLS and removes the need to wire a
tls.Config instance.
Signed-off-by: Matthew Sykes <matthew.sykes@fmr.com>
* Avoid unnecessary use of customized transport
Unless specific options are used with http.send that necessitate a
custom http.Transport configuration, the default transport should be
used to perform the operation.
Under load, policies that use http.send were encountering errors and
high throughput variability. While troubleshooting, we noted a high
number of sockets in the TIME_WAIT state. This indicated that client
connections were not getting reused during policy evaluation. We
attempted to address this by increasing MaxIdleConns and
MaxIdleConnsPerHost on the default transport. Unfortunately, this did
not help.
When we looked through the code, we discovered that nearly every path
through http.send creates a new transport. When transports aren't
reused, connection pooling is effectively disabled. These new transports
also disable keep-alive. Either one of these effectively forces a new
client connections and associated TLS handshake for each request.
We found one existing path through the code that constructs an
http.Client without a transport: explicitly setting tls_use_system_certs
to false. When we made this change to our policy, the default transport
was used, we no longer encountered errors, and the throughput was higher
and more stable.
This change removes the block that explicitly set tlsUseSystemCerts to
true when tls_use_system_certs was unset and no other CA cert options
were used. This results in the use of the default transport when TLS
options are not set and, by default, the default transport will use the
system certificate pool.
Signed-off-by: Matthew Sykes <matthew.sykes@fmr.com>
---------
Signed-off-by: Matthew Sykes <matthew.sykes@fmr.com>
Have done this some time in the past, but there was a few
new issues this would highlight now that we're on Go 1.24.
Mostly:
- Use `b.Loop()` in benchmarks
- Use `strings.SplitSeq` where possible
- Remove `omitempty` tag for types that can't be empty
Signed-off-by: Anders Eknert <anders@eknert.com>
This PR adds interning of strings representing common integer values,
which greatly speeds up "to string" operations on numbers, and updates
some built-ins commonly used for this to make use of interned values
where possible.
This is "light" version of a previous PR that did this more aggressively,
but also came with more caveats. Importantly, interning of new strings is
now never done at "runtime", but only allowed at init time. The API for
interning is marked experimental and should not relied upon by anyone
who expects a stable API.
Signed-off-by: Anders Eknert <anders@styra.com>
Following up on #7566, and now applying the more exciting
modernizations. fmt.Appendf was new to me! But especially
the contains checks are so much better IMHO. I have reviewed
all changes myself and did a few manual changes where it
became obvious that things could be improved a little further.
(the modernize analyzer still has some issues running against
OPA, and I have manually worked around those for the time being)
Signed-off-by: Anders Eknert <anders@styra.com>
Brace yourselves! For there are many touched files here. No changes
in semantics however.
Spent a long time trying out the various optional rules gocritic
provides, and settled for a few of them. There are more I really
like, but that would take many hours to address across the codebase.
Perhaps others find gocritic too pedantic? If so, we can merge the
fixes without enabling the rule.
Signed-off-by: Anders Eknert <anders@styra.com>
This is a simpler version of util.TypedHashMap where the keys
implement a `.Hash()` method and as such won't need one to be passed
in, and where the values are largely ignored by the map. These maps
are smaller / more performant, but most importantly, they are nicer
to work with.
Perf wise, this saves about 600k+ allocs and 40 MB allocated memory
in `regal lint bundle`:
```
1207614875 ns/op 3293454016 B/op 64802095 allocs/op
1197978125 ns/op 3256960504 B/op 64164871 allocs/op
```
Also:
- Use `strings.Builder` instead of `fmt.Sprintf` in one location
- Remove `ValueMap.Copy` as it was only used in a test
Signed-off-by: Anders Eknert <anders@styra.com>
And update code to conform to the rule.
- Replace unnecessary fmt.Sprintf with string concatenation
- Replace fmt.Sprint with more efficient strconv.Itoa
- Replace static fmt.Errorf calls with more efficient errors.New
Thanks @srenatus for pushing me down this rabbit hole!
Signed-off-by: Anders Eknert <anders@styra.com>
Having worked on performance improvements in OPA on the side for almost
a month now, there's a lot of code piling up 😅 So much that a single PR
would be way too much to review. Instead, I'm splitting the work into
chunks, and will submit the next PR as soon as this one is merged. Using
the same benchmark as before — Regal linting itself, these new changes
in total reduce the number of allocations by ~13 million, and quite a
substantial amount of evaluation time saved as well.
This first PR is isolated to improvements to terms, values and
built-ins, and saves ~3M allocations. The details can be found below for
each change, and of course in the code :)
**BenchmarkRegalLintingItself-10 Before**
```
1885978209 ns/op 3497157312 B/op 69064779 allocs/op
```
**BenchmarkRegalLintingItself-10 After**
```
1796255084 ns/op 3452379408 B/op 66126623 allocs/op
```
**Terms**
- Use pointer receivers consistently for object and set types. This allows
changing the sortGuard once lock from a pointer to a non-pointer type, which
is really the biggest win performance-wise in this PR.
- Comparisons happen all the time, so make sure these take the shortest path
possible whenever, possible, such as when one type is compared to another
value of the same type.
Built-in functions:
**Arrays**
- Both `array.concat` and `array.slice` will now return the operand on operations
where the result isn't different from the input operand (like when concatenating
an empty array) instead of allocating a new term/value.
**Strings**
- Return operand on unchanged result rather than allocating new term/value.
- Where applicable, have functions take a cheaper path when string is ASCII
and we can avoid the cost of rune conversion.
**Crypto**
- Hashing functions now optimized, spending less than half the time compared to
previously.
**Objects**
- Avoid heap allocating result boolean escaping its scope, and instead use the
return value of the `Until` function.
**HTTP**
- Use interned terms for keys in configuration object, avoding allocating these
each time `http.send` is invoked.
**Globs**
- Use read/write lock to avoid contention. Use package level vars for "constant"
values, avoiding them to escape to the heap each invocation.
**Not directly/only related to built-in functions**
- Add `ValueName` function replacing the previous `TypeName` functions for
getting the name of Value's without paying for `any` interface allocations.
- Add a few more interned terms.
Signed-off-by: Anders Eknert <anders@styra.com>
Anders Eknert