docs: Add notes about use of GOMEMLIMIT (#8771)

Documents issue in memory constrained environments following the change
in typical heap size following the change from 1 to 2 max procs in
https://github.com/open-policy-agent/opa/pull/8696

---------

Signed-off-by: Charlie Egan <charlie_egan@apple.com>

docs/docs/deploy/k8s/index.mdx

@@ -221,6 +221,11 @@ Deployment manifest. This deployment has the following features:
 - Runs with 3 replicas. This number can be adjusted to suit your needs.
 - Runs in a guaranteed service class with 1 CPU and 1Gi of memory. This again
   should be tuned to your needs.
+- Sets `GOMEMLIMIT` to a value below `resources.limits.memory` so the Go GC has
+  headroom for non-heap runtime memory (stacks, runtime metadata, etc.). The exact
+  value is a tradeoff: closer to the limit risks Out Of Memory (OOM), further below increases GC
+  CPU overhead and request latency.
+  See [Resource Utilization](../policy-performance/#resource-utilization) for more information.
 - Loads the OPA configuration from the secret created above.
 - Has health and liveness checks using the OPA `/health` endpoints.
 - Uses pod anti-affinity to prefer that OPA pods are not scheduled on the same
@@ -261,6 +266,11 @@ spec:
           - "--config-file=/run/secrets/opa-config.yaml"
           ports:
             - containerPort: 8181
+          env:
+            # Soft cap a little below the hard memory limit imposed by the
+            # container runtime, so the Go GC has headroom.
+            - name: GOMEMLIMIT
+              value: "900MiB"
           resources:
             requests:
               memory: "1Gi"

docs/docs/kubernetes/tutorial.md

@@ -247,6 +247,15 @@ docker run --rm --name bundle-server -d -p 8888:80 -v ${PWD}:/usr/share/nginx/ht
 
 Next, use the file below to deploy OPA as an admission controller.
 
+:::tip
+If you set a memory _limit_ in a containerized deployment environment, you may
+also want to set `GOMEMLIMIT` to a value below that limit so the Go garbage
+collector treats it as a soft cap. This is a tradeoff: a tighter `GOMEMLIMIT`
+reduces Out Of Memory (OOM) risk but increases GC CPU overhead and request latency. Benchmark
+with your policies, data, and load to find the right value. See the
+[Kubernetes deployment guide](../deploy/k8s) for an example.
+:::
+
 <EvergreenCodeBlock>
 ```
 # Grant OPA/kube-mgmt read-only access to resources. This lets kube-mgmt

docs/docs/policy-performance.md

@@ -897,8 +897,11 @@ likely becomes I/O-bound.) Policy evaluation is currently single-threaded. If yo
 are embedding OPA as a library, it is your responsibility to dispatch concurrent queries
 to different Goroutines/threads. If you are running the OPA server, it will parallelize
 concurrent requests and use as many cores as possible. You can limit the number of
-cores that OPA can consume by starting OPA with the [`GOMAXPROCS`](https://pkg.go.dev/runtime)
-environment variable.
+cores that OPA can consume by setting the [`GOMAXPROCS`](https://pkg.go.dev/runtime)
+environment variable. Since Go 1.25, OPA auto-detects the cgroup CPU quota and sets
+`GOMAXPROCS` accordingly, with a minimum of 2. For containers with very low CPU
+limits, if setting `GOMAXPROCS=1`, benchmark OPA's memory and GC performance to
+ensure your latency requirements are met.
 
 Memory usage scales with the size of the policy (i.e., Rego) and data (e.g., JSON) that you
 load into OPA. Raw JSON data loaded into OPA uses approximately 20x more memory compared to the
@@ -915,6 +918,19 @@ consumes approximately 1.1GB of RAM.
 
 By default, OPA stores policy and data in-memory. OPA's disk storage feature allows policy and data to be stored on disk. See [this](./storage/#disk) for more details.
 
+Memory usage depends on your specific policies, data, and request load. Benchmark
+OPA with [`opa bench`](#benchmarking-queries) to determine the memory needed for your
+deployment. When running OPA in a container with a hard memory limit, set the
+[`GOMEMLIMIT`](https://pkg.go.dev/runtime) environment variable to a value slightly
+below that limit. `GOMEMLIMIT` is a soft cap the Go garbage collector treats as a
+target. Without it set, the heap can grow to roughly twice the live heap before GC
+runs (the `GOGC=100` default). When OPA holds a large amount of data, that permanent
+heap can be a significant fraction of the memory limit, and the doubling can cause an
+Out Of Memory (OOM) kill before GC catches up. Setting `GOMEMLIMIT` too close to the live heap has the
+opposite cost. The GC runs more often, increasing CPU usage and request latency. The
+right value is a tradeoff for your workload. See the
+[Kubernetes deployment guide](./deploy/k8s) for an example.
+
 ## Optimization Levels
 
 The `--optimize` (or `-O`) flag on the `opa build` command controls how bundles are optimized.