mirror of
https://github.com/open-policy-agent/opa.git
synced 2026-08-25 01:35:18 -06:00
topdown: Add raw_body parameter to http.send
Instead of changing the behaviour of the body parameter (which is always sent as JSON) just add a new parameter that gives the caller complete control over the message body. Also, refactor the docs for the function to be a bit more readable. Fixes #1903 Signed-off-by: Torin Sandall <torinsandall@gmail.com>
This commit is contained in:
Torin Sandall
@@ -424,36 +424,51 @@ Note that the opa executable will need access to the timezone files in the envir
|
||||
|
||||
| Built-in | Description |
|
||||
| ------- |-------------|
|
||||
| <span class="opa-keep-it-together">``http.send(request, output)``</span> | ``http.send`` executes a HTTP request and returns the response.``request`` is an object containing keys ``method``, ``url`` and optionally ``body``, ``enable_redirect``, ``force_json_decode``, ``headers``, ``tls_use_system_certs``, ``tls_ca_cert_file``, ``tls_ca_cert_env_variable``, ``tls_client_cert_env_variable``, ``tls_client_key_env_variable`` or ``tls_client_cert_file``, ``tls_client_key_file`` . For example, ``http.send({"method": "get", "url": "http://www.openpolicyagent.org/", "headers": {"X-Foo":"bar", "X-Opa": "rules"}}, output)``. ``output`` is an object containing keys ``status``, ``status_code``, ``body`` and ``raw_body`` which represent the HTTP status, status code, JSON value from the response body and response body as string respectively. Sample output, ``{"status": "200 OK", "status_code": 200, "body": {"hello": "world"}, "raw_body": "{\"hello\": \"world\"}"``}. By default, HTTP redirects are not enabled. To enable, set ``enable_redirect`` to ``true``. Also ``force_json_decode`` is set to ``false`` by default. This means if the HTTP server response does not specify the ``Content-type`` as ``application/json``, the response body will not be JSON decoded ie. output's ``body`` field will be ``null``. To change this behaviour, set ``force_json_decode`` to ``true``.|
|
||||
| <span class="opa-keep-it-together">``response := http.send(request)``</span> | ``http.send`` executes an HTTP `request` and returns a `response`. |
|
||||
|
||||
#### HTTPs Usage
|
||||
The `request` object parameter may contain the following fields:
|
||||
|
||||
The following table explains the HTTPs objects
|
||||
| Field | Required | Type | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `url` | yes | `string` | HTTP URL to specify in the request (e.g., `"https://www.openpolicyagent.org"`). |
|
||||
| `method` | yes | `string` | HTTP method to specify in request (e.g., `"GET"`, `"POST"`, `"PUT"`, etc.) |
|
||||
| `body` | no | `any` | HTTP message body to include in request. The value will be serialized to JSON. |
|
||||
| `raw_body` | no | `string` | HTTP message body to include in request. The value WILL NOT be serialized. Use this for non-JSON messages. |
|
||||
| `headers` | no | `object` | HTTP headers to include in the request (e.g,. `{"X-Opa": "rules"}`). |
|
||||
| `enable_redirect` | no | `boolean` | Follow HTTP redirects. Default: `false`. |
|
||||
| `force_json_decode` | no | `boolean` | Decode the HTTP response message body as JSON even if the `Content-Type` header is missing. Default: `false`. |
|
||||
| `tls_use_system_certs` | no | `boolean` | Use system certificate pool. |
|
||||
| `tls_ca_cert_file` | no | `string` | Path to file containing a root certificate in PEM encoded format. |
|
||||
| `tls_ca_cert_env_variable` | no | `string` | Environment variable containing a root certificate in PEM encoded format. |
|
||||
| `tls_client_cert_env_variable` | no | `string` | Environment variable containing a client certificate in PEM encoded format. |
|
||||
| `tls_client_key_env_variable` | no | `string` | Environment variable containing a client key in PEM encoded format. |
|
||||
| `tls_client_cert_file` | no | `string` | Path to file containing a client certificate in PEM encoded format. |
|
||||
| `tls_client_key_file` | no | `string` | Path to file containing a key in PEM encoded format. |
|
||||
|
||||
| Object | Definition | Value|
|
||||
| -------- |-----------|------|
|
||||
| tls_use_system_certs | Use system certificate pool | true or false
|
||||
| tls_ca_cert_file | Path to file containing a root certificate in PEM encoded format | double-quoted string
|
||||
| tls_ca_cert_env_variable | Environment variable containing a root certificate in PEM encoded format | double-quoted string
|
||||
| tls_client_cert_env_variable | Environment variable containing a client certificate in PEM encoded format | double-quoted string
|
||||
| tls_client_key_env_variable | Environment variable containing a client key in PEM encoded format | double-quoted string
|
||||
| tls_client_cert_file | Path to file containing a client certificate in PEM encoded format | double-quoted string
|
||||
| tls_client_key_file | Path to file containing a key in PEM encoded format | double-quoted string
|
||||
|
||||
In order to trigger the use of HTTPs the user must provide one of the following combinations:
|
||||
To trigger the use of HTTPs the user must provide one of the following combinations:
|
||||
|
||||
* ``tls_client_cert_file``, ``tls_client_key_file``
|
||||
* ``tls_client_cert_env_variable``, ``tls_client_key_env_variable``
|
||||
|
||||
The user must also provide a trusted root CA through tls_ca_cert_file or tls_ca_cert_env_variable. Alternatively the user could set tls_use_system_certs to ``true`` and the system certificate pool will be used.
|
||||
|
||||
#### HTTPs Examples
|
||||
The `response` object parameter will contain the following fields:
|
||||
|
||||
| Examples | Comments |
|
||||
| Field | Type | Description |
|
||||
| --- | --- | --- |
|
||||
| `status` | `string` | HTTP status message (e.g., `"200 OK"`). |
|
||||
| `status_code` | `number` | HTTP status code (e.g., `200`). |
|
||||
| `body` | `any` | Any JSON value. If the HTTP response message body was not deserialized from JSON, this field is set to `null`. |
|
||||
| `raw_body` | `string` | The entire raw HTTP resposne message body represented as a string. |
|
||||
|
||||
The table below shows examples of calling `http.send`:
|
||||
|
||||
| Example | Comments |
|
||||
| -------- |-----------|
|
||||
| Files containing TLS material | ``http.send({"method": "get", "url": "https://127.0.0.1:65331", "tls_ca_cert_file": "testdata/ca.pem", "tls_client_cert_file": "testdata/client-cert.pem", "tls_client_key_file": "testdata/client-key.pem"}, output)``.
|
||||
|Environment variables containing TLS material | ``http.send({"method": "get", "url": "https://127.0.0.1:65360", "tls_ca_cert_env_variable": "CLIENT_CA_ENV", "tls_client_cert_env_variable": "CLIENT_CERT_ENV", "tls_client_key_env_variable": "CLIENT_KEY_ENV"}, output)``.|
|
||||
| Accessing Google using System Cert Pool | ``http.send({"method": "get", "url": "https://www.google.com", "tls_use_system_certs": true, "tls_client_cert_file": "testdata/client-cert.pem", "tls_client_key_file": "testdata/client-key.pem"}, output)``
|
||||
| Files containing TLS material | ``http.send({"method": "get", "url": "https://127.0.0.1:65331", "tls_ca_cert_file": "testdata/ca.pem", "tls_client_cert_file": "testdata/client-cert.pem", "tls_client_key_file": "testdata/client-key.pem"})`` |
|
||||
|Environment variables containing TLS material | ``http.send({"method": "get", "url": "https://127.0.0.1:65360", "tls_ca_cert_env_variable": "CLIENT_CA_ENV", "tls_client_cert_env_variable": "CLIENT_CERT_ENV", "tls_client_key_env_variable": "CLIENT_KEY_ENV"})`` |
|
||||
| Accessing Google using System Cert Pool | ``http.send({"method": "get", "url": "https://www.google.com", "tls_use_system_certs": true, "tls_client_cert_file": "testdata/client-cert.pem", "tls_client_key_file": "testdata/client-key.pem"})`` |
|
||||
|
||||
### Net
|
||||
| Built-in | Description |
|
||||
|
||||
+12
-2
@@ -33,6 +33,7 @@ var allowedKeyNames = [...]string{
|
||||
"enable_redirect",
|
||||
"force_json_decode",
|
||||
"headers",
|
||||
"raw_body",
|
||||
"tls_use_system_certs",
|
||||
"tls_ca_cert_file",
|
||||
"tls_ca_cert_env_variable",
|
||||
@@ -131,6 +132,7 @@ func executeHTTPRequest(bctx BuiltinContext, obj ast.Object) (ast.Value, error)
|
||||
var tlsClientCertFile string
|
||||
var tlsClientKeyFile string
|
||||
var body *bytes.Buffer
|
||||
var rawBody *bytes.Buffer
|
||||
var enableRedirect bool
|
||||
var forceJSONDecode bool
|
||||
var tlsUseSystemCerts bool
|
||||
@@ -173,6 +175,12 @@ func executeHTTPRequest(bctx BuiltinContext, obj ast.Object) (ast.Value, error)
|
||||
return nil, err
|
||||
}
|
||||
body = bytes.NewBuffer(bodyValBytes)
|
||||
case "raw_body":
|
||||
s, ok := obj.Get(val).Value.(ast.String)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("raw_body must be a string")
|
||||
}
|
||||
rawBody = bytes.NewBuffer([]byte(s))
|
||||
case "tls_use_system_certs":
|
||||
tlsUseSystemCerts, err = strconv.ParseBool(obj.Get(val).String())
|
||||
if err != nil {
|
||||
@@ -211,7 +219,7 @@ func executeHTTPRequest(bctx BuiltinContext, obj ast.Object) (ast.Value, error)
|
||||
return nil, fmt.Errorf("invalid type for headers key")
|
||||
}
|
||||
default:
|
||||
return nil, fmt.Errorf("Invalid Key %v", key)
|
||||
return nil, fmt.Errorf("invalid parameter %q", key)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -249,7 +257,9 @@ func executeHTTPRequest(bctx BuiltinContext, obj ast.Object) (ast.Value, error)
|
||||
client.CheckRedirect = nil
|
||||
}
|
||||
|
||||
if body == nil {
|
||||
if rawBody != nil {
|
||||
body = rawBody
|
||||
} else if body == nil {
|
||||
body = bytes.NewBufferString("")
|
||||
}
|
||||
|
||||
|
||||
+73
-35
@@ -9,6 +9,7 @@ import (
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/ioutil"
|
||||
"net/http"
|
||||
@@ -185,59 +186,96 @@ func TestHTTPCustomHeaders(t *testing.T) {
|
||||
// TestHTTPostRequest adds a new person
|
||||
func TestHTTPostRequest(t *testing.T) {
|
||||
|
||||
// test server
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
var person Person
|
||||
if r.Body == nil {
|
||||
http.Error(w, "Please send a request body", 400)
|
||||
return
|
||||
}
|
||||
err := json.NewDecoder(r.Body).Decode(&person)
|
||||
contentType := r.Header.Get("Content-Type")
|
||||
|
||||
bs, err := ioutil.ReadAll(r.Body)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), 400)
|
||||
return
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.Header().Set("Content-Type", contentType)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
json.NewEncoder(w).Encode(person)
|
||||
w.Write(bs)
|
||||
}))
|
||||
|
||||
defer ts.Close()
|
||||
|
||||
// expected result
|
||||
expectedResult := map[string]interface{}{
|
||||
"status": "200 OK",
|
||||
"status_code": http.StatusOK,
|
||||
"body": map[string]string{"id": "2", "firstname": "Joe"},
|
||||
"raw_body": "{\"id\":\"2\",\"firstname\":\"Joe\"}\n",
|
||||
}
|
||||
|
||||
resultObj, err := ast.InterfaceToValue(expectedResult)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// create a new person object
|
||||
person2 := Person{ID: "2", Firstname: "Joe"}
|
||||
b := new(bytes.Buffer)
|
||||
json.NewEncoder(b).Encode(person2)
|
||||
|
||||
// run the test
|
||||
tests := []struct {
|
||||
note string
|
||||
rules []string
|
||||
params string
|
||||
expected interface{}
|
||||
}{
|
||||
{"http.send", []string{fmt.Sprintf(
|
||||
`p = x { http.send({"method": "post", "url": "%s", "body": %s}, x) }`, ts.URL, b)}, resultObj.String()},
|
||||
|
||||
{
|
||||
note: "basic",
|
||||
params: `{
|
||||
"method": "post",
|
||||
"headers": {"Content-Type": "application/json"},
|
||||
"body": {"id": "2", "firstname": "Joe"}
|
||||
}`,
|
||||
expected: `{
|
||||
"status": "200 OK",
|
||||
"status_code": 200,
|
||||
"body": {"id": "2", "firstname": "Joe"},
|
||||
"raw_body": "{\"firstname\":\"Joe\",\"id\":\"2\"}"
|
||||
}`,
|
||||
},
|
||||
{
|
||||
note: "raw_body",
|
||||
params: `{
|
||||
"method": "post",
|
||||
"headers": {"Content-Type": "application/x-www-form-encoded"},
|
||||
"raw_body": "username=foobar&password=baz"
|
||||
}`,
|
||||
expected: `{
|
||||
"status": "200 OK",
|
||||
"status_code": 200,
|
||||
"body": null,
|
||||
"raw_body": "username=foobar&password=baz"
|
||||
}`,
|
||||
},
|
||||
{
|
||||
note: "raw_body overrides body",
|
||||
params: `{
|
||||
"method": "post",
|
||||
"headers": {"Content-Type": "application/x-www-form-encoded"},
|
||||
"body": {"foo": 1},
|
||||
"raw_body": "username=foobar&password=baz"
|
||||
}`,
|
||||
expected: `{
|
||||
"status": "200 OK",
|
||||
"status_code": 200,
|
||||
"body": null,
|
||||
"raw_body": "username=foobar&password=baz"
|
||||
}`,
|
||||
},
|
||||
{
|
||||
note: "raw_body bad type",
|
||||
params: `{
|
||||
"method": "post",
|
||||
"headers": {"Content-Type": "application/x-www-form-encoded"},
|
||||
"raw_body": {"bar": "bar"}
|
||||
}`,
|
||||
expected: errors.New("raw_body must be a string"),
|
||||
},
|
||||
}
|
||||
|
||||
data := loadSmallTestData()
|
||||
data := map[string]interface{}{}
|
||||
|
||||
for _, tc := range tests {
|
||||
runTopDownTestCase(t, data, tc.note, tc.rules, tc.expected)
|
||||
|
||||
// Automatically set the URL because it's generated when the test server
|
||||
// is started. If needed, the test cases can override in the future.
|
||||
term := ast.MustParseTerm(tc.params)
|
||||
term.Value.(ast.Object).Insert(ast.StringTerm("url"), ast.StringTerm(ts.URL))
|
||||
|
||||
rules := []string{
|
||||
fmt.Sprintf(`p = x { http.send(%s, x) }`, term),
|
||||
}
|
||||
|
||||
runTopDownTestCase(t, data, tc.note, rules, tc.expected)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -2979,7 +2979,7 @@ func assertTopDownWithPath(t *testing.T, compiler *ast.Compiler, store storage.S
|
||||
}
|
||||
|
||||
if util.Compare(expected, result) != 0 {
|
||||
t.Fatalf("Unexpected result:\nGot: %v\nExp:\n%v", result, expected)
|
||||
t.Fatalf("Unexpected result:\nGot: %+v\nExp:\n%+v", result, expected)
|
||||
}
|
||||
|
||||
// If the test case involved the input document, re-run it with partial
|
||||
|
||||
Reference in New Issue
Block a user