diff --git a/docs/content/policy-reference.md b/docs/content/policy-reference.md index 088c6b8e7b..c4b40e1798 100644 --- a/docs/content/policy-reference.md +++ b/docs/content/policy-reference.md @@ -424,36 +424,51 @@ Note that the opa executable will need access to the timezone files in the envir | Built-in | Description | | ------- |-------------| -| ``http.send(request, output)`` | ``http.send`` executes a HTTP request and returns the response.``request`` is an object containing keys ``method``, ``url`` and optionally ``body``, ``enable_redirect``, ``force_json_decode``, ``headers``, ``tls_use_system_certs``, ``tls_ca_cert_file``, ``tls_ca_cert_env_variable``, ``tls_client_cert_env_variable``, ``tls_client_key_env_variable`` or ``tls_client_cert_file``, ``tls_client_key_file`` . For example, ``http.send({"method": "get", "url": "http://www.openpolicyagent.org/", "headers": {"X-Foo":"bar", "X-Opa": "rules"}}, output)``. ``output`` is an object containing keys ``status``, ``status_code``, ``body`` and ``raw_body`` which represent the HTTP status, status code, JSON value from the response body and response body as string respectively. Sample output, ``{"status": "200 OK", "status_code": 200, "body": {"hello": "world"}, "raw_body": "{\"hello\": \"world\"}"``}. By default, HTTP redirects are not enabled. To enable, set ``enable_redirect`` to ``true``. Also ``force_json_decode`` is set to ``false`` by default. This means if the HTTP server response does not specify the ``Content-type`` as ``application/json``, the response body will not be JSON decoded ie. output's ``body`` field will be ``null``. To change this behaviour, set ``force_json_decode`` to ``true``.| +| ``response := http.send(request)`` | ``http.send`` executes an HTTP `request` and returns a `response`. | -#### HTTPs Usage +The `request` object parameter may contain the following fields: -The following table explains the HTTPs objects +| Field | Required | Type | Description | +| --- | --- | --- | --- | +| `url` | yes | `string` | HTTP URL to specify in the request (e.g., `"https://www.openpolicyagent.org"`). | +| `method` | yes | `string` | HTTP method to specify in request (e.g., `"GET"`, `"POST"`, `"PUT"`, etc.) | +| `body` | no | `any` | HTTP message body to include in request. The value will be serialized to JSON. | +| `raw_body` | no | `string` | HTTP message body to include in request. The value WILL NOT be serialized. Use this for non-JSON messages. | +| `headers` | no | `object` | HTTP headers to include in the request (e.g,. `{"X-Opa": "rules"}`). | +| `enable_redirect` | no | `boolean` | Follow HTTP redirects. Default: `false`. | +| `force_json_decode` | no | `boolean` | Decode the HTTP response message body as JSON even if the `Content-Type` header is missing. Default: `false`. | +| `tls_use_system_certs` | no | `boolean` | Use system certificate pool. | +| `tls_ca_cert_file` | no | `string` | Path to file containing a root certificate in PEM encoded format. | +| `tls_ca_cert_env_variable` | no | `string` | Environment variable containing a root certificate in PEM encoded format. | +| `tls_client_cert_env_variable` | no | `string` | Environment variable containing a client certificate in PEM encoded format. | +| `tls_client_key_env_variable` | no | `string` | Environment variable containing a client key in PEM encoded format. | +| `tls_client_cert_file` | no | `string` | Path to file containing a client certificate in PEM encoded format. | +| `tls_client_key_file` | no | `string` | Path to file containing a key in PEM encoded format. | -| Object | Definition | Value| -| -------- |-----------|------| -| tls_use_system_certs | Use system certificate pool | true or false -| tls_ca_cert_file | Path to file containing a root certificate in PEM encoded format | double-quoted string -| tls_ca_cert_env_variable | Environment variable containing a root certificate in PEM encoded format | double-quoted string -| tls_client_cert_env_variable | Environment variable containing a client certificate in PEM encoded format | double-quoted string -| tls_client_key_env_variable | Environment variable containing a client key in PEM encoded format | double-quoted string -| tls_client_cert_file | Path to file containing a client certificate in PEM encoded format | double-quoted string -| tls_client_key_file | Path to file containing a key in PEM encoded format | double-quoted string -In order to trigger the use of HTTPs the user must provide one of the following combinations: +To trigger the use of HTTPs the user must provide one of the following combinations: * ``tls_client_cert_file``, ``tls_client_key_file`` * ``tls_client_cert_env_variable``, ``tls_client_key_env_variable`` The user must also provide a trusted root CA through tls_ca_cert_file or tls_ca_cert_env_variable. Alternatively the user could set tls_use_system_certs to ``true`` and the system certificate pool will be used. -#### HTTPs Examples +The `response` object parameter will contain the following fields: -| Examples | Comments | +| Field | Type | Description | +| --- | --- | --- | +| `status` | `string` | HTTP status message (e.g., `"200 OK"`). | +| `status_code` | `number` | HTTP status code (e.g., `200`). | +| `body` | `any` | Any JSON value. If the HTTP response message body was not deserialized from JSON, this field is set to `null`. | +| `raw_body` | `string` | The entire raw HTTP resposne message body represented as a string. | + +The table below shows examples of calling `http.send`: + +| Example | Comments | | -------- |-----------| -| Files containing TLS material | ``http.send({"method": "get", "url": "https://127.0.0.1:65331", "tls_ca_cert_file": "testdata/ca.pem", "tls_client_cert_file": "testdata/client-cert.pem", "tls_client_key_file": "testdata/client-key.pem"}, output)``. -|Environment variables containing TLS material | ``http.send({"method": "get", "url": "https://127.0.0.1:65360", "tls_ca_cert_env_variable": "CLIENT_CA_ENV", "tls_client_cert_env_variable": "CLIENT_CERT_ENV", "tls_client_key_env_variable": "CLIENT_KEY_ENV"}, output)``.| -| Accessing Google using System Cert Pool | ``http.send({"method": "get", "url": "https://www.google.com", "tls_use_system_certs": true, "tls_client_cert_file": "testdata/client-cert.pem", "tls_client_key_file": "testdata/client-key.pem"}, output)`` +| Files containing TLS material | ``http.send({"method": "get", "url": "https://127.0.0.1:65331", "tls_ca_cert_file": "testdata/ca.pem", "tls_client_cert_file": "testdata/client-cert.pem", "tls_client_key_file": "testdata/client-key.pem"})`` | +|Environment variables containing TLS material | ``http.send({"method": "get", "url": "https://127.0.0.1:65360", "tls_ca_cert_env_variable": "CLIENT_CA_ENV", "tls_client_cert_env_variable": "CLIENT_CERT_ENV", "tls_client_key_env_variable": "CLIENT_KEY_ENV"})`` | +| Accessing Google using System Cert Pool | ``http.send({"method": "get", "url": "https://www.google.com", "tls_use_system_certs": true, "tls_client_cert_file": "testdata/client-cert.pem", "tls_client_key_file": "testdata/client-key.pem"})`` | ### Net | Built-in | Description | diff --git a/topdown/http.go b/topdown/http.go index f1b3d6a0b4..695bd5d100 100644 --- a/topdown/http.go +++ b/topdown/http.go @@ -33,6 +33,7 @@ var allowedKeyNames = [...]string{ "enable_redirect", "force_json_decode", "headers", + "raw_body", "tls_use_system_certs", "tls_ca_cert_file", "tls_ca_cert_env_variable", @@ -131,6 +132,7 @@ func executeHTTPRequest(bctx BuiltinContext, obj ast.Object) (ast.Value, error) var tlsClientCertFile string var tlsClientKeyFile string var body *bytes.Buffer + var rawBody *bytes.Buffer var enableRedirect bool var forceJSONDecode bool var tlsUseSystemCerts bool @@ -173,6 +175,12 @@ func executeHTTPRequest(bctx BuiltinContext, obj ast.Object) (ast.Value, error) return nil, err } body = bytes.NewBuffer(bodyValBytes) + case "raw_body": + s, ok := obj.Get(val).Value.(ast.String) + if !ok { + return nil, fmt.Errorf("raw_body must be a string") + } + rawBody = bytes.NewBuffer([]byte(s)) case "tls_use_system_certs": tlsUseSystemCerts, err = strconv.ParseBool(obj.Get(val).String()) if err != nil { @@ -211,7 +219,7 @@ func executeHTTPRequest(bctx BuiltinContext, obj ast.Object) (ast.Value, error) return nil, fmt.Errorf("invalid type for headers key") } default: - return nil, fmt.Errorf("Invalid Key %v", key) + return nil, fmt.Errorf("invalid parameter %q", key) } } @@ -249,7 +257,9 @@ func executeHTTPRequest(bctx BuiltinContext, obj ast.Object) (ast.Value, error) client.CheckRedirect = nil } - if body == nil { + if rawBody != nil { + body = rawBody + } else if body == nil { body = bytes.NewBufferString("") } diff --git a/topdown/http_test.go b/topdown/http_test.go index 21decbd6a6..4e8a968aa8 100644 --- a/topdown/http_test.go +++ b/topdown/http_test.go @@ -9,6 +9,7 @@ import ( "crypto/tls" "crypto/x509" "encoding/json" + "errors" "fmt" "io/ioutil" "net/http" @@ -185,59 +186,96 @@ func TestHTTPCustomHeaders(t *testing.T) { // TestHTTPostRequest adds a new person func TestHTTPostRequest(t *testing.T) { - // test server ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - var person Person - if r.Body == nil { - http.Error(w, "Please send a request body", 400) - return - } - err := json.NewDecoder(r.Body).Decode(&person) + contentType := r.Header.Get("Content-Type") + + bs, err := ioutil.ReadAll(r.Body) if err != nil { - http.Error(w, err.Error(), 400) - return + t.Fatal(err) } - w.Header().Set("Content-Type", "application/json") + w.Header().Set("Content-Type", contentType) w.WriteHeader(http.StatusOK) - json.NewEncoder(w).Encode(person) + w.Write(bs) })) defer ts.Close() - // expected result - expectedResult := map[string]interface{}{ - "status": "200 OK", - "status_code": http.StatusOK, - "body": map[string]string{"id": "2", "firstname": "Joe"}, - "raw_body": "{\"id\":\"2\",\"firstname\":\"Joe\"}\n", - } - - resultObj, err := ast.InterfaceToValue(expectedResult) - if err != nil { - panic(err) - } - - // create a new person object - person2 := Person{ID: "2", Firstname: "Joe"} - b := new(bytes.Buffer) - json.NewEncoder(b).Encode(person2) - - // run the test tests := []struct { note string - rules []string + params string expected interface{} }{ - {"http.send", []string{fmt.Sprintf( - `p = x { http.send({"method": "post", "url": "%s", "body": %s}, x) }`, ts.URL, b)}, resultObj.String()}, + + { + note: "basic", + params: `{ + "method": "post", + "headers": {"Content-Type": "application/json"}, + "body": {"id": "2", "firstname": "Joe"} + }`, + expected: `{ + "status": "200 OK", + "status_code": 200, + "body": {"id": "2", "firstname": "Joe"}, + "raw_body": "{\"firstname\":\"Joe\",\"id\":\"2\"}" + }`, + }, + { + note: "raw_body", + params: `{ + "method": "post", + "headers": {"Content-Type": "application/x-www-form-encoded"}, + "raw_body": "username=foobar&password=baz" + }`, + expected: `{ + "status": "200 OK", + "status_code": 200, + "body": null, + "raw_body": "username=foobar&password=baz" + }`, + }, + { + note: "raw_body overrides body", + params: `{ + "method": "post", + "headers": {"Content-Type": "application/x-www-form-encoded"}, + "body": {"foo": 1}, + "raw_body": "username=foobar&password=baz" + }`, + expected: `{ + "status": "200 OK", + "status_code": 200, + "body": null, + "raw_body": "username=foobar&password=baz" + }`, + }, + { + note: "raw_body bad type", + params: `{ + "method": "post", + "headers": {"Content-Type": "application/x-www-form-encoded"}, + "raw_body": {"bar": "bar"} + }`, + expected: errors.New("raw_body must be a string"), + }, } - data := loadSmallTestData() + data := map[string]interface{}{} for _, tc := range tests { - runTopDownTestCase(t, data, tc.note, tc.rules, tc.expected) + + // Automatically set the URL because it's generated when the test server + // is started. If needed, the test cases can override in the future. + term := ast.MustParseTerm(tc.params) + term.Value.(ast.Object).Insert(ast.StringTerm("url"), ast.StringTerm(ts.URL)) + + rules := []string{ + fmt.Sprintf(`p = x { http.send(%s, x) }`, term), + } + + runTopDownTestCase(t, data, tc.note, rules, tc.expected) } } diff --git a/topdown/topdown_test.go b/topdown/topdown_test.go index 062b056ba9..d71f88780d 100644 --- a/topdown/topdown_test.go +++ b/topdown/topdown_test.go @@ -2979,7 +2979,7 @@ func assertTopDownWithPath(t *testing.T, compiler *ast.Compiler, store storage.S } if util.Compare(expected, result) != 0 { - t.Fatalf("Unexpected result:\nGot: %v\nExp:\n%v", result, expected) + t.Fatalf("Unexpected result:\nGot: %+v\nExp:\n%+v", result, expected) } // If the test case involved the input document, re-run it with partial