ewilliams28/releases
1
0
Fork 0
mirror of https://github.com/open-policy-agent/opa.git synced 2026-08-13 03:42:35 -06:00
Code Issues Packages Projects Releases Wiki Activity

build(deps): upgrade github.com/dgraph-io/badger to v4

Browse Source 
Despite the major version bump, there are no breaking changes that
affects our existing code.

From changelog of BadgerDB 4.0.0:

  "This is a major release because we are making a switch to SemVer in
order to make it easier for the community to understand when breaking
API and data format changes are made."

Reference: https://github.com/dgraph-io/badger/blob/ec1a9f3ae8c41edb50703493f1e4991c0641e6a8/CHANGELOG.md

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
This commit is contained in:
Eng Zer Jun
2025-01-18 04:25:27 +08:00
committed by GitHub
parent 2f3a6fb8b8
commit 304768b4e1
259 changed files with 36115 additions and 26373 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+4
View File
@@ -5,6 +5,10 @@ project adheres to [Semantic Versioning](http://semver.org/).
## Unreleased
### Dependency Updates
- github.com/dgraph-io/badger: v3.2103.5 -> v4.5.0 by @Juneezee in [#7239](https://github.com/open-policy-agent/opa/pull/7239)
## 1.0.0
> **_NOTES:_**
go.mod
+5 -10
View File
@@ -11,7 +11,7 @@ require (
github.com/bytecodealliance/wasmtime-go/v3 v3.0.2
github.com/containerd/containerd v1.7.25
github.com/containerd/errdefs v1.0.0
github.com/dgraph-io/badger/v3 v3.2103.5
github.com/dgraph-io/badger/v4 v4.5.0
github.com/fortytw2/leaktest v1.3.0
github.com/foxcpp/go-mockdns v1.1.0
github.com/fsnotify/fsnotify v1.8.0
@@ -57,25 +57,20 @@ require (
github.com/AdaLogics/go-fuzz-headers v0.0.0-20230811130428-ced1acdcaa24 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/cespare/xxhash v1.1.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/containerd/platforms v0.2.1 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.4 // indirect
github.com/dgraph-io/ristretto v0.1.1 // indirect
github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 // indirect
github.com/dustin/go-humanize v1.0.0 // indirect
github.com/dgraph-io/ristretto/v2 v2.0.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/glog v1.2.2 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/flatbuffers v1.12.1 // indirect
github.com/google/flatbuffers v24.3.25+incompatible // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.24.0 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/klauspost/compress v1.17.9 // indirect
github.com/klauspost/compress v1.17.11 // indirect
github.com/kr/pretty v0.3.1 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
go.sum
+10 -71
View File
@@ -6,7 +6,6 @@ github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERo
github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
github.com/Microsoft/hcsshim v0.11.7 h1:vl/nj3Bar/CvJSYo7gIQPyRWc9f3c6IeSNavBTSZNZQ=
github.com/Microsoft/hcsshim v0.11.7/go.mod h1:MV8xMfmECjl5HdO7U/3/hFVnkmSBjAjmA09d4bExKcU=
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
github.com/OneOfOne/xxhash v1.2.8 h1:31czK/TI9sNkxIKfaUfGlU47BAxQ0ztGgd9vPyqimf8=
github.com/OneOfOne/xxhash v1.2.8/go.mod h1:eZbhyaAYD41SGSSsnmcpxVoRiQ/MPUTjUdIIOT9Um7Q=
github.com/agnivade/levenshtein v1.2.0 h1:U9L4IOT0Y3i0TIlUIDJ7rVUziKi/zPbrJGaFrtYH3SY=
@@ -15,7 +14,6 @@ github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883 h1:bvNMNQO63//z+xNg
github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883/go.mod h1:rCTlJbsFo29Kk6CurOXKm700vrz8f0KW0JNfpkRJY/8=
github.com/arbovm/levenshtein v0.0.0-20160628152529-48b4e1c0c4d0 h1:jfIu9sQUG6Ig+0+Ap1h4unLjW6YQJpKZVmUzxsD4E/Q=
github.com/arbovm/levenshtein v0.0.0-20160628152529-48b4e1c0c4d0/go.mod h1:t2tdKJDJF9BV14lnkjHmOQgcvEKgtqs5a1N3LNdJhGE=
github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/bytecodealliance/wasmtime-go/v3 v3.0.2 h1:3uZCA/BLTIu+DqCfguByNMJa2HVHpXvjfy0Dy7g6fuA=
@@ -23,9 +21,6 @@ github.com/bytecodealliance/wasmtime-go/v3 v3.0.2/go.mod h1:RnUjnIXxEJcL6BgCvNyz
github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
@@ -42,10 +37,6 @@ github.com/containerd/log v0.1.0 h1:TCJt7ioM2cr/tfR8GPbGf9/VRAX8D2B4PjzCpfX540I=
github.com/containerd/log v0.1.0/go.mod h1:VRRf09a7mHDIRezVKTRCrOq78v577GXq3bSa3EhrzVo=
github.com/containerd/platforms v0.2.1 h1:zvwtM3rz2YHPQsF2CHYM8+KtB5dvhISiXh5ZpSBQv6A=
github.com/containerd/platforms v0.2.1/go.mod h1:XHCb+2/hzowdiut9rkudds9bE5yJ7npe7dG/wG+uFPw=
github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=
github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
github.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE=
github.com/cpuguy83/go-md2man/v2 v2.0.4 h1:wfIWP927BUkWJb2NmU/kNDYIBTh/ziUX91+lVfRxZq4=
github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
@@ -53,17 +44,16 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dgraph-io/badger/v3 v3.2103.5 h1:ylPa6qzbjYRQMU6jokoj4wzcaweHylt//CH0AKt0akg=
github.com/dgraph-io/badger/v3 v3.2103.5/go.mod h1:4MPiseMeDQ3FNCYwRbbcBOGJLf5jsE0PPFzRiKjtcdw=
github.com/dgraph-io/ristretto v0.1.1 h1:6CWw5tJNgpegArSHpNHJKldNeq03FQCwYvfMVWajOK8=
github.com/dgraph-io/ristretto v0.1.1/go.mod h1:S1GPSBCYCIhmVNfcth17y2zZtQT6wzkzgwUve0VDWWA=
github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw=
github.com/dgraph-io/badger/v4 v4.5.0 h1:TeJE3I1pIWLBjYhIYCA1+uxrjWEoJXImFBMEBVSm16g=
github.com/dgraph-io/badger/v4 v4.5.0/go.mod h1:ysgYmIeG8dS/E8kwxT7xHyc7MkmwNYLRoYnFbr7387A=
github.com/dgraph-io/ristretto/v2 v2.0.0 h1:l0yiSOtlJvc0otkqyMaDNysg8E9/F/TYZwMbxscNOAQ=
github.com/dgraph-io/ristretto/v2 v2.0.0/go.mod h1:FVFokF2dRqXyPyeMnK1YDy8Fc6aTe0IKgbcd03CYeEk=
github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 h1:fAjc9m62+UWV/WAFKLNi6ZS0675eEUC9y3AlwSbQu1Y=
github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw=
github.com/dgryski/trifles v0.0.0-20230903005119-f50d829f2e54 h1:SG7nF6SRlWhcT7cNTs5R6Hk4V2lcmLz2NsG2VnInyNo=
github.com/dgryski/trifles v0.0.0-20230903005119-f50d829f2e54/go.mod h1:if7Fbed8SFyPtHLHbg49SI7NAdJiC5WIA09pe59rfAA=
github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
@@ -76,7 +66,6 @@ github.com/foxcpp/go-mockdns v1.1.0 h1:jI0rD8M0wuYAxL7r/ynTrCQQq0BVqfB99Vgk7Dlme
github.com/foxcpp/go-mockdns v1.1.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.8.0 h1:dAwr6QBTBZIkG8roQaJjGof0pp0EeF+tNV7YBP3F/8M=
github.com/fsnotify/fsnotify v1.8.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
github.com/go-ini/ini v1.67.0 h1:z6ZrTEZqSWOTyH2FlglNbNgARyHG8oLW9gMELqKr06A=
@@ -91,15 +80,11 @@ github.com/gobwas/glob v0.2.3/go.mod h1:d3Ez4x06l9bZtSvzIay5+Yzi0fmZzPgnTbPcKjJA
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/glog v1.2.2 h1:1+mZ9upx1Dh6FmUTFR1naJ77miKiXgALjWOZ3NVFPmY=
github.com/golang/glog v1.2.2/go.mod h1:6AhwSGph0fcJtXVM/PEHPqZlFeoLxhs7/t5UDAwmO+w=
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da h1:oI5xCqsCo564l8iNU+DwB5epxmsaqB+rhGL0m5jtYqE=
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
@@ -110,18 +95,14 @@ github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QD
github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/golang/snappy v0.0.3/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM=
github.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/google/flatbuffers v1.12.1 h1:MVlul7pQNoDzWRLTw5imwYsl+usrS1TXG2H4jg6ImGw=
github.com/google/flatbuffers v1.12.1/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
github.com/google/flatbuffers v24.3.25+incompatible h1:CX395cjN9Kke9mmalRoL3d81AtFUxJM+yDthflgJGkI=
github.com/google/flatbuffers v24.3.25+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.3/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
@@ -134,14 +115,10 @@ github.com/grpc-ecosystem/grpc-gateway/v2 v2.24.0 h1:TmHmbvxPmaegwhDubVz0lICL0J5
github.com/grpc-ecosystem/grpc-gateway/v2 v2.24.0/go.mod h1:qztMSjm835F2bXf+5HKAPIS5qsmQDqZna/PgVt4rWtI=
github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.12.3/go.mod h1:8dP1Hq4DHOhN9w426knH3Rhby4rFm6D8eO+e+Dq5Gzg=
github.com/klauspost/compress v1.17.9 h1:6KIumPrER1LHsvBVuDa0r5xaG0Es51mhhB9BQB2qeMA=
github.com/klauspost/compress v1.17.9/go.mod h1:Di0epgTjJY877eYKx5yC51cX2A2Vl2ibi7bDH9ttBbw=
github.com/klauspost/compress v1.17.11 h1:In6xLpyWOi1+C7tXUUWv2ot1QvBjxevKAaI6IXrJmUc=
github.com/klauspost/compress v1.17.11/go.mod h1:pMDklpSncoRMuLFrf1W9Ss9KT+0rH90U12bZKk7uwG0=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
@@ -152,7 +129,6 @@ github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
github.com/magiconair/properties v1.8.7 h1:IeQXZAiQcpL9mgcAe1Nu6cX9LLw6ExEHKjN0VQdvPDY=
github.com/magiconair/properties v1.8.7/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
github.com/mattn/go-runewidth v0.0.3/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
@@ -160,8 +136,6 @@ github.com/mattn/go-runewidth v0.0.9 h1:Lm995f3rfxdpd6TSmuVCHVb/QhupuXlYr8sCI/Qd
github.com/mattn/go-runewidth v0.0.9/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
github.com/miekg/dns v1.1.57 h1:Jzi7ApEIzwEPLHWRcafCN9LZSBbqQpxjt/wpgvg7wcM=
github.com/miekg/dns v1.1.57/go.mod h1:uqRjCRUuEAA6qsOiJvDd+CFo/vW+y5WR6SNmHE55hZk=
github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY=
github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
github.com/moby/locker v1.0.1 h1:fOXqR41zeveg4fFODix+1Ch4mj/gT0NE1XJbp/epuBg=
@@ -178,7 +152,6 @@ github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
github.com/opencontainers/image-spec v1.1.0 h1:8SG7/vwALn54lVB/0yZ/MMwhFrPYtpEHQb2IpWsCzug=
github.com/opencontainers/image-spec v1.1.0/go.mod h1:W4s4sFTMaBeK1BQLXbG4AdM2szdn85PY75RI83NrTrM=
github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
github.com/pelletier/go-toml/v2 v2.1.0 h1:FnwAJ4oYMvbT/34k9zzHuZNrhlz48GB3/s6at6/MHO4=
github.com/pelletier/go-toml/v2 v2.1.0/go.mod h1:tJU2Z3ZkXwnxa4DPO899bsyIoywizdUvyaeZurnPPDc=
github.com/peterh/liner v1.2.2 h1:aJ4AOodmL+JxOZZEL2u9iJf8omNRpqHc/EbrK+3mAXw=
@@ -205,7 +178,6 @@ github.com/rcrowley/go-metrics v0.0.0-20200313005456-10cdbea86bc0/go.mod h1:bCqn
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII=
github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o=
github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
github.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/sagikazarmark/locafero v0.4.0 h1:HApY1R9zGo4DBgr7dqsTH/JJxLTTsOt7u6keLGt6kNQ=
@@ -218,29 +190,19 @@ github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/sourcegraph/conc v0.3.0 h1:OQTbbt6P72L20UqAkXXuLOj79LfEanQ+YQFNpLA9ySo=
github.com/sourcegraph/conc v0.3.0/go.mod h1:Sdozi7LEKbFPqYX2/J+iBAM6HpqSLTASQIKqDmF7Mt0=
github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI=
github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
github.com/spf13/afero v1.11.0 h1:WJQKhtpdm3v2IzqG8VMqrr6Rf3UYpEF239Jy9wNepM8=
github.com/spf13/afero v1.11.0/go.mod h1:GH9Y3pIexgf1MTIWtNGyogA5MwRIDXGUr+hbWNoBjkY=
github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
github.com/spf13/cast v1.6.0 h1:GEiTHELF+vaR5dhz3VqZfFSzZjYbgeKDpBxQVS4GYJ0=
github.com/spf13/cast v1.6.0/go.mod h1:ancEpBxwJDODSW/UG4rDrAqiKolqNNh2DX3mk86cAdo=
github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU=
github.com/spf13/cobra v1.8.1 h1:e5/vxKd/rZsfSJMUX1agtjeTDf+qv1/JdBF8gg5k9ZM=
github.com/spf13/cobra v1.8.1/go.mod h1:wHxEcudfqmLYa8iTfL+OuZPbBZkmvliBWKIezN3kD9Y=
github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=
github.com/spf13/viper v1.18.2 h1:LUXCnvUvSM6FXAsj6nnfc8Q2tp1dIgUfY9Kc8GsSOiQ=
github.com/spf13/viper v1.18.2/go.mod h1:EKmWIqdnk5lOcmR72yw6hS+8OPYcwD0jteitLMVB+yk=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
@@ -254,18 +216,13 @@ github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8
github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU=
github.com/tchap/go-patricia/v2 v2.3.2 h1:xTHFutuitO2zqKAQ5rCROYgUb7Or/+IC3fts9/Yc7nM=
github.com/tchap/go-patricia/v2 v2.3.2/go.mod h1:VZRHKAb53DLaG+nA9EaYYiaEx6YztwDlLElMsnSHD4k=
github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb h1:zGWFAtiMcyryUHoUjUJX0/lt1H2+i2Ka2n+D3DImSNo=
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU=
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 h1:EzJWgHovont7NscjpAxXsDA8S8BMYve8Y5+7cuRE7R0=
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415/go.mod h1:GwrjFmJcFw6At/Gs6z4yjiIwzuJ1/+UwLxMQDVQXShQ=
github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
github.com/yashtewari/glob-intersection v0.2.0 h1:8iuHdN88yYuCzCdjt0gDe+6bAhUwBeEWqThExu54RFg=
github.com/yashtewari/glob-intersection v0.2.0/go.mod h1:LK7pIC3piUjovexikBbJ26Yml7g8xa5bsjfx2v1fwok=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk=
go.opencensus.io v0.24.0 h1:y73uSU6J157QMP2kn2r30vwW1A2W2WFwSCGnAVxeaD0=
go.opencensus.io v0.24.0/go.mod h1:vNK8G9p7aAivkbmorf4v+7Hgx+Zs0yY+0fOtgBfjQKo=
go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=
@@ -296,9 +253,7 @@ go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.uber.org/multierr v1.9.0 h1:7fIwc/ZtS0q++VgcfqFDxSBZVv/Xo49/SYnDFupUwlI=
go.uber.org/multierr v1.9.0/go.mod h1:X2jQV1h+kxSjClGpnseKVIxpmcjrj7MNnI0bnlfKTVQ=
golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc=
@@ -310,8 +265,6 @@ golang.org/x/exp v0.0.0-20230905200255-921286631fa9/go.mod h1:S2oDrQGGwySpoQPVqR
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
@@ -324,8 +277,6 @@ golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73r
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
@@ -339,10 +290,7 @@ golang.org/x/net v0.34.0/go.mod h1:di0qlW3YNM5oh6GqDGQr92MyTozJPmybPK4Ev/Gm31k=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
@@ -351,10 +299,8 @@ golang.org/x/sync v0.5.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.10.0 h1:3NQrjDixjgGwUOCaF8w2+VYHv0Ve/vGYSbdkTa98gmQ=
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
@@ -362,7 +308,6 @@ golang.org/x/sys v0.0.0-20211117180635-dee7805ff2e1/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20221010170243-090e33056c14/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
@@ -394,8 +339,6 @@ golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58=
@@ -403,13 +346,10 @@ golang.org/x/tools v0.15.0/go.mod h1:hpksKq4dtpQWS1uQ61JkdqWM3LscIS6Slf+VVkm+wQk
golang.org/x/tools v0.22.0 h1:gqSGLZqv+AI9lIQzniJ0nZDRG5GBPsSi+DRNHWNz6yA=
golang.org/x/tools v0.22.0/go.mod h1:aCwcsjqvq7Yqt6TNyX7QMU2enbQ/Gt0bo6krSeEri+c=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/genproto/googleapis/api v0.0.0-20241209162323-e6fa225c2576 h1:CkkIfIt50+lT6NHAVoRYEyAvQGFM7xEwXUUywFvEb3Q=
@@ -417,7 +357,6 @@ google.golang.org/genproto/googleapis/api v0.0.0-20241209162323-e6fa225c2576/go.
google.golang.org/genproto/googleapis/rpc v0.0.0-20241209162323-e6fa225c2576 h1:8ZmaLZE4XWrtU3MyClkYqqtl6Oegr3235h7jxsDyqCY=
google.golang.org/genproto/googleapis/rpc v0.0.0-20241209162323-e6fa225c2576/go.mod h1:5uTbfoYQed2U9p3KIj2/Zzm02PYhndfdmML0qC3q3FU=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
v1/storage/disk/config.go
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"fmt"
"os"
badger "github.com/dgraph-io/badger/v3"
badger "github.com/dgraph-io/badger/v4"
"github.com/open-policy-agent/opa/v1/config"
"github.com/open-policy-agent/opa/v1/storage"
"github.com/open-policy-agent/opa/v1/util"
v1/storage/disk/config_test.go
+1 -1
View File
@@ -11,7 +11,7 @@ import (
"path/filepath"
"testing"
"github.com/dgraph-io/badger/v3"
"github.com/dgraph-io/badger/v4"
"github.com/open-policy-agent/opa/v1/logging"
)
v1/storage/disk/disk.go
+1 -1
View File
@@ -71,7 +71,7 @@ import (
"sync/atomic"
"time"
badger "github.com/dgraph-io/badger/v3"
badger "github.com/dgraph-io/badger/v4"
"github.com/prometheus/client_golang/prometheus"
"github.com/open-policy-agent/opa/v1/logging"
v1/storage/disk/disk_test.go
+1 -1
View File
@@ -19,7 +19,7 @@ import (
"github.com/open-policy-agent/opa/v1/bundle"
badger "github.com/dgraph-io/badger/v3"
badger "github.com/dgraph-io/badger/v4"
"github.com/open-policy-agent/opa/v1/logging"
"github.com/open-policy-agent/opa/v1/storage"
v1/storage/disk/txn.go
+1 -1
View File
@@ -11,7 +11,7 @@ import (
"fmt"
"strconv"
badger "github.com/dgraph-io/badger/v3"
badger "github.com/dgraph-io/badger/v4"
"github.com/open-policy-agent/opa/v1/metrics"
"github.com/open-policy-agent/opa/v1/storage"
v1/storage/disk/txn_test.go
+1 -1
View File
@@ -11,7 +11,7 @@ import (
"math/rand"
"testing"
"github.com/dgraph-io/badger/v3"
"github.com/dgraph-io/badger/v4"
"github.com/open-policy-agent/opa/v1/logging"
"github.com/open-policy-agent/opa/v1/storage"
"github.com/open-policy-agent/opa/v1/util/test"
vendor/github.com/cespare/xxhash/LICENSE.txt
Generated Vendored
-22
View File
@@ -1,22 +0,0 @@
Copyright (c) 2016 Caleb Spare
MIT License
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
vendor/github.com/cespare/xxhash/README.md
Generated Vendored
-50
View File
@@ -1,50 +0,0 @@
# xxhash
[![GoDoc](https://godoc.org/github.com/cespare/xxhash?status.svg)](https://godoc.org/github.com/cespare/xxhash)
xxhash is a Go implementation of the 64-bit
[xxHash](http://cyan4973.github.io/xxHash/) algorithm, XXH64. This is a
high-quality hashing algorithm that is much faster than anything in the Go
standard library.
The API is very small, taking its cue from the other hashing packages in the
standard library:
$ go doc github.com/cespare/xxhash !
package xxhash // import "github.com/cespare/xxhash"
Package xxhash implements the 64-bit variant of xxHash (XXH64) as described
at http://cyan4973.github.io/xxHash/.
func New() hash.Hash64
func Sum64(b []byte) uint64
func Sum64String(s string) uint64
This implementation provides a fast pure-Go implementation and an even faster
assembly implementation for amd64.
## Benchmarks
Here are some quick benchmarks comparing the pure-Go and assembly
implementations of Sum64 against another popular Go XXH64 implementation,
[github.com/OneOfOne/xxhash](https://github.com/OneOfOne/xxhash):
| input size | OneOfOne | cespare (purego) | cespare |
| --- | --- | --- | --- |
| 5 B | 416 MB/s | 720 MB/s | 872 MB/s |
| 100 B | 3980 MB/s | 5013 MB/s | 5252 MB/s |
| 4 KB | 12727 MB/s | 12999 MB/s | 13026 MB/s |
| 10 MB | 9879 MB/s | 10775 MB/s | 10913 MB/s |
These numbers were generated with:
```
$ go test -benchtime 10s -bench '/OneOfOne,'
$ go test -tags purego -benchtime 10s -bench '/xxhash,'
$ go test -benchtime 10s -bench '/xxhash,'
```
## Projects using this package
- [InfluxDB](https://github.com/influxdata/influxdb)
- [Prometheus](https://github.com/prometheus/prometheus)
vendor/github.com/cespare/xxhash/rotate.go
Generated Vendored
-14
View File
@@ -1,14 +0,0 @@
// +build !go1.9
package xxhash
// TODO(caleb): After Go 1.10 comes out, remove this fallback code.
func rol1(x uint64) uint64 { return (x << 1) | (x >> (64 - 1)) }
func rol7(x uint64) uint64 { return (x << 7) | (x >> (64 - 7)) }
func rol11(x uint64) uint64 { return (x << 11) | (x >> (64 - 11)) }
func rol12(x uint64) uint64 { return (x << 12) | (x >> (64 - 12)) }
func rol18(x uint64) uint64 { return (x << 18) | (x >> (64 - 18)) }
func rol23(x uint64) uint64 { return (x << 23) | (x >> (64 - 23)) }
func rol27(x uint64) uint64 { return (x << 27) | (x >> (64 - 27)) }
func rol31(x uint64) uint64 { return (x << 31) | (x >> (64 - 31)) }
vendor/github.com/cespare/xxhash/rotate19.go
Generated Vendored
-14
View File
@@ -1,14 +0,0 @@
// +build go1.9
package xxhash
import "math/bits"
func rol1(x uint64) uint64 { return bits.RotateLeft64(x, 1) }
func rol7(x uint64) uint64 { return bits.RotateLeft64(x, 7) }
func rol11(x uint64) uint64 { return bits.RotateLeft64(x, 11) }
func rol12(x uint64) uint64 { return bits.RotateLeft64(x, 12) }
func rol18(x uint64) uint64 { return bits.RotateLeft64(x, 18) }
func rol23(x uint64) uint64 { return bits.RotateLeft64(x, 23) }
func rol27(x uint64) uint64 { return bits.RotateLeft64(x, 27) }
func rol31(x uint64) uint64 { return bits.RotateLeft64(x, 31) }
vendor/github.com/cespare/xxhash/xxhash.go
Generated Vendored
-168
View File
@@ -1,168 +0,0 @@
// Package xxhash implements the 64-bit variant of xxHash (XXH64) as described
// at http://cyan4973.github.io/xxHash/.
package xxhash
import (
"encoding/binary"
"hash"
)
const (
prime1 uint64 = 11400714785074694791
prime2 uint64 = 14029467366897019727
prime3 uint64 = 1609587929392839161
prime4 uint64 = 9650029242287828579
prime5 uint64 = 2870177450012600261
)
// NOTE(caleb): I'm using both consts and vars of the primes. Using consts where
// possible in the Go code is worth a small (but measurable) performance boost
// by avoiding some MOVQs. Vars are needed for the asm and also are useful for
// convenience in the Go code in a few places where we need to intentionally
// avoid constant arithmetic (e.g., v1 := prime1 + prime2 fails because the
// result overflows a uint64).
var (
prime1v = prime1
prime2v = prime2
prime3v = prime3
prime4v = prime4
prime5v = prime5
)
type xxh struct {
v1 uint64
v2 uint64
v3 uint64
v4 uint64
total int
mem [32]byte
n int // how much of mem is used
}
// New creates a new hash.Hash64 that implements the 64-bit xxHash algorithm.
func New() hash.Hash64 {
var x xxh
x.Reset()
return &x
}
func (x *xxh) Reset() {
x.n = 0
x.total = 0
x.v1 = prime1v + prime2
x.v2 = prime2
x.v3 = 0
x.v4 = -prime1v
}
func (x *xxh) Size() int { return 8 }
func (x *xxh) BlockSize() int { return 32 }
// Write adds more data to x. It always returns len(b), nil.
func (x *xxh) Write(b []byte) (n int, err error) {
n = len(b)
x.total += len(b)
if x.n+len(b) < 32 {
// This new data doesn't even fill the current block.
copy(x.mem[x.n:], b)
x.n += len(b)
return
}
if x.n > 0 {
// Finish off the partial block.
copy(x.mem[x.n:], b)
x.v1 = round(x.v1, u64(x.mem[0:8]))
x.v2 = round(x.v2, u64(x.mem[8:16]))
x.v3 = round(x.v3, u64(x.mem[16:24]))
x.v4 = round(x.v4, u64(x.mem[24:32]))
b = b[32-x.n:]
x.n = 0
}
if len(b) >= 32 {
// One or more full blocks left.
b = writeBlocks(x, b)
}
// Store any remaining partial block.
copy(x.mem[:], b)
x.n = len(b)
return
}
func (x *xxh) Sum(b []byte) []byte {
s := x.Sum64()
return append(
b,
byte(s>>56),
byte(s>>48),
byte(s>>40),
byte(s>>32),
byte(s>>24),
byte(s>>16),
byte(s>>8),
byte(s),
)
}
func (x *xxh) Sum64() uint64 {
var h uint64
if x.total >= 32 {
v1, v2, v3, v4 := x.v1, x.v2, x.v3, x.v4
h = rol1(v1) + rol7(v2) + rol12(v3) + rol18(v4)
h = mergeRound(h, v1)
h = mergeRound(h, v2)
h = mergeRound(h, v3)
h = mergeRound(h, v4)
} else {
h = x.v3 + prime5
}
h += uint64(x.total)
i, end := 0, x.n
for ; i+8 <= end; i += 8 {
k1 := round(0, u64(x.mem[i:i+8]))
h ^= k1
h = rol27(h)*prime1 + prime4
}
if i+4 <= end {
h ^= uint64(u32(x.mem[i:i+4])) * prime1
h = rol23(h)*prime2 + prime3
i += 4
}
for i < end {
h ^= uint64(x.mem[i]) * prime5
h = rol11(h) * prime1
i++
}
h ^= h >> 33
h *= prime2
h ^= h >> 29
h *= prime3
h ^= h >> 32
return h
}
func u64(b []byte) uint64 { return binary.LittleEndian.Uint64(b) }
func u32(b []byte) uint32 { return binary.LittleEndian.Uint32(b) }
func round(acc, input uint64) uint64 {
acc += input * prime2
acc = rol31(acc)
acc *= prime1
return acc
}
func mergeRound(acc, val uint64) uint64 {
val = round(0, val)
acc ^= val
acc = acc*prime1 + prime4
return acc
}
vendor/github.com/cespare/xxhash/xxhash_amd64.go
Generated Vendored
-12
View File
@@ -1,12 +0,0 @@
// +build !appengine
// +build gc
// +build !purego
package xxhash
// Sum64 computes the 64-bit xxHash digest of b.
//
//go:noescape
func Sum64(b []byte) uint64
func writeBlocks(x *xxh, b []byte) []byte
vendor/github.com/cespare/xxhash/xxhash_amd64.s
Generated Vendored
-233
View File
@@ -1,233 +0,0 @@
// +build !appengine
// +build gc
// +build !purego
#include "textflag.h"
// Register allocation:
// AX h
// CX pointer to advance through b
// DX n
// BX loop end
// R8 v1, k1
// R9 v2
// R10 v3
// R11 v4
// R12 tmp
// R13 prime1v
// R14 prime2v
// R15 prime4v
// round reads from and advances the buffer pointer in CX.
// It assumes that R13 has prime1v and R14 has prime2v.
#define round(r) \
MOVQ (CX), R12 \
ADDQ $8, CX \
IMULQ R14, R12 \
ADDQ R12, r \
ROLQ $31, r \
IMULQ R13, r
// mergeRound applies a merge round on the two registers acc and val.
// It assumes that R13 has prime1v, R14 has prime2v, and R15 has prime4v.
#define mergeRound(acc, val) \
IMULQ R14, val \
ROLQ $31, val \
IMULQ R13, val \
XORQ val, acc \
IMULQ R13, acc \
ADDQ R15, acc
// func Sum64(b []byte) uint64
TEXT ·Sum64(SB), NOSPLIT, $0-32
// Load fixed primes.
MOVQ ·prime1v(SB), R13
MOVQ ·prime2v(SB), R14
MOVQ ·prime4v(SB), R15
// Load slice.
MOVQ b_base+0(FP), CX
MOVQ b_len+8(FP), DX
LEAQ (CX)(DX*1), BX
// The first loop limit will be len(b)-32.
SUBQ $32, BX
// Check whether we have at least one block.
CMPQ DX, $32
JLT noBlocks
// Set up initial state (v1, v2, v3, v4).
MOVQ R13, R8
ADDQ R14, R8
MOVQ R14, R9
XORQ R10, R10
XORQ R11, R11
SUBQ R13, R11
// Loop until CX > BX.
blockLoop:
round(R8)
round(R9)
round(R10)
round(R11)
CMPQ CX, BX
JLE blockLoop
MOVQ R8, AX
ROLQ $1, AX
MOVQ R9, R12
ROLQ $7, R12
ADDQ R12, AX
MOVQ R10, R12
ROLQ $12, R12
ADDQ R12, AX
MOVQ R11, R12
ROLQ $18, R12
ADDQ R12, AX
mergeRound(AX, R8)
mergeRound(AX, R9)
mergeRound(AX, R10)
mergeRound(AX, R11)
JMP afterBlocks
noBlocks:
MOVQ ·prime5v(SB), AX
afterBlocks:
ADDQ DX, AX
// Right now BX has len(b)-32, and we want to loop until CX > len(b)-8.
ADDQ $24, BX
CMPQ CX, BX
JG fourByte
wordLoop:
// Calculate k1.
MOVQ (CX), R8
ADDQ $8, CX
IMULQ R14, R8
ROLQ $31, R8
IMULQ R13, R8
XORQ R8, AX
ROLQ $27, AX
IMULQ R13, AX
ADDQ R15, AX
CMPQ CX, BX
JLE wordLoop
fourByte:
ADDQ $4, BX
CMPQ CX, BX
JG singles
MOVL (CX), R8
ADDQ $4, CX
IMULQ R13, R8
XORQ R8, AX
ROLQ $23, AX
IMULQ R14, AX
ADDQ ·prime3v(SB), AX
singles:
ADDQ $4, BX
CMPQ CX, BX
JGE finalize
singlesLoop:
MOVBQZX (CX), R12
ADDQ $1, CX
IMULQ ·prime5v(SB), R12
XORQ R12, AX
ROLQ $11, AX
IMULQ R13, AX
CMPQ CX, BX
JL singlesLoop
finalize:
MOVQ AX, R12
SHRQ $33, R12
XORQ R12, AX
IMULQ R14, AX
MOVQ AX, R12
SHRQ $29, R12
XORQ R12, AX
IMULQ ·prime3v(SB), AX
MOVQ AX, R12
SHRQ $32, R12
XORQ R12, AX
MOVQ AX, ret+24(FP)
RET
// writeBlocks uses the same registers as above except that it uses AX to store
// the x pointer.
// func writeBlocks(x *xxh, b []byte) []byte
TEXT ·writeBlocks(SB), NOSPLIT, $0-56
// Load fixed primes needed for round.
MOVQ ·prime1v(SB), R13
MOVQ ·prime2v(SB), R14
// Load slice.
MOVQ b_base+8(FP), CX
MOVQ CX, ret_base+32(FP) // initialize return base pointer; see NOTE below
MOVQ b_len+16(FP), DX
LEAQ (CX)(DX*1), BX
SUBQ $32, BX
// Load vN from x.
MOVQ x+0(FP), AX
MOVQ 0(AX), R8 // v1
MOVQ 8(AX), R9 // v2
MOVQ 16(AX), R10 // v3
MOVQ 24(AX), R11 // v4
// We don't need to check the loop condition here; this function is
// always called with at least one block of data to process.
blockLoop:
round(R8)
round(R9)
round(R10)
round(R11)
CMPQ CX, BX
JLE blockLoop
// Copy vN back to x.
MOVQ R8, 0(AX)
MOVQ R9, 8(AX)
MOVQ R10, 16(AX)
MOVQ R11, 24(AX)
// Construct return slice.
// NOTE: It's important that we don't construct a slice that has a base
// pointer off the end of the original slice, as in Go 1.7+ this will
// cause runtime crashes. (See discussion in, for example,
// https://github.com/golang/go/issues/16772.)
// Therefore, we calculate the length/cap first, and if they're zero, we
// keep the old base. This is what the compiler does as well if you
// write code like
// b = b[len(b):]
// New length is 32 - (CX - BX) -> BX+32 - CX.
ADDQ $32, BX
SUBQ CX, BX
JZ afterSetBase
MOVQ CX, ret_base+32(FP)
afterSetBase:
MOVQ BX, ret_len+40(FP)
MOVQ BX, ret_cap+48(FP) // set cap == len
RET
vendor/github.com/cespare/xxhash/xxhash_other.go
Generated Vendored
-75
View File
@@ -1,75 +0,0 @@
// +build !amd64 appengine !gc purego
package xxhash
// Sum64 computes the 64-bit xxHash digest of b.
func Sum64(b []byte) uint64 {
// A simpler version would be
// x := New()
// x.Write(b)
// return x.Sum64()
// but this is faster, particularly for small inputs.
n := len(b)
var h uint64
if n >= 32 {
v1 := prime1v + prime2
v2 := prime2
v3 := uint64(0)
v4 := -prime1v
for len(b) >= 32 {
v1 = round(v1, u64(b[0:8:len(b)]))
v2 = round(v2, u64(b[8:16:len(b)]))
v3 = round(v3, u64(b[16:24:len(b)]))
v4 = round(v4, u64(b[24:32:len(b)]))
b = b[32:len(b):len(b)]
}
h = rol1(v1) + rol7(v2) + rol12(v3) + rol18(v4)
h = mergeRound(h, v1)
h = mergeRound(h, v2)
h = mergeRound(h, v3)
h = mergeRound(h, v4)
} else {
h = prime5
}
h += uint64(n)
i, end := 0, len(b)
for ; i+8 <= end; i += 8 {
k1 := round(0, u64(b[i:i+8:len(b)]))
h ^= k1
h = rol27(h)*prime1 + prime4
}
if i+4 <= end {
h ^= uint64(u32(b[i:i+4:len(b)])) * prime1
h = rol23(h)*prime2 + prime3
i += 4
}
for ; i < end; i++ {
h ^= uint64(b[i]) * prime5
h = rol11(h) * prime1
}
h ^= h >> 33
h *= prime2
h ^= h >> 29
h *= prime3
h ^= h >> 32
return h
}
func writeBlocks(x *xxh, b []byte) []byte {
v1, v2, v3, v4 := x.v1, x.v2, x.v3, x.v4
for len(b) >= 32 {
v1 = round(v1, u64(b[0:8:len(b)]))
v2 = round(v2, u64(b[8:16:len(b)]))
v3 = round(v3, u64(b[16:24:len(b)]))
v4 = round(v4, u64(b[24:32:len(b)]))
b = b[32:len(b):len(b)]
}
x.v1, x.v2, x.v3, x.v4 = v1, v2, v3, v4
return b
}
vendor/github.com/cespare/xxhash/xxhash_safe.go
Generated Vendored
-10
View File
@@ -1,10 +0,0 @@
// +build appengine
// This file contains the safe implementations of otherwise unsafe-using code.
package xxhash
// Sum64String computes the 64-bit xxHash digest of s.
func Sum64String(s string) uint64 {
return Sum64([]byte(s))
}
vendor/github.com/cespare/xxhash/xxhash_unsafe.go
Generated Vendored
-30
View File
@@ -1,30 +0,0 @@
// +build !appengine
// This file encapsulates usage of unsafe.
// xxhash_safe.go contains the safe implementations.
package xxhash
import (
"reflect"
"unsafe"
)
// Sum64String computes the 64-bit xxHash digest of s.
// It may be faster than Sum64([]byte(s)) by avoiding a copy.
//
// TODO(caleb): Consider removing this if an optimization is ever added to make
// it unnecessary: https://golang.org/issue/2205.
//
// TODO(caleb): We still have a function call; we could instead write Go/asm
// copies of Sum64 for strings to squeeze out a bit more speed.
func Sum64String(s string) uint64 {
// See https://groups.google.com/d/msg/golang-nuts/dcjzJy-bSpw/tcZYBzQqAQAJ
// for some discussion about this unsafe conversion.
var b []byte
bh := (*reflect.SliceHeader)(unsafe.Pointer(&b))
bh.Data = (*reflect.StringHeader)(unsafe.Pointer(&s)).Data
bh.Len = len(s)
bh.Cap = len(s)
return Sum64(b)
}
vendor/github.com/dgraph-io/badger/v3/.go-version
Generated Vendored
-1
View File
@@ -1 +0,0 @@
1.18
vendor/github.com/dgraph-io/badger/v3/.golangci.yml
Generated Vendored
-21
View File
@@ -1,21 +0,0 @@
run:
tests: false
linters-settings:
lll:
line-length: 120
linters:
disable-all: true
enable:
# - errcheck
# - ineffassign
# - gas
- gofmt
# - golint
- gosimple
- govet
- lll
# - varcheck
# - unused
# - gosec
vendor/github.com/dgraph-io/badger/v3/appveyor.yml
Generated Vendored
-49
View File
@@ -1,49 +0,0 @@
# version format
version: "{build}"
# Operating system (build VM template)
os: Windows Server 2012 R2
# Platform.
platform: x64
clone_folder: c:\gopath\src\github.com\dgraph-io\badger
# Environment variables
environment:
GOVERSION: 1.12
GOPATH: c:\gopath
GO111MODULE: on
# scripts that run after cloning repository
install:
- set PATH=%GOPATH%\bin;c:\go\bin;c:\msys64\mingw64\bin;%PATH%
- go version
- go env
- python --version
- gcc --version
# To run your custom scripts instead of automatic MSBuild
build_script:
# We need to disable firewall - https://github.com/appveyor/ci/issues/1579#issuecomment-309830648
- ps: Disable-NetFirewallRule -DisplayName 'File and Printer Sharing (SMB-Out)'
- cd c:\gopath\src\github.com\dgraph-io\badger
- git branch
- go get -t ./...
# To run your custom scripts instead of automatic tests
test_script:
# Unit tests
- ps: Add-AppveyorTest "Unit Tests" -Outcome Running
- go test -v github.com/dgraph-io/badger/...
- ps: Update-AppveyorTest "Unit Tests" -Outcome Passed
notifications:
- provider: Email
to:
- pawan@dgraph.io
on_build_failure: true
on_build_status_changed: true
# to disable deployment
deploy: off
vendor/github.com/dgraph-io/badger/v3/pb/badgerpb3.pb.go
Generated Vendored
-2164
View File
File diff suppressed because it is too large Load Diff
vendor/github.com/dgraph-io/badger/v3/pb/gen.sh
Generated Vendored
-8
View File
@@ -1,8 +0,0 @@
#!/bin/bash
# Run this script from its directory, so that badgerpb2.proto is where it's expected to
# be.
# You might need to go get -v github.com/gogo/protobuf/...
go get -v github.com/gogo/protobuf/protoc-gen-gogofaster
protoc --gogofaster_out=. --gogofaster_opt=paths=source_relative -I=. badgerpb3.proto
vendor/github.com/dgraph-io/badger/v3/y/metrics.go
Generated Vendored
-169
View File
@@ -1,169 +0,0 @@
/*
* Copyright (C) 2017 Dgraph Labs, Inc. and Contributors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package y
import (
"expvar"
)
var (
// lsmSize has size of the LSM in bytes
lsmSize *expvar.Map
// vlogSize has size of the value log in bytes
vlogSize *expvar.Map
// pendingWrites tracks the number of pending writes.
pendingWrites *expvar.Map
// These are cumulative
// numReads has cumulative number of reads
numReads *expvar.Int
// numWrites has cumulative number of writes
numWrites *expvar.Int
// numBytesRead has cumulative number of bytes read
numBytesRead *expvar.Int
// numBytesWritten has cumulative number of bytes written
numBytesWritten *expvar.Int
// numLSMGets is number of LMS gets
numLSMGets *expvar.Map
// numLSMBloomHits is number of LMS bloom hits
numLSMBloomHits *expvar.Map
// numGets is number of gets
numGets *expvar.Int
// numPuts is number of puts
numPuts *expvar.Int
// numBlockedPuts is number of blocked puts
numBlockedPuts *expvar.Int
// numMemtableGets is number of memtable gets
numMemtableGets *expvar.Int
// numCompactionTables is the number of tables being compacted
numCompactionTables *expvar.Int
)
// These variables are global and have cumulative values for all kv stores.
func init() {
numReads = expvar.NewInt("badger_v3_disk_reads_total")
numWrites = expvar.NewInt("badger_v3_disk_writes_total")
numBytesRead = expvar.NewInt("badger_v3_read_bytes")
numBytesWritten = expvar.NewInt("badger_v3_written_bytes")
numLSMGets = expvar.NewMap("badger_v3_lsm_level_gets_total")
numLSMBloomHits = expvar.NewMap("badger_v3_lsm_bloom_hits_total")
numGets = expvar.NewInt("badger_v3_gets_total")
numPuts = expvar.NewInt("badger_v3_puts_total")
numBlockedPuts = expvar.NewInt("badger_v3_blocked_puts_total")
numMemtableGets = expvar.NewInt("badger_v3_memtable_gets_total")
lsmSize = expvar.NewMap("badger_v3_lsm_size_bytes")
vlogSize = expvar.NewMap("badger_v3_vlog_size_bytes")
pendingWrites = expvar.NewMap("badger_v3_pending_writes_total")
numCompactionTables = expvar.NewInt("badger_v3_compactions_current")
}
func NumReadsAdd(enabled bool, val int64) {
addInt(enabled, numReads, val)
}
func NumWritesAdd(enabled bool, val int64) {
addInt(enabled, numWrites, val)
}
func NumBytesReadAdd(enabled bool, val int64) {
addInt(enabled, numBytesRead, val)
}
func NumBytesWrittenAdd(enabled bool, val int64) {
addInt(enabled, numBytesWritten, val)
}
func NumGetsAdd(enabled bool, val int64) {
addInt(enabled, numGets, val)
}
func NumPutsAdd(enabled bool, val int64) {
addInt(enabled, numPuts, val)
}
func NumBlockedPutsAdd(enabled bool, val int64) {
addInt(enabled, numBlockedPuts, val)
}
func NumMemtableGetsAdd(enabled bool, val int64) {
addInt(enabled, numMemtableGets, val)
}
func NumCompactionTablesAdd(enabled bool, val int64) {
addInt(enabled, numCompactionTables, val)
}
func LSMSizeSet(enabled bool, key string, val expvar.Var) {
storeToMap(enabled, lsmSize, key, val)
}
func VlogSizeSet(enabled bool, key string, val expvar.Var) {
storeToMap(enabled, vlogSize, key, val)
}
func PendingWritesSet(enabled bool, key string, val expvar.Var) {
storeToMap(enabled, pendingWrites, key, val)
}
func NumLSMBloomHitsAdd(enabled bool, key string, val int64) {
addToMap(enabled, numLSMBloomHits, key, val)
}
func NumLSMGetsAdd(enabled bool, key string, val int64) {
addToMap(enabled, numLSMGets, key, val)
}
func LSMSizeGet(enabled bool, key string) expvar.Var {
return getFromMap(enabled, lsmSize, key)
}
func VlogSizeGet(enabled bool, key string) expvar.Var {
return getFromMap(enabled, vlogSize, key)
}
func addInt(enabled bool, metric *expvar.Int, val int64) {
if !enabled {
return
}
metric.Add(val)
}
func addToMap(enabled bool, metric *expvar.Map, key string, val int64) {
if !enabled {
return
}
metric.Add(key, val)
}
func storeToMap(enabled bool, metric *expvar.Map, key string, val expvar.Var) {
if !enabled {
return
}
metric.Set(key, val)
}
func getFromMap(enabled bool, metric *expvar.Map, key string) expvar.Var {
if !enabled {
return nil
}
return metric.Get(key)
}
vendor/github.com/dgraph-io/badger/v3/.deepsource.toml → vendor/github.com/dgraph-io/badger/v4/.deepsource.toml
Generated Vendored
View File
vendor/github.com/dgraph-io/badger/v3/.gitignore → vendor/github.com/dgraph-io/badger/v4/.gitignore
Generated
View File
vendor/github.com/dgraph-io/badger/v4/.golangci.yml
Generated Vendored
+32
View File
@@ -0,0 +1,32 @@
run:
skip-dirs:
skip-files:
linters-settings:
lll:
line-length: 120
staticcheck:
checks:
- all
- '-SA1019' # it is okay to use math/rand at times.
gosec:
excludes: # these are not relevant for us right now
- G114
- G204
- G306
- G404
linters:
disable-all: true
enable:
- errcheck
- gofmt
- goimports
- gosec
- gosimple
- govet
- ineffassign
- lll
- staticcheck
- unconvert
- unused
vendor/github.com/dgraph-io/badger/v3/CHANGELOG.md → vendor/github.com/dgraph-io/badger/v4/CHANGELOG.md
Generated Vendored
+229 -7
View File
@@ -3,6 +3,205 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
## [4.4.0] - 2024-10-26
- retract v4.3.0 due to #2121 and #2113, upgrade to Go v1.23, use ristretto v2 in https://github.com/dgraph-io/badger/pull/2122
- Allow stream custom maxsize per batch in https://github.com/dgraph-io/badger/pull/2063
- chore(deps): bump github.com/klauspost/compress from 1.17.10 to 1.17.11 in the patch group in https://github.com/dgraph-io/badger/pull/2120
- fix: sentinel errors should not have stack traces in https://github.com/dgraph-io/badger/pull/2042
- chore(deps): bump the minor group with 2 updates in https://github.com/dgraph-io/badger/pull/2119
**Full Changelog**: https://github.com/dgraph-io/badger/compare/v4.3.1...v4.4.0
## [4.3.1] - 2024-10-06
- chore: update docs links by @ryanfoxtyler in https://github.com/dgraph-io/badger/pull/2097
- chore(deps): bump golang.org/x/sys from 0.24.0 to 0.25.0 in the minor group by @dependabot in https://github.com/dgraph-io/badger/pull/2100
- chore(deps): bump golang.org/x/net from 0.28.0 to 0.29.0 in the minor group by @dependabot in https://github.com/dgraph-io/badger/pull/2106
- fix: fix reverse iterator broken by seek by @harshil-goel in https://github.com/dgraph-io/badger/pull/2109
- chore(deps): bump github.com/klauspost/compress from 1.17.9 to 1.17.10 in the patch group by @dependabot in https://github.com/dgraph-io/badger/pull/2114
- chore(deps): bump github.com/dgraph-io/ristretto from 0.1.2-0.20240116140435-c67e07994f91 to 1.0.0 by @dependabot in https://github.com/dgraph-io/badger/pull/2112
**Full Changelog**: https://github.com/dgraph-io/badger/compare/v4.3.0...v4.3.1
## [4.3.0] - 2024-08-29
> **Warning**
> The tag v4.3.0 has been retracted due to an issue go.sum.
> Use v4.3.1 (see #2121 and #2113)
### Fixes
- chore(changelog): add a missed entry in CHANGELOG for v4.2.0 by @mangalaman93 in https://github.com/dgraph-io/badger/pull/1988
- update README with project KVS using badger by @tauraamui in https://github.com/dgraph-io/badger/pull/1989
- fix edge case for watermark when index is zero by @mangalaman93 in https://github.com/dgraph-io/badger/pull/1999
- upgrade spf13/cobra to version v1.7.0 by @mangalaman93 in https://github.com/dgraph-io/badger/pull/2001
- chore: update readme by @joshua-goldstein in https://github.com/dgraph-io/badger/pull/2011
- perf: upgrade compress package test and benchmark. by @siddhant2001 in https://github.com/dgraph-io/badger/pull/2009
- fix(Transactions): Fix resource consumption on empty write transaction by @Zach-Johnson in https://github.com/dgraph-io/badger/pull/2018
- chore(deps): bump golang.org/x/net from 0.7.0 to 0.17.0 by @dependabot in https://github.com/dgraph-io/badger/pull/2017
- perf(compactor): optimize allocations: use buffer for priorities by @deff7 in https://github.com/dgraph-io/badger/pull/2006
- fix(Transaction): discard empty transactions on CommitWith by @Wondertan in https://github.com/dgraph-io/badger/pull/2031
- fix(levelHandler): use lock for levelHandler sort tables instead of rlock by @xgzlucario in https://github.com/dgraph-io/badger/pull/2034
- Docs: update README with project LLS using badger by @Boc-chi-no in https://github.com/dgraph-io/badger/pull/2032
- chore: MaxTableSize has been renamed to BaseTableSize by @mitar in https://github.com/dgraph-io/badger/pull/2038
- Update CODEOWNERS by @ryanfoxtyler in https://github.com/dgraph-io/badger/pull/2043
- Chore(): add Stale Action by @ryanfoxtyler in https://github.com/dgraph-io/badger/pull/2070
- Update ristretto and refactor for use of generics by @paralin in https://github.com/dgraph-io/badger/pull/2047
- chore: Remove obsolete comment by @mitar in https://github.com/dgraph-io/badger/pull/2039
- chore(Docs): Update jQuery 3.2.1 to 3.7.1 by @kokizzu in https://github.com/dgraph-io/badger/pull/2023
- chore(deps): bump the go_modules group with 3 updates by @dependabot in https://github.com/dgraph-io/badger/pull/2074
- docs(): update docs path by @ryanfoxtyler in https://github.com/dgraph-io/badger/pull/2076
- perf: fix operation in seek by @harshil-goel in https://github.com/dgraph-io/badger/pull/2077
- Add lakeFS to README.md by @N-o-Z in https://github.com/dgraph-io/badger/pull/2078
- chore(): add Dependabot by @ryanfoxtyler in https://github.com/dgraph-io/badger/pull/2080
- chore(deps): bump golangci/golangci-lint-action from 4 to 6 by @dependabot in https://github.com/dgraph-io/badger/pull/2083
- chore(deps): bump actions/upload-artifact from 3 to 4 by @dependabot in https://github.com/dgraph-io/badger/pull/2081
- chore(deps): bump github/codeql-action from 2 to 3 by @dependabot in https://github.com/dgraph-io/badger/pull/2082
- chore(deps): bump the minor group with 7 updates by @dependabot in https://github.com/dgraph-io/badger/pull/2089
- Action Manager by @madhu72 in https://github.com/dgraph-io/badger/pull/2050
- chore(deps): bump golang.org/x/sys from 0.23.0 to 0.24.0 in the minor group by @dependabot in https://github.com/dgraph-io/badger/pull/2091
- chore(deps): bump github.com/golang/protobuf from 1.5.3 to 1.5.4 in the patch group by @dependabot in https://github.com/dgraph-io/badger/pull/2090
- chore: fix some comments by @dufucun in https://github.com/dgraph-io/badger/pull/2092
- chore(deps): bump github.com/google/flatbuffers from 1.12.1 to 24.3.25+incompatible by @dependabot in https://github.com/dgraph-io/badger/pull/2084
### CI
- ci: change cron frequency to fix ghost jobs by @joshua-goldstein in https://github.com/dgraph-io/badger/pull/2010
- fix(CI): Update to pull_request trigger by @ryanfoxtyler in https://github.com/dgraph-io/badger/pull/2056
- ci/cd optimization by @ryanfoxtyler in https://github.com/dgraph-io/badger/pull/2051
- fix(cd): fixed cd pipeline by @harshil-goel in https://github.com/dgraph-io/badger/pull/2093
- fix(cd): change name by @harshil-goel in https://github.com/dgraph-io/badger/pull/2094
- fix(cd): added more debug things to cd by @harshil-goel in https://github.com/dgraph-io/badger/pull/2095
- fix(cd): removing some debug items by @harshil-goel in https://github.com/dgraph-io/badger/pull/2096
**Full Changelog**: https://github.com/dgraph-io/badger/compare/v4.2.0...v4.3.0
## [4.2.0] - 2023-08-03
### Breaking
- feat(metrics): fix and update metrics in badger (#1948)
- fix(metrics): remove badger version in the metrics name (#1982)
### Fixed
- fix(db): avoid panic in parallel reads after closing DB (#1987)
- fix(logging): fix direct access to logger (#1980)
- fix(sec): bump google.golang.org/grpc from 1.20.1 to 1.53.0 (#1977)
- fix(sec): update gopkg.in/yaml.v2 package (#1969)
- fix(test): fix flakiness of TestPersistLFDiscardStats (#1963)
- fix(stream): setup oracle correctly in stream writer (#1968) (#1904)
- fix(stream): improve incremental stream writer (#1901)
- fix(test): improve the params in BenchmarkDbGrowth (#1967)
- fix(sync): sync active memtable and value log on Db.Sync (#1847) (#1953)
- fix(test): handle draining of closed channel, speed up test. (#1957)
- fix(test): fix table checksum test. Test on uncompressed. (#1952)
- fix(level): change split key range right key to use ts=0 (#1932)
- fix(test): the new test case PagebufferReader5 introduced an error. (#1936)
- fix(test): add missing unlock in TestPersistLFDiscardStats (#1924)
- fix(PageBufferReader): should conform to io.Reader interface (#1935)
- fix(publisher): publish updates after persistence in WAL (#1917)
### CI
- chore(ci): split off coverage workflow (#1944)
- chore(ci): adding trivy scanning workflow (#1925)
## [4.1.0] - 2023-03-30
This release adds support for incremental stream writer. We also do some cleanup in the docs
and resolve some CI issues for community PR's. We resolve high and medium CVE's and fix
[#1833](https://github.com/dgraph-io/badger/issues/1833).
### Features
- feat(stream): add support for incremental stream writer (#1722) (#1874)
### Fixes
- chore: upgrade xxhash from v1.1.0 to v2.1.2 (#1910) (fixes [#1833](https://github.com/dgraph-io/badger/issues/1833))
### Security
- chore(deps): bump golang.org/x/net from 0.0.0-20201021035429-f5854403a974 to 0.7.0 (#1885)
### CVE's
- [CVE-2021-31525](https://github.com/dgraph-io/badger/security/dependabot/7)
- [CVE-2022-41723](https://github.com/dgraph-io/badger/security/dependabot/4)
- [CVE-2022-27664](https://github.com/dgraph-io/badger/security/dependabot/5)
- [CVE-2021-33194](https://github.com/dgraph-io/badger/security/dependabot/9)
- [CVE-2022-41723](https://github.com/dgraph-io/badger/security/dependabot/13)
- [CVE-2021-33194](https://github.com/dgraph-io/badger/security/dependabot/16)
- [CVE-2021-38561](https://github.com/dgraph-io/badger/security/dependabot/8)
### Chores
- fix(docs): update README (#1915)
- cleanup sstable file after tests (#1912)
- chore(ci): add dgraph regression tests (#1908)
- docs: fix the default value in docs (#1909)
- chore: update URL for unsupported manifest version error (#1905)
- docs(README): add raft-badger to projects using badger (#1902)
- sync the docs with README with projects using badger (#1903)
- fix: update code comments for WithNumCompactors (#1900)
- docs: add loggie to projects using badger (#1882)
- chore(memtable): refactor code for memtable flush (#1866)
- resolve coveralls issue for community PR's (#1892, #1894, #1896)
## [4.0.1] - 2023-02-28
We issue a follow up release in order to resolve a bug in subscriber. We also generate updated protobufs for Badger v4.
### Fixed
- fix(pb): fix generated protos #1888
- fix(publisher): initialize the atomic variable #1889
### Chores
- chore(cd): tag based deployments #1887
- chore(ci): fail fast when testing #1890
## [4.0.0] - 2023-02-27
> **Warning**
> The tag v4.0.0 has been retracted due to a bug in publisher.
> Use v4.0.1 (see #1889)
This release fixes a bug in the maxHeaderSize parameter that could lead
to panics. We introduce an external magic number to keep track of external
dependencies. We bump up the minimum required Go version to 1.19. No changes
were made to the format of data on disk. This is a major release because
we are making a switch to SemVer in order to make it easier for the community
to understand when breaking API and data format changes are made.
### Fixed
- fix: update maxHeaderSize #1877
- feat(externalMagic): Introduce external magic number (#1745) #1852
- fix(bench): bring in benchmark fixes from main #1863
### Chores
- upgrade go to 1.19 #1868
- enable linters (gosimple, govet, lll, unused, staticcheck, errcheck, ineffassign, gofmt) #1871 #1870 #1876
- remove dependency on io/ioutil #1879
- various doc and comment fixes #1857
- moving from CalVer to SemVer
## [3.2103.5] - 2022-12-15
We release Badger CLI tool binaries for amd64 and now arm64. This release does not involve any core code changes to Badger. We add a CD job for building Badger for arm64.
## [3.2103.4] - 2022-11-04
### Fixed
- fix(manifest): fix manifest corruption due to race condition in concurrent compactions (#1756)
### Chores
- We bring the release branch to parity with main by updating the CI/CD jobs, Readme, Codeowners, PR and issue templates, etc.
## [3.2103.3] - 2022-10-14
### Remarks
@@ -199,6 +398,29 @@ This release is not backward compatible with Badger v2.x.x
- WithTruncate (#1555)
- WithValueLogLoadingMode (#1555)
## [2.2007.4] - 2021-08-25
### Fixed
- Fix build on Plan 9 (#1451) (#1508) (#1738)
### Features
- feat(zstd): backport replacement of DataDog's zstd with Klauspost's zstd (#1736)
## [2.2007.3] - 2021-07-21
### Fixed
- fix(maxVersion): Use choosekey instead of KeyToList (#1532) #1533
- fix(flatten): Add --num_versions flag (#1518) #1520
- fix(build): Fix integer overflow on 32-bit architectures #1558
- fix(pb): avoid protobuf warning due to common filename (#1519)
### Features
- Add command to stream contents of DB into another DB. (#1486)
### New APIs
- DB.StreamDB
- DB.MaxVersion
## [2.2007.2] - 2020-08-31
### Fixed
@@ -363,17 +585,17 @@ This sections lists the changes which were reverted because of non-reproducible
- Improve write stalling on level 0 and 1. (#1186)
- Disable compression and set ZSTD Compression Level to 1. (#1191)
## [2.0.1] - 2020-01-02
## [2.0.1] - 2020-01-02
### New APIs
- badger.Options
- WithInMemory (f5b6321)
- WithZSTDCompressionLevel (3eb4e72)
- Badger.TableInfo
- EstimatedSz (f46f8ea)
### Features
- Introduce in-memory mode in badger. (#1113)
@@ -386,12 +608,12 @@ This sections lists the changes which were reverted because of non-reproducible
- Fix windows dataloss issue. (#1134)
- Fix VerifyValueChecksum checks. (#1138)
- Fix encryption in stream writer. (#1146)
- Fix segmentation fault in vlog.Read. (header.Decode) (#1150)
- Fix segmentation fault in vlog.Read. (header.Decode) (#1150)
- Fix merge iterator duplicates issue. (#1157)
### Performance
- Set level 15 as default compression level in Zstd. (#1111)
- Set level 15 as default compression level in Zstd. (#1111)
- Optimize createTable in stream_writer.go. (#1132)
## [2.0.0] - 2019-11-12
@@ -413,7 +635,7 @@ This sections lists the changes which were reverted because of non-reproducible
- WithEncryptionKey (a425b0e)
- WithEncryptionKeyRotationDuration (a425b0e)
- WithChecksumVerificationMode (7b4083d)
### Features
- Data cache to speed up lookups and iterations. (#1066)
@@ -493,7 +715,7 @@ _Note_: The hashes in parentheses correspond to the commits that impacted the gi
- NewEntry
- WithMeta
- WithDiscard
- WithTTL
- WithTTL
- badger.Item
- KeySize (fd59907)
vendor/github.com/dgraph-io/badger/v3/CODE_OF_CONDUCT.md → vendor/github.com/dgraph-io/badger/v4/CODE_OF_CONDUCT.md
Generated Vendored
View File
vendor/github.com/dgraph-io/badger/v3/CONTRIBUTING.md → vendor/github.com/dgraph-io/badger/v4/CONTRIBUTING.md
Generated Vendored
View File
vendor/github.com/dgraph-io/badger/v3/LICENSE → vendor/github.com/dgraph-io/badger/v4/LICENSE
Generated Vendored
View File
vendor/github.com/dgraph-io/badger/v3/Makefile → vendor/github.com/dgraph-io/badger/v4/Makefile
Generated Vendored
-1
View File
@@ -50,7 +50,6 @@ jemalloc:
dependency:
@echo "Installing dependencies..."
@sudo apt-get update
@sudo apt-get -y upgrade
@sudo apt-get -y install \
ca-certificates \
curl \
vendor/github.com/dgraph-io/badger/v3/README.md → vendor/github.com/dgraph-io/badger/v4/README.md
Generated Vendored
+36 -25
View File
@@ -1,7 +1,7 @@
# BadgerDB
# BadgerDB
[![Go Reference](https://pkg.go.dev/badge/github.com/dgraph-io/badger/v3.svg)](https://pkg.go.dev/github.com/dgraph-io/badger/v3)
[![Go Report Card](https://goreportcard.com/badge/github.com/dgraph-io/badger/v3)](https://goreportcard.com/report/github.com/dgraph-io/badger/v3)
[![Go Reference](https://pkg.go.dev/badge/github.com/dgraph-io/badger/v4.svg)](https://pkg.go.dev/github.com/dgraph-io/badger/v4)
[![Go Report Card](https://goreportcard.com/badge/github.com/dgraph-io/badger/v4)](https://goreportcard.com/report/github.com/dgraph-io/badger/v4)
[![Sourcegraph](https://sourcegraph.com/github.com/dgraph-io/badger/-/badge.svg)](https://sourcegraph.com/github.com/dgraph-io/badger?badge)
[![ci-badger-tests](https://github.com/dgraph-io/badger/actions/workflows/ci-badger-tests.yml/badge.svg)](https://github.com/dgraph-io/badger/actions/workflows/ci-badger-tests.yml)
[![ci-badger-bank-tests](https://github.com/dgraph-io/badger/actions/workflows/ci-badger-bank-tests.yml/badge.svg)](https://github.com/dgraph-io/badger/actions/workflows/ci-badger-bank-tests.yml)
@@ -39,43 +39,46 @@ Please consult the [Changelog] for more detailed information on releases.
For more details on our version naming schema please read [Choosing a version](#choosing-a-version).
[Changelog]:https://github.com/dgraph-io/badger/blob/master/CHANGELOG.md
[Changelog]:https://github.com/dgraph-io/badger/blob/main/CHANGELOG.md
## Table of Contents
* [Getting Started](#getting-started)
+ [Installing](#installing)
- [BadgerDB](#badgerdb)
- [Project Status](#project-status)
- [Table of Contents](#table-of-contents)
- [Getting Started](#getting-started)
- [Installing](#installing)
- [Installing Badger Command Line Tool](#installing-badger-command-line-tool)
- [Choosing a version](#choosing-a-version)
* [Badger Documentation](#badger-documentation)
* [Resources](#resources)
+ [Blog Posts](#blog-posts)
* [Design](#design)
+ [Comparisons](#comparisons)
+ [Benchmarks](#benchmarks)
* [Projects Using Badger](#projects-using-badger)
* [Contributing](#contributing)
* [Contact](#contact)
- [Badger Documentation](#badger-documentation)
- [Resources](#resources)
- [Blog Posts](#blog-posts)
- [Design](#design)
- [Comparisons](#comparisons)
- [Benchmarks](#benchmarks)
- [Projects Using Badger](#projects-using-badger)
- [Contributing](#contributing)
- [Contact](#contact)
## Getting Started
### Installing
To start using Badger, install Go 1.12 or above. Badger v3 needs go modules. From your project, run the following command
To start using Badger, install Go 1.21 or above. Badger v3 and above needs go modules. From your project, run the following command
```sh
$ go get github.com/dgraph-io/badger/v3
$ go get github.com/dgraph-io/badger/v4
```
This will retrieve the library.
#### Installing Badger Command Line Tool
Badger provides a CLI tool which can perform certain operations like offline backup/restore. To install the Badger CLI,
Badger provides a CLI tool which can perform certain operations like offline backup/restore. To install the Badger CLI,
retrieve the repository and checkout the desired version. Then run
```sh
$ cd badger
$ go install .
```
This will install the badger command line utility into your $GOBIN path.
This will install the badger command line utility into your $GOBIN path.
#### Choosing a version
@@ -176,7 +179,6 @@ Below is a list of known projects that use Badger:
* [Sandglass](https://github.com/celrenheit/sandglass) - distributed, horizontally scalable, persistent, time sorted message queue.
* [TalariaDB](https://github.com/grab/talaria) - Grab's Distributed, low latency time-series database.
* [Sloop](https://github.com/salesforce/sloop) - Salesforce's Kubernetes History Visualization Project.
* [Immudb](https://github.com/codenotary/immudb) - Lightweight, high-speed immutable database for systems and applications.
* [Usenet Express](https://usenetexpress.com/) - Serving over 300TB of data with Badger.
* [gorush](https://github.com/appleboy/gorush) - A push notification server written in Go.
* [0-stor](https://github.com/zero-os/0-stor) - Single device object store.
@@ -196,11 +198,11 @@ Below is a list of known projects that use Badger:
* [BadgerHold](https://github.com/timshannon/badgerhold) - An embeddable NoSQL store for querying Go types built on Badger
* [Goblero](https://github.com/didil/goblero) - Pure Go embedded persistent job queue backed by BadgerDB
* [Surfline](https://www.surfline.com) - Serving global wave and weather forecast data with Badger.
* [Cete](https://github.com/mosuka/cete) - Simple and highly available distributed key-value store built on Badger. Makes it easy bringing up a cluster of Badger with Raft consensus algorithm by hashicorp/raft.
* [Cete](https://github.com/mosuka/cete) - Simple and highly available distributed key-value store built on Badger. Makes it easy bringing up a cluster of Badger with Raft consensus algorithm by hashicorp/raft.
* [Volument](https://volument.com/) - A new take on website analytics backed by Badger.
* [KVdb](https://kvdb.io/) - Hosted key-value store and serverless platform built on top of Badger.
* [Terminotes](https://gitlab.com/asad-awadia/terminotes) - Self hosted notes storage and search server - storage powered by BadgerDB
* [Pyroscope](https://github.com/pyroscope-io/pyroscope) - Open source confinuous profiling platform built with BadgerDB
* [Pyroscope](https://github.com/pyroscope-io/pyroscope) - Open source continuous profiling platform built with BadgerDB
* [Veri](https://github.com/bgokden/veri) - A distributed feature store optimized for Search and Recommendation tasks.
* [bIter](https://github.com/MikkelHJuul/bIter) - A library and Iterator interface for working with the `badger.Iterator`, simplifying from-to, and prefix mechanics.
* [ld](https://github.com/MikkelHJuul/ld) - (Lean Database) A very simple gRPC-only key-value database, exposing BadgerDB with key-range scanning semantics.
@@ -211,6 +213,15 @@ Below is a list of known projects that use Badger:
* [vxdb](https://github.com/vitalvas/vxdb) - Simple schema-less Key-Value NoSQL database with simplest API interface.
* [Opacity](https://github.com/opacity/storage-node) - Backend implementation for the Opacity storage project
* [Vephar](https://github.com/vaccovecrana/vephar) - A minimal key/value store using hashicorp-raft for cluster coordination and Badger for data storage.
* [gowarcserver](https://github.com/nlnwa/gowarcserver) - Open-source server for warc files. Can be used in conjunction with pywb
* [flow-go](https://github.com/onflow/flow-go) - A fast, secure, and developer-friendly blockchain built to support the next generation of games, apps and the digital assets that power them.
* [Wrgl](https://www.wrgl.co) - A data version control system that works like Git but specialized to store and diff CSV.
* [Loggie](https://github.com/loggie-io/loggie) - A lightweight, cloud-native data transfer agent and aggregator.
* [raft-badger](https://github.com/rfyiamcool/raft-badger) - raft-badger implements LogStore and StableStore Interface of hashcorp/raft. it is used to store raft log and metadata of hashcorp/raft.
* [DVID](https://github.com/janelia-flyem/dvid) - A dataservice for branched versioning of a variety of data types. Originally created for large-scale brain reconstructions in Connectomics.
* [KVS](https://github.com/tauraamui/kvs) - A library for making it easy to persist, load and query full structs into BadgerDB, using an ownership hierarchy model.
* [LLS](https://github.com/Boc-chi-no/LLS) - LLS is an efficient URL Shortener that can be used to shorten links and track link usage. Support for BadgerDB and MongoDB. Improved performance by more than 30% when using BadgerDB
* [lakeFS](https://github.com/treeverse/lakeFS) - lakeFS is an open-source data version control that transforms your object storage to Git-like repositories. lakeFS uses BadgerDB for its underlying local metadata KV store implementation.
If you are using Badger in a project please send a pull request to add it to the list.
@@ -219,6 +230,6 @@ If you are using Badger in a project please send a pull request to add it to the
If you're interested in contributing to Badger see [CONTRIBUTING](./CONTRIBUTING.md).
## Contact
- Please use [discuss.dgraph.io](https://discuss.dgraph.io) for questions, feature requests and discussions.
- Please use [discuss.dgraph.io](https://discuss.dgraph.io) for filing bugs or feature requests.
- Follow us on Twitter [@dgraphlabs](https://twitter.com/dgraphlabs).
- Please use [Github issues](https://github.com/dgraph-io/badger/issues) for filing bugs.
- Please use [discuss.dgraph.io](https://discuss.dgraph.io) for questions, discussions, and feature requests.
- Follow us on Twitter [@dgraphlabs](https://twitter.com/dgraphlabs).
vendor/github.com/dgraph-io/badger/v3/VERSIONING.md → vendor/github.com/dgraph-io/badger/v4/VERSIONING.md
Generated Vendored
+1 -1
View File
@@ -43,5 +43,5 @@ For more background on our decision to adopt Serialization Versioning, read the
[Semantic Versioning, Go Modules, and Databases][blog] and the original proposal on
[this comment on Dgraph's Discuss forum][discuss].
[blog]: https://blog.dgraph.io/post/serialization-versioning/
[blog]: https://open.dgraph.io/post/serialization-versioning/
[discuss]: https://discuss.dgraph.io/t/go-modules-on-badger-and-dgraph/4662/7
vendor/github.com/dgraph-io/badger/v3/backup.go → vendor/github.com/dgraph-io/badger/v4/backup.go
Generated Vendored
+6 -6
View File
@@ -23,11 +23,12 @@ import (
"encoding/binary"
"io"
"github.com/dgraph-io/badger/v3/pb"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/ristretto/z"
"github.com/golang/protobuf/proto"
"github.com/pkg/errors"
"google.golang.org/protobuf/proto"
"github.com/dgraph-io/badger/v4/pb"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2/z"
)
// flushThreshold determines when a buffer will be flushed. When performing a
@@ -78,8 +79,7 @@ func (stream *Stream) Backup(w io.Writer, since uint64) (uint64, error) {
var valCopy []byte
if !item.IsDeletedOrExpired() {
// No need to copy value, if item is deleted or expired.
var err error
err = item.Value(func(val []byte) error {
err := item.Value(func(val []byte) error {
valCopy = a.Copy(val)
return nil
})
vendor/github.com/dgraph-io/badger/v3/batch.go → vendor/github.com/dgraph-io/badger/v4/batch.go
Generated Vendored
+6 -4
View File
@@ -20,10 +20,12 @@ import (
"sync"
"sync/atomic"
"github.com/dgraph-io/badger/v3/pb"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/ristretto/z"
"github.com/pkg/errors"
"google.golang.org/protobuf/proto"
"github.com/dgraph-io/badger/v4/pb"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2/z"
)
// WriteBatch holds the necessary info to perform batched writes.
@@ -112,7 +114,7 @@ func (wb *WriteBatch) Write(buf *z.Buffer) error {
err := buf.SliceIterate(func(s []byte) error {
kv := &pb.KV{}
if err := kv.Unmarshal(s); err != nil {
if err := proto.Unmarshal(s, kv); err != nil {
return err
}
return wb.writeKV(kv)
vendor/github.com/dgraph-io/badger/v3/changes.sh → vendor/github.com/dgraph-io/badger/v4/changes.sh
Generated Vendored
View File
vendor/github.com/dgraph-io/badger/v3/compaction.go → vendor/github.com/dgraph-io/badger/v4/compaction.go
Generated Vendored
+2 -2
View File
@@ -23,8 +23,8 @@ import (
"math"
"sync"
"github.com/dgraph-io/badger/v3/table"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/badger/v4/table"
"github.com/dgraph-io/badger/v4/y"
)
type keyRange struct {
vendor/github.com/dgraph-io/badger/v3/db.go → vendor/github.com/dgraph-io/badger/v4/db.go
Generated Vendored
+151 -120
View File
@@ -20,6 +20,7 @@ import (
"bytes"
"context"
"encoding/binary"
stderrors "errors"
"expvar"
"fmt"
"math"
@@ -31,15 +32,17 @@ import (
"sync/atomic"
"time"
"github.com/dgraph-io/badger/v3/options"
"github.com/dgraph-io/badger/v3/pb"
"github.com/dgraph-io/badger/v3/skl"
"github.com/dgraph-io/badger/v3/table"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/ristretto"
"github.com/dgraph-io/ristretto/z"
humanize "github.com/dustin/go-humanize"
"github.com/pkg/errors"
"github.com/dgraph-io/badger/v4/fb"
"github.com/dgraph-io/badger/v4/options"
"github.com/dgraph-io/badger/v4/pb"
"github.com/dgraph-io/badger/v4/skl"
"github.com/dgraph-io/badger/v4/table"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2"
"github.com/dgraph-io/ristretto/v2/z"
)
var (
@@ -48,10 +51,6 @@ var (
bannedNsKey = []byte("!badger!banned") // For storing the banned namespaces.
)
const (
maxNumSplits = 128
)
type closers struct {
updateSize *z.Closer
compactors *z.Closer
@@ -93,6 +92,8 @@ func (lk *lockedKeys) all() []uint64 {
// DB provides the various functions required to interact with Badger.
// DB is thread-safe.
type DB struct {
testOnlyDBExtensions
lock sync.RWMutex // Guards list of inmemory tables, not individual reads and writes.
dirLockGuard *directoryLockGuard
@@ -112,11 +113,11 @@ type DB struct {
lc *levelsController
vlog valueLog
writeCh chan *request
flushChan chan flushTask // For flushing memtables.
flushChan chan *memTable // For flushing memtables.
closeOnce sync.Once // For closing DB only once.
blockWrites int32
isClosed uint32
blockWrites atomic.Int32
isClosed atomic.Uint32
orc *oracle
bannedNamespaces *lockedKeys
@@ -124,8 +125,8 @@ type DB struct {
pub *publisher
registry *KeyRegistry
blockCache *ristretto.Cache
indexCache *ristretto.Cache
blockCache *ristretto.Cache[[]byte, *table.Block]
indexCache *ristretto.Cache[uint64, *fb.TableIndex]
allocPool *z.AllocatorPool
}
@@ -163,10 +164,10 @@ func checkAndSetOptions(opt *Options) error {
// the transaction APIs. Transaction batches entries into batches of size opt.maxBatchSize.
if opt.ValueThreshold > opt.maxBatchSize {
return errors.Errorf("Valuethreshold %d greater than max batch size of %d. Either "+
"reduce opt.ValueThreshold or increase opt.MaxTableSize.",
"reduce opt.ValueThreshold or increase opt.BaseTableSize.",
opt.ValueThreshold, opt.maxBatchSize)
}
// ValueLogFileSize should be stricly LESS than 2<<30 otherwise we will
// ValueLogFileSize should be strictly LESS than 2<<30 otherwise we will
// overflow the uint32 when we mmap it in OpenMemtable.
if !(opt.ValueLogFileSize < 2<<30 && opt.ValueLogFileSize >= 1<<20) {
return ErrValueLogSize
@@ -243,7 +244,7 @@ func Open(opt Options) (*DB, error) {
db := &DB{
imm: make([]*memTable, 0, opt.NumMemtables),
flushChan: make(chan flushTask, opt.NumMemtables),
flushChan: make(chan *memTable, opt.NumMemtables),
writeCh: make(chan *request, kvWriteChCapacity),
opt: opt,
manifest: manifestFile,
@@ -255,6 +256,9 @@ func Open(opt Options) (*DB, error) {
bannedNamespaces: &lockedKeys{keys: make(map[uint64]struct{})},
threshold: initVlogThreshold(&opt),
}
db.syncChan = opt.syncChan
// Cleanup all the goroutines started by badger in case of an error.
defer func() {
if err != nil {
@@ -272,14 +276,14 @@ func Open(opt Options) (*DB, error) {
numInCache = 1
}
config := ristretto.Config{
config := ristretto.Config[[]byte, *table.Block]{
NumCounters: numInCache * 8,
MaxCost: opt.BlockCacheSize,
BufferItems: 64,
Metrics: true,
OnExit: table.BlockEvictHandler,
}
db.blockCache, err = ristretto.NewCache(&config)
db.blockCache, err = ristretto.NewCache[[]byte, *table.Block](&config)
if err != nil {
return nil, y.Wrap(err, "failed to create data cache")
}
@@ -295,7 +299,7 @@ func Open(opt Options) (*DB, error) {
numInCache = 1
}
config := ristretto.Config{
config := ristretto.Config[uint64, *fb.TableIndex]{
NumCounters: numInCache * 8,
MaxCost: opt.IndexCacheSize,
BufferItems: 64,
@@ -354,11 +358,11 @@ func Open(opt Options) (*DB, error) {
db.closers.memtable = z.NewCloser(1)
go func() {
_ = db.flushMemtable(db.closers.memtable) // Need levels controller to be up.
db.flushMemtable(db.closers.memtable) // Need levels controller to be up.
}()
// Flush them to disk asap.
for _, mt := range db.imm {
db.flushChan <- flushTask{mt: mt}
db.flushChan <- mt
}
}
// We do increment nextTxnTs below. So, no need to do it here.
@@ -400,7 +404,7 @@ func Open(opt Options) (*DB, error) {
return db, nil
}
// initBannedNamespaces retrieves the banned namepsaces from the DB and updates in-memory structure.
// initBannedNamespaces retrieves the banned namespaces from the DB and updates in-memory structure.
func (db *DB) initBannedNamespaces() error {
if db.opt.NamespaceOffset < 0 {
return nil
@@ -534,16 +538,17 @@ func (db *DB) Close() error {
// IsClosed denotes if the badger DB is closed or not. A DB instance should not
// be used after closing it.
func (db *DB) IsClosed() bool {
return atomic.LoadUint32(&db.isClosed) == 1
return db.isClosed.Load() == 1
}
func (db *DB) close() (err error) {
defer db.allocPool.Release()
db.opt.Debugf("Closing database")
db.opt.Infof("Lifetime L0 stalled for: %s\n", time.Duration(atomic.LoadInt64(&db.lc.l0stallsMs)))
db.opt.Infof("Lifetime L0 stalled for: %s\n", time.Duration(db.lc.l0stallsMs.Load()))
atomic.StoreInt32(&db.blockWrites, 1)
db.blockWrites.Store(1)
db.isClosed.Store(1)
if !db.opt.InMemory {
// Stop value GC first.
@@ -571,12 +576,12 @@ func (db *DB) close() (err error) {
} else {
db.opt.Debugf("Flushing memtable")
for {
pushedFlushTask := func() bool {
pushedMemTable := func() bool {
db.lock.Lock()
defer db.lock.Unlock()
y.AssertTrue(db.mt != nil)
select {
case db.flushChan <- flushTask{mt: db.mt}:
case db.flushChan <- db.mt:
db.imm = append(db.imm, db.mt) // Flusher will attempt to remove this from s.imm.
db.mt = nil // Will segfault if we try writing!
db.opt.Debugf("pushed to flush chan\n")
@@ -589,7 +594,7 @@ func (db *DB) close() (err error) {
}
return false
}()
if pushedFlushTask {
if pushedMemTable {
break
}
time.Sleep(10 * time.Millisecond)
@@ -629,7 +634,6 @@ func (db *DB) close() (err error) {
db.blockCache.Close()
db.indexCache.Close()
atomic.StoreUint32(&db.isClosed, 1)
db.threshold.close()
if db.opt.InMemory {
@@ -679,7 +683,38 @@ const (
// Sync syncs database content to disk. This function provides
// more control to user to sync data whenever required.
func (db *DB) Sync() error {
return db.vlog.sync()
/**
Make an attempt to sync both the logs, the active memtable's WAL and the vLog (1847).
Cases:
- All_ok :: If both the logs sync successfully.
- Entry_Lost :: If an entry with a value pointer was present in the active memtable's WAL,
:: and the WAL was synced but there was an error in syncing the vLog.
:: The entry will be considered lost and this case will need to be handled during recovery.
- Entries_Lost :: If there were errors in syncing both the logs, multiple entries would be lost.
- Entries_Lost :: If the active memtable's WAL is not synced but the vLog is synced, it will
:: result in entries being lost because recovery of the active memtable is done from its WAL.
:: Check `UpdateSkipList` in memtable.go.
- Nothing_lost :: If an entry with its value was present in the active memtable's WAL, and the WAL was synced,
:: but there was an error in syncing the vLog.
:: Nothing is lost for this very specific entry because the entry is completely present in the memtable's WAL.
- Partially_lost :: If entries were written partially in either of the logs,
:: the logs will be truncated during recovery.
:: As a result of truncation, some entries might be lost.
:: Assume that 4KB of data is to be synced and invoking `Sync` results only in syncing 3KB
:: of data and then the machine shuts down or the disk failure happens,
:: this will result in partial writes. [[This case needs verification]]
*/
db.lock.RLock()
memtableSyncError := db.mt.SyncWAL()
db.lock.RUnlock()
vLogSyncError := db.vlog.sync()
return y.CombineErrors(memtableSyncError, vLogSyncError)
}
// getMemtables returns the current memtables and get references.
@@ -742,6 +777,7 @@ func (db *DB) get(key []byte) (y.ValueStruct, error) {
}
// Found the required version of the key, return immediately.
if vs.Version == version {
y.NumGetsWithResultsAdd(db.opt.MetricsEnabled, 1)
return vs, nil
}
if maxVs.Version < vs.Version {
@@ -819,8 +855,6 @@ func (db *DB) writeRequests(reqs []*request) error {
return err
}
db.opt.Debugf("Sending updates to subscribers")
db.pub.sendUpdates(reqs)
db.opt.Debugf("Writing to memtable")
var count int
for _, b := range reqs {
@@ -829,6 +863,7 @@ func (db *DB) writeRequests(reqs []*request) error {
}
count += len(b.Entries)
var i uint64
var err error
for err = db.ensureRoomForWrite(); err == errNoRoom; err = db.ensureRoomForWrite() {
i++
if i%100 == 0 {
@@ -848,13 +883,17 @@ func (db *DB) writeRequests(reqs []*request) error {
return y.Wrap(err, "writeRequests")
}
}
db.opt.Debugf("Sending updates to subscribers")
db.pub.sendUpdates(reqs)
done(nil)
db.opt.Debugf("%d entries written", count)
return nil
}
func (db *DB) sendToWriteCh(entries []*Entry) (*request, error) {
if atomic.LoadInt32(&db.blockWrites) == 1 {
if db.blockWrites.Load() == 1 {
return nil, ErrBlockedWrites
}
var count, size int64
@@ -862,11 +901,12 @@ func (db *DB) sendToWriteCh(entries []*Entry) (*request, error) {
size += e.estimateSizeAndSetThreshold(db.valueThreshold())
count++
}
y.NumBytesWrittenUserAdd(db.opt.MetricsEnabled, size)
if count >= db.opt.maxBatchCount || size >= db.opt.maxBatchSize {
return nil, ErrTxnTooBig
}
// We can only service one request because we need each txn to be stored in a contigous section.
// We can only service one request because we need each txn to be stored in a contiguous section.
// Txns should not interleave among other txns or rewrites.
req := requestPool.Get().(*request)
req.reset()
@@ -945,7 +985,8 @@ func (db *DB) doWrites(lc *z.Closer) {
// batchSet applies a list of badger.Entry. If a request level error occurs it
// will be returned.
// Check(kv.BatchSet(entries))
//
// Check(kv.BatchSet(entries))
func (db *DB) batchSet(entries []*Entry) error {
req, err := db.sendToWriteCh(entries)
if err != nil {
@@ -958,9 +999,10 @@ func (db *DB) batchSet(entries []*Entry) error {
// batchSetAsync is the asynchronous version of batchSet. It accepts a callback
// function which is called when all the sets are complete. If a request level
// error occurs, it will be passed back via the callback.
// err := kv.BatchSetAsync(entries, func(err error)) {
// Check(err)
// }
//
// err := kv.BatchSetAsync(entries, func(err error)) {
// Check(err)
// }
func (db *DB) batchSetAsync(entries []*Entry, f func(error)) error {
req, err := db.sendToWriteCh(entries)
if err != nil {
@@ -974,7 +1016,7 @@ func (db *DB) batchSetAsync(entries []*Entry, f func(error)) error {
return nil
}
var errNoRoom = errors.New("No room for write")
var errNoRoom = stderrors.New("No room for write")
// ensureRoomForWrite is always called serially.
func (db *DB) ensureRoomForWrite() error {
@@ -988,7 +1030,7 @@ func (db *DB) ensureRoomForWrite() error {
}
select {
case db.flushChan <- flushTask{mt: db.mt}:
case db.flushChan <- db.mt:
db.opt.Debugf("Flushing memtable, mt.size=%d size of flushChan: %d\n",
db.mt.sl.MemSize(), len(db.flushChan))
// We manage to push this task. Let's modify imm.
@@ -1010,12 +1052,12 @@ func arenaSize(opt Options) int64 {
}
// buildL0Table builds a new table from the memtable.
func buildL0Table(ft flushTask, bopts table.Options) *table.Builder {
iter := ft.mt.sl.NewIterator()
func buildL0Table(iter y.Iterator, dropPrefixes [][]byte, bopts table.Options) *table.Builder {
defer iter.Close()
b := table.NewTableBuilder(bopts)
for iter.SeekToFirst(); iter.Valid(); iter.Next() {
if len(ft.dropPrefixes) > 0 && hasAnyPrefixes(iter.Key(), ft.dropPrefixes) {
for iter.Rewind(); iter.Valid(); iter.Next() {
if len(dropPrefixes) > 0 && hasAnyPrefixes(iter.Key(), dropPrefixes) {
continue
}
vs := iter.Value()
@@ -1025,23 +1067,15 @@ func buildL0Table(ft flushTask, bopts table.Options) *table.Builder {
}
b.Add(iter.Key(), iter.Value(), vp.Len)
}
return b
}
type flushTask struct {
mt *memTable
dropPrefixes [][]byte
}
// handleFlushTask must be run serially.
func (db *DB) handleFlushTask(ft flushTask) error {
// There can be a scenario, when empty memtable is flushed.
if ft.mt.sl.Empty() {
return nil
}
// handleMemTableFlush must be run serially.
func (db *DB) handleMemTableFlush(mt *memTable, dropPrefixes [][]byte) error {
bopts := buildTableOptions(db)
builder := buildL0Table(ft, bopts)
itr := mt.sl.NewUniIterator(false)
builder := buildL0Table(itr, nil, bopts)
defer builder.Close()
// buildL0Table can return nil if the none of the items in the skiplist are
@@ -1070,39 +1104,39 @@ func (db *DB) handleFlushTask(ft flushTask) error {
return err
}
// flushMemtable must keep running until we send it an empty flushTask. If there
// are errors during handling the flush task, we'll retry indefinitely.
func (db *DB) flushMemtable(lc *z.Closer) error {
// flushMemtable must keep running until we send it an empty memtable. If there
// are errors during handling the memtable flush, we'll retry indefinitely.
func (db *DB) flushMemtable(lc *z.Closer) {
defer lc.Done()
for ft := range db.flushChan {
if ft.mt == nil {
// We close db.flushChan now, instead of sending a nil ft.mt.
for mt := range db.flushChan {
if mt == nil {
continue
}
for {
err := db.handleFlushTask(ft)
if err == nil {
// Update s.imm. Need a lock.
db.lock.Lock()
// This is a single-threaded operation. ft.mt corresponds to the head of
// db.imm list. Once we flush it, we advance db.imm. The next ft.mt
// which would arrive here would match db.imm[0], because we acquire a
// lock over DB when pushing to flushChan.
// TODO: This logic is dirty AF. Any change and this could easily break.
y.AssertTrue(ft.mt == db.imm[0])
db.imm = db.imm[1:]
ft.mt.DecrRef() // Return memory.
db.lock.Unlock()
break
for {
if err := db.handleMemTableFlush(mt, nil); err != nil {
// Encountered error. Retry indefinitely.
db.opt.Errorf("error flushing memtable to disk: %v, retrying", err)
time.Sleep(time.Second)
continue
}
// Encountered error. Retry indefinitely.
db.opt.Errorf("Failure while flushing memtable to disk: %v. Retrying...\n", err)
time.Sleep(time.Second)
// Update s.imm. Need a lock.
db.lock.Lock()
// This is a single-threaded operation. mt corresponds to the head of
// db.imm list. Once we flush it, we advance db.imm. The next mt
// which would arrive here would match db.imm[0], because we acquire a
// lock over DB when pushing to flushChan.
// TODO: This logic is dirty AF. Any change and this could easily break.
y.AssertTrue(mt == db.imm[0])
db.imm = db.imm[1:]
mt.DecrRef() // Return memory.
// unlock
db.lock.Unlock()
break
}
}
return nil
}
func exists(path string) (bool, error) {
@@ -1522,10 +1556,10 @@ func (db *DB) startCompactions() {
func (db *DB) startMemoryFlush() {
// Start memory fluhser.
if db.closers.memtable != nil {
db.flushChan = make(chan flushTask, db.opt.NumMemtables)
db.flushChan = make(chan *memTable, db.opt.NumMemtables)
db.closers.memtable = z.NewCloser(1)
go func() {
_ = db.flushMemtable(db.closers.memtable)
db.flushMemtable(db.closers.memtable)
}()
}
}
@@ -1585,7 +1619,7 @@ func (db *DB) Flatten(workers int) error {
}
}
if len(levels) <= 1 {
prios := db.lc.pickCompactLevels()
prios := db.lc.pickCompactLevels(nil)
if len(prios) == 0 || prios[0].score <= 1.0 {
db.opt.Infof("All tables consolidated into one level. Flattening done.\n")
return nil
@@ -1605,7 +1639,7 @@ func (db *DB) Flatten(workers int) error {
func (db *DB) blockWrite() error {
// Stop accepting new writes.
if !atomic.CompareAndSwapInt32(&db.blockWrites, 0, 1) {
if !db.blockWrites.CompareAndSwap(0, 1) {
return ErrBlockedWrites
}
@@ -1620,7 +1654,7 @@ func (db *DB) unblockWrite() {
go db.doWrites(db.closers.writes)
// Resume writes.
atomic.StoreInt32(&db.blockWrites, 0)
db.blockWrites.Store(0)
}
func (db *DB) prepareToDrop() (func(), error) {
@@ -1628,10 +1662,10 @@ func (db *DB) prepareToDrop() (func(), error) {
panic("Attempting to drop data in read-only mode.")
}
// In order prepare for drop, we need to block the incoming writes and
// write it to db. Then, flush all the pending flushtask. So that, we
// write it to db. Then, flush all the pending memtable. So that, we
// don't miss any entries.
if err := db.blockWrite(); err != nil {
return nil, err
return func() {}, err
}
reqs := make([]*request, 0, 10)
for {
@@ -1677,7 +1711,7 @@ func (db *DB) dropAll() (func(), error) {
if err != nil {
return f, err
}
// prepareToDrop will stop all the incomming write and flushes any pending flush tasks.
// prepareToDrop will stop all the incoming write and flushes any pending memtables.
// Before we drop, we'll stop the compaction because anyways all the datas are going to
// be deleted.
db.stopCompactions()
@@ -1710,7 +1744,7 @@ func (db *DB) dropAll() (func(), error) {
if err != nil {
return resume, err
}
db.lc.nextFileID = 1
db.lc.nextFileID.Store(1)
db.opt.Infof("Deleted %d value log files. DropAll done.\n", num)
db.blockCache.Clear()
db.indexCache.Clear()
@@ -1719,16 +1753,16 @@ func (db *DB) dropAll() (func(), error) {
}
// DropPrefix would drop all the keys with the provided prefix. It does this in the following way:
// - Stop accepting new writes.
// - Stop memtable flushes before acquiring lock. Because we're acquring lock here
// and memtable flush stalls for lock, which leads to deadlock
// - Flush out all memtables, skipping over keys with the given prefix, Kp.
// - Write out the value log header to memtables when flushing, so we don't accidentally bring Kp
// back after a restart.
// - Stop compaction.
// - Compact L0->L1, skipping over Kp.
// - Compact rest of the levels, Li->Li, picking tables which have Kp.
// - Resume memtable flushes, compactions and writes.
// - Stop accepting new writes.
// - Stop memtable flushes before acquiring lock. Because we're acquiring lock here
// and memtable flush stalls for lock, which leads to deadlock
// - Flush out all memtables, skipping over keys with the given prefix, Kp.
// - Write out the value log header to memtables when flushing, so we don't accidentally bring Kp
// back after a restart.
// - Stop compaction.
// - Compact L0->L1, skipping over Kp.
// - Compact rest of the levels, Li->Li, picking tables which have Kp.
// - Resume memtable flushes, compactions and writes.
func (db *DB) DropPrefix(prefixes ...[]byte) error {
if len(prefixes) == 0 {
return nil
@@ -1759,13 +1793,8 @@ func (db *DB) DropPrefix(prefixes ...[]byte) error {
memtable.DecrRef()
continue
}
task := flushTask{
mt: memtable,
// Ensure that the head of value log gets persisted to disk.
dropPrefixes: filtered,
}
db.opt.Debugf("Flushing memtable")
if err := db.handleFlushTask(task); err != nil {
if err := db.handleMemTableFlush(memtable, filtered); err != nil {
db.opt.Errorf("While trying to flush memtable: %v", err)
return err
}
@@ -1869,7 +1898,10 @@ func (db *DB) Subscribe(ctx context.Context, cb func(kv *KVList) error, matches
}
c := z.NewCloser(1)
s := db.pub.newSubscriber(c, matches)
s, err := db.pub.newSubscriber(c, matches)
if err != nil {
return y.Wrapf(err, "while creating a new subscriber")
}
slurp := func(batch *pb.KVList) error {
for {
select {
@@ -1887,7 +1919,11 @@ func (db *DB) Subscribe(ctx context.Context, cb func(kv *KVList) error, matches
drain := func() {
for {
select {
case <-s.sendCh:
case _, ok := <-s.sendCh:
if !ok {
// Channel is closed.
return
}
default:
return
}
@@ -1904,7 +1940,7 @@ func (db *DB) Subscribe(ctx context.Context, cb func(kv *KVList) error, matches
return err
case <-ctx.Done():
c.Done()
atomic.StoreUint64(s.active, 0)
s.active.Store(0)
drain()
db.pub.deleteSubscriber(s.id)
// Delete the subscriber to avoid further updates.
@@ -1913,7 +1949,7 @@ func (db *DB) Subscribe(ctx context.Context, cb func(kv *KVList) error, matches
err := slurp(batch)
if err != nil {
c.Done()
atomic.StoreUint64(s.active, 0)
s.active.Store(0)
drain()
// Delete the subscriber if there is an error by the callback.
db.pub.deleteSubscriber(s.id)
@@ -1923,11 +1959,6 @@ func (db *DB) Subscribe(ctx context.Context, cb func(kv *KVList) error, matches
}
}
// shouldEncrypt returns bool, which tells whether to encrypt or not.
func (db *DB) shouldEncrypt() bool {
return len(db.opt.EncryptionKey) > 0
}
func (db *DB) syncDir(dir string) error {
if db.opt.InMemory {
return nil
@@ -1968,7 +1999,7 @@ func (db *DB) StreamDB(outOptions Options) error {
defer outDB.Close()
writer := outDB.NewStreamWriter()
if err := writer.Prepare(); err != nil {
y.Wrapf(err, "cannot create stream writer in out DB at %s", outDir)
return y.Wrapf(err, "cannot create stream writer in out DB at %s", outDir)
}
// Stream contents of DB to the output DB.
vendor/github.com/dgraph-io/badger/v3/dir_plan9.go → vendor/github.com/dgraph-io/badger/v4/dir_plan9.go
Generated Vendored
+4 -4
View File
@@ -22,7 +22,7 @@ import (
"path/filepath"
"strings"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/badger/v4/y"
)
// directoryLockGuard holds a lock on a directory and a pid file inside. The pid file isn't part
@@ -123,9 +123,9 @@ func syncDir(dir string) error {
// Opening an exclusive-use file returns an error.
// The expected error strings are:
//
// - "open/create -- file is locked" (cwfs, kfs)
// - "exclusive lock" (fossil)
// - "exclusive use file already open" (ramfs)
// - "open/create -- file is locked" (cwfs, kfs)
// - "exclusive lock" (fossil)
// - "exclusive use file already open" (ramfs)
//
// See https://github.com/golang/go/blob/go1.15rc1/src/cmd/go/internal/lockedfile/lockedfile_plan9.go#L16
var lockedErrStrings = [...]string{
vendor/github.com/dgraph-io/badger/v3/dir_unix.go → vendor/github.com/dgraph-io/badger/v4/dir_unix.go
Generated Vendored
+4 -3
View File
@@ -1,3 +1,4 @@
//go:build !windows && !plan9
// +build !windows,!plan9
/*
@@ -20,12 +21,12 @@ package badger
import (
"fmt"
"io/ioutil"
"os"
"path/filepath"
"github.com/dgraph-io/badger/v3/y"
"golang.org/x/sys/unix"
"github.com/dgraph-io/badger/v4/y"
)
// directoryLockGuard holds a lock on a directory and a pid file inside. The pid file isn't part
@@ -70,7 +71,7 @@ func acquireDirectoryLock(dirPath string, pidFileName string, readOnly bool) (
if !readOnly {
// Yes, we happily overwrite a pre-existing pid file. We're the
// only read-write badger process using this directory.
err = ioutil.WriteFile(absPidFilePath, []byte(fmt.Sprintf("%d\n", os.Getpid())), 0666)
err = os.WriteFile(absPidFilePath, []byte(fmt.Sprintf("%d\n", os.Getpid())), 0666)
if err != nil {
f.Close()
return nil, y.Wrapf(err,
vendor/github.com/dgraph-io/badger/v3/dir_windows.go → vendor/github.com/dgraph-io/badger/v4/dir_windows.go
Generated Vendored
+2 -1
View File
@@ -1,3 +1,4 @@
//go:build windows
// +build windows
/*
@@ -24,7 +25,7 @@ import (
"path/filepath"
"syscall"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/badger/v4/y"
)
// FILE_ATTRIBUTE_TEMPORARY - A file that is being used for temporary storage.
vendor/github.com/dgraph-io/badger/v3/discard.go → vendor/github.com/dgraph-io/badger/v4/discard.go
Generated Vendored
+6 -6
View File
@@ -23,8 +23,8 @@ import (
"sort"
"sync"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/ristretto/z"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2/z"
)
// discardStats keeps track of the amount of data that could be discarded for
@@ -42,14 +42,14 @@ const discardFname string = "DISCARD"
func InitDiscardStats(opt Options) (*discardStats, error) {
fname := filepath.Join(opt.ValueDir, discardFname)
// 1GB file can store 67M discard entries. Each entry is 16 bytes.
// 1MB file can store 65.536 discard entries. Each entry is 16 bytes.
mf, err := z.OpenMmapFile(fname, os.O_CREATE|os.O_RDWR, 1<<20)
lf := &discardStats{
MmapFile: mf,
opt: opt,
}
if err == z.NewFile {
// We don't need to zero out the entire 1GB.
// We don't need to zero out the entire 1MB.
lf.zeroOut()
} else if err != nil {
@@ -131,7 +131,7 @@ func (lf *discardStats) Update(fidu uint32, discard int64) int64 {
// Could not find the fid. Add the entry.
idx = lf.nextEmptySlot
lf.set(idx*16, uint64(fid))
lf.set(idx*16, fid)
lf.set(idx*16+8, uint64(discard))
// Move to next slot.
@@ -142,7 +142,7 @@ func (lf *discardStats) Update(fidu uint32, discard int64) int64 {
lf.zeroOut()
sort.Sort(lf)
return int64(discard)
return discard
}
func (lf *discardStats) Iterate(f func(fid, stats uint64)) {
vendor/github.com/dgraph-io/badger/v3/doc.go → vendor/github.com/dgraph-io/badger/v4/doc.go
Generated Vendored
+1 -2
View File
@@ -10,8 +10,7 @@ hence reducing both write amplification and the size of the LSM tree. This
allows LSM tree to be served entirely from RAM, while the values are served
from SSD.
Usage
# Usage
Badger has the following main types: DB, Txn, Item and Iterator. DB contains
keys that are associated with values. It must be opened with the appropriate
vendor/github.com/dgraph-io/badger/v3/errors.go → vendor/github.com/dgraph-io/badger/v4/errors.go
Generated Vendored
+28 -29
View File
@@ -17,9 +17,8 @@
package badger
import (
stderrors "errors"
"math"
"github.com/pkg/errors"
)
const (
@@ -30,97 +29,97 @@ const (
var (
// ErrValueLogSize is returned when opt.ValueLogFileSize option is not within the valid
// range.
ErrValueLogSize = errors.New("Invalid ValueLogFileSize, must be in range [1MB, 2GB)")
ErrValueLogSize = stderrors.New("Invalid ValueLogFileSize, must be in range [1MB, 2GB)")
// ErrKeyNotFound is returned when key isn't found on a txn.Get.
ErrKeyNotFound = errors.New("Key not found")
ErrKeyNotFound = stderrors.New("Key not found")
// ErrTxnTooBig is returned if too many writes are fit into a single transaction.
ErrTxnTooBig = errors.New("Txn is too big to fit into one request")
ErrTxnTooBig = stderrors.New("Txn is too big to fit into one request")
// ErrConflict is returned when a transaction conflicts with another transaction. This can
// happen if the read rows had been updated concurrently by another transaction.
ErrConflict = errors.New("Transaction Conflict. Please retry")
ErrConflict = stderrors.New("Transaction Conflict. Please retry")
// ErrReadOnlyTxn is returned if an update function is called on a read-only transaction.
ErrReadOnlyTxn = errors.New("No sets or deletes are allowed in a read-only transaction")
ErrReadOnlyTxn = stderrors.New("No sets or deletes are allowed in a read-only transaction")
// ErrDiscardedTxn is returned if a previously discarded transaction is re-used.
ErrDiscardedTxn = errors.New("This transaction has been discarded. Create a new one")
ErrDiscardedTxn = stderrors.New("This transaction has been discarded. Create a new one")
// ErrEmptyKey is returned if an empty key is passed on an update function.
ErrEmptyKey = errors.New("Key cannot be empty")
ErrEmptyKey = stderrors.New("Key cannot be empty")
// ErrInvalidKey is returned if the key has a special !badger! prefix,
// reserved for internal usage.
ErrInvalidKey = errors.New("Key is using a reserved !badger! prefix")
ErrInvalidKey = stderrors.New("Key is using a reserved !badger! prefix")
// ErrBannedKey is returned if the read/write key belongs to any banned namespace.
ErrBannedKey = errors.New("Key is using the banned prefix")
ErrBannedKey = stderrors.New("Key is using the banned prefix")
// ErrThresholdZero is returned if threshold is set to zero, and value log GC is called.
// In such a case, GC can't be run.
ErrThresholdZero = errors.New(
ErrThresholdZero = stderrors.New(
"Value log GC can't run because threshold is set to zero")
// ErrNoRewrite is returned if a call for value log GC doesn't result in a log file rewrite.
ErrNoRewrite = errors.New(
ErrNoRewrite = stderrors.New(
"Value log GC attempt didn't result in any cleanup")
// ErrRejected is returned if a value log GC is called either while another GC is running, or
// after DB::Close has been called.
ErrRejected = errors.New("Value log GC request rejected")
ErrRejected = stderrors.New("Value log GC request rejected")
// ErrInvalidRequest is returned if the user request is invalid.
ErrInvalidRequest = errors.New("Invalid request")
ErrInvalidRequest = stderrors.New("Invalid request")
// ErrManagedTxn is returned if the user tries to use an API which isn't
// allowed due to external management of transactions, when using ManagedDB.
ErrManagedTxn = errors.New(
ErrManagedTxn = stderrors.New(
"Invalid API request. Not allowed to perform this action using ManagedDB")
// ErrNamespaceMode is returned if the user tries to use an API which is allowed only when
// NamespaceOffset is non-negative.
ErrNamespaceMode = errors.New(
ErrNamespaceMode = stderrors.New(
"Invalid API request. Not allowed to perform this action when NamespaceMode is not set.")
// ErrInvalidDump if a data dump made previously cannot be loaded into the database.
ErrInvalidDump = errors.New("Data dump cannot be read")
ErrInvalidDump = stderrors.New("Data dump cannot be read")
// ErrZeroBandwidth is returned if the user passes in zero bandwidth for sequence.
ErrZeroBandwidth = errors.New("Bandwidth must be greater than zero")
ErrZeroBandwidth = stderrors.New("Bandwidth must be greater than zero")
// ErrWindowsNotSupported is returned when opt.ReadOnly is used on Windows
ErrWindowsNotSupported = errors.New("Read-only mode is not supported on Windows")
ErrWindowsNotSupported = stderrors.New("Read-only mode is not supported on Windows")
// ErrPlan9NotSupported is returned when opt.ReadOnly is used on Plan 9
ErrPlan9NotSupported = errors.New("Read-only mode is not supported on Plan 9")
ErrPlan9NotSupported = stderrors.New("Read-only mode is not supported on Plan 9")
// ErrTruncateNeeded is returned when the value log gets corrupt, and requires truncation of
// corrupt data to allow Badger to run properly.
ErrTruncateNeeded = errors.New(
ErrTruncateNeeded = stderrors.New(
"Log truncate required to run DB. This might result in data loss")
// ErrBlockedWrites is returned if the user called DropAll. During the process of dropping all
// data from Badger, we stop accepting new writes, by returning this error.
ErrBlockedWrites = errors.New("Writes are blocked, possibly due to DropAll or Close")
ErrBlockedWrites = stderrors.New("Writes are blocked, possibly due to DropAll or Close")
// ErrNilCallback is returned when subscriber's callback is nil.
ErrNilCallback = errors.New("Callback cannot be nil")
ErrNilCallback = stderrors.New("Callback cannot be nil")
// ErrEncryptionKeyMismatch is returned when the storage key is not
// matched with the key previously given.
ErrEncryptionKeyMismatch = errors.New("Encryption key mismatch")
ErrEncryptionKeyMismatch = stderrors.New("Encryption key mismatch")
// ErrInvalidDataKeyID is returned if the datakey id is invalid.
ErrInvalidDataKeyID = errors.New("Invalid datakey id")
ErrInvalidDataKeyID = stderrors.New("Invalid datakey id")
// ErrInvalidEncryptionKey is returned if length of encryption keys is invalid.
ErrInvalidEncryptionKey = errors.New("Encryption key's length should be" +
ErrInvalidEncryptionKey = stderrors.New("Encryption key's length should be" +
"either 16, 24, or 32 bytes")
// ErrGCInMemoryMode is returned when db.RunValueLogGC is called in in-memory mode.
ErrGCInMemoryMode = errors.New("Cannot run value log GC when DB is opened in InMemory mode")
ErrGCInMemoryMode = stderrors.New("Cannot run value log GC when DB is opened in InMemory mode")
// ErrDBClosed is returned when a get operation is performed after closing the DB.
ErrDBClosed = errors.New("DB Closed")
ErrDBClosed = stderrors.New("DB Closed")
)
vendor/github.com/dgraph-io/badger/v3/fb/BlockOffset.go → vendor/github.com/dgraph-io/badger/v4/fb/BlockOffset.go
Generated Vendored
View File
vendor/github.com/dgraph-io/badger/v3/fb/TableIndex.go → vendor/github.com/dgraph-io/badger/v4/fb/TableIndex.go
Generated Vendored
View File
vendor/github.com/dgraph-io/badger/v3/fb/flatbuffer.fbs → vendor/github.com/dgraph-io/badger/v4/fb/flatbuffer.fbs
Generated Vendored
View File
vendor/github.com/dgraph-io/badger/v3/fb/gen.sh → vendor/github.com/dgraph-io/badger/v4/fb/gen.sh
Generated Vendored
View File
vendor/github.com/dgraph-io/badger/v3/fb/install_flatbuffers.sh → vendor/github.com/dgraph-io/badger/v4/fb/install_flatbuffers.sh
Generated Vendored
View File
vendor/github.com/dgraph-io/badger/v3/histogram.go → vendor/github.com/dgraph-io/badger/v4/histogram.go
Generated Vendored
+1 -1
View File
@@ -106,7 +106,7 @@ func (histogram *histogramData) Update(value int64) {
}
// Check if the value should be added to the "index" bin
if value < int64(histogram.bins[index]) {
if value < histogram.bins[index] {
histogram.countPerBin[index]++
break
}
vendor/github.com/dgraph-io/badger/v3/iterator.go → vendor/github.com/dgraph-io/badger/v4/iterator.go
Generated Vendored
+26 -18
View File
@@ -23,13 +23,11 @@ import (
"math"
"sort"
"sync"
"sync/atomic"
"time"
"github.com/dgraph-io/badger/v3/table"
"github.com/dgraph-io/ristretto/z"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/badger/v4/table"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2/z"
)
type prefetchStatus uint8
@@ -170,7 +168,7 @@ func (item *Item) yieldItemValue() ([]byte, func(), error) {
db := item.txn.db
result, cb, err := db.vlog.Read(vp, item.slice)
if err != nil {
db.opt.Logger.Errorf("Unable to read: Key: %v, Version : %v, meta: %v, userMeta: %v"+
db.opt.Errorf("Unable to read: Key: %v, Version : %v, meta: %v, userMeta: %v"+
" Error: %v", key, item.version, item.meta, item.userMeta, err)
var txn *Txn
if db.opt.managedTxns {
@@ -193,7 +191,7 @@ func (item *Item) yieldItemValue() ([]byte, func(), error) {
if item.meta&bitValuePointer > 0 {
vp.Decode(item.vptr)
}
db.opt.Logger.Errorf("Key: %v, Version : %v, meta: %v, userMeta: %v valuePointer: %+v",
db.opt.Errorf("Key: %v, Version : %v, meta: %v, userMeta: %v valuePointer: %+v",
item.Key(), item.version, item.meta, item.userMeta, vp)
}
}
@@ -464,24 +462,25 @@ type Iterator struct {
// Using prefetch is recommended if you're doing a long running iteration, for performance.
//
// Multiple Iterators:
// For a read-only txn, multiple iterators can be running simultaneously. However, for a read-write
// For a read-only txn, multiple iterators can be running simultaneously. However, for a read-write
// txn, iterators have the nuance of being a snapshot of the writes for the transaction at the time
// iterator was created. If writes are performed after an iterator is created, then that iterator
// will not be able to see those writes. Only writes performed before an iterator was created can be
// viewed.
func (txn *Txn) NewIterator(opt IteratorOptions) *Iterator {
if txn.discarded {
panic("Transaction has already been discarded")
panic(ErrDiscardedTxn)
}
if txn.db.IsClosed() {
panic(ErrDBClosed.Error())
panic(ErrDBClosed)
}
// Keep track of the number of active iterators.
atomic.AddInt32(&txn.numIterators, 1)
y.NumIteratorsCreatedAdd(txn.db.opt.MetricsEnabled, 1)
// TODO: If Prefix is set, only pick those memtables which have keys with
// the prefix.
// Keep track of the number of active iterators.
txn.numIterators.Add(1)
// TODO: If Prefix is set, only pick those memtables which have keys with the prefix.
tables, decr := txn.db.getMemTables()
defer decr()
txn.db.vlog.incrIteratorCount()
@@ -555,7 +554,7 @@ func (it *Iterator) Close() {
}
it.closed = true
if it.iitr == nil {
atomic.AddInt32(&it.txn.numIterators, -1)
it.txn.numIterators.Add(-1)
return
}
@@ -574,7 +573,7 @@ func (it *Iterator) Close() {
// TODO: We could handle this error.
_ = it.txn.db.vlog.decrIteratorCount()
atomic.AddInt32(&it.txn.numIterators, -1)
it.txn.numIterators.Add(-1)
}
// Next would advance the iterator by one. Always check it.Valid() after a Next()
@@ -590,7 +589,7 @@ func (it *Iterator) Next() {
// Set next item to current
it.item = it.data.pop()
for it.iitr.Valid() {
for it.iitr.Valid() && hasPrefix(it) {
if it.parseItem() {
// parseItem calls one extra next.
// This is used to deal with the complexity of reverse iteration.
@@ -726,6 +725,15 @@ func (it *Iterator) fill(item *Item) {
}
}
func hasPrefix(it *Iterator) bool {
// We shouldn't check prefix in case the iterator is going in reverse. Since in reverse we expect
// people to append items to the end of prefix.
if !it.opt.Reverse && len(it.opt.Prefix) > 0 {
return bytes.HasPrefix(y.ParseKey(it.iitr.Key()), it.opt.Prefix)
}
return true
}
func (it *Iterator) prefetch() {
prefetchSize := 2
if it.opt.PrefetchValues && it.opt.PrefetchSize > 1 {
@@ -735,7 +743,7 @@ func (it *Iterator) prefetch() {
i := it.iitr
var count int
it.item = nil
for i.Valid() {
for i.Valid() && hasPrefix(it) {
if !it.parseItem() {
continue
}
vendor/github.com/dgraph-io/badger/v3/key_registry.go → vendor/github.com/dgraph-io/badger/v4/key_registry.go
Generated Vendored
+5 -4
View File
@@ -28,8 +28,9 @@ import (
"sync"
"time"
"github.com/dgraph-io/badger/v3/pb"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/badger/v4/pb"
"github.com/dgraph-io/badger/v4/y"
"google.golang.org/protobuf/proto"
)
const (
@@ -195,7 +196,7 @@ func (kri *keyRegistryIterator) next() (*pb.DataKey, error) {
return nil, y.Wrapf(y.ErrChecksumMismatch, "Error while checking checksum for data key.")
}
dataKey := &pb.DataKey{}
if err = dataKey.Unmarshal(data); err != nil {
if err = proto.Unmarshal(data, dataKey); err != nil {
return nil, y.Wrapf(err, "While unmarshal of datakey in keyRegistryIterator.next")
}
if len(kri.encryptionKey) > 0 {
@@ -404,7 +405,7 @@ func storeDataKey(buf *bytes.Buffer, storageKey []byte, k *pb.DataKey) error {
return y.Wrapf(err, "Error while encrypting datakey in storeDataKey")
}
var data []byte
if data, err = k.Marshal(); err != nil {
if data, err = proto.Marshal(k); err != nil {
err = y.Wrapf(err, "Error while marshaling datakey in storeDataKey")
var err2 error
// decrypting the datakey back.
vendor/github.com/dgraph-io/badger/v3/level_handler.go → vendor/github.com/dgraph-io/badger/v4/level_handler.go
Generated Vendored
+4 -4
View File
@@ -21,8 +21,8 @@ import (
"sort"
"sync"
"github.com/dgraph-io/badger/v3/table"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/badger/v4/table"
"github.com/dgraph-io/badger/v4/y"
)
type levelHandler struct {
@@ -165,8 +165,8 @@ func (s *levelHandler) addTable(t *table.Table) {
// sortTables sorts tables of levelHandler based on table.Smallest.
// Normally it should be called after all addTable calls.
func (s *levelHandler) sortTables() {
s.RLock()
defer s.RUnlock()
s.Lock()
defer s.Unlock()
sort.Slice(s.tables, func(i, j int) bool {
return y.CompareKeys(s.tables[i].Smallest(), s.tables[j].Smallest()) < 0
vendor/github.com/dgraph-io/badger/v3/levels.go → vendor/github.com/dgraph-io/badger/v4/levels.go
Generated Vendored
+66 -39
View File
@@ -20,6 +20,7 @@ import (
"bytes"
"context"
"encoding/hex"
stderrors "errors"
"fmt"
"math"
"math/rand"
@@ -30,18 +31,18 @@ import (
"sync/atomic"
"time"
"github.com/pkg/errors"
otrace "go.opencensus.io/trace"
"github.com/dgraph-io/badger/v3/pb"
"github.com/dgraph-io/badger/v3/table"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/ristretto/z"
"github.com/pkg/errors"
"github.com/dgraph-io/badger/v4/pb"
"github.com/dgraph-io/badger/v4/table"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2/z"
)
type levelsController struct {
nextFileID uint64 // Atomic
l0stallsMs int64 // Atomic
nextFileID atomic.Uint64
l0stallsMs atomic.Int64
// The following are initialized once and const.
levels []*levelHandler
@@ -107,7 +108,7 @@ func newLevelsController(db *DB, mf *Manifest) (*levelsController, error) {
throttle := y.NewThrottle(3)
start := time.Now()
var numOpened int32
var numOpened atomic.Int32
tick := time.NewTicker(3 * time.Second)
defer tick.Stop()
@@ -115,7 +116,7 @@ func newLevelsController(db *DB, mf *Manifest) (*levelsController, error) {
fname := table.NewFilename(fileID, db.opt.Dir)
select {
case <-tick.C:
db.opt.Infof("%d tables out of %d opened in %s\n", atomic.LoadInt32(&numOpened),
db.opt.Infof("%d tables out of %d opened in %s\n", numOpened.Load(),
len(mf.Tables), time.Since(start).Round(time.Millisecond))
default:
}
@@ -130,7 +131,7 @@ func newLevelsController(db *DB, mf *Manifest) (*levelsController, error) {
var rerr error
defer func() {
throttle.Done(rerr)
atomic.AddInt32(&numOpened, 1)
numOpened.Add(1)
}()
dk, err := db.registry.DataKey(tf.KeyID)
if err != nil {
@@ -168,9 +169,9 @@ func newLevelsController(db *DB, mf *Manifest) (*levelsController, error) {
closeAllTables(tables)
return nil, err
}
db.opt.Infof("All %d tables opened in %s\n", atomic.LoadInt32(&numOpened),
db.opt.Infof("All %d tables opened in %s\n", numOpened.Load(),
time.Since(start).Round(time.Millisecond))
s.nextFileID = maxFileID + 1
s.nextFileID.Store(maxFileID + 1)
for i, tbls := range tables {
s.levels[i].initTables(tbls)
}
@@ -473,8 +474,13 @@ func (s *levelsController) runCompactor(id int, lc *z.Closer) {
}
return false
}
var priosBuffer []compactionPriority
runOnce := func() bool {
prios := s.pickCompactLevels()
prios := s.pickCompactLevels(priosBuffer)
defer func() {
priosBuffer = prios
}()
if id == 0 {
// Worker ID zero prefers to compact L0 always.
prios = moveL0toFront(prios)
@@ -536,7 +542,9 @@ func (s *levelsController) lastLevel() *levelHandler {
// pickCompactLevel determines which level to compact.
// Based on: https://github.com/facebook/rocksdb/wiki/Leveled-Compaction
func (s *levelsController) pickCompactLevels() (prios []compactionPriority) {
// It tries to reuse priosBuffer to reduce memory allocation,
// passing nil is acceptable, then new memory will be allocated.
func (s *levelsController) pickCompactLevels(priosBuffer []compactionPriority) (prios []compactionPriority) {
t := s.levelTargets()
addPriority := func(level int, score float64) {
pri := compactionPriority{
@@ -548,6 +556,12 @@ func (s *levelsController) pickCompactLevels() (prios []compactionPriority) {
prios = append(prios, pri)
}
// Grow buffer to fit all levels.
if cap(priosBuffer) < len(s.levels) {
priosBuffer = make([]compactionPriority, 0, len(s.levels))
}
prios = priosBuffer[:0]
// Add L0 priority based on the number of tables.
addPriority(0, float64(s.levels[0].numTables())/float64(s.kv.opt.NumLevelZeroTables))
@@ -998,20 +1012,14 @@ func containsPrefix(table *table.Table, prefix []byte) bool {
// In table iterator's Seek, we assume that key has version in last 8 bytes. We set
// version=0 (ts=math.MaxUint64), so that we don't skip the key prefixed with prefix.
ti.Seek(y.KeyWithTs(prefix, math.MaxUint64))
if bytes.HasPrefix(ti.Key(), prefix) {
return true
}
return false
return bytes.HasPrefix(ti.Key(), prefix)
}
if bytes.Compare(prefix, smallValue) > 0 &&
bytes.Compare(prefix, largeValue) < 0 {
// There may be a case when table contains [0x0000,...., 0xffff]. If we are searching for
// k=0x0011, we should not directly infer that k is present. It may not be present.
if !isPresent() {
return false
}
return true
return isPresent()
}
return false
@@ -1079,14 +1087,13 @@ func (s *levelsController) addSplits(cd *compactDef) {
return
}
if i%width == width-1 {
// Right should always have ts=maxUint64 otherwise we'll lose keys
// in subcompaction. Consider the following.
// Right is assigned ts=0. The encoding ts bytes take MaxUint64-ts,
// so, those with smaller TS will be considered larger for the same key.
// Consider the following.
// Top table is [A1...C3(deleted)]
// bot table is [B1....C2]
// This will generate splits like [A1 ... C2] . Notice that we
// dropped the C3 which is the last key of the top table.
// See TestCompaction/with_split test.
right := y.KeyWithTs(y.ParseKey(t.Biggest()), math.MaxUint64)
// It will generate a split [A1 ... C0], including any records of Key C.
right := y.KeyWithTs(y.ParseKey(t.Biggest()), 0)
addRange(right)
}
}
@@ -1426,8 +1433,8 @@ func (s *levelsController) runCompactDef(id, l int, cd compactDef) (err error) {
cd.splits = append(cd.splits, keyRange{})
}
// Table should never be moved directly between levels, always be rewritten to allow discarding
// invalid versions.
// Table should never be moved directly between levels,
// always be rewritten to allow discarding invalid versions.
newTables, decr, err := s.compactBuildTables(l, cd)
if err != nil {
@@ -1446,6 +1453,22 @@ func (s *levelsController) runCompactDef(id, l int, cd compactDef) (err error) {
return err
}
getSizes := func(tables []*table.Table) int64 {
size := int64(0)
for _, i := range tables {
size += i.Size()
}
return size
}
sizeNewTables := int64(0)
sizeOldTables := int64(0)
if s.kv.opt.MetricsEnabled {
sizeNewTables = getSizes(newTables)
sizeOldTables = getSizes(cd.bot) + getSizes(cd.top)
y.NumBytesCompactionWrittenAdd(s.kv.opt.MetricsEnabled, nextLevel.strLevel, sizeNewTables)
}
// See comment earlier in this function about the ordering of these ops, and the order in which
// we access levels when reading.
if err := nextLevel.replaceTables(cd.bot, newTables); err != nil {
@@ -1466,16 +1489,16 @@ func (s *levelsController) runCompactDef(id, l int, cd compactDef) (err error) {
expensive = " [E]"
}
s.kv.opt.Infof("[%d]%s LOG Compact %d->%d (%d, %d -> %d tables with %d splits)."+
" [%s] -> [%s], took %v\n",
" [%s] -> [%s], took %v\n, deleted %d bytes",
id, expensive, thisLevel.level, nextLevel.level, len(cd.top), len(cd.bot),
len(newTables), len(cd.splits), strings.Join(from, " "), strings.Join(to, " "),
dur.Round(time.Millisecond))
dur.Round(time.Millisecond), sizeOldTables-sizeNewTables)
}
if cd.thisLevel.level != 0 && len(newTables) > 2*s.kv.opt.LevelSizeMultiplier {
s.kv.opt.Debugf("This Range (numTables: %d)\nLeft:\n%s\nRight:\n%s\n",
s.kv.opt.Infof("This Range (numTables: %d)\nLeft:\n%s\nRight:\n%s\n",
len(cd.top), hex.Dump(cd.thisRange.left), hex.Dump(cd.thisRange.right))
s.kv.opt.Debugf("Next Range (numTables: %d)\nLeft:\n%s\nRight:\n%s\n",
s.kv.opt.Infof("Next Range (numTables: %d)\nLeft:\n%s\nRight:\n%s\n",
len(cd.bot), hex.Dump(cd.nextRange.left), hex.Dump(cd.nextRange.right))
}
return nil
@@ -1490,7 +1513,7 @@ func tablesToString(tables []*table.Table) []string {
return res
}
var errFillTables = errors.New("Unable to fill tables")
var errFillTables = stderrors.New("Unable to fill tables")
// doCompact picks some table on level l and compacts it away to the next level.
func (s *levelsController) doCompact(id int, p compactionPriority) error {
@@ -1568,7 +1591,7 @@ func (s *levelsController) addLevel0Table(t *table.Table) error {
if dur > time.Second {
s.kv.opt.Infof("L0 was stalled for %s\n", dur.Round(time.Millisecond))
}
atomic.AddInt64(&s.l0stallsMs, int64(dur.Round(time.Millisecond)))
s.l0stallsMs.Add(int64(dur.Round(time.Millisecond)))
}
return nil
@@ -1580,8 +1603,8 @@ func (s *levelsController) close() error {
}
// get searches for a given key in all the levels of the LSM tree. It returns
// key version <= the expected version (maxVs). If not found, it returns an empty
// y.ValueStruct.
// key version <= the expected version (version in key). If not found,
// it returns an empty y.ValueStruct.
func (s *levelsController) get(key []byte, maxVs y.ValueStruct, startLevel int) (
y.ValueStruct, error) {
if s.kv.IsClosed() {
@@ -1605,6 +1628,7 @@ func (s *levelsController) get(key []byte, maxVs y.ValueStruct, startLevel int)
if vs.Value == nil && vs.Meta == 0 {
continue
}
y.NumBytesReadsLSMAdd(s.kv.opt.MetricsEnabled, int64(len(vs.Value)))
if vs.Version == version {
return vs, nil
}
@@ -1612,6 +1636,9 @@ func (s *levelsController) get(key []byte, maxVs y.ValueStruct, startLevel int)
maxVs = vs
}
}
if len(maxVs.Value) > 0 {
y.NumGetsWithResultsAdd(s.kv.opt.MetricsEnabled, 1)
}
return maxVs, nil
}
@@ -1694,7 +1721,7 @@ type LevelInfo struct {
func (s *levelsController) getLevelInfo() []LevelInfo {
t := s.levelTargets()
prios := s.pickCompactLevels()
prios := s.pickCompactLevels(nil)
result := make([]LevelInfo, len(s.levels))
for i, l := range s.levels {
l.RLock()
vendor/github.com/dgraph-io/badger/v3/logger.go → vendor/github.com/dgraph-io/badger/v4/logger.go
Generated Vendored
View File
vendor/github.com/dgraph-io/badger/v3/managed_db.go → vendor/github.com/dgraph-io/badger/v4/managed_db.go
Generated Vendored
View File
vendor/github.com/dgraph-io/badger/v3/manifest.go → vendor/github.com/dgraph-io/badger/v4/manifest.go
Generated Vendored
+46 -21
View File
@@ -20,18 +20,21 @@ import (
"bufio"
"bytes"
"encoding/binary"
stderrors "errors"
"fmt"
"hash/crc32"
"io"
"math"
"os"
"path/filepath"
"sync"
"github.com/dgraph-io/badger/v3/options"
"github.com/dgraph-io/badger/v3/pb"
"github.com/dgraph-io/badger/v3/y"
"github.com/golang/protobuf/proto"
"github.com/pkg/errors"
"google.golang.org/protobuf/proto"
"github.com/dgraph-io/badger/v4/options"
"github.com/dgraph-io/badger/v4/pb"
"github.com/dgraph-io/badger/v4/y"
)
// Manifest represents the contents of the MANIFEST file in a Badger store.
@@ -79,6 +82,10 @@ type TableManifest struct {
type manifestFile struct {
fp *os.File
directory string
// The external magic number used by the application running badger.
externalMagic uint16
// We make this configurable so that unit tests can hit rewrite() code quickly
deletionsRewriteThreshold int
@@ -124,11 +131,12 @@ func openOrCreateManifestFile(opt Options) (
if opt.InMemory {
return &manifestFile{inMemory: true}, Manifest{}, nil
}
return helpOpenOrCreateManifestFile(opt.Dir, opt.ReadOnly, manifestDeletionsRewriteThreshold)
return helpOpenOrCreateManifestFile(opt.Dir, opt.ReadOnly, opt.ExternalMagicVersion,
manifestDeletionsRewriteThreshold)
}
func helpOpenOrCreateManifestFile(dir string, readOnly bool, deletionsThreshold int) (
*manifestFile, Manifest, error) {
func helpOpenOrCreateManifestFile(dir string, readOnly bool, extMagic uint16,
deletionsThreshold int) (*manifestFile, Manifest, error) {
path := filepath.Join(dir, ManifestFilename)
var flags y.Flags
@@ -144,7 +152,7 @@ func helpOpenOrCreateManifestFile(dir string, readOnly bool, deletionsThreshold
return nil, Manifest{}, fmt.Errorf("no manifest found, required for read-only db")
}
m := createManifest()
fp, netCreations, err := helpRewrite(dir, &m)
fp, netCreations, err := helpRewrite(dir, &m, extMagic)
if err != nil {
return nil, Manifest{}, err
}
@@ -152,13 +160,14 @@ func helpOpenOrCreateManifestFile(dir string, readOnly bool, deletionsThreshold
mf := &manifestFile{
fp: fp,
directory: dir,
externalMagic: extMagic,
manifest: m.clone(),
deletionsRewriteThreshold: deletionsThreshold,
}
return mf, m, nil
}
manifest, truncOffset, err := ReplayManifestFile(fp)
manifest, truncOffset, err := ReplayManifestFile(fp, extMagic)
if err != nil {
_ = fp.Close()
return nil, Manifest{}, err
@@ -179,6 +188,7 @@ func helpOpenOrCreateManifestFile(dir string, readOnly bool, deletionsThreshold
mf := &manifestFile{
fp: fp,
directory: dir,
externalMagic: extMagic,
manifest: manifest.clone(),
deletionsRewriteThreshold: deletionsThreshold,
}
@@ -237,10 +247,10 @@ var syncFunc = func(f *os.File) error { return f.Sync() }
// Has to be 4 bytes. The value can never change, ever, anyway.
var magicText = [4]byte{'B', 'd', 'g', 'r'}
// The magic version number.
const magicVersion = 8
// The magic version number. It is allocated 2 bytes, so it's value must be <= math.MaxUint16
const badgerMagicVersion = 8
func helpRewrite(dir string, m *Manifest) (*os.File, int, error) {
func helpRewrite(dir string, m *Manifest, extMagic uint16) (*os.File, int, error) {
rewritePath := filepath.Join(dir, manifestRewriteFilename)
// We explicitly sync.
fp, err := y.OpenTruncFile(rewritePath, false)
@@ -248,9 +258,16 @@ func helpRewrite(dir string, m *Manifest) (*os.File, int, error) {
return nil, 0, err
}
// magic bytes are structured as
// +---------------------+-------------------------+-----------------------+
// | magicText (4 bytes) | externalMagic (2 bytes) | badgerMagic (2 bytes) |
// +---------------------+-------------------------+-----------------------+
y.AssertTrue(badgerMagicVersion <= math.MaxUint16)
buf := make([]byte, 8)
copy(buf[0:4], magicText[:])
binary.BigEndian.PutUint32(buf[4:8], magicVersion)
binary.BigEndian.PutUint16(buf[4:6], extMagic)
binary.BigEndian.PutUint16(buf[6:8], badgerMagicVersion)
netCreations := len(m.Tables)
changes := m.asChanges()
@@ -305,7 +322,7 @@ func (mf *manifestFile) rewrite() error {
if err := mf.fp.Close(); err != nil {
return err
}
fp, netCreations, err := helpRewrite(mf.directory, &mf.manifest)
fp, netCreations, err := helpRewrite(mf.directory, &mf.manifest, mf.externalMagic)
if err != nil {
return err
}
@@ -336,8 +353,8 @@ func (r *countingReader) ReadByte() (b byte, err error) {
}
var (
errBadMagic = errors.New("manifest has bad magic")
errBadChecksum = errors.New("manifest has checksum mismatch")
errBadMagic = stderrors.New("manifest has bad magic")
errBadChecksum = stderrors.New("manifest has checksum mismatch")
)
// ReplayManifestFile reads the manifest file and constructs two manifest objects. (We need one
@@ -345,7 +362,7 @@ var (
// Also, returns the last offset after a completely read manifest entry -- the file must be
// truncated at that point before further appends are made (if there is a partial entry after
// that). In normal conditions, truncOffset is the file size.
func ReplayManifestFile(fp *os.File) (Manifest, int64, error) {
func ReplayManifestFile(fp *os.File, extMagic uint16) (Manifest, int64, error) {
r := countingReader{wrapped: bufio.NewReader(fp)}
var magicBuf [8]byte
@@ -355,14 +372,22 @@ func ReplayManifestFile(fp *os.File) (Manifest, int64, error) {
if !bytes.Equal(magicBuf[0:4], magicText[:]) {
return Manifest{}, 0, errBadMagic
}
version := y.BytesToU32(magicBuf[4:8])
if version != magicVersion {
extVersion := y.BytesToU16(magicBuf[4:6])
version := y.BytesToU16(magicBuf[6:8])
if version != badgerMagicVersion {
return Manifest{}, 0,
//nolint:lll
fmt.Errorf("manifest has unsupported version: %d (we support %d).\n"+
"Please see https://github.com/dgraph-io/badger/blob/master/README.md#i-see-manifest-has-unsupported-version-x-we-support-y-error"+
"Please see https://dgraph.io/docs/badger/faq/#i-see-manifest-has-unsupported-version-x-we-support-y-error"+
" on how to fix this.",
version, magicVersion)
version, badgerMagicVersion)
}
if extVersion != extMagic {
return Manifest{}, 0,
fmt.Errorf("Cannot open DB because the external magic number doesn't match. "+
"Expected: %d, version present in manifest: %d\n", extMagic, extVersion)
}
stat, err := fp.Stat()
vendor/github.com/dgraph-io/badger/v3/memtable.go → vendor/github.com/dgraph-io/badger/v4/memtable.go
Generated Vendored
+17 -22
View File
@@ -25,7 +25,6 @@ import (
"fmt"
"hash/crc32"
"io"
"io/ioutil"
"os"
"path/filepath"
"sort"
@@ -34,11 +33,12 @@ import (
"sync"
"sync/atomic"
"github.com/dgraph-io/badger/v3/pb"
"github.com/dgraph-io/badger/v3/skl"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/ristretto/z"
"github.com/pkg/errors"
"github.com/dgraph-io/badger/v4/pb"
"github.com/dgraph-io/badger/v4/skl"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2/z"
)
// memTable structure stores a skiplist and a corresponding WAL. Writes to memTable are written
@@ -58,7 +58,7 @@ func (db *DB) openMemTables(opt Options) error {
if db.opt.InMemory {
return nil
}
files, err := ioutil.ReadDir(db.opt.Dir)
files, err := os.ReadDir(db.opt.Dir)
if err != nil {
return errFile(err, db.opt.Dir, "Unable to open mem dir.")
}
@@ -147,8 +147,6 @@ func (db *DB) openMemTable(fid, flags int) (*memTable, error) {
return mt, y.Wrapf(err, "while updating skiplist")
}
var errExpectingNewFile = errors.New("Expecting to create a new file, but found an existing file")
func (db *DB) newMemTable() (*memTable, error) {
mt, err := db.openMemTable(db.nextMemFid, os.O_CREATE|os.O_RDWR)
if err == z.NewFile {
@@ -209,6 +207,7 @@ func (mt *memTable) Put(key []byte, value y.ValueStruct) error {
if ts := y.ParseTs(entry.Key); ts > mt.maxVersion {
mt.maxVersion = ts
}
y.NumBytesWrittenToL0Add(mt.opt.MetricsEnabled, entry.estimateSizeAndSetThreshold(mt.opt.ValueThreshold))
return nil
}
@@ -220,8 +219,8 @@ func (mt *memTable) UpdateSkipList() error {
if err != nil {
return y.Wrapf(err, "while iterating wal: %s", mt.wal.Fd.Name())
}
if endOff < mt.wal.size && mt.opt.ReadOnly {
return y.Wrapf(ErrTruncateNeeded, "end offset: %d < size: %d", endOff, mt.wal.size)
if endOff < mt.wal.size.Load() && mt.opt.ReadOnly {
return y.Wrapf(ErrTruncateNeeded, "end offset: %d < size: %d", endOff, mt.wal.size.Load())
}
return mt.wal.Truncate(int64(endOff))
}
@@ -270,7 +269,7 @@ type logFile struct {
// exclusive ownership to open/close the descriptor, unmap or remove the file.
lock sync.RWMutex
fid uint32
size uint32
size atomic.Uint32
dataKey *pb.DataKey
baseIV []byte
registry *KeyRegistry
@@ -285,7 +284,7 @@ func (lf *logFile) Truncate(end int64) error {
return nil
}
y.AssertTrue(!lf.opt.ReadOnly)
lf.size = uint32(end)
lf.size.Store(uint32(end))
return lf.MmapFile.Truncate(end)
}
@@ -390,14 +389,13 @@ func (lf *logFile) encryptionEnabled() bool {
// Acquire lock on mmap/file if you are calling this
func (lf *logFile) read(p valuePointer) (buf []byte, err error) {
var nbr int64
offset := p.Offset
// Do not convert size to uint32, because the lf.Data can be of size
// 4GB, which overflows the uint32 during conversion to make the size 0,
// causing the read to fail with ErrEOF. See issue #585.
size := int64(len(lf.Data))
valsz := p.Len
lfsz := atomic.LoadUint32(&lf.size)
lfsz := lf.size.Load()
if int64(offset) >= size || int64(offset+valsz) > size ||
// Ensure that the read is within the file's actual size. It might be possible that
// the offset+valsz length is beyond the file's actual size. This could happen when
@@ -406,10 +404,7 @@ func (lf *logFile) read(p valuePointer) (buf []byte, err error) {
err = y.ErrEOF
} else {
buf = lf.Data[offset : offset+valsz]
nbr = int64(valsz)
}
y.NumReadsAdd(lf.opt.MetricsEnabled, 1)
y.NumBytesReadAdd(lf.opt.MetricsEnabled, nbr)
return buf, err
}
@@ -489,7 +484,7 @@ loop:
}
var vp valuePointer
vp.Len = uint32(int(e.hlen) + len(e.Key) + len(e.Value) + crc32.Size)
vp.Len = uint32(e.hlen + len(e.Key) + len(e.Value) + crc32.Size)
read.recordOffset += vp.Len
vp.Offset = e.offset
@@ -561,14 +556,14 @@ func (lf *logFile) open(path string, flags int, fsize int64) error {
os.Remove(path)
return err
}
lf.size = vlogHeaderSize
lf.size.Store(vlogHeaderSize)
} else if ferr != nil {
return y.Wrapf(ferr, "while opening file: %s", path)
}
lf.size = uint32(len(lf.Data))
lf.size.Store(uint32(len(lf.Data)))
if lf.size < vlogHeaderSize {
if lf.size.Load() < vlogHeaderSize {
// Every vlog file should have at least vlogHeaderSize. If it is less than vlogHeaderSize
// then it must have been corrupted. But no need to handle here. log replayer will truncate
// and bootstrap the logfile. So ignoring here.
@@ -579,7 +574,7 @@ func (lf *logFile) open(path string, flags int, fsize int64) error {
buf := make([]byte, vlogHeaderSize)
y.AssertTruef(vlogHeaderSize == copy(buf, lf.Data),
"Unable to copy from %s, size %d", path, lf.size)
"Unable to copy from %s, size %d", path, lf.size.Load())
keyID := binary.BigEndian.Uint64(buf[:8])
// retrieve datakey.
if dk, err := lf.registry.DataKey(keyID); err != nil {
vendor/github.com/dgraph-io/badger/v3/merge.go → vendor/github.com/dgraph-io/badger/v4/merge.go
Generated Vendored
+4 -4
View File
@@ -17,12 +17,12 @@
package badger
import (
stderrors "errors"
"sync"
"time"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/ristretto/z"
"github.com/pkg/errors"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2/z"
)
// MergeOperator represents a Badger merge operator.
@@ -57,7 +57,7 @@ func (db *DB) GetMergeOperator(key []byte,
return op
}
var errNoMerge = errors.New("No need for merge")
var errNoMerge = stderrors.New("No need for merge")
func (op *MergeOperator) iterateAndMerge() (newVal []byte, latest uint64, err error) {
txn := op.db.NewTransaction(false)
vendor/github.com/dgraph-io/badger/v3/options.go → vendor/github.com/dgraph-io/badger/v4/options.go
Generated Vendored
+29 -16
View File
@@ -24,12 +24,12 @@ import (
"strings"
"time"
"github.com/dgraph-io/ristretto/z"
"github.com/pkg/errors"
"github.com/dgraph-io/badger/v3/options"
"github.com/dgraph-io/badger/v3/table"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/badger/v4/options"
"github.com/dgraph-io/badger/v4/table"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2/z"
)
// Note: If you add a new option X make sure you also add a WithX method on Options.
@@ -42,6 +42,8 @@ import (
//
// Each option X is documented on the WithX method.
type Options struct {
testOnlyOptions
// Required options.
Dir string
@@ -112,6 +114,10 @@ type Options struct {
// NamespaceOffset specifies the offset from where the next 8 bytes contains the namespace.
NamespaceOffset int
// Magic version used by the application using badger to ensure that it doesn't open the DB
// with incompatible data format.
ExternalMagicVersion uint16
// Transaction start and commit timestamps are managed by end-user.
// This is only useful for databases built on top of Badger (like Dgraph).
// Not recommended for most users.
@@ -167,8 +173,6 @@ func DefaultOptions(path string) Options {
// Benchmark code can be found in table/builder_test.go file
ZSTDCompressionLevel: 1,
// Nothing to read/write value log using standard File I/O
// MemoryMap to mmap() the value log files
// (2^30 - 1)*2 when mmapping < 2^31 - 1, max int32.
// -1 so 2*ValueLogFileSize won't overflow on 32-bit systems.
ValueLogFileSize: 1<<30 - 1,
@@ -328,7 +332,7 @@ func (opt Options) FromSuperFlag(superflag string) Options {
case reflect.Int, reflect.Int64:
field.SetInt(flags.GetInt64(name))
case reflect.Uint32, reflect.Uint64:
field.SetUint(uint64(flags.GetUint64(name)))
field.SetUint(flags.GetUint64(name))
case reflect.Float64:
field.SetFloat(flags.GetFloat64(name))
case reflect.String:
@@ -457,7 +461,7 @@ func (opt Options) WithLoggingLevel(val loggingLevel) Options {
return opt
}
// WithBaseTableSize returns a new Options value with MaxTableSize set to the given value.
// WithBaseTableSize returns a new Options value with BaseTableSize set to the given value.
//
// BaseTableSize sets the maximum size in bytes for LSM table or file in the base level.
//
@@ -472,7 +476,7 @@ func (opt Options) WithBaseTableSize(val int64) Options {
//
// LevelSizeMultiplier sets the ratio between the maximum sizes of contiguous levels in the LSM.
// Once a level grows to be larger than this ratio allowed, the compaction process will be
// triggered.
// triggered.
//
// The default value of LevelSizeMultiplier is 10.
func (opt Options) WithLevelSizeMultiplier(val int) Options {
@@ -495,7 +499,8 @@ func (opt Options) WithMaxLevels(val int) Options {
// ValueThreshold sets the threshold used to decide whether a value is stored directly in the LSM
// tree or separately in the log value files.
//
// The default value of ValueThreshold is 1 MB, but LSMOnlyOptions sets it to maxValueThreshold.
// The default value of ValueThreshold is 1 MB, and LSMOnlyOptions sets it to maxValueThreshold
// which is set to 1 MB too.
func (opt Options) WithValueThreshold(val int64) Options {
opt.ValueThreshold = val
return opt
@@ -510,7 +515,7 @@ func (opt Options) WithValueThreshold(val int64) Options {
// and only 1 percent in vlog. The value threshold will be dynamically updated within the range of
// [ValueThreshold, Options.maxValueThreshold]
//
// Say VLogPercentile with 1.0 means threshold will eventually set to Options.maxValueThreshold
// # Say VLogPercentile with 1.0 means threshold will eventually set to Options.maxValueThreshold
//
// The default value of VLogPercentile is 0.0.
func (opt Options) WithVLogPercentile(t float64) Options {
@@ -576,7 +581,7 @@ func (opt Options) WithNumLevelZeroTables(val int) Options {
// WithNumLevelZeroTablesStall sets the number of Level 0 tables that once reached causes the DB to
// stall until compaction succeeds.
//
// The default value of NumLevelZeroTablesStall is 10.
// The default value of NumLevelZeroTablesStall is 15.
func (opt Options) WithNumLevelZeroTablesStall(val int) Options {
opt.NumLevelZeroTablesStall = val
return opt
@@ -611,7 +616,7 @@ func (opt Options) WithValueLogMaxEntries(val uint32) Options {
// WithNumCompactors sets the number of compaction workers to run concurrently. Setting this to
// zero stops compactions, which could eventually cause writes to block forever.
//
// The default value of NumCompactors is 2. One is dedicated just for L0 and L1.
// The default value of NumCompactors is 4. One is dedicated just for L0 and L1.
func (opt Options) WithNumCompactors(val int) Options {
opt.NumCompactors = val
return opt
@@ -639,6 +644,8 @@ func (opt Options) WithEncryptionKey(key []byte) Options {
//
// Key Registry will use this duration to create new keys. If the previous generated
// key exceed the given duration. Then the key registry will create new key.
// The default value is set to 10 days.
func (opt Options) WithEncryptionKeyRotationDuration(d time.Duration) Options {
opt.EncryptionKeyRotationDuration = d
return opt
@@ -648,8 +655,7 @@ func (opt Options) WithEncryptionKeyRotationDuration(d time.Duration) Options {
// block will be compressed using the specified algorithm. This option doesn't affect existing
// tables. Only the newly created tables will be compressed.
//
// The default compression algorithm used is zstd when built with Cgo. Without Cgo, the default is
// snappy. Compression is enabled by default.
// The default compression algorithm used is snappy. Compression is enabled by default.
func (opt Options) WithCompression(cType options.CompressionType) Options {
opt.Compression = cType
return opt
@@ -685,7 +691,7 @@ func (opt Options) WithChecksumVerificationMode(cvMode options.ChecksumVerificat
// unnecessary overhead which will affect the read performance. Setting size to
// zero disables the cache altogether.
//
// Default value of BlockCacheSize is zero.
// Default value of BlockCacheSize is 256 MB.
func (opt Options) WithBlockCacheSize(size int64) Options {
opt.BlockCacheSize = size
return opt
@@ -779,6 +785,13 @@ func (opt Options) WithNamespaceOffset(offset int) Options {
return opt
}
// WithExternalMagic returns a new Options value with ExternalMagicVersion set to the given value.
// The DB would fail to start if either the internal or the external magic number fails validated.
func (opt Options) WithExternalMagic(magic uint16) Options {
opt.ExternalMagicVersion = magic
return opt
}
func (opt Options) getFileFlags() int {
var flags int
// opt.SyncWrites would be using msync to sync. All writes go through mmap.
vendor/github.com/dgraph-io/badger/v3/options/options.go → vendor/github.com/dgraph-io/badger/v4/options/options.go
Generated Vendored
View File
vendor/github.com/dgraph-io/badger/v4/pb/badgerpb4.pb.go
Generated Vendored
+870
View File
@@ -0,0 +1,870 @@
//
// Copyright (C) 2017 Dgraph Labs, Inc. and Contributors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Use protos/gen.sh to generate .pb.go files.
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.31.0
// protoc v3.21.12
// source: badgerpb4.proto
package pb
import (
protoreflect "google.golang.org/protobuf/reflect/protoreflect"
protoimpl "google.golang.org/protobuf/runtime/protoimpl"
reflect "reflect"
sync "sync"
)
const (
// Verify that this generated code is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
// Verify that runtime/protoimpl is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
)
type EncryptionAlgo int32
const (
EncryptionAlgo_aes EncryptionAlgo = 0
)
// Enum value maps for EncryptionAlgo.
var (
EncryptionAlgo_name = map[int32]string{
0: "aes",
}
EncryptionAlgo_value = map[string]int32{
"aes": 0,
}
)
func (x EncryptionAlgo) Enum() *EncryptionAlgo {
p := new(EncryptionAlgo)
*p = x
return p
}
func (x EncryptionAlgo) String() string {
return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x))
}
func (EncryptionAlgo) Descriptor() protoreflect.EnumDescriptor {
return file_badgerpb4_proto_enumTypes[0].Descriptor()
}
func (EncryptionAlgo) Type() protoreflect.EnumType {
return &file_badgerpb4_proto_enumTypes[0]
}
func (x EncryptionAlgo) Number() protoreflect.EnumNumber {
return protoreflect.EnumNumber(x)
}
// Deprecated: Use EncryptionAlgo.Descriptor instead.
func (EncryptionAlgo) EnumDescriptor() ([]byte, []int) {
return file_badgerpb4_proto_rawDescGZIP(), []int{0}
}
type ManifestChange_Operation int32
const (
ManifestChange_CREATE ManifestChange_Operation = 0
ManifestChange_DELETE ManifestChange_Operation = 1
)
// Enum value maps for ManifestChange_Operation.
var (
ManifestChange_Operation_name = map[int32]string{
0: "CREATE",
1: "DELETE",
}
ManifestChange_Operation_value = map[string]int32{
"CREATE": 0,
"DELETE": 1,
}
)
func (x ManifestChange_Operation) Enum() *ManifestChange_Operation {
p := new(ManifestChange_Operation)
*p = x
return p
}
func (x ManifestChange_Operation) String() string {
return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x))
}
func (ManifestChange_Operation) Descriptor() protoreflect.EnumDescriptor {
return file_badgerpb4_proto_enumTypes[1].Descriptor()
}
func (ManifestChange_Operation) Type() protoreflect.EnumType {
return &file_badgerpb4_proto_enumTypes[1]
}
func (x ManifestChange_Operation) Number() protoreflect.EnumNumber {
return protoreflect.EnumNumber(x)
}
// Deprecated: Use ManifestChange_Operation.Descriptor instead.
func (ManifestChange_Operation) EnumDescriptor() ([]byte, []int) {
return file_badgerpb4_proto_rawDescGZIP(), []int{3, 0}
}
type Checksum_Algorithm int32
const (
Checksum_CRC32C Checksum_Algorithm = 0
Checksum_XXHash64 Checksum_Algorithm = 1
)
// Enum value maps for Checksum_Algorithm.
var (
Checksum_Algorithm_name = map[int32]string{
0: "CRC32C",
1: "XXHash64",
}
Checksum_Algorithm_value = map[string]int32{
"CRC32C": 0,
"XXHash64": 1,
}
)
func (x Checksum_Algorithm) Enum() *Checksum_Algorithm {
p := new(Checksum_Algorithm)
*p = x
return p
}
func (x Checksum_Algorithm) String() string {
return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x))
}
func (Checksum_Algorithm) Descriptor() protoreflect.EnumDescriptor {
return file_badgerpb4_proto_enumTypes[2].Descriptor()
}
func (Checksum_Algorithm) Type() protoreflect.EnumType {
return &file_badgerpb4_proto_enumTypes[2]
}
func (x Checksum_Algorithm) Number() protoreflect.EnumNumber {
return protoreflect.EnumNumber(x)
}
// Deprecated: Use Checksum_Algorithm.Descriptor instead.
func (Checksum_Algorithm) EnumDescriptor() ([]byte, []int) {
return file_badgerpb4_proto_rawDescGZIP(), []int{4, 0}
}
type KV struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Key []byte `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty"`
Value []byte `protobuf:"bytes,2,opt,name=value,proto3" json:"value,omitempty"`
UserMeta []byte `protobuf:"bytes,3,opt,name=user_meta,json=userMeta,proto3" json:"user_meta,omitempty"`
Version uint64 `protobuf:"varint,4,opt,name=version,proto3" json:"version,omitempty"`
ExpiresAt uint64 `protobuf:"varint,5,opt,name=expires_at,json=expiresAt,proto3" json:"expires_at,omitempty"`
Meta []byte `protobuf:"bytes,6,opt,name=meta,proto3" json:"meta,omitempty"`
// Stream id is used to identify which stream the KV came from.
StreamId uint32 `protobuf:"varint,10,opt,name=stream_id,json=streamId,proto3" json:"stream_id,omitempty"`
// Stream done is used to indicate end of stream.
StreamDone bool `protobuf:"varint,11,opt,name=stream_done,json=streamDone,proto3" json:"stream_done,omitempty"`
}
func (x *KV) Reset() {
*x = KV{}
if protoimpl.UnsafeEnabled {
mi := &file_badgerpb4_proto_msgTypes[0]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *KV) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*KV) ProtoMessage() {}
func (x *KV) ProtoReflect() protoreflect.Message {
mi := &file_badgerpb4_proto_msgTypes[0]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use KV.ProtoReflect.Descriptor instead.
func (*KV) Descriptor() ([]byte, []int) {
return file_badgerpb4_proto_rawDescGZIP(), []int{0}
}
func (x *KV) GetKey() []byte {
if x != nil {
return x.Key
}
return nil
}
func (x *KV) GetValue() []byte {
if x != nil {
return x.Value
}
return nil
}
func (x *KV) GetUserMeta() []byte {
if x != nil {
return x.UserMeta
}
return nil
}
func (x *KV) GetVersion() uint64 {
if x != nil {
return x.Version
}
return 0
}
func (x *KV) GetExpiresAt() uint64 {
if x != nil {
return x.ExpiresAt
}
return 0
}
func (x *KV) GetMeta() []byte {
if x != nil {
return x.Meta
}
return nil
}
func (x *KV) GetStreamId() uint32 {
if x != nil {
return x.StreamId
}
return 0
}
func (x *KV) GetStreamDone() bool {
if x != nil {
return x.StreamDone
}
return false
}
type KVList struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Kv []*KV `protobuf:"bytes,1,rep,name=kv,proto3" json:"kv,omitempty"`
// alloc_ref used internally for memory management.
AllocRef uint64 `protobuf:"varint,10,opt,name=alloc_ref,json=allocRef,proto3" json:"alloc_ref,omitempty"`
}
func (x *KVList) Reset() {
*x = KVList{}
if protoimpl.UnsafeEnabled {
mi := &file_badgerpb4_proto_msgTypes[1]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *KVList) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*KVList) ProtoMessage() {}
func (x *KVList) ProtoReflect() protoreflect.Message {
mi := &file_badgerpb4_proto_msgTypes[1]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use KVList.ProtoReflect.Descriptor instead.
func (*KVList) Descriptor() ([]byte, []int) {
return file_badgerpb4_proto_rawDescGZIP(), []int{1}
}
func (x *KVList) GetKv() []*KV {
if x != nil {
return x.Kv
}
return nil
}
func (x *KVList) GetAllocRef() uint64 {
if x != nil {
return x.AllocRef
}
return 0
}
type ManifestChangeSet struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
// A set of changes that are applied atomically.
Changes []*ManifestChange `protobuf:"bytes,1,rep,name=changes,proto3" json:"changes,omitempty"`
}
func (x *ManifestChangeSet) Reset() {
*x = ManifestChangeSet{}
if protoimpl.UnsafeEnabled {
mi := &file_badgerpb4_proto_msgTypes[2]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *ManifestChangeSet) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*ManifestChangeSet) ProtoMessage() {}
func (x *ManifestChangeSet) ProtoReflect() protoreflect.Message {
mi := &file_badgerpb4_proto_msgTypes[2]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use ManifestChangeSet.ProtoReflect.Descriptor instead.
func (*ManifestChangeSet) Descriptor() ([]byte, []int) {
return file_badgerpb4_proto_rawDescGZIP(), []int{2}
}
func (x *ManifestChangeSet) GetChanges() []*ManifestChange {
if x != nil {
return x.Changes
}
return nil
}
type ManifestChange struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Id uint64 `protobuf:"varint,1,opt,name=Id,proto3" json:"Id,omitempty"` // Table ID.
Op ManifestChange_Operation `protobuf:"varint,2,opt,name=Op,proto3,enum=badgerpb4.ManifestChange_Operation" json:"Op,omitempty"`
Level uint32 `protobuf:"varint,3,opt,name=Level,proto3" json:"Level,omitempty"` // Only used for CREATE.
KeyId uint64 `protobuf:"varint,4,opt,name=key_id,json=keyId,proto3" json:"key_id,omitempty"`
EncryptionAlgo EncryptionAlgo `protobuf:"varint,5,opt,name=encryption_algo,json=encryptionAlgo,proto3,enum=badgerpb4.EncryptionAlgo" json:"encryption_algo,omitempty"`
Compression uint32 `protobuf:"varint,6,opt,name=compression,proto3" json:"compression,omitempty"` // Only used for CREATE Op.
}
func (x *ManifestChange) Reset() {
*x = ManifestChange{}
if protoimpl.UnsafeEnabled {
mi := &file_badgerpb4_proto_msgTypes[3]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *ManifestChange) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*ManifestChange) ProtoMessage() {}
func (x *ManifestChange) ProtoReflect() protoreflect.Message {
mi := &file_badgerpb4_proto_msgTypes[3]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use ManifestChange.ProtoReflect.Descriptor instead.
func (*ManifestChange) Descriptor() ([]byte, []int) {
return file_badgerpb4_proto_rawDescGZIP(), []int{3}
}
func (x *ManifestChange) GetId() uint64 {
if x != nil {
return x.Id
}
return 0
}
func (x *ManifestChange) GetOp() ManifestChange_Operation {
if x != nil {
return x.Op
}
return ManifestChange_CREATE
}
func (x *ManifestChange) GetLevel() uint32 {
if x != nil {
return x.Level
}
return 0
}
func (x *ManifestChange) GetKeyId() uint64 {
if x != nil {
return x.KeyId
}
return 0
}
func (x *ManifestChange) GetEncryptionAlgo() EncryptionAlgo {
if x != nil {
return x.EncryptionAlgo
}
return EncryptionAlgo_aes
}
func (x *ManifestChange) GetCompression() uint32 {
if x != nil {
return x.Compression
}
return 0
}
type Checksum struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Algo Checksum_Algorithm `protobuf:"varint,1,opt,name=algo,proto3,enum=badgerpb4.Checksum_Algorithm" json:"algo,omitempty"` // For storing type of Checksum algorithm used
Sum uint64 `protobuf:"varint,2,opt,name=sum,proto3" json:"sum,omitempty"`
}
func (x *Checksum) Reset() {
*x = Checksum{}
if protoimpl.UnsafeEnabled {
mi := &file_badgerpb4_proto_msgTypes[4]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *Checksum) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*Checksum) ProtoMessage() {}
func (x *Checksum) ProtoReflect() protoreflect.Message {
mi := &file_badgerpb4_proto_msgTypes[4]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use Checksum.ProtoReflect.Descriptor instead.
func (*Checksum) Descriptor() ([]byte, []int) {
return file_badgerpb4_proto_rawDescGZIP(), []int{4}
}
func (x *Checksum) GetAlgo() Checksum_Algorithm {
if x != nil {
return x.Algo
}
return Checksum_CRC32C
}
func (x *Checksum) GetSum() uint64 {
if x != nil {
return x.Sum
}
return 0
}
type DataKey struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
KeyId uint64 `protobuf:"varint,1,opt,name=key_id,json=keyId,proto3" json:"key_id,omitempty"`
Data []byte `protobuf:"bytes,2,opt,name=data,proto3" json:"data,omitempty"`
Iv []byte `protobuf:"bytes,3,opt,name=iv,proto3" json:"iv,omitempty"`
CreatedAt int64 `protobuf:"varint,4,opt,name=created_at,json=createdAt,proto3" json:"created_at,omitempty"`
}
func (x *DataKey) Reset() {
*x = DataKey{}
if protoimpl.UnsafeEnabled {
mi := &file_badgerpb4_proto_msgTypes[5]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *DataKey) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*DataKey) ProtoMessage() {}
func (x *DataKey) ProtoReflect() protoreflect.Message {
mi := &file_badgerpb4_proto_msgTypes[5]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use DataKey.ProtoReflect.Descriptor instead.
func (*DataKey) Descriptor() ([]byte, []int) {
return file_badgerpb4_proto_rawDescGZIP(), []int{5}
}
func (x *DataKey) GetKeyId() uint64 {
if x != nil {
return x.KeyId
}
return 0
}
func (x *DataKey) GetData() []byte {
if x != nil {
return x.Data
}
return nil
}
func (x *DataKey) GetIv() []byte {
if x != nil {
return x.Iv
}
return nil
}
func (x *DataKey) GetCreatedAt() int64 {
if x != nil {
return x.CreatedAt
}
return 0
}
type Match struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Prefix []byte `protobuf:"bytes,1,opt,name=prefix,proto3" json:"prefix,omitempty"`
IgnoreBytes string `protobuf:"bytes,2,opt,name=ignore_bytes,json=ignoreBytes,proto3" json:"ignore_bytes,omitempty"` // Comma separated with dash to represent ranges "1, 2-3, 4-7, 9"
}
func (x *Match) Reset() {
*x = Match{}
if protoimpl.UnsafeEnabled {
mi := &file_badgerpb4_proto_msgTypes[6]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *Match) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*Match) ProtoMessage() {}
func (x *Match) ProtoReflect() protoreflect.Message {
mi := &file_badgerpb4_proto_msgTypes[6]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use Match.ProtoReflect.Descriptor instead.
func (*Match) Descriptor() ([]byte, []int) {
return file_badgerpb4_proto_rawDescGZIP(), []int{6}
}
func (x *Match) GetPrefix() []byte {
if x != nil {
return x.Prefix
}
return nil
}
func (x *Match) GetIgnoreBytes() string {
if x != nil {
return x.IgnoreBytes
}
return ""
}
var File_badgerpb4_proto protoreflect.FileDescriptor
var file_badgerpb4_proto_rawDesc = []byte{
0x0a, 0x0f, 0x62, 0x61, 0x64, 0x67, 0x65, 0x72, 0x70, 0x62, 0x34, 0x2e, 0x70, 0x72, 0x6f, 0x74,
0x6f, 0x12, 0x09, 0x62, 0x61, 0x64, 0x67, 0x65, 0x72, 0x70, 0x62, 0x34, 0x22, 0xd4, 0x01, 0x0a,
0x02, 0x4b, 0x56, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0c,
0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02,
0x20, 0x01, 0x28, 0x0c, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x12, 0x1b, 0x0a, 0x09, 0x75,
0x73, 0x65, 0x72, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x08,
0x75, 0x73, 0x65, 0x72, 0x4d, 0x65, 0x74, 0x61, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, 0x72, 0x73,
0x69, 0x6f, 0x6e, 0x18, 0x04, 0x20, 0x01, 0x28, 0x04, 0x52, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69,
0x6f, 0x6e, 0x12, 0x1d, 0x0a, 0x0a, 0x65, 0x78, 0x70, 0x69, 0x72, 0x65, 0x73, 0x5f, 0x61, 0x74,
0x18, 0x05, 0x20, 0x01, 0x28, 0x04, 0x52, 0x09, 0x65, 0x78, 0x70, 0x69, 0x72, 0x65, 0x73, 0x41,
0x74, 0x12, 0x12, 0x0a, 0x04, 0x6d, 0x65, 0x74, 0x61, 0x18, 0x06, 0x20, 0x01, 0x28, 0x0c, 0x52,
0x04, 0x6d, 0x65, 0x74, 0x61, 0x12, 0x1b, 0x0a, 0x09, 0x73, 0x74, 0x72, 0x65, 0x61, 0x6d, 0x5f,
0x69, 0x64, 0x18, 0x0a, 0x20, 0x01, 0x28, 0x0d, 0x52, 0x08, 0x73, 0x74, 0x72, 0x65, 0x61, 0x6d,
0x49, 0x64, 0x12, 0x1f, 0x0a, 0x0b, 0x73, 0x74, 0x72, 0x65, 0x61, 0x6d, 0x5f, 0x64, 0x6f, 0x6e,
0x65, 0x18, 0x0b, 0x20, 0x01, 0x28, 0x08, 0x52, 0x0a, 0x73, 0x74, 0x72, 0x65, 0x61, 0x6d, 0x44,
0x6f, 0x6e, 0x65, 0x22, 0x44, 0x0a, 0x06, 0x4b, 0x56, 0x4c, 0x69, 0x73, 0x74, 0x12, 0x1d, 0x0a,
0x02, 0x6b, 0x76, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x0d, 0x2e, 0x62, 0x61, 0x64, 0x67,
0x65, 0x72, 0x70, 0x62, 0x34, 0x2e, 0x4b, 0x56, 0x52, 0x02, 0x6b, 0x76, 0x12, 0x1b, 0x0a, 0x09,
0x61, 0x6c, 0x6c, 0x6f, 0x63, 0x5f, 0x72, 0x65, 0x66, 0x18, 0x0a, 0x20, 0x01, 0x28, 0x04, 0x52,
0x08, 0x61, 0x6c, 0x6c, 0x6f, 0x63, 0x52, 0x65, 0x66, 0x22, 0x48, 0x0a, 0x11, 0x4d, 0x61, 0x6e,
0x69, 0x66, 0x65, 0x73, 0x74, 0x43, 0x68, 0x61, 0x6e, 0x67, 0x65, 0x53, 0x65, 0x74, 0x12, 0x33,
0x0a, 0x07, 0x63, 0x68, 0x61, 0x6e, 0x67, 0x65, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32,
0x19, 0x2e, 0x62, 0x61, 0x64, 0x67, 0x65, 0x72, 0x70, 0x62, 0x34, 0x2e, 0x4d, 0x61, 0x6e, 0x69,
0x66, 0x65, 0x73, 0x74, 0x43, 0x68, 0x61, 0x6e, 0x67, 0x65, 0x52, 0x07, 0x63, 0x68, 0x61, 0x6e,
0x67, 0x65, 0x73, 0x22, 0x8d, 0x02, 0x0a, 0x0e, 0x4d, 0x61, 0x6e, 0x69, 0x66, 0x65, 0x73, 0x74,
0x43, 0x68, 0x61, 0x6e, 0x67, 0x65, 0x12, 0x0e, 0x0a, 0x02, 0x49, 0x64, 0x18, 0x01, 0x20, 0x01,
0x28, 0x04, 0x52, 0x02, 0x49, 0x64, 0x12, 0x33, 0x0a, 0x02, 0x4f, 0x70, 0x18, 0x02, 0x20, 0x01,
0x28, 0x0e, 0x32, 0x23, 0x2e, 0x62, 0x61, 0x64, 0x67, 0x65, 0x72, 0x70, 0x62, 0x34, 0x2e, 0x4d,
0x61, 0x6e, 0x69, 0x66, 0x65, 0x73, 0x74, 0x43, 0x68, 0x61, 0x6e, 0x67, 0x65, 0x2e, 0x4f, 0x70,
0x65, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x02, 0x4f, 0x70, 0x12, 0x14, 0x0a, 0x05, 0x4c,
0x65, 0x76, 0x65, 0x6c, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0d, 0x52, 0x05, 0x4c, 0x65, 0x76, 0x65,
0x6c, 0x12, 0x15, 0x0a, 0x06, 0x6b, 0x65, 0x79, 0x5f, 0x69, 0x64, 0x18, 0x04, 0x20, 0x01, 0x28,
0x04, 0x52, 0x05, 0x6b, 0x65, 0x79, 0x49, 0x64, 0x12, 0x42, 0x0a, 0x0f, 0x65, 0x6e, 0x63, 0x72,
0x79, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x61, 0x6c, 0x67, 0x6f, 0x18, 0x05, 0x20, 0x01, 0x28,
0x0e, 0x32, 0x19, 0x2e, 0x62, 0x61, 0x64, 0x67, 0x65, 0x72, 0x70, 0x62, 0x34, 0x2e, 0x45, 0x6e,
0x63, 0x72, 0x79, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x41, 0x6c, 0x67, 0x6f, 0x52, 0x0e, 0x65, 0x6e,
0x63, 0x72, 0x79, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x41, 0x6c, 0x67, 0x6f, 0x12, 0x20, 0x0a, 0x0b,
0x63, 0x6f, 0x6d, 0x70, 0x72, 0x65, 0x73, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x06, 0x20, 0x01, 0x28,
0x0d, 0x52, 0x0b, 0x63, 0x6f, 0x6d, 0x70, 0x72, 0x65, 0x73, 0x73, 0x69, 0x6f, 0x6e, 0x22, 0x23,
0x0a, 0x09, 0x4f, 0x70, 0x65, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x0a, 0x0a, 0x06, 0x43,
0x52, 0x45, 0x41, 0x54, 0x45, 0x10, 0x00, 0x12, 0x0a, 0x0a, 0x06, 0x44, 0x45, 0x4c, 0x45, 0x54,
0x45, 0x10, 0x01, 0x22, 0x76, 0x0a, 0x08, 0x43, 0x68, 0x65, 0x63, 0x6b, 0x73, 0x75, 0x6d, 0x12,
0x31, 0x0a, 0x04, 0x61, 0x6c, 0x67, 0x6f, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x1d, 0x2e,
0x62, 0x61, 0x64, 0x67, 0x65, 0x72, 0x70, 0x62, 0x34, 0x2e, 0x43, 0x68, 0x65, 0x63, 0x6b, 0x73,
0x75, 0x6d, 0x2e, 0x41, 0x6c, 0x67, 0x6f, 0x72, 0x69, 0x74, 0x68, 0x6d, 0x52, 0x04, 0x61, 0x6c,
0x67, 0x6f, 0x12, 0x10, 0x0a, 0x03, 0x73, 0x75, 0x6d, 0x18, 0x02, 0x20, 0x01, 0x28, 0x04, 0x52,
0x03, 0x73, 0x75, 0x6d, 0x22, 0x25, 0x0a, 0x09, 0x41, 0x6c, 0x67, 0x6f, 0x72, 0x69, 0x74, 0x68,
0x6d, 0x12, 0x0a, 0x0a, 0x06, 0x43, 0x52, 0x43, 0x33, 0x32, 0x43, 0x10, 0x00, 0x12, 0x0c, 0x0a,
0x08, 0x58, 0x58, 0x48, 0x61, 0x73, 0x68, 0x36, 0x34, 0x10, 0x01, 0x22, 0x63, 0x0a, 0x07, 0x44,
0x61, 0x74, 0x61, 0x4b, 0x65, 0x79, 0x12, 0x15, 0x0a, 0x06, 0x6b, 0x65, 0x79, 0x5f, 0x69, 0x64,
0x18, 0x01, 0x20, 0x01, 0x28, 0x04, 0x52, 0x05, 0x6b, 0x65, 0x79, 0x49, 0x64, 0x12, 0x12, 0x0a,
0x04, 0x64, 0x61, 0x74, 0x61, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x04, 0x64, 0x61, 0x74,
0x61, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x76, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x02, 0x69,
0x76, 0x12, 0x1d, 0x0a, 0x0a, 0x63, 0x72, 0x65, 0x61, 0x74, 0x65, 0x64, 0x5f, 0x61, 0x74, 0x18,
0x04, 0x20, 0x01, 0x28, 0x03, 0x52, 0x09, 0x63, 0x72, 0x65, 0x61, 0x74, 0x65, 0x64, 0x41, 0x74,
0x22, 0x42, 0x0a, 0x05, 0x4d, 0x61, 0x74, 0x63, 0x68, 0x12, 0x16, 0x0a, 0x06, 0x70, 0x72, 0x65,
0x66, 0x69, 0x78, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x06, 0x70, 0x72, 0x65, 0x66, 0x69,
0x78, 0x12, 0x21, 0x0a, 0x0c, 0x69, 0x67, 0x6e, 0x6f, 0x72, 0x65, 0x5f, 0x62, 0x79, 0x74, 0x65,
0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0b, 0x69, 0x67, 0x6e, 0x6f, 0x72, 0x65, 0x42,
0x79, 0x74, 0x65, 0x73, 0x2a, 0x19, 0x0a, 0x0e, 0x45, 0x6e, 0x63, 0x72, 0x79, 0x70, 0x74, 0x69,
0x6f, 0x6e, 0x41, 0x6c, 0x67, 0x6f, 0x12, 0x07, 0x0a, 0x03, 0x61, 0x65, 0x73, 0x10, 0x00, 0x42,
0x23, 0x5a, 0x21, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x64, 0x67,
0x72, 0x61, 0x70, 0x68, 0x2d, 0x69, 0x6f, 0x2f, 0x62, 0x61, 0x64, 0x67, 0x65, 0x72, 0x2f, 0x76,
0x34, 0x2f, 0x70, 0x62, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
}
var (
file_badgerpb4_proto_rawDescOnce sync.Once
file_badgerpb4_proto_rawDescData = file_badgerpb4_proto_rawDesc
)
func file_badgerpb4_proto_rawDescGZIP() []byte {
file_badgerpb4_proto_rawDescOnce.Do(func() {
file_badgerpb4_proto_rawDescData = protoimpl.X.CompressGZIP(file_badgerpb4_proto_rawDescData)
})
return file_badgerpb4_proto_rawDescData
}
var file_badgerpb4_proto_enumTypes = make([]protoimpl.EnumInfo, 3)
var file_badgerpb4_proto_msgTypes = make([]protoimpl.MessageInfo, 7)
var file_badgerpb4_proto_goTypes = []interface{}{
(EncryptionAlgo)(0), // 0: badgerpb4.EncryptionAlgo
(ManifestChange_Operation)(0), // 1: badgerpb4.ManifestChange.Operation
(Checksum_Algorithm)(0), // 2: badgerpb4.Checksum.Algorithm
(*KV)(nil), // 3: badgerpb4.KV
(*KVList)(nil), // 4: badgerpb4.KVList
(*ManifestChangeSet)(nil), // 5: badgerpb4.ManifestChangeSet
(*ManifestChange)(nil), // 6: badgerpb4.ManifestChange
(*Checksum)(nil), // 7: badgerpb4.Checksum
(*DataKey)(nil), // 8: badgerpb4.DataKey
(*Match)(nil), // 9: badgerpb4.Match
}
var file_badgerpb4_proto_depIdxs = []int32{
3, // 0: badgerpb4.KVList.kv:type_name -> badgerpb4.KV
6, // 1: badgerpb4.ManifestChangeSet.changes:type_name -> badgerpb4.ManifestChange
1, // 2: badgerpb4.ManifestChange.Op:type_name -> badgerpb4.ManifestChange.Operation
0, // 3: badgerpb4.ManifestChange.encryption_algo:type_name -> badgerpb4.EncryptionAlgo
2, // 4: badgerpb4.Checksum.algo:type_name -> badgerpb4.Checksum.Algorithm
5, // [5:5] is the sub-list for method output_type
5, // [5:5] is the sub-list for method input_type
5, // [5:5] is the sub-list for extension type_name
5, // [5:5] is the sub-list for extension extendee
0, // [0:5] is the sub-list for field type_name
}
func init() { file_badgerpb4_proto_init() }
func file_badgerpb4_proto_init() {
if File_badgerpb4_proto != nil {
return
}
if !protoimpl.UnsafeEnabled {
file_badgerpb4_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*KV); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_badgerpb4_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*KVList); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_badgerpb4_proto_msgTypes[2].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*ManifestChangeSet); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_badgerpb4_proto_msgTypes[3].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*ManifestChange); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_badgerpb4_proto_msgTypes[4].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*Checksum); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_badgerpb4_proto_msgTypes[5].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*DataKey); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_badgerpb4_proto_msgTypes[6].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*Match); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
}
type x struct{}
out := protoimpl.TypeBuilder{
File: protoimpl.DescBuilder{
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: file_badgerpb4_proto_rawDesc,
NumEnums: 3,
NumMessages: 7,
NumExtensions: 0,
NumServices: 0,
},
GoTypes: file_badgerpb4_proto_goTypes,
DependencyIndexes: file_badgerpb4_proto_depIdxs,
EnumInfos: file_badgerpb4_proto_enumTypes,
MessageInfos: file_badgerpb4_proto_msgTypes,
}.Build()
File_badgerpb4_proto = out.File
file_badgerpb4_proto_rawDesc = nil
file_badgerpb4_proto_goTypes = nil
file_badgerpb4_proto_depIdxs = nil
}
vendor/github.com/dgraph-io/badger/v3/pb/badgerpb3.proto → vendor/github.com/dgraph-io/badger/v4/pb/badgerpb4.proto
Generated Vendored
+4 -4
View File
@@ -17,9 +17,9 @@
// Use protos/gen.sh to generate .pb.go files.
syntax = "proto3";
package badgerpb3;
package badgerpb4;
option go_package = "github.com/dgraph-io/badger/v3/pb";
option go_package = "github.com/dgraph-io/badger/v4/pb";
message KV {
bytes key = 1;
@@ -54,8 +54,8 @@ enum EncryptionAlgo {
message ManifestChange {
uint64 Id = 1; // Table ID.
enum Operation {
CREATE = 0;
DELETE = 1;
CREATE = 0;
DELETE = 1;
}
Operation Op = 2;
uint32 Level = 3; // Only used for CREATE.
vendor/github.com/dgraph-io/badger/v4/pb/gen.sh
Generated Vendored
+7
View File
@@ -0,0 +1,7 @@
#!/bin/bash
# Run this script from its directory, so that badgerpb4.proto is where it's expected to
# be.
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.31.0
protoc --go_out=. --go_opt=paths=source_relative badgerpb4.proto
vendor/github.com/dgraph-io/badger/v3/publisher.go → vendor/github.com/dgraph-io/badger/v4/publisher.go
Generated Vendored
+18 -15
View File
@@ -20,20 +20,20 @@ import (
"sync"
"sync/atomic"
"github.com/dgraph-io/badger/v3/pb"
"github.com/dgraph-io/badger/v3/trie"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/ristretto/z"
"github.com/dgraph-io/badger/v4/pb"
"github.com/dgraph-io/badger/v4/trie"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2/z"
)
type subscriber struct {
id uint64
id uint64
matches []pb.Match
sendCh chan *pb.KVList
subCloser *z.Closer
// this will be atomic pointer which will be used to
// track whether the subscriber is active or not
active *uint64
active *atomic.Uint64
}
type publisher struct {
@@ -111,32 +111,35 @@ func (p *publisher) publishUpdates(reqs requests) {
}
for id, kvs := range batchedUpdates {
if atomic.LoadUint64(p.subscribers[id].active) == 1 {
if p.subscribers[id].active.Load() == 1 {
p.subscribers[id].sendCh <- kvs
}
}
}
func (p *publisher) newSubscriber(c *z.Closer, matches []pb.Match) subscriber {
func (p *publisher) newSubscriber(c *z.Closer, matches []pb.Match) (subscriber, error) {
p.Lock()
defer p.Unlock()
ch := make(chan *pb.KVList, 1000)
id := p.nextID
// Increment next ID.
p.nextID++
active := uint64(1)
s := subscriber{
active: &active,
id: id,
id: id,
matches: matches,
sendCh: ch,
subCloser: c,
active: new(atomic.Uint64),
}
s.active.Store(1)
p.subscribers[id] = s
for _, m := range matches {
p.indexer.AddMatch(m, id)
if err := p.indexer.AddMatch(m, id); err != nil {
return subscriber{}, err
}
}
return s
return s, nil
}
// cleanSubscribers stops all the subscribers. Ideally, It should be called while closing DB.
@@ -145,7 +148,7 @@ func (p *publisher) cleanSubscribers() {
defer p.Unlock()
for id, s := range p.subscribers {
for _, m := range s.matches {
p.indexer.DeleteMatch(m, id)
_ = p.indexer.DeleteMatch(m, id)
}
delete(p.subscribers, id)
s.subCloser.SignalAndWait()
@@ -157,7 +160,7 @@ func (p *publisher) deleteSubscriber(id uint64) {
defer p.Unlock()
if s, ok := p.subscribers[id]; ok {
for _, m := range s.matches {
p.indexer.DeleteMatch(m, id)
_ = p.indexer.DeleteMatch(m, id)
}
}
delete(p.subscribers, id)
vendor/github.com/dgraph-io/badger/v3/skl/README.md → vendor/github.com/dgraph-io/badger/v4/skl/README.md
Generated Vendored
View File
vendor/github.com/dgraph-io/badger/v3/skl/arena.go → vendor/github.com/dgraph-io/badger/v4/skl/arena.go
Generated Vendored
+9 -11
View File
@@ -20,7 +20,7 @@ import (
"sync/atomic"
"unsafe"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/badger/v4/y"
)
const (
@@ -35,7 +35,7 @@ const (
// Arena should be lock-free.
type Arena struct {
n uint32
n atomic.Uint32
buf []byte
}
@@ -43,15 +43,13 @@ type Arena struct {
func newArena(n int64) *Arena {
// Don't store data at position 0 in order to reserve offset=0 as a kind
// of nil pointer.
out := &Arena{
n: 1,
buf: make([]byte, n),
}
out := &Arena{buf: make([]byte, n)}
out.n.Store(1)
return out
}
func (s *Arena) size() int64 {
return int64(atomic.LoadUint32(&s.n))
return int64(s.n.Load())
}
// putNode allocates a node in the arena. The node is aligned on a pointer-sized
@@ -63,7 +61,7 @@ func (s *Arena) putNode(height int) uint32 {
// Pad the allocation with enough bytes to ensure pointer alignment.
l := uint32(MaxNodeSize - unusedSize + nodeAlign)
n := atomic.AddUint32(&s.n, l)
n := s.n.Add(l)
y.AssertTruef(int(n) <= len(s.buf),
"Arena too small, toWrite:%d newTotal:%d limit:%d",
l, n, len(s.buf))
@@ -78,8 +76,8 @@ func (s *Arena) putNode(height int) uint32 {
// size of val. We could also store this size inside arena but the encoding and
// decoding will incur some overhead.
func (s *Arena) putVal(v y.ValueStruct) uint32 {
l := uint32(v.EncodedSize())
n := atomic.AddUint32(&s.n, l)
l := v.EncodedSize()
n := s.n.Add(l)
y.AssertTruef(int(n) <= len(s.buf),
"Arena too small, toWrite:%d newTotal:%d limit:%d",
l, n, len(s.buf))
@@ -90,7 +88,7 @@ func (s *Arena) putVal(v y.ValueStruct) uint32 {
func (s *Arena) putKey(key []byte) uint32 {
l := uint32(len(key))
n := atomic.AddUint32(&s.n, l)
n := s.n.Add(l)
y.AssertTruef(int(n) <= len(s.buf),
"Arena too small, toWrite:%d newTotal:%d limit:%d",
l, n, len(s.buf))
vendor/github.com/dgraph-io/badger/v3/skl/skl.go → vendor/github.com/dgraph-io/badger/v4/skl/skl.go
Generated Vendored
+21 -23
View File
@@ -37,8 +37,8 @@ import (
"sync/atomic"
"unsafe"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/ristretto/z"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2/z"
)
const (
@@ -54,7 +54,7 @@ type node struct {
// can be atomically loaded and stored:
// value offset: uint32 (bits 0-31)
// value size : uint16 (bits 32-63)
value uint64
value atomic.Uint64
// A byte slice is 24 bytes. We are trying to save space here.
keyOffset uint32 // Immutable. No need to lock to access key.
@@ -70,25 +70,25 @@ type node struct {
// is deliberately truncated to not include unneeded tower elements.
//
// All accesses to elements should use CAS operations, with no need to lock.
tower [maxHeight]uint32
tower [maxHeight]atomic.Uint32
}
type Skiplist struct {
height int32 // Current height. 1 <= height <= kMaxHeight. CAS.
height atomic.Int32 // Current height. 1 <= height <= kMaxHeight. CAS.
head *node
ref int32
ref atomic.Int32
arena *Arena
OnClose func()
}
// IncrRef increases the refcount
func (s *Skiplist) IncrRef() {
atomic.AddInt32(&s.ref, 1)
s.ref.Add(1)
}
// DecrRef decrements the refcount, deallocating the Skiplist when done using it
func (s *Skiplist) DecrRef() {
newRef := atomic.AddInt32(&s.ref, -1)
newRef := s.ref.Add(-1)
if newRef > 0 {
return
}
@@ -111,7 +111,7 @@ func newNode(arena *Arena, key []byte, v y.ValueStruct, height int) *node {
node.keyOffset = arena.putKey(key)
node.keySize = uint16(len(key))
node.height = uint16(height)
node.value = encodeValue(arena.putVal(v), v.EncodedSize())
node.value.Store(encodeValue(arena.putVal(v), v.EncodedSize()))
return node
}
@@ -129,16 +129,14 @@ func decodeValue(value uint64) (valOffset uint32, valSize uint32) {
func NewSkiplist(arenaSize int64) *Skiplist {
arena := newArena(arenaSize)
head := newNode(arena, nil, y.ValueStruct{}, maxHeight)
return &Skiplist{
height: 1,
head: head,
arena: arena,
ref: 1,
}
s := &Skiplist{head: head, arena: arena}
s.height.Store(1)
s.ref.Store(1)
return s
}
func (s *node) getValueOffset() (uint32, uint32) {
value := atomic.LoadUint64(&s.value)
value := s.value.Load()
return decodeValue(value)
}
@@ -149,15 +147,15 @@ func (s *node) key(arena *Arena) []byte {
func (s *node) setValue(arena *Arena, v y.ValueStruct) {
valOffset := arena.putVal(v)
value := encodeValue(valOffset, v.EncodedSize())
atomic.StoreUint64(&s.value, value)
s.value.Store(value)
}
func (s *node) getNextOffset(h int) uint32 {
return atomic.LoadUint32(&s.tower[h])
return s.tower[h].Load()
}
func (s *node) casNextOffset(h int, old, val uint32) bool {
return atomic.CompareAndSwapUint32(&s.tower[h], old, val)
return s.tower[h].CompareAndSwap(old, val)
}
// Returns true if key is strictly > n.key.
@@ -279,7 +277,7 @@ func (s *Skiplist) findSpliceForLevel(key []byte, before *node, level int) (*nod
}
func (s *Skiplist) getHeight() int32 {
return atomic.LoadInt32(&s.height)
return s.height.Load()
}
// Put inserts the key-value pair.
@@ -308,7 +306,7 @@ func (s *Skiplist) Put(key []byte, v y.ValueStruct) {
// Try to increase s.height via CAS.
listHeight = s.getHeight()
for height > int(listHeight) {
if atomic.CompareAndSwapInt32(&s.height, listHeight, int32(height)) {
if s.height.CompareAndSwap(listHeight, int32(height)) {
// Successfully increased skiplist.height.
break
}
@@ -329,7 +327,7 @@ func (s *Skiplist) Put(key []byte, v y.ValueStruct) {
y.AssertTrue(prev[i] != next[i])
}
nextOffset := s.arena.getNodeOffset(next[i])
x.tower[i] = nextOffset
x.tower[i].Store(nextOffset)
if prev[i].casNextOffset(i, nextOffset, s.arena.getNodeOffset(x)) {
// Managed to insert x between prev[i] and next[i]. Go to the next level.
break
@@ -431,7 +429,7 @@ func (s *Iterator) Value() y.ValueStruct {
// ValueUint64 returns the uint64 value of the current node.
func (s *Iterator) ValueUint64() uint64 {
return s.n.value
return s.n.value.Load()
}
// Next advances to the next position.
vendor/github.com/dgraph-io/badger/v3/stream.go → vendor/github.com/dgraph-io/badger/v4/stream.go
Generated Vendored
+31 -19
View File
@@ -24,10 +24,12 @@ import (
"sync/atomic"
"time"
"github.com/dgraph-io/badger/v3/pb"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/ristretto/z"
humanize "github.com/dustin/go-humanize"
"google.golang.org/protobuf/proto"
"github.com/dgraph-io/badger/v4/pb"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2/z"
)
const batchSize = 16 << 20 // 16 MB
@@ -61,6 +63,14 @@ type Stream struct {
// Note: Calls to ChooseKey are concurrent.
ChooseKey func(item *Item) bool
// MaxSize is the maximum allowed size of a stream batch. This is a soft limit
// as a single list that is still over the limit will have to be sent as is since it
// cannot be split further. This limit prevents the framework from creating batches
// so big that sending them causes issues (e.g running into the max size gRPC limit).
// If necessary, set it up before the Stream starts synchronisation
// This is not a concurrency-safe setting
MaxSize uint64
// KeyToList, similar to ChooseKey, is only invoked on the highest version of the value. It
// is upto the caller to iterate over the versions and generate zero, one or more KVs. It
// is expected that the user would advance the iterator to go through the versions of the
@@ -87,10 +97,10 @@ type Stream struct {
db *DB
rangeCh chan keyRange
kvChan chan *z.Buffer
nextStreamId uint32
nextStreamId atomic.Uint32
doneMarkers bool
scanned uint64 // used to estimate the ETA for data scan.
numProducers int32
scanned atomic.Uint64 // used to estimate the ETA for data scan.
numProducers atomic.Int32
}
// SendDoneMarkers when true would send out done markers on the stream. False by default.
@@ -164,8 +174,8 @@ func (st *Stream) produceRanges(ctx context.Context) {
// produceKVs picks up ranges from rangeCh, generates KV lists and sends them to kvChan.
func (st *Stream) produceKVs(ctx context.Context, threadId int) error {
atomic.AddInt32(&st.numProducers, 1)
defer atomic.AddInt32(&st.numProducers, -1)
st.numProducers.Add(1)
defer st.numProducers.Add(-1)
var txn *Txn
if st.readTs > 0 {
@@ -180,7 +190,7 @@ func (st *Stream) produceKVs(ctx context.Context, threadId int) error {
defer func() {
// The outList variable changes. So, we need to evaluate the variable in the defer. DO NOT
// call `defer outList.Release()`.
outList.Release()
_ = outList.Release()
}()
iterate := func(kr keyRange) error {
@@ -197,14 +207,14 @@ func (st *Stream) produceKVs(ctx context.Context, threadId int) error {
defer itr.Alloc.Release()
// This unique stream id is used to identify all the keys from this iteration.
streamId := atomic.AddUint32(&st.nextStreamId, 1)
streamId := st.nextStreamId.Add(1)
var scanned int
sendIt := func() error {
select {
case st.kvChan <- outList:
outList = z.NewBuffer(2*batchSize, "Stream.ProduceKVs")
atomic.AddUint64(&st.scanned, uint64(itr.scanned-scanned))
st.scanned.Add(uint64(itr.scanned - scanned))
scanned = itr.scanned
case <-ctx.Done():
return ctx.Err()
@@ -294,7 +304,7 @@ func (st *Stream) streamKVs(ctx context.Context) error {
now := time.Now()
sendBatch := func(batch *z.Buffer) error {
defer batch.Release()
defer func() { _ = batch.Release() }()
sz := uint64(batch.LenNoPadding())
if sz == 0 {
return nil
@@ -314,7 +324,7 @@ func (st *Stream) streamKVs(ctx context.Context) error {
// Send the batch immediately if it already exceeds the maximum allowed size.
// If the size of the batch exceeds maxStreamSize, break from the loop to
// avoid creating a batch that is so big that certain limits are reached.
if batch.LenNoPadding() > int(maxStreamSize) {
if uint64(batch.LenNoPadding()) > st.MaxSize {
break loop
}
select {
@@ -346,9 +356,9 @@ outer:
// Instead of calculating speed over the entire lifetime, we average the speed over
// ticker duration.
writeRate.Capture(bytesSent)
scanned := atomic.LoadUint64(&st.scanned)
scanned := st.scanned.Load()
scanRate.Capture(scanned)
numProducers := atomic.LoadInt32(&st.numProducers)
numProducers := st.numProducers.Load()
st.db.opt.Infof("%s [%s] Scan (%d): ~%s/%s at %s/sec. Sent: %s at %s/sec."+
" jemalloc: %s\n",
@@ -431,7 +441,7 @@ func (st *Stream) Orchestrate(ctx context.Context) error {
defer func() {
// If due to some error, we have buffers left in kvChan, we should release them.
for buf := range st.kvChan {
buf.Release()
_ = buf.Release()
}
}()
@@ -451,6 +461,7 @@ func (db *DB) newStream() *Stream {
db: db,
NumGo: db.opt.NumGoroutines,
LogPrefix: "Badger.Stream",
MaxSize: maxStreamSize,
}
}
@@ -476,7 +487,7 @@ func BufferToKVList(buf *z.Buffer) (*pb.KVList, error) {
var list pb.KVList
err := buf.SliceIterate(func(s []byte) error {
kv := new(pb.KV)
if err := kv.Unmarshal(s); err != nil {
if err := proto.Unmarshal(s, kv); err != nil {
return err
}
list.Kv = append(list.Kv, kv)
@@ -486,6 +497,7 @@ func BufferToKVList(buf *z.Buffer) (*pb.KVList, error) {
}
func KVToBuffer(kv *pb.KV, buf *z.Buffer) {
out := buf.SliceAllocate(kv.Size())
y.Check2(kv.MarshalToSizedBuffer(out))
in := buf.SliceAllocate(proto.Size(kv))[:0]
_, err := proto.MarshalOptions{}.MarshalAppend(in, kv)
y.AssertTrue(err == nil)
}
vendor/github.com/dgraph-io/badger/v3/stream_writer.go → vendor/github.com/dgraph-io/badger/v4/stream_writer.go
Generated Vendored
+86 -13
View File
@@ -21,12 +21,14 @@ import (
"fmt"
"sync"
"github.com/dgraph-io/badger/v3/pb"
"github.com/dgraph-io/badger/v3/table"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/ristretto/z"
humanize "github.com/dustin/go-humanize"
"github.com/pkg/errors"
"google.golang.org/protobuf/proto"
"github.com/dgraph-io/badger/v4/pb"
"github.com/dgraph-io/badger/v4/table"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2/z"
)
// StreamWriter is used to write data coming from multiple streams. The streams must not have any
@@ -47,6 +49,7 @@ type StreamWriter struct {
throttle *y.Throttle
maxVersion uint64
writers map[uint32]*sortedWriter
prevLevel int
}
// NewStreamWriter creates a StreamWriter. Right after creating StreamWriter, Prepare must be
@@ -66,20 +69,74 @@ func (db *DB) NewStreamWriter() *StreamWriter {
// Prepare should be called before writing any entry to StreamWriter. It deletes all data present in
// existing DB, stops compactions and any writes being done by other means. Be very careful when
// calling Prepare, because it could result in permanent data loss. Not calling Prepare would result
// in a corrupt Badger instance.
// in a corrupt Badger instance. Use PrepareIncremental to do incremental stream write.
func (sw *StreamWriter) Prepare() error {
sw.writeLock.Lock()
defer sw.writeLock.Unlock()
done, err := sw.db.dropAll()
// Ensure that done() is never called more than once.
var once sync.Once
sw.done = func() { once.Do(done) }
return err
}
// PrepareIncremental should be called before writing any entry to StreamWriter incrementally.
// In incremental stream write, the tables are written at one level above the current base level.
func (sw *StreamWriter) PrepareIncremental() error {
sw.writeLock.Lock()
defer sw.writeLock.Unlock()
// Ensure that done() is never called more than once.
var once sync.Once
// prepareToDrop will stop all the incoming writes and process any pending flush tasks.
// Before we start writing, we'll stop the compactions because no one else should be writing to
// the same level as the stream writer is writing to.
f, err := sw.db.prepareToDrop()
if err != nil {
sw.done = func() { once.Do(f) }
return err
}
sw.db.stopCompactions()
done := func() {
sw.db.startCompactions()
f()
}
sw.done = func() { once.Do(done) }
mts, decr := sw.db.getMemTables()
defer decr()
for _, m := range mts {
if !m.sl.Empty() {
return fmt.Errorf("Unable to do incremental writes because MemTable has data")
}
}
isEmptyDB := true
for _, level := range sw.db.Levels() {
if level.NumTables > 0 {
sw.prevLevel = level.Level
isEmptyDB = false
break
}
}
if isEmptyDB {
// If DB is empty, we should allow doing incremental stream write.
return nil
}
if sw.prevLevel == 0 {
// It seems that data is present in all levels from Lmax to L0. If we call flatten
// on the tree, all the data will go to Lmax. All the levels above will be empty
// after flatten call. Now, we should be able to use incremental stream writer again.
if err := sw.db.Flatten(3); err != nil {
return errors.Wrapf(err, "error during flatten in StreamWriter")
}
sw.prevLevel = len(sw.db.Levels()) - 1
}
return nil
}
// Write writes KVList to DB. Each KV within the list contains the stream id which StreamWriter
// would use to demux the writes. Write is thread safe and can be called concurrently by multiple
// goroutines.
@@ -96,7 +153,7 @@ func (sw *StreamWriter) Write(buf *z.Buffer) error {
err := buf.SliceIterate(func(s []byte) error {
var kv pb.KV
if err := kv.Unmarshal(s); err != nil {
if err := proto.Unmarshal(s, &kv); err != nil {
return err
}
if kv.StreamDone {
@@ -109,6 +166,18 @@ func (sw *StreamWriter) Write(buf *z.Buffer) error {
panic(fmt.Sprintf("write performed on closed stream: %d", kv.StreamId))
}
sw.writeLock.Lock()
if sw.maxVersion < kv.Version {
sw.maxVersion = kv.Version
}
if sw.prevLevel == 0 {
// If prevLevel is 0, that means that we have not written anything yet.
// So, we can write to the maxLevel. newWriter writes to prevLevel - 1,
// so we can set prevLevel to len(levels).
sw.prevLevel = len(sw.db.lc.levels)
}
sw.writeLock.Unlock()
var meta, userMeta byte
if len(kv.Meta) > 0 {
meta = kv.Meta[0]
@@ -116,9 +185,6 @@ func (sw *StreamWriter) Write(buf *z.Buffer) error {
if len(kv.UserMeta) > 0 {
userMeta = kv.UserMeta[0]
}
if sw.maxVersion < kv.Version {
sw.maxVersion = kv.Version
}
e := &Entry{
Key: y.KeyWithTs(kv.Key, kv.Version),
Value: y.Copy(kv.Value),
@@ -178,7 +244,7 @@ func (sw *StreamWriter) Write(buf *z.Buffer) error {
for streamId := range closedStreams {
writer, ok := sw.writers[streamId]
if !ok {
sw.db.opt.Logger.Warningf("Trying to close stream: %d, but no sorted "+
sw.db.opt.Warningf("Trying to close stream: %d, but no sorted "+
"writer found for it", streamId)
continue
}
@@ -220,6 +286,11 @@ func (sw *StreamWriter) Flush() error {
if sw.db.orc != nil {
sw.db.orc.Stop()
}
if curMax := sw.db.orc.readTs(); curMax >= sw.maxVersion {
sw.maxVersion = curMax
}
sw.db.orc = newOracle(sw.db.opt)
sw.db.orc.nextTxnTs = sw.maxVersion
sw.db.orc.txnMark.Done(sw.maxVersion)
@@ -284,6 +355,7 @@ type sortedWriter struct {
builder *table.Builder
lastKey []byte
level int
streamID uint32
reqCh chan *request
// Have separate closer for each writer, as it can be closed at any time.
@@ -303,6 +375,7 @@ func (sw *StreamWriter) newWriter(streamID uint32) (*sortedWriter, error) {
builder: table.NewTableBuilder(bopts),
reqCh: make(chan *request, 3),
closer: z.NewCloser(1),
level: sw.prevLevel - 1, // Write at the level just above the one we were writing to.
}
go w.handleRequests()
@@ -434,7 +507,7 @@ func (w *sortedWriter) createTable(builder *table.Builder) error {
}
lc := w.db.lc
lhandler := lc.levels[len(lc.levels)-1]
lhandler := lc.levels[w.level]
// Now that table can be opened successfully, let's add this to the MANIFEST.
change := &pb.ManifestChange{
Id: tbl.ID(),
vendor/github.com/dgraph-io/badger/v3/structs.go → vendor/github.com/dgraph-io/badger/v4/structs.go
Generated Vendored
+3 -2
View File
@@ -72,8 +72,8 @@ type header struct {
const (
// Maximum possible size of the header. The maximum size of header struct will be 18 but the
// maximum size of varint encoded header will be 21.
maxHeaderSize = 21
// maximum size of varint encoded header will be 22.
maxHeaderSize = 22
)
// Encode encodes the header into []byte. The provided []byte should be atleast 5 bytes. The
@@ -174,6 +174,7 @@ func (e *Entry) skipVlogAndSetThreshold(threshold int64) bool {
return int64(len(e.Value)) < e.valThreshold
}
//nolint:unused
func (e Entry) print(prefix string) {
fmt.Printf("%s Key: %s Meta: %d UserMeta: %d Offset: %d len(val)=%d",
prefix, e.Key, e.meta, e.UserMeta, e.offset, len(e.Value))
vendor/github.com/dgraph-io/badger/v3/table/README.md → vendor/github.com/dgraph-io/badger/v4/table/README.md
Generated Vendored
View File
vendor/github.com/dgraph-io/badger/v3/table/builder.go → vendor/github.com/dgraph-io/badger/v4/table/builder.go
Generated Vendored
+28 -20
View File
@@ -24,16 +24,16 @@ import (
"sync/atomic"
"unsafe"
"github.com/dgraph-io/badger/v3/fb"
"github.com/golang/protobuf/proto"
"github.com/golang/snappy"
fbs "github.com/google/flatbuffers/go"
"github.com/klauspost/compress/s2"
"github.com/pkg/errors"
"google.golang.org/protobuf/proto"
"github.com/dgraph-io/badger/v3/options"
"github.com/dgraph-io/badger/v3/pb"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/ristretto/z"
"github.com/dgraph-io/badger/v4/fb"
"github.com/dgraph-io/badger/v4/options"
"github.com/dgraph-io/badger/v4/pb"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2/z"
)
const (
@@ -80,11 +80,10 @@ type Builder struct {
// Typically tens or hundreds of meg. This is for one single file.
alloc *z.Allocator
curBlock *bblock
compressedSize uint32
uncompressedSize uint32
compressedSize atomic.Uint32
uncompressedSize atomic.Uint32
lenOffsets uint32
estimatedSize uint32
keyHashes []uint32 // Used for building the bloomfilter.
opts *Options
maxVersion uint64
@@ -157,6 +156,16 @@ func NewTableBuilder(opts Options) *Builder {
return b
}
func maxEncodedLen(ctype options.CompressionType, sz int) int {
switch ctype {
case options.Snappy:
return s2.MaxEncodedLen(sz)
case options.ZSTD:
return y.ZSTDCompressBound(sz)
}
return sz
}
func (b *Builder) handleBlock() {
defer b.wg.Done()
@@ -179,13 +188,13 @@ func (b *Builder) handleBlock() {
// BlockBuf should always less than or equal to allocated space. If the blockBuf is greater
// than allocated space that means the data from this block cannot be stored in its
// existing location.
allocatedSpace := (item.end) + padding + 1
allocatedSpace := maxEncodedLen(b.opts.Compression, (item.end)) + padding + 1
y.AssertTrue(len(blockBuf) <= allocatedSpace)
// blockBuf was allocated on allocator. So, we don't need to copy it over.
item.data = blockBuf
item.end = len(blockBuf)
atomic.AddUint32(&b.compressedSize, uint32(len(blockBuf)))
b.compressedSize.Add(uint32(len(blockBuf)))
}
}
@@ -276,7 +285,7 @@ func (b *Builder) finishBlock() {
b.append(y.U32ToBytes(uint32(len(checksum))))
b.blockList = append(b.blockList, b.curBlock)
atomic.AddUint32(&b.uncompressedSize, uint32(b.curBlock.end))
b.uncompressedSize.Add(uint32(b.curBlock.end))
// Add length of baseKey (rounded to next multiple of 4 because of alignment).
// Add another 40 Bytes, these additional 40 bytes consists of
@@ -291,7 +300,6 @@ func (b *Builder) finishBlock() {
if b.blockChan != nil {
b.blockChan <- b.curBlock
}
return
}
func (b *Builder) shouldFinishBlock(key []byte, value y.ValueStruct) bool {
@@ -308,7 +316,7 @@ func (b *Builder) shouldFinishBlock(key []byte, value y.ValueStruct) bool {
8 + // Sum64 in checksum proto
4) // checksum length
estimatedSize := uint32(b.curBlock.end) + uint32(6 /*header size for entry*/) +
uint32(len(key)) + uint32(value.EncodedSize()) + entriesOffsetsSize
uint32(len(key)) + value.EncodedSize() + entriesOffsetsSize
if b.shouldEncrypt() {
// IV is added at the end of the block, while encrypting.
@@ -360,9 +368,9 @@ func (b *Builder) addInternal(key []byte, value y.ValueStruct, valueLen uint32,
// ReachedCapacity returns true if we... roughly (?) reached capacity?
func (b *Builder) ReachedCapacity() bool {
// If encryption/compression is enabled then use the compresssed size.
sumBlockSizes := atomic.LoadUint32(&b.compressedSize)
sumBlockSizes := b.compressedSize.Load()
if b.opts.Compression == options.None && b.opts.DataKey == nil {
sumBlockSizes = b.uncompressedSize
sumBlockSizes = b.uncompressedSize.Load()
}
blocksSize := sumBlockSizes + // actual length of current buffer
uint32(len(b.curBlock.entryOffsets)*4) + // all entry offsets size
@@ -515,9 +523,9 @@ func (b *Builder) compressData(data []byte) ([]byte, error) {
case options.None:
return data, nil
case options.Snappy:
sz := snappy.MaxEncodedLen(len(data))
sz := s2.MaxEncodedLen(len(data))
dst := b.alloc.Allocate(sz)
return snappy.Encode(dst, data), nil
return s2.EncodeSnappy(dst, data), nil
case options.ZSTD:
sz := y.ZSTDCompressBound(len(data))
dst := b.alloc.Allocate(sz)
@@ -549,7 +557,7 @@ func (b *Builder) buildIndex(bloom []byte) ([]byte, uint32) {
fb.TableIndexAddOffsets(builder, boEnd)
fb.TableIndexAddBloomFilter(builder, bfoff)
fb.TableIndexAddMaxVersion(builder, b.maxVersion)
fb.TableIndexAddUncompressedSize(builder, b.uncompressedSize)
fb.TableIndexAddUncompressedSize(builder, b.uncompressedSize.Load())
fb.TableIndexAddKeyCount(builder, uint32(len(b.keyHashes)))
fb.TableIndexAddOnDiskSize(builder, b.onDiskSize)
fb.TableIndexAddStaleDataSize(builder, uint32(b.staleDataSize))
vendor/github.com/dgraph-io/badger/v3/table/iterator.go → vendor/github.com/dgraph-io/badger/v4/table/iterator.go
Generated Vendored
+4 -4
View File
@@ -22,8 +22,8 @@ import (
"io"
"sort"
"github.com/dgraph-io/badger/v3/fb"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/badger/v4/fb"
"github.com/dgraph-io/badger/v4/y"
)
type blockIterator struct {
@@ -34,7 +34,7 @@ type blockIterator struct {
key []byte
val []byte
entryOffsets []uint32
block *block
block *Block
tableID uint64
blockID int
@@ -43,7 +43,7 @@ type blockIterator struct {
prevOverlap uint16
}
func (itr *blockIterator) setBlock(b *block) {
func (itr *blockIterator) setBlock(b *Block) {
// Decrement the ref for the old block. If the old block was compressed, we
// might be able to reuse it.
itr.block.decrRef()
vendor/github.com/dgraph-io/badger/v3/table/merge_iterator.go → vendor/github.com/dgraph-io/badger/v4/table/merge_iterator.go
Generated Vendored
+2 -2
View File
@@ -19,7 +19,7 @@ package table
import (
"bytes"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/badger/v4/y"
)
// MergeIterator merges multiple iterators.
@@ -115,7 +115,7 @@ func (mi *MergeIterator) fix() {
case cmp < 0: // Small is less than bigger().
if mi.reverse {
mi.swapSmall()
} else {
} else { //nolint:staticcheck
// we don't need to do anything. Small already points to the smallest.
}
return
vendor/github.com/dgraph-io/badger/v3/table/table.go → vendor/github.com/dgraph-io/badger/v4/table/table.go
Generated Vendored
+48 -44
View File
@@ -31,16 +31,17 @@ import (
"time"
"unsafe"
"github.com/golang/protobuf/proto"
"github.com/golang/snappy"
"github.com/klauspost/compress/snappy"
"github.com/klauspost/compress/zstd"
"github.com/pkg/errors"
"google.golang.org/protobuf/proto"
"github.com/dgraph-io/badger/v3/fb"
"github.com/dgraph-io/badger/v3/options"
"github.com/dgraph-io/badger/v3/pb"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/ristretto"
"github.com/dgraph-io/ristretto/z"
"github.com/dgraph-io/badger/v4/fb"
"github.com/dgraph-io/badger/v4/options"
"github.com/dgraph-io/badger/v4/pb"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2"
"github.com/dgraph-io/ristretto/v2/z"
)
const fileSuffix = ".sst"
@@ -76,8 +77,8 @@ type Options struct {
Compression options.CompressionType
// Block cache is used to cache decompressed and decrypted blocks.
BlockCache *ristretto.Cache
IndexCache *ristretto.Cache
BlockCache *ristretto.Cache[[]byte, *Block]
IndexCache *ristretto.Cache[uint64, *fb.TableIndex]
AllocPool *z.AllocatorPool
@@ -102,7 +103,7 @@ type Table struct {
_index *fb.TableIndex // Nil if encryption is enabled. Use fetchIndex to access.
_cheap *cheapIndex
ref int32 // For file garbage collection. Atomic.
ref atomic.Int32 // For file garbage collection
// The following are initialized once and const.
smallest, biggest []byte // Smallest and largest keys (with timestamps).
@@ -155,12 +156,12 @@ func (t *Table) CompressionType() options.CompressionType {
// IncrRef increments the refcount (having to do with whether the file should be deleted)
func (t *Table) IncrRef() {
atomic.AddInt32(&t.ref, 1)
t.ref.Add(1)
}
// DecrRef decrements the refcount and possibly deletes the table
func (t *Table) DecrRef() error {
newRef := atomic.AddInt32(&t.ref, -1)
newRef := t.ref.Add(-1)
if newRef == 0 {
// We can safely delete this file, because for all the current files, we always have
// at least one reference pointing to them.
@@ -177,13 +178,11 @@ func (t *Table) DecrRef() error {
}
// BlockEvictHandler is used to reuse the byte slice stored in the block on cache eviction.
func BlockEvictHandler(value interface{}) {
if b, ok := value.(*block); ok {
b.decrRef()
}
func BlockEvictHandler(b *Block) {
b.decrRef()
}
type block struct {
type Block struct {
offset int
data []byte
checksum []byte
@@ -191,19 +190,19 @@ type block struct {
entryOffsets []uint32 // used to binary search an entry in the block.
chkLen int // checksum length.
freeMe bool // used to determine if the blocked should be reused.
ref int32
ref atomic.Int32
}
var NumBlocks int32
var NumBlocks atomic.Int32
// incrRef increments the ref of a block and return a bool indicating if the
// increment was successful. A true value indicates that the block can be used.
func (b *block) incrRef() bool {
func (b *Block) incrRef() bool {
for {
// We can't blindly add 1 to ref. We need to check whether it has
// reached zero first, because if it did, then we should absolutely not
// use this block.
ref := atomic.LoadInt32(&b.ref)
ref := b.ref.Load()
// The ref would not be equal to 0 unless the existing
// block get evicted before this line. If the ref is zero, it means that
// the block is already added the the blockPool and cannot be used
@@ -216,12 +215,12 @@ func (b *block) incrRef() bool {
// Increment the ref only if it is not zero and has not changed between
// the time we read it and we're updating it.
//
if atomic.CompareAndSwapInt32(&b.ref, ref, ref+1) {
if b.ref.CompareAndSwap(ref, ref+1) {
return true
}
}
}
func (b *block) decrRef() {
func (b *Block) decrRef() {
if b == nil {
return
}
@@ -232,21 +231,21 @@ func (b *block) decrRef() {
// In case of an uncompressed block, the []byte is a reference to the
// table.mmap []byte slice. Any attempt to write data to the mmap []byte
// will lead to SEGFAULT.
if atomic.AddInt32(&b.ref, -1) == 0 {
if b.ref.Add(-1) == 0 {
if b.freeMe {
z.Free(b.data)
}
atomic.AddInt32(&NumBlocks, -1)
NumBlocks.Add(-1)
// blockPool.Put(&b.data)
}
y.AssertTrue(atomic.LoadInt32(&b.ref) >= 0)
y.AssertTrue(b.ref.Load() >= 0)
}
func (b *block) size() int64 {
func (b *Block) size() int64 {
return int64(3*intSize /* Size of the offset, entriesIndexStart and chkLen */ +
cap(b.data) + cap(b.checksum) + cap(b.entryOffsets)*4)
}
func (b block) verifyCheckSum() error {
func (b *Block) verifyCheckSum() error {
cs := &pb.Checksum{}
if err := proto.Unmarshal(b.checksum, cs); err != nil {
return y.Wrapf(err, "unable to unmarshal checksum for block")
@@ -297,13 +296,14 @@ func OpenTable(mf *z.MmapFile, opts Options) (*Table, error) {
}
t := &Table{
MmapFile: mf,
ref: 1, // Caller is given one reference.
id: id,
opt: &opts,
IsInmemory: false,
tableSize: int(fileInfo.Size()),
CreatedAt: fileInfo.ModTime(),
}
// Caller is given one reference.
t.ref.Store(1)
if err := t.initBiggestAndSmallest(); err != nil {
return nil, y.Wrapf(err, "failed to initialize table")
@@ -328,12 +328,13 @@ func OpenInMemoryTable(data []byte, id uint64, opt *Options) (*Table, error) {
}
t := &Table{
MmapFile: mf,
ref: 1, // Caller is given one reference.
opt: opt,
tableSize: len(data),
IsInmemory: true,
id: id, // It is important that each table gets a unique ID.
}
// Caller is given one reference.
t.ref.Store(1)
if err := t.initBiggestAndSmallest(); err != nil {
return nil, err
@@ -379,7 +380,7 @@ func (t *Table) initBiggestAndSmallest() error {
checksum := &pb.Checksum{}
readPos -= checksumLen
buf = t.readNoFail(readPos, checksumLen)
proto.Unmarshal(buf, checksum)
_ = proto.Unmarshal(buf, checksum)
fmt.Fprintf(&debugBuf, "checksum: %+v ", checksum)
// Read index size from the footer.
@@ -518,7 +519,7 @@ func (t *Table) fetchIndex() *fb.TableIndex {
panic("Index Cache must be set for encrypted workloads")
}
if val, ok := t.opt.IndexCache.Get(t.indexKey()); ok && val != nil {
return val.(*fb.TableIndex)
return val
}
index, err := t.readTableIndex()
@@ -534,7 +535,7 @@ func (t *Table) offsets(ko *fb.BlockOffset, i int) bool {
// block function return a new block. Each block holds a ref and the byte
// slice stored in the block will be reused when the ref becomes zero. The
// caller should release the block by calling block.decrRef() on it.
func (t *Table) block(idx int, useCache bool) (*block, error) {
func (t *Table) block(idx int, useCache bool) (*Block, error) {
y.AssertTruef(idx >= 0, "idx=%d", idx)
if idx >= t.offsetsLength() {
return nil, errors.New("block out of index")
@@ -546,20 +547,18 @@ func (t *Table) block(idx int, useCache bool) (*block, error) {
// Use the block only if the increment was successful. The block
// could get evicted from the cache between the Get() call and the
// incrRef() call.
if b := blk.(*block); b.incrRef() {
return b, nil
if blk.incrRef() {
return blk, nil
}
}
}
var ko fb.BlockOffset
y.AssertTrue(t.offsets(&ko, idx))
blk := &block{
offset: int(ko.Offset()),
ref: 1,
}
blk := &Block{offset: int(ko.Offset())}
blk.ref.Store(1)
defer blk.decrRef() // Deal with any errors, where blk would not be returned.
atomic.AddInt32(&NumBlocks, 1)
NumBlocks.Add(1)
var err error
if blk.data, err = t.read(blk.offset, int(ko.Len())); err != nil {
@@ -794,7 +793,7 @@ func NewFilename(id uint64, dir string) string {
}
// decompress decompresses the data stored in a block.
func (t *Table) decompress(b *block) error {
func (t *Table) decompress(b *Block) error {
var dst []byte
var err error
@@ -809,7 +808,7 @@ func (t *Table) decompress(b *block) error {
if sz, err := snappy.DecodedLen(b.data); err == nil {
dst = z.Calloc(sz, "Table.Decompress")
} else {
dst = z.Calloc(len(b.data) * 4, "Table.Decompress") // Take a guess.
dst = z.Calloc(len(b.data)*4, "Table.Decompress") // Take a guess.
}
b.data, err = snappy.Decode(dst, b.data)
if err != nil {
@@ -818,6 +817,11 @@ func (t *Table) decompress(b *block) error {
}
case options.ZSTD:
sz := int(float64(t.opt.BlockSize) * 1.2)
// Get frame content size from header.
var hdr zstd.Header
if err := hdr.Decode(b.data); err == nil && hdr.HasFCS && hdr.FrameContentSize < uint64(t.opt.BlockSize*2) {
sz = int(hdr.FrameContentSize)
}
dst = z.Calloc(sz, "Table.Decompress")
b.data, err = y.ZSTDDecompress(dst, b.data)
if err != nil {
@@ -828,7 +832,7 @@ func (t *Table) decompress(b *block) error {
return errors.New("Unsupported compression type")
}
if b.freeMe == true {
if b.freeMe {
z.Free(src)
b.freeMe = false
}
vendor/github.com/dgraph-io/badger/v3/test.sh → vendor/github.com/dgraph-io/badger/v4/test.sh
Generated Vendored
+17 -18
View File
@@ -16,7 +16,7 @@ fi
# Run `go list` BEFORE setting GOFLAGS so that the output is in the right
# format for grep.
# export packages because the test will run in a sub process.
export packages=$(go list ./... | grep "github.com/dgraph-io/badger/v3/")
export packages=$(go list ./... | grep "github.com/dgraph-io/badger/v4/")
tags="-tags=jemalloc"
@@ -32,7 +32,7 @@ manual() {
set -e
for pkg in $packages; do
echo "===> Testing $pkg"
go test $tags -timeout=25m $covermode $coverprofile -race -parallel 16 $pkg && write_coverage
go test $tags -timeout=25m $covermode $coverprofile -failfast -race -parallel 16 $pkg && write_coverage || return 1
done
echo "==> DONE package tests"
@@ -40,10 +40,10 @@ manual() {
# Run the special Truncate test.
rm -rf p
set -e
go test $tags $timeout $covermode $coverprofile -run='TestTruncateVlogNoClose$' --manual=true && write_coverage
go test $tags $timeout $covermode $coverprofile -run='TestTruncateVlogNoClose$' -failfast --manual=true && write_coverage || return 1
truncate --size=4096 p/000000.vlog
go test $tags $timeout $covermode $coverprofile -run='TestTruncateVlogNoClose2$' --manual=true && write_coverage
go test $tags $timeout $covermode $coverprofile -run='TestTruncateVlogNoClose3$' --manual=true && write_coverage
go test $tags $timeout $covermode $coverprofile -run='TestTruncateVlogNoClose2$' -failfast --manual=true && write_coverage || return 1
go test $tags $timeout $covermode $coverprofile -run='TestTruncateVlogNoClose3$' -failfast --manual=true && write_coverage || return 1
rm -rf p
# TODO(ibrahim): Let's make these tests have Manual prefix.
@@ -52,25 +52,24 @@ manual() {
# TestValueGCManaged
# TestDropPrefix
# TestDropAllManaged
go test $tags $timeout $covermode $coverprofile -run='TestBigKeyValuePairs$' --manual=true && write_coverage
go test $tags $timeout $covermode $coverprofile -run='TestPushValueLogLimit' --manual=true && write_coverage
go test $tags $timeout $covermode $coverprofile -run='TestKeyCount' --manual=true && write_coverage
go test $tags $timeout $covermode $coverprofile -run='TestIteratePrefix' --manual=true && write_coverage
go test $tags $timeout $covermode $coverprofile -run='TestIterateParallel' --manual=true && write_coverage
go test $tags $timeout $covermode $coverprofile -run='TestBigStream' --manual=true && write_coverage
go test $tags $timeout $covermode $coverprofile -run='TestGoroutineLeak' --manual=true && write_coverage
go test $tags $timeout $covermode $coverprofile -run='TestGetMore' --manual=true && write_coverage
go test $tags $timeout $covermode $coverprofile -failfast -run='TestBigKeyValuePairs$' --manual=true && write_coverage || return 1
go test $tags $timeout $covermode $coverprofile -failfast -run='TestPushValueLogLimit' --manual=true && write_coverage || return 1
go test $tags $timeout $covermode $coverprofile -failfast -run='TestKeyCount' --manual=true && write_coverage || return 1
go test $tags $timeout $covermode $coverprofile -failfast -run='TestIteratePrefix' --manual=true && write_coverage || return 1
go test $tags $timeout $covermode $coverprofile -failfast -run='TestIterateParallel' --manual=true && write_coverage || return 1
go test $tags $timeout $covermode $coverprofile -failfast -run='TestBigStream' --manual=true && write_coverage || return 1
go test $tags $timeout $covermode $coverprofile -failfast -run='TestGoroutineLeak' --manual=true && write_coverage || return 1
go test $tags $timeout $covermode $coverprofile -failfast -run='TestGetMore' --manual=true && write_coverage || return 1
echo "==> DONE manual tests"
}
root() {
# Run the normal tests.
# go test -timeout=25m -v -race github.com/dgraph-io/badger/v3/...
# go test -timeout=25m -v -race github.com/dgraph-io/badger/v4/...
echo "==> Running root level tests."
set -e
go test $tags -v -race -parallel=16 -timeout=25m $covermode $coverprofile . && write_coverage
go test $tags -v -race -parallel=16 -timeout=25m -failfast $covermode $coverprofile . && write_coverage || return 1
echo "==> DONE root level tests"
}
@@ -94,8 +93,8 @@ stream() {
}
write_coverage() {
if [ $CI = "true" ]; then
if [ -f cover_tmp.out ]; then
if [[ $CI = "true" ]]; then
if [[ -f cover_tmp.out ]]; then
sed -i '1d' cover_tmp.out
cat cover_tmp.out >> cover.out && rm cover_tmp.out
fi
vendor/github.com/dgraph-io/badger/v4/test_extensions.go
Generated Vendored
+79
View File
@@ -0,0 +1,79 @@
/*
* Copyright 2023 Dgraph Labs, Inc. and Contributors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package badger
// Important: Do NOT import the "testing" package, as otherwise, that
// will pull in imports into the production class that we do not want.
// TODO: Consider using this with specific compilation tags so that it only
// shows up when performing testing (e.g., specify build tag=unit).
// We are not yet ready to do that, as it may impact customer usage as
// well as requiring us to update the CI build flags. Moreover, the
// current model does not actually incur any significant cost.
// If we do this, we will also want to introduce a parallel file that
// overrides some of these structs and functions with empty contents.
// String constants for messages to be pushed to syncChan.
const (
updateDiscardStatsMsg = "updateDiscardStats iteration done"
endVLogInitMsg = "End: vlog.init(db)"
)
// testOnlyOptions specifies an extension to the type Options that we want to
// use only in the context of testing.
type testOnlyOptions struct {
// syncChan is used to listen for specific messages related to activities
// that can occur in a DB instance. Currently, this is only used in
// testing activities.
syncChan chan string
}
// testOnlyDBExtensions specifies an extension to the type DB that we want to
// use only in the context of testing.
type testOnlyDBExtensions struct {
syncChan chan string
// onCloseDiscardCapture will be populated by a DB instance during the
// process of performing the Close operation. Currently, we only consider
// using this during testing.
onCloseDiscardCapture map[uint64]uint64
}
// logToSyncChan sends a message to the DB's syncChan. Note that we expect
// that the DB never closes this channel; the responsibility for
// allocating and closing the channel belongs to the test module.
// if db.syncChan is nil or has never been initialized, ths will be
// silently ignored.
func (db *DB) logToSyncChan(msg string) {
if db.syncChan != nil {
db.syncChan <- msg
}
}
// captureDiscardStats will copy the contents of the discardStats file
// maintained by vlog to the onCloseDiscardCapture map specified by
// db.opt. Of couse, if db.opt.onCloseDiscardCapture is nil (as expected
// for a production system as opposed to a test system), this is a no-op.
func (db *DB) captureDiscardStats() {
if db.onCloseDiscardCapture != nil {
db.vlog.discardStats.Lock()
db.vlog.discardStats.Iterate(func(id, val uint64) {
db.onCloseDiscardCapture[id] = val
})
db.vlog.discardStats.Unlock()
}
}
vendor/github.com/dgraph-io/badger/v3/trie/trie.go → vendor/github.com/dgraph-io/badger/v4/trie/trie.go
Generated Vendored
+4 -3
View File
@@ -21,9 +21,10 @@ import (
"strconv"
"strings"
"github.com/dgraph-io/badger/v3/pb"
"github.com/dgraph-io/badger/v3/y"
"github.com/pkg/errors"
"github.com/dgraph-io/badger/v4/pb"
"github.com/dgraph-io/badger/v4/y"
)
type node struct {
@@ -68,7 +69,7 @@ func parseIgnoreBytes(ig string) ([]bool, error) {
if len(r) == 0 || len(r) > 2 {
return out, fmt.Errorf("Invalid range: %s", each)
}
start, end := -1, -1
start, end := -1, -1 //nolint:ineffassign
if len(r) == 2 {
idx, err := strconv.Atoi(strings.TrimSpace(r[1]))
if err != nil {
vendor/github.com/dgraph-io/badger/v3/txn.go → vendor/github.com/dgraph-io/badger/v4/txn.go
Generated Vendored
+13 -8
View File
@@ -26,9 +26,10 @@ import (
"sync"
"sync/atomic"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/ristretto/z"
"github.com/pkg/errors"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2/z"
)
type oracle struct {
@@ -261,7 +262,7 @@ type Txn struct {
pendingWrites map[string]*Entry // cache stores any writes done by txn.
duplicateWrites []*Entry // Used in managed mode to store duplicate entries.
numIterators int32
numIterators atomic.Int32
discarded bool
doneRead bool
update bool // update is used to conditionally keep track of reads.
@@ -519,7 +520,7 @@ func (txn *Txn) Discard() {
if txn.discarded { // Avoid a re-run.
return
}
if atomic.LoadInt32(&txn.numIterators) > 0 {
if txn.numIterators.Load() > 0 {
panic("Unclosed iterator at time of Txn.Discard.")
}
txn.discarded = true
@@ -660,7 +661,9 @@ func (txn *Txn) Commit() error {
// txn.conflictKeys can be zero if conflict detection is turned off. So we
// should check txn.pendingWrites.
if len(txn.pendingWrites) == 0 {
return nil // Nothing to do.
// Discard the transaction so that the read is marked done.
txn.Discard()
return nil
}
// Precheck before discarding txn.
if err := txn.commitPrecheck(); err != nil {
@@ -715,6 +718,8 @@ func (txn *Txn) CommitWith(cb func(error)) {
// callback might be acquiring the same locks. Instead run the callback
// from another goroutine.
go runTxnCallback(&txnCb{user: cb, err: nil})
// Discard the transaction so that the read is marked done.
txn.Discard()
return
}
@@ -757,9 +762,9 @@ func (txn *Txn) ReadTs() uint64 {
// to. Commit API internally runs Discard, but running it twice wouldn't cause
// any issues.
//
// txn := db.NewTransaction(false)
// defer txn.Discard()
// // Call various APIs.
// txn := db.NewTransaction(false)
// defer txn.Discard()
// // Call various APIs.
func (db *DB) NewTransaction(update bool) *Txn {
return db.newTransaction(update, false)
}
vendor/github.com/dgraph-io/badger/v3/util.go → vendor/github.com/dgraph-io/badger/v4/util.go
Generated Vendored
+6 -6
View File
@@ -18,14 +18,14 @@ package badger
import (
"encoding/hex"
"io/ioutil"
"math/rand"
"sync/atomic"
"os"
"time"
"github.com/dgraph-io/badger/v3/table"
"github.com/dgraph-io/badger/v3/y"
"github.com/pkg/errors"
"github.com/dgraph-io/badger/v4/table"
"github.com/dgraph-io/badger/v4/y"
)
func (s *levelsController) validate() error {
@@ -91,12 +91,12 @@ func (s *levelHandler) validate() error {
// reserveFileID reserves a unique file id.
func (s *levelsController) reserveFileID() uint64 {
id := atomic.AddUint64(&s.nextFileID, 1)
id := s.nextFileID.Add(1)
return id - 1
}
func getIDMap(dir string) map[uint64]struct{} {
fileInfos, err := ioutil.ReadDir(dir)
fileInfos, err := os.ReadDir(dir)
y.Check(err)
idMap := make(map[uint64]struct{})
for _, info := range fileInfos {
vendor/github.com/dgraph-io/badger/v3/value.go → vendor/github.com/dgraph-io/badger/v4/value.go
Generated Vendored
+54 -49
View File
@@ -19,11 +19,11 @@ package badger
import (
"bytes"
"context"
stderrors "errors"
"fmt"
"hash"
"hash/crc32"
"io"
"io/ioutil"
"math"
"os"
"sort"
@@ -32,10 +32,11 @@ import (
"sync"
"sync/atomic"
"github.com/dgraph-io/badger/v3/y"
"github.com/dgraph-io/ristretto/z"
"github.com/pkg/errors"
otrace "go.opencensus.io/trace"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2/z"
)
// maxVlogFileSize is the maximum size of the vlog file which can be created. Vlog Offset is of
@@ -54,7 +55,7 @@ const (
bitTxn byte = 1 << 6 // Set if the entry is part of a txn.
bitFinTxn byte = 1 << 7 // Set if the entry is to indicate end of txn in value log.
mi int64 = 1 << 20
mi int64 = 1 << 20 //nolint:unused
// size of vlog header.
// +----------------+------------------+
@@ -63,9 +64,8 @@ const (
vlogHeaderSize = 20
)
var errStop = errors.New("Stop iteration")
var errTruncate = errors.New("Do truncate")
var errDeleteVlogFile = errors.New("Delete vlog file")
var errStop = stderrors.New("Stop iteration")
var errTruncate = stderrors.New("Do truncate")
type logEntry func(e Entry, vp valuePointer) error
@@ -179,7 +179,7 @@ func (vlog *valueLog) rewrite(f *logFile) error {
}
}
maxFid := vlog.maxFid
y.AssertTruef(uint32(f.fid) < maxFid, "fid to move: %d. Current max fid: %d", f.fid, maxFid)
y.AssertTruef(f.fid < maxFid, "fid to move: %d. Current max fid: %d", f.fid, maxFid)
vlog.filesLock.RUnlock()
vlog.opt.Infof("Rewriting fid: %d", f.fid)
@@ -250,7 +250,7 @@ func (vlog *valueLog) rewrite(f *logFile) error {
}
wb = append(wb, ne)
size += es
} else {
} else { //nolint:staticcheck
// It might be possible that the entry read from LSM Tree points to
// an older vlog file. This can happen in the following situation.
// Assume DB is opened with
@@ -364,15 +364,15 @@ func (vlog *valueLog) rewrite(f *logFile) error {
}
func (vlog *valueLog) incrIteratorCount() {
atomic.AddInt32(&vlog.numActiveIterators, 1)
vlog.numActiveIterators.Add(1)
}
func (vlog *valueLog) iteratorCount() int {
return int(atomic.LoadInt32(&vlog.numActiveIterators))
return int(vlog.numActiveIterators.Load())
}
func (vlog *valueLog) decrIteratorCount() error {
num := atomic.AddInt32(&vlog.numActiveIterators, -1)
num := vlog.numActiveIterators.Add(-1)
if num != 0 {
return nil
}
@@ -439,7 +439,7 @@ func (vlog *valueLog) dropAll() (int, error) {
}
func (db *DB) valueThreshold() int64 {
return atomic.LoadInt64(&db.threshold.valueThreshold)
return db.threshold.valueThreshold.Load()
}
type valueLog struct {
@@ -451,10 +451,10 @@ type valueLog struct {
maxFid uint32
filesToBeDeleted []uint32
// A refcount of iterators -- when this hits zero, we can delete the filesToBeDeleted.
numActiveIterators int32
numActiveIterators atomic.Int32
db *DB
writableLogOffset uint32 // read by read, written by write. Must access via atomics.
writableLogOffset atomic.Uint32 // read by read, written by write
numEntriesWritten uint32
opt Options
@@ -473,7 +473,7 @@ func (vlog *valueLog) fpath(fid uint32) string {
func (vlog *valueLog) populateFilesMap() error {
vlog.filesMap = make(map[uint32]*logFile)
files, err := ioutil.ReadDir(vlog.dirPath)
files, err := os.ReadDir(vlog.dirPath)
if err != nil {
return errFile(err, vlog.dirPath, "Unable to open log dir.")
}
@@ -528,7 +528,7 @@ func (vlog *valueLog) createVlogFile() (*logFile, error) {
// writableLogOffset is only written by write func, by read by Read func.
// To avoid a race condition, all reads and updates to this variable must be
// done via atomics.
atomic.StoreUint32(&vlog.writableLogOffset, vlogHeaderSize)
vlog.writableLogOffset.Store(vlogHeaderSize)
vlog.numEntriesWritten = 0
vlog.filesLock.Unlock()
@@ -555,6 +555,8 @@ func (vlog *valueLog) init(db *DB) {
lf, err := InitDiscardStats(vlog.opt)
y.Check(err)
vlog.discardStats = lf
// See TestPersistLFDiscardStats for purpose of statement below.
db.logToSyncChan(endVLogInitMsg)
}
func (vlog *valueLog) open(db *DB) error {
@@ -587,7 +589,7 @@ func (vlog *valueLog) open(db *DB) error {
return y.Wrapf(err, "Open existing file: %q", lf.path)
}
// We shouldn't delete the maxFid file.
if lf.size == vlogHeaderSize && fid != vlog.maxFid {
if lf.size.Load() == vlogHeaderSize && fid != vlog.maxFid {
vlog.opt.Infof("Deleting empty file: %s", lf.path)
if err := lf.Delete(); err != nil {
return y.Wrapf(err, "while trying to delete empty file: %s", lf.path)
@@ -641,6 +643,7 @@ func (vlog *valueLog) Close() error {
}
}
if vlog.discardStats != nil {
vlog.db.captureDiscardStats()
if terr := vlog.discardStats.Close(-1); terr != nil && err == nil {
err = terr
}
@@ -674,7 +677,7 @@ type request struct {
Ptrs []valuePointer
Wg sync.WaitGroup
Err error
ref int32
ref atomic.Int32
}
func (req *request) reset() {
@@ -682,15 +685,15 @@ func (req *request) reset() {
req.Ptrs = req.Ptrs[:0]
req.Wg = sync.WaitGroup{}
req.Err = nil
req.ref = 0
req.ref.Store(0)
}
func (req *request) IncrRef() {
atomic.AddInt32(&req.ref, 1)
req.ref.Add(1)
}
func (req *request) DecrRef() {
nRef := atomic.AddInt32(&req.ref, -1)
nRef := req.ref.Add(-1)
if nRef > 0 {
return
}
@@ -748,7 +751,7 @@ func (vlog *valueLog) sync() error {
}
func (vlog *valueLog) woffset() uint32 {
return atomic.LoadUint32(&vlog.writableLogOffset)
return vlog.writableLogOffset.Load()
}
// validateWrites will check whether the given requests can fit into 4GB vlog file.
@@ -816,16 +819,19 @@ func (vlog *valueLog) write(reqs []*request) error {
}
n := uint32(buf.Len())
endOffset := atomic.AddUint32(&vlog.writableLogOffset, n)
endOffset := vlog.writableLogOffset.Add(n)
// Increase the file size if we cannot accommodate this entry.
// [Aman] Should this be >= or just >? Doesn't make sense to extend the file if it big enough already.
if int(endOffset) >= len(curlf.Data) {
curlf.Truncate(int64(endOffset))
if err := curlf.Truncate(int64(endOffset)); err != nil {
return err
}
}
start := int(endOffset - n)
y.AssertTrue(copy(curlf.Data[start:], buf.Bytes()) == int(n))
atomic.StoreUint32(&curlf.size, endOffset)
curlf.size.Store(endOffset)
return nil
}
@@ -888,8 +894,8 @@ func (vlog *valueLog) write(reqs []*request) error {
bytesWritten += buf.Len()
// No need to flush anything, we write to file directly via mmap.
}
y.NumWritesAdd(vlog.opt.MetricsEnabled, int64(written))
y.NumBytesWrittenAdd(vlog.opt.MetricsEnabled, int64(bytesWritten))
y.NumWritesVlogAdd(vlog.opt.MetricsEnabled, int64(written))
y.NumBytesWrittenVlogAdd(vlog.opt.MetricsEnabled, int64(bytesWritten))
vlog.numEntriesWritten += uint32(written)
vlog.db.threshold.update(valueSizes)
@@ -964,7 +970,7 @@ func (vlog *valueLog) Read(vp valuePointer, _ *y.Slice) ([]byte, func(), error)
}
}
if uint32(len(kv)) < h.klen+h.vlen {
vlog.db.opt.Logger.Errorf("Invalid read: vp: %+v", vp)
vlog.db.opt.Errorf("Invalid read: vp: %+v", vp)
return nil, nil, errors.Errorf("Invalid read: Len: %d read at:[%d:%d]",
len(kv), h.klen, h.klen+h.vlen)
}
@@ -989,6 +995,8 @@ func (vlog *valueLog) readValueBytes(vp valuePointer) ([]byte, *logFile, error)
}
buf, err := lf.read(vp)
y.NumReadsVlogAdd(vlog.db.opt.MetricsEnabled, 1)
y.NumBytesReadsVlogAdd(vlog.db.opt.MetricsEnabled, int64(len(buf)))
return buf, lf, err
}
@@ -1024,8 +1032,7 @@ LOOP:
discard, thr, fi.Name())
return nil
}
maxFid := atomic.LoadUint32(&vlog.maxFid)
if fid < maxFid {
if fid < vlog.maxFid {
vlog.opt.Infof("Found value log max discard fid: %d discard: %d\n", fid, discard)
lf, ok := vlog.filesMap[fid]
y.AssertTrue(ok)
@@ -1055,12 +1062,6 @@ func discardEntry(e Entry, vs y.ValueStruct, db *DB) bool {
return false
}
type reason struct {
total float64
discard float64
count int
}
func (vlog *valueLog) doRunGC(lf *logFile) error {
_, span := otrace.StartSpan(context.Background(), "Badger.GC")
span.Annotatef(nil, "GC rewrite for: %v", lf.path)
@@ -1108,12 +1109,15 @@ func (vlog *valueLog) updateDiscardStats(stats map[uint32]int64) {
for fid, discard := range stats {
vlog.discardStats.Update(fid, discard)
}
// The following is to coordinate with some test cases where we want to
// verify that at least one iteration of updateDiscardStats has been completed.
vlog.db.logToSyncChan(updateDiscardStatsMsg)
}
type vlogThreshold struct {
logger Logger
percentile float64
valueThreshold int64
valueThreshold atomic.Int64
valueCh chan []int64
clearCh chan bool
closer *z.Closer
@@ -1142,19 +1146,20 @@ func initVlogThreshold(opt *Options) *vlogThreshold {
}
return bounds
}
return &vlogThreshold{
logger: opt.Logger,
percentile: opt.VLogPercentile,
valueThreshold: opt.ValueThreshold,
valueCh: make(chan []int64, 1000),
clearCh: make(chan bool, 1),
closer: z.NewCloser(1),
vlMetrics: z.NewHistogramData(getBounds()),
lt := &vlogThreshold{
logger: opt.Logger,
percentile: opt.VLogPercentile,
valueCh: make(chan []int64, 1000),
clearCh: make(chan bool, 1),
closer: z.NewCloser(1),
vlMetrics: z.NewHistogramData(getBounds()),
}
lt.valueThreshold.Store(opt.ValueThreshold)
return lt
}
func (v *vlogThreshold) Clear(opt Options) {
atomic.StoreInt64(&v.valueThreshold, opt.ValueThreshold)
v.valueThreshold.Store(opt.ValueThreshold)
v.clearCh <- true
}
@@ -1180,11 +1185,11 @@ func (v *vlogThreshold) listenForValueThresholdUpdate() {
// in range of Options.VlogPercentile will make it to the LSM tree and rest to the
// value log file.
p := int64(v.vlMetrics.Percentile(v.percentile))
if atomic.LoadInt64(&v.valueThreshold) != p {
if v.valueThreshold.Load() != p {
if v.logger != nil {
v.logger.Infof("updating value of threshold to: %d", p)
}
atomic.StoreInt64(&v.valueThreshold, p)
v.valueThreshold.Store(p)
}
case <-v.clearCh:
v.vlMetrics.Clear()
vendor/github.com/dgraph-io/badger/v3/y/bloom.go → vendor/github.com/dgraph-io/badger/v4/y/bloom.go
Generated Vendored
+1 -1
View File
@@ -67,7 +67,7 @@ func appendFilter(buf []byte, keys []uint32, bitsPerKey int) []byte {
k = 30
}
nBits := len(keys) * int(bitsPerKey)
nBits := len(keys) * bitsPerKey
// For small len(keys), we can see a very high false positive rate. Fix it
// by enforcing a minimum bloom filter length.
if nBits < 64 {
vendor/github.com/dgraph-io/badger/v3/y/checksum.go → vendor/github.com/dgraph-io/badger/v4/y/checksum.go
Generated Vendored
+4 -4
View File
@@ -17,16 +17,16 @@
package y
import (
stderrors "errors"
"hash/crc32"
"github.com/dgraph-io/badger/v3/pb"
"github.com/cespare/xxhash/v2"
"github.com/cespare/xxhash"
"github.com/pkg/errors"
"github.com/dgraph-io/badger/v4/pb"
)
// ErrChecksumMismatch is returned at checksum mismatch.
var ErrChecksumMismatch = errors.New("checksum mismatch")
var ErrChecksumMismatch = stderrors.New("checksum mismatch")
// CalculateChecksum calculates checksum for data using ct checksum type.
func CalculateChecksum(data []byte, ct pb.Checksum_Algorithm) uint64 {
vendor/github.com/dgraph-io/badger/v3/y/encrypt.go → vendor/github.com/dgraph-io/badger/v4/y/encrypt.go
Generated Vendored
View File
vendor/github.com/dgraph-io/badger/v3/y/error.go → vendor/github.com/dgraph-io/badger/v4/y/error.go
Generated Vendored
+13
View File
@@ -84,3 +84,16 @@ func Wrapf(err error, format string, args ...interface{}) error {
}
return errors.Wrapf(err, format, args...)
}
func CombineErrors(one, other error) error {
if one != nil && other != nil {
return fmt.Errorf("%v; %v", one, other)
}
if one != nil && other == nil {
return fmt.Errorf("%v", one)
}
if one == nil && other != nil {
return fmt.Errorf("%v", other)
}
return nil
}
vendor/github.com/dgraph-io/badger/v3/y/event_log.go → vendor/github.com/dgraph-io/badger/v4/y/event_log.go
Generated Vendored
View File
vendor/github.com/dgraph-io/badger/v3/y/file_dsync.go → vendor/github.com/dgraph-io/badger/v4/y/file_dsync.go
Generated Vendored
+1
View File
@@ -1,3 +1,4 @@
//go:build !dragonfly && !freebsd && !windows && !plan9
// +build !dragonfly,!freebsd,!windows,!plan9
/*
vendor/github.com/dgraph-io/badger/v3/y/file_nodsync.go → vendor/github.com/dgraph-io/badger/v4/y/file_nodsync.go
Generated Vendored
+1
View File
@@ -1,3 +1,4 @@
//go:build dragonfly || freebsd || windows || plan9
// +build dragonfly freebsd windows plan9
/*
vendor/github.com/dgraph-io/badger/v3/y/iterator.go → vendor/github.com/dgraph-io/badger/v4/y/iterator.go
Generated Vendored
View File
vendor/github.com/dgraph-io/badger/v4/y/metrics.go
Generated Vendored
+223
View File
@@ -0,0 +1,223 @@
/*
* Copyright (C) 2017 Dgraph Labs, Inc. and Contributors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package y
import (
"expvar"
)
const (
BADGER_METRIC_PREFIX = "badger_"
)
var (
// lsmSize has size of the LSM in bytes
lsmSize *expvar.Map
// vlogSize has size of the value log in bytes
vlogSize *expvar.Map
// pendingWrites tracks the number of pending writes.
pendingWrites *expvar.Map
// These are cumulative
// VLOG METRICS
// numReads has cumulative number of reads from vlog
numReadsVlog *expvar.Int
// numWrites has cumulative number of writes into vlog
numWritesVlog *expvar.Int
// numBytesRead has cumulative number of bytes read from VLOG
numBytesReadVlog *expvar.Int
// numBytesVlogWritten has cumulative number of bytes written into VLOG
numBytesVlogWritten *expvar.Int
// LSM METRICS
// numBytesRead has cumulative number of bytes read from LSM tree
numBytesReadLSM *expvar.Int
// numBytesWrittenToL0 has cumulative number of bytes written into LSM Tree
numBytesWrittenToL0 *expvar.Int
// numLSMGets is number of LSM gets
numLSMGets *expvar.Map
// numBytesCompactionWritten is the number of bytes written in the lsm tree due to compaction
numBytesCompactionWritten *expvar.Map
// numLSMBloomHits is number of LMS bloom hits
numLSMBloomHits *expvar.Map
// DB METRICS
// numGets is number of gets -> Number of get requests made
numGets *expvar.Int
// number of get queries in which we actually get a result
numGetsWithResults *expvar.Int
// number of iterators created, these would be the number of range queries
numIteratorsCreated *expvar.Int
// numPuts is number of puts -> Number of puts requests made
numPuts *expvar.Int
// numMemtableGets is number of memtable gets -> Number of get requests made on memtable
numMemtableGets *expvar.Int
// numCompactionTables is the number of tables being compacted
numCompactionTables *expvar.Int
// Total writes by a user in bytes
numBytesWrittenUser *expvar.Int
)
// These variables are global and have cumulative values for all kv stores.
// Naming convention of metrics: {badger_version}_{singular operation}_{granularity}_{component}
func init() {
numReadsVlog = expvar.NewInt(BADGER_METRIC_PREFIX + "read_num_vlog")
numBytesReadVlog = expvar.NewInt(BADGER_METRIC_PREFIX + "read_bytes_vlog")
numWritesVlog = expvar.NewInt(BADGER_METRIC_PREFIX + "write_num_vlog")
numBytesVlogWritten = expvar.NewInt(BADGER_METRIC_PREFIX + "write_bytes_vlog")
numBytesReadLSM = expvar.NewInt(BADGER_METRIC_PREFIX + "read_bytes_lsm")
numBytesWrittenToL0 = expvar.NewInt(BADGER_METRIC_PREFIX + "write_bytes_l0")
numBytesCompactionWritten = expvar.NewMap(BADGER_METRIC_PREFIX + "write_bytes_compaction")
numLSMGets = expvar.NewMap(BADGER_METRIC_PREFIX + "get_num_lsm")
numLSMBloomHits = expvar.NewMap(BADGER_METRIC_PREFIX + "hit_num_lsm_bloom_filter")
numMemtableGets = expvar.NewInt(BADGER_METRIC_PREFIX + "get_num_memtable")
// User operations
numGets = expvar.NewInt(BADGER_METRIC_PREFIX + "get_num_user")
numPuts = expvar.NewInt(BADGER_METRIC_PREFIX + "put_num_user")
numBytesWrittenUser = expvar.NewInt(BADGER_METRIC_PREFIX + "write_bytes_user")
// Required for Enabled
numGetsWithResults = expvar.NewInt(BADGER_METRIC_PREFIX + "get_with_result_num_user")
numIteratorsCreated = expvar.NewInt(BADGER_METRIC_PREFIX + "iterator_num_user")
// Sizes
lsmSize = expvar.NewMap(BADGER_METRIC_PREFIX + "size_bytes_lsm")
vlogSize = expvar.NewMap(BADGER_METRIC_PREFIX + "size_bytes_vlog")
pendingWrites = expvar.NewMap(BADGER_METRIC_PREFIX + "write_pending_num_memtable")
numCompactionTables = expvar.NewInt(BADGER_METRIC_PREFIX + "compaction_current_num_lsm")
}
func NumIteratorsCreatedAdd(enabled bool, val int64) {
addInt(enabled, numIteratorsCreated, val)
}
func NumGetsWithResultsAdd(enabled bool, val int64) {
addInt(enabled, numGetsWithResults, val)
}
func NumReadsVlogAdd(enabled bool, val int64) {
addInt(enabled, numReadsVlog, val)
}
func NumBytesWrittenUserAdd(enabled bool, val int64) {
addInt(enabled, numBytesWrittenUser, val)
}
func NumWritesVlogAdd(enabled bool, val int64) {
addInt(enabled, numWritesVlog, val)
}
func NumBytesReadsVlogAdd(enabled bool, val int64) {
addInt(enabled, numBytesReadVlog, val)
}
func NumBytesReadsLSMAdd(enabled bool, val int64) {
addInt(enabled, numBytesReadLSM, val)
}
func NumBytesWrittenVlogAdd(enabled bool, val int64) {
addInt(enabled, numBytesVlogWritten, val)
}
func NumBytesWrittenToL0Add(enabled bool, val int64) {
addInt(enabled, numBytesWrittenToL0, val)
}
func NumBytesCompactionWrittenAdd(enabled bool, key string, val int64) {
addToMap(enabled, numBytesCompactionWritten, key, val)
}
func NumGetsAdd(enabled bool, val int64) {
addInt(enabled, numGets, val)
}
func NumPutsAdd(enabled bool, val int64) {
addInt(enabled, numPuts, val)
}
func NumMemtableGetsAdd(enabled bool, val int64) {
addInt(enabled, numMemtableGets, val)
}
func NumCompactionTablesAdd(enabled bool, val int64) {
addInt(enabled, numCompactionTables, val)
}
func LSMSizeSet(enabled bool, key string, val expvar.Var) {
storeToMap(enabled, lsmSize, key, val)
}
func VlogSizeSet(enabled bool, key string, val expvar.Var) {
storeToMap(enabled, vlogSize, key, val)
}
func PendingWritesSet(enabled bool, key string, val expvar.Var) {
storeToMap(enabled, pendingWrites, key, val)
}
func NumLSMBloomHitsAdd(enabled bool, key string, val int64) {
addToMap(enabled, numLSMBloomHits, key, val)
}
func NumLSMGetsAdd(enabled bool, key string, val int64) {
addToMap(enabled, numLSMGets, key, val)
}
func LSMSizeGet(enabled bool, key string) expvar.Var {
return getFromMap(enabled, lsmSize, key)
}
func VlogSizeGet(enabled bool, key string) expvar.Var {
return getFromMap(enabled, vlogSize, key)
}
func addInt(enabled bool, metric *expvar.Int, val int64) {
if !enabled {
return
}
metric.Add(val)
}
func addToMap(enabled bool, metric *expvar.Map, key string, val int64) {
if !enabled {
return
}
metric.Add(key, val)
}
func storeToMap(enabled bool, metric *expvar.Map, key string, val expvar.Var) {
if !enabled {
return
}
metric.Set(key, val)
}
func getFromMap(enabled bool, metric *expvar.Map, key string) expvar.Var {
if !enabled {
return nil
}
return metric.Get(key)
}
vendor/github.com/dgraph-io/badger/v3/y/watermark.go → vendor/github.com/dgraph-io/badger/v4/y/watermark.go
Generated Vendored
+14 -13
View File
@@ -21,7 +21,7 @@ import (
"context"
"sync/atomic"
"github.com/dgraph-io/ristretto/z"
"github.com/dgraph-io/ristretto/v2/z"
)
type uint64Heap []uint64
@@ -51,8 +51,8 @@ type mark struct {
// WaterMark is used to keep track of the minimum un-finished index. Typically, an index k becomes
// finished or "done" according to a WaterMark once Done(k) has been called
// 1. as many times as Begin(k) has, AND
// 2. a positive number of times.
// 1. as many times as Begin(k) has, AND
// 2. a positive number of times.
//
// An index may also become "done" by calling SetDoneUntil at a time such that it is not
// inter-mingled with Begin/Done calls.
@@ -60,8 +60,8 @@ type mark struct {
// Since doneUntil and lastIndex addresses are passed to sync/atomic packages, we ensure that they
// are 64-bit aligned by putting them at the beginning of the structure.
type WaterMark struct {
doneUntil uint64
lastIndex uint64
doneUntil atomic.Uint64
lastIndex atomic.Uint64
Name string
markCh chan mark
}
@@ -74,13 +74,13 @@ func (w *WaterMark) Init(closer *z.Closer) {
// Begin sets the last index to the given value.
func (w *WaterMark) Begin(index uint64) {
atomic.StoreUint64(&w.lastIndex, index)
w.lastIndex.Store(index)
w.markCh <- mark{index: index, done: false}
}
// BeginMany works like Begin but accepts multiple indices.
func (w *WaterMark) BeginMany(indices []uint64) {
atomic.StoreUint64(&w.lastIndex, indices[len(indices)-1])
w.lastIndex.Store(indices[len(indices)-1])
w.markCh <- mark{index: 0, indices: indices, done: false}
}
@@ -97,18 +97,18 @@ func (w *WaterMark) DoneMany(indices []uint64) {
// DoneUntil returns the maximum index that has the property that all indices
// less than or equal to it are done.
func (w *WaterMark) DoneUntil() uint64 {
return atomic.LoadUint64(&w.doneUntil)
return w.doneUntil.Load()
}
// SetDoneUntil sets the maximum index that has the property that all indices
// less than or equal to it are done.
func (w *WaterMark) SetDoneUntil(val uint64) {
atomic.StoreUint64(&w.doneUntil, val)
w.doneUntil.Store(val)
}
// LastIndex returns the last index for which Begin has been called.
func (w *WaterMark) LastIndex() uint64 {
return atomic.LoadUint64(&w.lastIndex)
return w.lastIndex.Load()
}
// WaitForMark waits until the given index is marked as done.
@@ -182,7 +182,7 @@ func (w *WaterMark) process(closer *z.Closer) {
}
if until != doneUntil {
AssertTrue(atomic.CompareAndSwapUint64(&w.doneUntil, doneUntil, until))
AssertTrue(w.doneUntil.CompareAndSwap(doneUntil, until))
}
notifyAndRemove := func(idx uint64, toNotify []chan struct{}) {
@@ -216,7 +216,7 @@ func (w *WaterMark) process(closer *z.Closer) {
return
case mark := <-w.markCh:
if mark.waiter != nil {
doneUntil := atomic.LoadUint64(&w.doneUntil)
doneUntil := w.doneUntil.Load()
if doneUntil >= mark.index {
close(mark.waiter)
} else {
@@ -228,7 +228,8 @@ func (w *WaterMark) process(closer *z.Closer) {
}
}
} else {
if mark.index > 0 {
// it is possible that mark.index is zero. We need to handle that case as well.
if mark.index > 0 || (mark.index == 0 && len(mark.indices) == 0) {
processOne(mark.index, mark.done)
}
for _, index := range mark.indices {
vendor/github.com/dgraph-io/badger/v3/y/y.go → vendor/github.com/dgraph-io/badger/v4/y/y.go
Generated Vendored
+18 -6
View File
@@ -19,6 +19,7 @@ package y
import (
"bytes"
"encoding/binary"
stderrors "errors"
"fmt"
"hash/crc32"
"io"
@@ -30,19 +31,18 @@ import (
"time"
"unsafe"
"github.com/dgraph-io/badger/v3/pb"
"github.com/dgraph-io/ristretto/z"
"github.com/pkg/errors"
"github.com/dgraph-io/badger/v4/pb"
"github.com/dgraph-io/ristretto/v2/z"
)
var (
// ErrEOF indicates an end of file when trying to read from a memory mapped file
// and encountering the end of slice.
ErrEOF = errors.New("ErrEOF: End of file")
ErrEOF = stderrors.New("ErrEOF: End of file")
// ErrCommitAfterFinish indicates that write batch commit was called after
// finish
ErrCommitAfterFinish = errors.New("Batch commit not permitted after finish")
ErrCommitAfterFinish = stderrors.New("Batch commit not permitted after finish")
)
type Flags int
@@ -257,6 +257,18 @@ func (t *Throttle) Finish() error {
return t.finishErr
}
// U16ToBytes converts the given Uint16 to bytes
func U16ToBytes(v uint16) []byte {
var uBuf [2]byte
binary.BigEndian.PutUint16(uBuf[:], v)
return uBuf[:]
}
// BytesToU16 converts the given byte slice to uint16
func BytesToU16(b []byte) uint16 {
return binary.BigEndian.Uint16(b)
}
// U32ToBytes converts the given Uint32 to bytes
func U32ToBytes(v uint32) []byte {
var uBuf [4]byte
@@ -495,7 +507,7 @@ func (r *PageBufferReader) Read(p []byte) (int, error) {
}
}
if read == 0 {
if read == 0 && len(p) > 0 {
return read, io.EOF
}
vendor/github.com/dgraph-io/badger/v3/y/zstd.go → vendor/github.com/dgraph-io/badger/v4/y/zstd.go
Generated Vendored
View File
vendor/github.com/dgraph-io/ristretto/.deepsource.toml
Generated Vendored
-17
View File
@@ -1,17 +0,0 @@
version = 1
test_patterns = [
'**/*_test.go'
]
exclude_patterns = [
]
[[analyzers]]
name = 'go'
enabled = true
[analyzers.meta]
import_path = 'github.com/dgraph-io/ristretto'
vendor/github.com/dgraph-io/ristretto/.go-version
Generated Vendored
-1
View File
@@ -1 +0,0 @@
1.17.11
vendor/github.com/dgraph-io/ristretto/.golangci.yml
Generated Vendored
-23
View File
@@ -1,23 +0,0 @@
run:
tests: false
skip-dirs:
- contrib
- sim
linters-settings:
lll:
line-length: 120
linters:
disable-all: true
enable:
#- errcheck
#- ineffassign
- gas
#- gofmt
#- golint
#- gosimple
#- govet
- lll
#- varcheck
#- unused
vendor/github.com/dgraph-io/ristretto/README.md
Generated Vendored
-220
View File
@@ -1,220 +0,0 @@
# Ristretto
[![Go Doc](https://img.shields.io/badge/godoc-reference-blue.svg)](http://godoc.org/github.com/dgraph-io/ristretto)
[![ci-ristretto-tests](https://github.com/dgraph-io/ristretto/actions/workflows/ci-ristretto-tests.yml/badge.svg)](https://github.com/dgraph-io/ristretto/actions/workflows/ci-ristretto-tests.yml)
[![ci-ristretto-lint](https://github.com/dgraph-io/ristretto/actions/workflows/ci-ristretto-lint.yml/badge.svg)](https://github.com/dgraph-io/ristretto/actions/workflows/ci-ristretto-lint.yml)
[![Coverage Status](https://coveralls.io/repos/github/dgraph-io/ristretto/badge.svg?branch=main)](https://coveralls.io/github/dgraph-io/ristretto?branch=main)
[![Go Report Card](https://img.shields.io/badge/go%20report-A%2B-brightgreen)](https://goreportcard.com/report/github.com/dgraph-io/ristretto)
Ristretto is a fast, concurrent cache library built with a focus on performance and correctness.
The motivation to build Ristretto comes from the need for a contention-free
cache in [Dgraph][].
[Dgraph]: https://github.com/dgraph-io/dgraph
## Features
* **High Hit Ratios** - with our unique admission/eviction policy pairing, Ristretto's performance is best in class.
* **Eviction: SampledLFU** - on par with exact LRU and better performance on Search and Database traces.
* **Admission: TinyLFU** - extra performance with little memory overhead (12 bits per counter).
* **Fast Throughput** - we use a variety of techniques for managing contention and the result is excellent throughput.
* **Cost-Based Eviction** - any large new item deemed valuable can evict multiple smaller items (cost could be anything).
* **Fully Concurrent** - you can use as many goroutines as you want with little throughput degradation.
* **Metrics** - optional performance metrics for throughput, hit ratios, and other stats.
* **Simple API** - just figure out your ideal `Config` values and you're off and running.
## Status
Ristretto is production-ready. See [Projects using Ristretto](#projects-using-ristretto).
## Table of Contents
* [Usage](#Usage)
* [Example](#Example)
* [Config](#Config)
* [NumCounters](#Config)
* [MaxCost](#Config)
* [BufferItems](#Config)
* [Metrics](#Config)
* [OnEvict](#Config)
* [KeyToHash](#Config)
* [Cost](#Config)
* [Benchmarks](#Benchmarks)
* [Hit Ratios](#Hit-Ratios)
* [Search](#Search)
* [Database](#Database)
* [Looping](#Looping)
* [CODASYL](#CODASYL)
* [Throughput](#Throughput)
* [Mixed](#Mixed)
* [Read](#Read)
* [Write](#Write)
* [Projects using Ristretto](#projects-using-ristretto)
* [FAQ](#FAQ)
## Usage
### Example
```go
func main() {
cache, err := ristretto.NewCache(&ristretto.Config{
NumCounters: 1e7, // number of keys to track frequency of (10M).
MaxCost: 1 << 30, // maximum cost of cache (1GB).
BufferItems: 64, // number of keys per Get buffer.
})
if err != nil {
panic(err)
}
// set a value with a cost of 1
cache.Set("key", "value", 1)
// wait for value to pass through buffers
time.Sleep(10 * time.Millisecond)
value, found := cache.Get("key")
if !found {
panic("missing value")
}
fmt.Println(value)
cache.Del("key")
}
```
### Config
The `Config` struct is passed to `NewCache` when creating Ristretto instances (see the example above).
**NumCounters** `int64`
NumCounters is the number of 4-bit access counters to keep for admission and eviction. We've seen good performance in setting this to 10x the number of items you expect to keep in the cache when full.
For example, if you expect each item to have a cost of 1 and MaxCost is 100, set NumCounters to 1,000. Or, if you use variable cost values but expect the cache to hold around 10,000 items when full, set NumCounters to 100,000. The important thing is the *number of unique items* in the full cache, not necessarily the MaxCost value.
**MaxCost** `int64`
MaxCost is how eviction decisions are made. For example, if MaxCost is 100 and a new item with a cost of 1 increases total cache cost to 101, 1 item will be evicted.
MaxCost can also be used to denote the max size in bytes. For example, if MaxCost is 1,000,000 (1MB) and the cache is full with 1,000 1KB items, a new item (that's accepted) would cause 5 1KB items to be evicted.
MaxCost could be anything as long as it matches how you're using the cost values when calling Set.
**BufferItems** `int64`
BufferItems is the size of the Get buffers. The best value we've found for this is 64.
If for some reason you see Get performance decreasing with lots of contention (you shouldn't), try increasing this value in increments of 64. This is a fine-tuning mechanism and you probably won't have to touch this.
**Metrics** `bool`
Metrics is true when you want real-time logging of a variety of stats. The reason this is a Config flag is because there's a 10% throughput performance overhead.
**OnEvict** `func(hashes [2]uint64, value interface{}, cost int64)`
OnEvict is called for every eviction.
**KeyToHash** `func(key interface{}) [2]uint64`
KeyToHash is the hashing algorithm used for every key. If this is nil, Ristretto has a variety of [defaults depending on the underlying interface type](https://github.com/dgraph-io/ristretto/blob/master/z/z.go#L19-L41).
Note that if you want 128bit hashes you should use the full `[2]uint64`,
otherwise just fill the `uint64` at the `0` position and it will behave like
any 64bit hash.
**Cost** `func(value interface{}) int64`
Cost is an optional function you can pass to the Config in order to evaluate
item cost at runtime, and only for the Set calls that aren't dropped (this is
useful if calculating item cost is particularly expensive and you don't want to
waste time on items that will be dropped anyways).
To signal to Ristretto that you'd like to use this Cost function:
1. Set the Cost field to a non-nil function.
2. When calling Set for new items or item updates, use a `cost` of 0.
## Benchmarks
The benchmarks can be found in https://github.com/dgraph-io/benchmarks/tree/master/cachebench/ristretto.
### Hit Ratios
#### Search
This trace is described as "disk read accesses initiated by a large commercial
search engine in response to various web search requests."
<p align="center">
<img src="https://raw.githubusercontent.com/dgraph-io/ristretto/master/benchmarks/Hit%20Ratios%20-%20Search%20(ARC-S3).svg">
</p>
#### Database
This trace is described as "a database server running at a commercial site
running an ERP application on top of a commercial database."
<p align="center">
<img src="https://raw.githubusercontent.com/dgraph-io/ristretto/master/benchmarks/Hit%20Ratios%20-%20Database%20(ARC-DS1).svg">
</p>
#### Looping
This trace demonstrates a looping access pattern.
<p align="center">
<img src="https://raw.githubusercontent.com/dgraph-io/ristretto/master/benchmarks/Hit%20Ratios%20-%20Glimpse%20(LIRS-GLI).svg">
</p>
#### CODASYL
This trace is described as "references to a CODASYL database for a one hour
period."
<p align="center">
<img src="https://raw.githubusercontent.com/dgraph-io/ristretto/master/benchmarks/Hit%20Ratios%20-%20CODASYL%20(ARC-OLTP).svg">
</p>
### Throughput
All throughput benchmarks were ran on an Intel Core i7-8700K (3.7GHz) with 16gb
of RAM.
#### Mixed
<p align="center">
<img src="https://raw.githubusercontent.com/dgraph-io/ristretto/master/benchmarks/Throughput%20-%20Mixed.svg">
</p>
#### Read
<p align="center">
<img src="https://raw.githubusercontent.com/dgraph-io/ristretto/master/benchmarks/Throughput%20-%20Read%20(Zipfian).svg">
</p>
#### Write
<p align="center">
<img src="https://raw.githubusercontent.com/dgraph-io/ristretto/master/benchmarks/Throughput%20-%20Write%20(Zipfian).svg">
</p>
## Projects Using Ristretto
Below is a list of known projects that use Ristretto:
- [Badger](https://github.com/dgraph-io/badger) - Embeddable key-value DB in Go
- [Dgraph](https://github.com/dgraph-io/dgraph) - Horizontally scalable and distributed GraphQL database with a graph backend
- [Vitess](https://github.com/vitessio/vitess) - Database clustering system for horizontal scaling of MySQL
- [SpiceDB](https://github.com/authzed/spicedb) - Horizontally scalable permissions database
## FAQ
### How are you achieving this performance? What shortcuts are you taking?
We go into detail in the [Ristretto blog post](https://blog.dgraph.io/post/introducing-ristretto-high-perf-go-cache/), but in short: our throughput performance can be attributed to a mix of batching and eventual consistency. Our hit ratio performance is mostly due to an excellent [admission policy](https://arxiv.org/abs/1512.00727) and SampledLFU eviction policy.
As for "shortcuts," the only thing Ristretto does that could be construed as one is dropping some Set calls. That means a Set call for a new item (updates are guaranteed) isn't guaranteed to make it into the cache. The new item could be dropped at two points: when passing through the Set buffer or when passing through the admission policy. However, this doesn't affect hit ratios much at all as we expect the most popular items to be Set multiple times and eventually make it in the cache.
### Is Ristretto distributed?
No, it's just like any other Go library that you can import into your project and use in a single process.

Some files were not shown because too many files have changed in this diff Show More