Skip to content

Commit

Permalink
Perf/multisymbol 4bw (#912)
Browse files Browse the repository at this point in the history
* feat: IsByteZero works

* fix: works on small test case

* bench: awful

* perf: use logderiv map

* fix: "last byte" bug

* style: remove redundant checks

* fix: some bugs

* feat: new data set and huffman estimations

* chore: gitignore

* chore: improved analytics

* more data and an lzss compressor

* feat: basic lzss decompressor

* test: a couple

* fix: compression works on the first 300b of calldata

* fix: 1B addresses seem to work

* test: some logging

* fix: works on 2c2964. performance awful

* perf: prefer actual backrefs for RLE; better but still bad

* fix: works on 3c2943 with symb 0

* test: print compressed file size

* feat: small tests work with indeterminate length

* test: more state machine tests. failing

* chore: comments/cleanup for lzss compression

* fix: can handle two consecutive backrefs

* test: more, failing "8zerosAfterNonzero"

* test: zeroAfterNonzero added, fixed

* test: twoZerosAfterNonzero

* fix: zerosAfterNonzero pass

* test: actually, 257zerosAfterNonzero fails

* test: twoBackrefsAfterNonzero fails

* fix: all "simple" tests pass

* feat: basic i/o funcs

* feat: preliminary snark decompressor impl

* test: decompression snark

* fix: write to the output table

* fix: all zeros tests pass

* fix: all tests pass, except for 3c2943: too slow

* chore: remove prints

* fix: bug with lone 0 in high indexes

* test: add huffman estimated gains

* perf: dfa search; actually makes things worse

* revert: dfa search was counterproductive

* refactor: compile 600KB

* perf: ite -> api.Select

* fix: bug with negative indexes

* refac: remove useless functions

* feat: some experiments with huffman coding

* feat: bzip2 (bzip would be better)

* fix: RLE bug

* feat: huffman Decode

* test: single symbol test

* fix: two symbols test

* revert: strange uppercase

* bench: huffman decoding

* perf: huffman improvement

* refactor: lots of cleanup. one failing test remaining

* fix: presumption of long negative space of zeros

* bench: 26KB

* update gitignore

* fix: failed

* safeguard

* test: failing test for cs loading

* bench: proving works

* perf: reduce 1 lookup per backref

* fix: read bugs

* style: remove prints

* perf: a few petty opts

* perf: a few little opts

* perf: custom constraint for inIDelta

* perf: custom constraint for advancing inI

* perf: custom constraint for copying

* feat: api.IsNonZero

* chore: clean up comments and prints

* fix: trailing backref bug

* refactor: massive simplification of lzssv1 compression

* refactor: remove log heads

* refactor: cleaning tests

* feat: some analytics

* fix: make builder private again

* chore: better logging, remove code from data folder

* fix remove extra zero from logs

* make staticcheck happy(er)

* feat: more analytics

* skip testCompareCoverage

* perf: "start at"

* bench: don't inflate the decompressed size too much

* test: added average batch test case

* test: added BenchmarkAverageBatch

* perf: make compress way faster

* test: reactivate other for cmp

* style: costmetics

* checkpoint

* fix perf: longer backref starting with runs of 0, but slower

* some simplification

* revert: remove TestCompressWithContext

* revert: uncrowd the pr

* chore: uncrowd the pr a bit more

* fix: make tests pass

* test skip calldata test temporarily

* feat: pack/unpack functions

* feat: bit-level alignment of compressed

* test: compress_tests pass

* test: pack

* test: decompression works; must go about packing differently

* feat: r/w num, (un)marshal for stream

* test: marshal test

* test: compression roundtrip passes w merged stream utils

* refactor: simplify packing

* revert overcomplicated packing

* fix: snark errors

* chore make staticcheck happy

* test: average batch

* fix: 18b offset - ave

* checkpoint

* perf: binary search of longest backref

* feat: started v2

* perf: one binary search only

* perf: add lazy match look ahead 1

* style: code cleaning

* perf: even better lookahead for lazy deflate

* perf: adjustement

* checkpoint

* perf: smaller backrefs

* perf: faster compression by reducing search space

* perf: reduce mem alloc when init suffix array

* perf: replace sort.Search

* fix: fuzzer bug

* test: add testdata/ fuzzer dir

* perf: don't use 0 as symbol delimiter

* ensure symbol is in dict and reachable

* feat: add multi symbol

* check against savings

* fix bug

* feat: new stream

* fix: small packing test works

* refactor: packing as stream feature

* bench: large

* refactor: no need for outAt

* remove duplicate bitio dep

* feat: read lengths!

* feat: offset, length and bytes tables

* feat: snark decompressor, all but eof logic done

* feat: snark decomp done, not yet tested

* fix: ReadIntoStream

* fix: some minor bugs

* test: trying to recreate the length bug

* fix: test with backrefs

* test: 253-254-255 fails

* fix: simple table lookup works

* test: actual calldata

* test: fuzz

* refactor: get rid of lzssv1

* feat: compile large circuit outside tests

* make sure all backref address lengths are multiples of 4

* fix: readIntoStream bug

* feat: forceDivisibleBy

* chore: minor changes to benchmark

* chore: set word size to 1

* chore: remove unused code

* chore: remove prints and all huffman code

* refactor: lzssv2 -> lzss

* refac: compression modes

* chore: clean up test cases

* chore: go.sum

* revert: map.keys/values to be private as before

* revert extra gitignore rules

* docs: BestCompression vs BestSnarkDecomposition

* minor fixes and removing prints

* build: get gopter

* fix: comment out go generate in suffixarray

* fix: actually remove the go generate line

* style: code cleaning

* test: updates fuzz test with new API

* fix ineffassign

* test: update FuzzSnark

* refactor: move _testing.go into _test.go file

* feat: cache lookup blueprint entries in solving phase

* clean: remove unused IsNonZero

* refactor: remove hintwithname

* refactor: remove gzwrite and gcd

* fix: NewCombination when at least one var is const

* fix: in test engine, commit should not return 0 with tiny field

* refactor: NewCombination -> EvaluatePlonkExpression

* fix: close zip writer

* style: remove some dead code

* fix: fixes #918 incorrect hash Sum usage

* style: costmetic changes

---------

Co-authored-by: Gautam Botrel <[email protected]>
  • Loading branch information
Tabaie and gbotrel authored Nov 17, 2023
1 parent acaefaf commit 51719fe
Show file tree
Hide file tree
Showing 33 changed files with 4,334 additions and 9 deletions.
5 changes: 5 additions & 0 deletions frontend/api.go
Original file line number Diff line number Diff line change
Expand Up @@ -154,3 +154,8 @@ type BatchInverter interface {
// This is a temporary API, do not use it in your circuit
BatchInvert(i1 []Variable) []Variable
}

type PlonkAPI interface {
// EvaluatePlonkExpression returns res = qL.a + qR.b + qM.ab + qC
EvaluatePlonkExpression(a, b Variable, qL, qR, qM, qC int) Variable
}
27 changes: 27 additions & 0 deletions frontend/cs/scs/api.go
Original file line number Diff line number Diff line change
Expand Up @@ -654,6 +654,33 @@ func (builder *builder) Commit(v ...frontend.Variable) (frontend.Variable, error
})
}

// EvaluatePlonkExpression in the form of res = qL.a + qR.b + qM.ab + qC
func (builder *builder) EvaluatePlonkExpression(a, b frontend.Variable, qL, qR, qM, qC int) frontend.Variable {
_, aConstant := builder.constantValue(a)
_, bConstant := builder.constantValue(b)
if aConstant || bConstant {
return builder.Add(
builder.Mul(a, qL),
builder.Mul(b, qR),
builder.Mul(a, b, qM),
qC,
)
}

res := builder.newInternalVariable()
builder.addPlonkConstraint(sparseR1C{
xa: a.(expr.Term).VID,
xb: b.(expr.Term).VID,
xc: res.VID,
qL: builder.cs.Mul(builder.cs.FromInterface(qL), a.(expr.Term).Coeff),
qR: builder.cs.Mul(builder.cs.FromInterface(qR), b.(expr.Term).Coeff),
qO: builder.tMinusOne,
qM: builder.cs.Mul(builder.cs.FromInterface(qM), builder.cs.Mul(a.(expr.Term).Coeff, b.(expr.Term).Coeff)),
qC: builder.cs.FromInterface(qC),
})
return res
}

func filterConstants(v []frontend.Variable) []frontend.Variable {
res := make([]frontend.Variable, 0, len(v))
for _, vI := range v {
Expand Down
4 changes: 3 additions & 1 deletion go.mod
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
module github.com/consensys/gnark

go 1.19
go 1.20

require (
github.com/bits-and-blooms/bitset v1.8.0
Expand All @@ -10,6 +10,8 @@ require (
github.com/fxamacker/cbor/v2 v2.5.0
github.com/google/go-cmp v0.5.9
github.com/google/pprof v0.0.0-20230817174616-7a8ec2ada47b
github.com/icza/bitio v1.1.0
github.com/klauspost/compress v1.17.1
github.com/ingonyama-zk/iciclegnark v0.1.0
github.com/leanovate/gopter v0.2.9
github.com/rs/zerolog v1.30.0
Expand Down
10 changes: 5 additions & 5 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,15 @@ github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeN
github.com/google/pprof v0.0.0-20230817174616-7a8ec2ada47b h1:h9U78+dx9a4BKdQkBBos92HalKpaGKHrp+3Uo6yTodo=
github.com/google/pprof v0.0.0-20230817174616-7a8ec2ada47b/go.mod h1:czg5+yv1E0ZGTi6S6vVK1mke0fV+FaUhNGcd6VRS9Ik=
github.com/google/subcommands v1.2.0/go.mod h1:ZjhPrFU+Olkh9WazFPsl27BQ4UPiG37m3yTrtFlrHVk=
github.com/icza/bitio v1.1.0 h1:ysX4vtldjdi3Ygai5m1cWy4oLkhWTAi+SyO6HC8L9T0=
github.com/icza/bitio v1.1.0/go.mod h1:0jGnlLAx8MKMr9VGnn/4YrvZiprkvBelsVIbA9Jjr9A=
github.com/icza/mighty v0.0.0-20180919140131-cfd07d671de6/go.mod h1:xQig96I1VNBDIWGCdTt54nHt6EeI639SmHycLYL7FkA=
github.com/ingonyama-zk/icicle v0.0.0-20230928131117-97f0079e5c71 h1:YxI1RTPzpFJ3MBmxPl3Bo0F7ume7CmQEC1M9jL6CT94=
github.com/ingonyama-zk/icicle v0.0.0-20230928131117-97f0079e5c71/go.mod h1:kAK8/EoN7fUEmakzgZIYdWy1a2rBnpCaZLqSHwZWxEk=
github.com/ingonyama-zk/iciclegnark v0.1.0 h1:88MkEghzjQBMjrYRJFxZ9oR9CTIpB8NG2zLeCJSvXKQ=
github.com/ingonyama-zk/iciclegnark v0.1.0/go.mod h1:wz6+IpyHKs6UhMMoQpNqz1VY+ddfKqC/gRwR/64W6WU=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/klauspost/compress v1.17.1 h1:NE3C767s2ak2bweCZo3+rdP4U/HoyVXLv/X9f2gPS5g=
github.com/klauspost/compress v1.17.1/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
github.com/leanovate/gopter v0.2.9 h1:fQjYxZaynp97ozCzfOyOuAGOU4aU/z37zf/tOujFk7c=
github.com/leanovate/gopter v0.2.9/go.mod h1:U2L/78B+KVFIx2VmW6onHJQzXtFb+p5y3y2Sh+Jxxv8=
github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4=
Expand All @@ -38,7 +41,6 @@ github.com/mmcloughlin/profile v0.1.1/go.mod h1:IhHD7q1ooxgwTgjxQYkACGA77oFTDdFV
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rogpeppe/go-internal v1.11.0 h1:cWPaGQEPrBb5/AsnsZesgZZ9yb1OQ+GOISoDNXVBh4M=
github.com/rogpeppe/go-internal v1.11.0/go.mod h1:ddIwULY96R17DhadqLgMfk9H9tvdUzkipdSkR5nkCZA=
github.com/rs/xid v1.5.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg=
github.com/rs/zerolog v1.30.0 h1:SymVODrcRsaRaSInD9yQtKbtWqwsfoPcRff/oRXLj4c=
Expand All @@ -60,8 +62,6 @@ golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.11.0 h1:eG7RXZHdqOJ1i+0lgLgCpSXAp6M3LYlAo6osgSi0xOM=
golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
rsc.io/tmplfunc v0.0.3 h1:53XFQh69AfOa8Tw0Jm7t+GV7KZhOi6jzsCzTtKbMvzU=
Expand Down
Binary file added profile001.pdf
Binary file not shown.
Binary file added profile002.pdf
Binary file not shown.
74 changes: 74 additions & 0 deletions std/compress/lzss/backref.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
package lzss

import (
"math"

"github.com/icza/bitio"
)

const (
maxInputSize = 1 << 21 // 2Mb
maxDictSize = 1 << 22 // 4Mb
)

type backrefType struct {
delimiter byte
nbBitsAddress uint8
nbBitsLength uint8
nbBitsBackRef uint8
nbBytesBackRef int
maxAddress int
maxLength int
dictOnly bool
}

func newBackRefType(symbol byte, nbBitsAddress, nbBitsLength uint8, dictOnly bool) backrefType {
return backrefType{
delimiter: symbol,
nbBitsAddress: nbBitsAddress,
nbBitsLength: nbBitsLength,
nbBitsBackRef: 8 + nbBitsAddress + nbBitsLength,
nbBytesBackRef: int(8+nbBitsAddress+nbBitsLength+7) / 8,
maxAddress: 1 << nbBitsAddress,
maxLength: 1 << nbBitsLength,
dictOnly: dictOnly,
}
}

const (
symbolDict = 0xFF
symbolShort = 0xFE
symbolLong = 0xFD
)

type backref struct {
address int
length int
bType backrefType
}

func (b *backref) writeTo(w *bitio.Writer, i int) {
w.TryWriteByte(b.bType.delimiter)
w.TryWriteBits(uint64(b.length-1), b.bType.nbBitsLength)
if b.bType.dictOnly {
w.TryWriteBits(uint64(b.address), b.bType.nbBitsAddress)
} else {
w.TryWriteBits(uint64(i-b.address-1), b.bType.nbBitsAddress)
}
}

func (b *backref) readFrom(r *bitio.Reader) {
b.length = int(r.TryReadBits(b.bType.nbBitsLength)) + 1
if b.bType.dictOnly {
b.address = int(r.TryReadBits(b.bType.nbBitsAddress))
} else {
b.address = int(r.TryReadBits(b.bType.nbBitsAddress)) + 1
}
}

func (b *backref) savings() int {
if b.length == -1 {
return math.MinInt // -1 is a special value
}
return b.length - b.bType.nbBytesBackRef
}
Loading

0 comments on commit 51719fe

Please sign in to comment.