161 Commits

Author	SHA1	Message	Date
dependabot-preview[bot]	5e1cd0e71a	build(deps): bump github.com/greatroar/blobloom from 0.1.1 to 0.2.0 (#6580) ... Bumps [github.com/greatroar/blobloom](https://github.com/greatroar/blobloom) from 0.1.1 to 0.2.0. - [Release notes](https://github.com/greatroar/blobloom/releases) - [Commits](https://github.com/greatroar/blobloom/compare/v0.1.1...v0.2.0)	2020-04-27 14:38:20 +02:00
greatroar	0e5ba3ca05	lib/db: Upgrade to Blobloom v0.1.1 (#6553) ... Now faster and Apache-licensed.	2020-04-20 14:23:36 +02:00
greatroar	44b0f0b456	lib/db: Switch to faster blobloom Bloom filter pkg (#6537)	2020-04-20 09:02:33 +02:00
Jakob Borg	7da898f2d6	go.mod: Use github.com/twmb/murmur3 for murmur3 (#6486) ... Let's try this again shall we	2020-04-01 23:51:31 +02:00
Jakob Borg	0d919bd79c	Revert "go.mod: Use github.com/twmb/murmur3 for murmur3" ... "I shall not commit to master without testing all architectures on the builder" * 100 on the black board	2020-04-01 21:16:15 +02:00
Jakob Borg	f91e90a94f	go.mod: Use github.com/twmb/murmur3 for murmur3 ... It seems, like, maintained and stuff.	2020-04-01 21:04:43 +02:00
Jakob Borg	9c67d57c28	lib/api: Update ldap package (fixes #6479) (#6481)	2020-03-31 09:56:04 +02:00
Alberto Donato	7709ac33a7	go.mod: Update jackpal/gateway dependency (fixes #5288) (#6469)	2020-03-30 11:11:55 +02:00
Simon Frei	a7de4c68e3	go.mod: Update quic-go to 0.14.4 (#6453)	2020-03-24 21:12:57 +01:00
Jakob Borg	c4b9046eaa	build: Forked github.com/spaolacci/murmur3 for unsafe (ref #6371)	2020-02-26 20:25:24 +01:00
Jakob Borg	8fc2dfad0c	lib/db: Deduplicate block lists in database (fixes #5898) (#6283) ... * lib/db: Deduplicate block lists in database (fixes #5898) This moves the block list in the database out from being just a field on the FileInfo to being an object of its own. When putting a FileInfo we marshal the block list separately and store it keyed by the sha256 of the marshalled block list. When getting, if we are not doing a "truncated" get, we do an extra read and unmarshal for the block list. Old block lists are cleared out by a periodic GC sweep. The alternative would be to use refcounting, but: - There is a larger risk of getting that wrong and either dropping a block list in error or keeping them around forever. - It's tricky with our current database, as we don't have dirty reads. This means that if we update two FileInfos with identical block lists in the same transaction we can't just do read/modify/write for the ref counters as we wouldn't see our own first update. See above about tracking this and risks about getting it wrong. GC uses a bloom filter for keys to avoid heavy RAM usage. GC can't run concurrently with FileInfo updates so there is a new lock around those operation at the lowlevel. The end result is a much more compact database, especially for setups with many peers where files get duplicated many times. This is per-key-class stats for a large database I'm currently working with, under the current schema: ``` 0x00: 9138161 items, 870876 KB keys + 7397482 KB data, 95 B + 809 B avg, 1637651 B max 0x01: 185656 items, 10388 KB keys + 1790909 KB data, 55 B + 9646 B avg, 924525 B max 0x02: 916890 items, 84795 KB keys + 3667 KB data, 92 B + 4 B avg, 192 B max 0x03: 384 items, 27 KB keys + 5 KB data, 72 B + 15 B avg, 87 B max 0x04: 1109 items, 17 KB keys + 17 KB data, 15 B + 15 B avg, 69 B max 0x06: 383 items, 3 KB keys + 0 KB data, 9 B + 2 B avg, 18 B max 0x07: 510 items, 4 KB keys + 12 KB data, 9 B + 24 B avg, 41 B max 0x08: 1349 items, 12 KB keys + 10 KB data, 9 B + 8 B avg, 17 B max 0x09: 194 items, 0 KB keys + 123 KB data, 5 B + 634 B avg, 11484 B max 0x0a: 3 items, 0 KB keys + 0 KB data, 14 B + 7 B avg, 30 B max 0x0b: 181836 items, 2363 KB keys + 10694 KB data, 13 B + 58 B avg, 173 B max Total 10426475 items, 968490 KB keys + 9202925 KB data. ``` Note 7.4 GB of data in class 00, total size 9.2 GB. After running the migration we get this instead: ``` 0x00: 9138161 items, 870876 KB keys + 2611392 KB data, 95 B + 285 B avg, 4788 B max 0x01: 185656 items, 10388 KB keys + 1790909 KB data, 55 B + 9646 B avg, 924525 B max 0x02: 916890 items, 84795 KB keys + 3667 KB data, 92 B + 4 B avg, 192 B max 0x03: 384 items, 27 KB keys + 5 KB data, 72 B + 15 B avg, 87 B max 0x04: 1109 items, 17 KB keys + 17 KB data, 15 B + 15 B avg, 69 B max 0x06: 383 items, 3 KB keys + 0 KB data, 9 B + 2 B avg, 18 B max 0x07: 510 items, 4 KB keys + 12 KB data, 9 B + 24 B avg, 41 B max 0x09: 194 items, 0 KB keys + 123 KB data, 5 B + 634 B avg, 11484 B max 0x0a: 3 items, 0 KB keys + 0 KB data, 14 B + 17 B avg, 51 B max 0x0b: 181836 items, 2363 KB keys + 10694 KB data, 13 B + 58 B avg, 173 B max 0x0d: 44282 items, 1461 KB keys + 61081 KB data, 33 B + 1379 B avg, 1637399 B max Total 10469408 items, 969939 KB keys + 4477905 KB data. ``` Class 00 is now down to 2.6 GB, with just 61 MB added in class 0d. There will be some additional reads in some cases which theoretically hurts performance, but this will be more than compensated for by smaller writes and better compaction. On my own home setup which just has three devices and a handful of folders the difference is smaller in absolute numbers of course, but still less than half the old size: ``` 0x00: 297122 items, 20894 KB keys + 306860 KB data, 70 B + 1032 B avg, 103237 B max 0x01: 115299 items, 7738 KB keys + 17542 KB data, 67 B + 152 B avg, 419 B max 0x02: 1430537 items, 121223 KB keys + 5722 KB data, 84 B + 4 B avg, 253 B max ... Total 1947412 items, 151268 KB keys + 337485 KB data. ``` to: ``` 0x00: 297122 items, 20894 KB keys + 37038 KB data, 70 B + 124 B avg, 520 B max 0x01: 115299 items, 7738 KB keys + 17542 KB data, 67 B + 152 B avg, 419 B max 0x02: 1430537 items, 121223 KB keys + 5722 KB data, 84 B + 4 B avg, 253 B max ... 0x0d: 18041 items, 595 KB keys + 71964 KB data, 33 B + 3988 B avg, 101109 B max Total 1965447 items, 151863 KB keys + 139628 KB data. ``` * wip * wip * wip * wip	2020-01-24 08:35:44 +01:00
dependabot-preview[bot]	ddd26f5c42	build(deps): bump github.com/pkg/errors from 0.9.0 to 0.9.1 (#6279) ... Bumps [github.com/pkg/errors](https://github.com/pkg/errors) from 0.9.0 to 0.9.1. - [Release notes](https://github.com/pkg/errors/releases) - [Commits](https://github.com/pkg/errors/compare/v0.9.0...v0.9.1) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>	2020-01-20 20:05:19 +01:00
Jakob Borg	f454e8b609	build: go mod tidy	2020-01-14 07:59:31 +01:00
dependabot-preview[bot]	c6cef168a5	build(deps): bump github.com/oschwald/geoip2-golang from 1.3.0 to 1.4.0 (#6245) ... Bumps [github.com/oschwald/geoip2-golang](https://github.com/oschwald/geoip2-golang) from 1.3.0 to 1.4.0. - [Release notes](https://github.com/oschwald/geoip2-golang/releases) - [Commits](https://github.com/oschwald/geoip2-golang/compare/v1.3.0...v1.4.0) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>	2020-01-14 09:57:08 +04:00
dependabot-preview[bot]	4de6b94de7	build(deps): bump github.com/pkg/errors from 0.8.1 to 0.9.0 (#6267) ... Bumps [github.com/pkg/errors](https://github.com/pkg/errors) from 0.8.1 to 0.9.0. - [Release notes](https://github.com/pkg/errors/releases) - [Commits](https://github.com/pkg/errors/compare/v0.8.1...v0.9.0) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>	2020-01-13 12:18:32 +04:00
Simon Frei	7a8e73d599	build, pmp: Replace fork with upstream for go-nat-pmp and tidy go.mod (#6247)	2020-01-03 12:39:59 +01:00
dependabot-preview[bot]	0f1e0eff05	build(deps): bump github.com/mattn/go-isatty from 0.0.10 to 0.0.11 (#6231) ... Bumps [github.com/mattn/go-isatty](https://github.com/mattn/go-isatty) from 0.0.10 to 0.0.11. - [Release notes](https://github.com/mattn/go-isatty/releases) - [Commits](https://github.com/mattn/go-isatty/compare/v0.0.10...v0.0.11) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>	2019-12-16 07:52:02 +00:00
dependabot-preview[bot]	999647b7d6	build(deps): bump github.com/urfave/cli from 1.22.1 to 1.22.2 (#6183) ... Bumps [github.com/urfave/cli](https://github.com/urfave/cli) from 1.22.1 to 1.22.2. - [Release notes](https://github.com/urfave/cli/releases) - [Changelog](https://github.com/urfave/cli/blob/master/docs/CHANGELOG.md) - [Commits](https://github.com/urfave/cli/compare/v1.22.1...v1.22.2) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>	2019-11-25 11:05:55 +01:00
Simon Frei	cf420e135e	gui: New folder state "Local Additions" for receive-only (fixes #5968) (#6117)	2019-11-01 20:44:23 +01:00
dependabot-preview[bot]	b1dbe925d4	build(deps): bump github.com/prometheus/client_golang (#6099) ... Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.1.0 to 1.2.1. - [Release notes](https://github.com/prometheus/client_golang/releases) - [Changelog](https://github.com/prometheus/client_golang/blob/master/CHANGELOG.md) - [Commits](https://github.com/prometheus/client_golang/compare/v1.1.0...v1.2.1) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>	2019-10-22 21:56:32 +02:00
Jakob Borg	ad2d3702ae	all: Upgrade github.com/gogo/protobuf and regenerate (fixes #6085)	2019-10-18 09:53:59 +02:00
dependabot-preview[bot]	aa4b918224	build(deps): bump github.com/lucas-clemente/quic-go (#6084) ... Bumps [github.com/lucas-clemente/quic-go](https://github.com/lucas-clemente/quic-go) from 0.12.0 to 0.12.1. - [Release notes](https://github.com/lucas-clemente/quic-go/releases) - [Changelog](https://github.com/lucas-clemente/quic-go/blob/master/Changelog.md) - [Commits](https://github.com/lucas-clemente/quic-go/compare/v0.12.0...v0.12.1) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>	2019-10-14 18:30:00 +01:00
dependabot-preview[bot]	7043b1fbba	build(deps): bump github.com/mattn/go-isatty from 0.0.9 to 0.0.10 (#6083) ... Bumps [github.com/mattn/go-isatty](https://github.com/mattn/go-isatty) from 0.0.9 to 0.0.10. - [Release notes](https://github.com/mattn/go-isatty/releases) - [Commits](https://github.com/mattn/go-isatty/compare/v0.0.9...v0.0.10) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>	2019-10-14 18:17:01 +01:00
Audrius Butkevicius	8fb576ed54	lib/model: Adjust blocks reported in usage reporting (fixes #5995) (#6037) ... * lib/model: Adjust blocks reported in usage reporting (fixes #5995) * Use variables, fix go.mod	2019-10-04 12:03:13 +01:00
Jakob Borg	1b5a61e03e	build: Upgrade github.com/syndtr/goleveldb ... Newer is always better. Always.	2019-10-03 17:45:45 +02:00
dependabot-preview[bot]	f653f540f8	build(deps): bump github.com/urfave/cli from 1.21.0 to 1.22.1 (#6015) ... Bumps [github.com/urfave/cli](https://github.com/urfave/cli) from 1.21.0 to 1.22.1. - [Release notes](https://github.com/urfave/cli/releases) - [Changelog](https://github.com/urfave/cli/blob/master/CHANGELOG.md) - [Commits](https://github.com/urfave/cli/compare/v1.21.0...v1.22.1) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>	2019-09-16 08:31:00 +01:00
dependabot-preview[bot]	078923bd1a	build(deps): bump github.com/minio/sha256-simd from 0.1.0 to 0.1.1 (#6014) ... Bumps [github.com/minio/sha256-simd](https://github.com/minio/sha256-simd) from 0.1.0 to 0.1.1. - [Release notes](https://github.com/minio/sha256-simd/releases) - [Commits](https://github.com/minio/sha256-simd/compare/v0.1.0...v0.1.1) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>	2019-09-16 08:28:46 +01:00
Jakob Borg	c0b3de2680	build: Correct hash for quic package	2019-09-11 15:31:43 +02:00
Jakob Borg	88482b29ee	build: Upgrade dependencies ... go get -u ./... go mod tidy	2019-09-05 15:13:51 +02:00
Jakob Borg	80894948f6	build: Upgrade github.com/gogo/protobuf (#5994) ... This is the result of: - Changing build.go to take the protobuf version from the modules instead of hardcoded - `go get github.com/gogo/protobuf@v1.3.0` to upgrade - `go run build.go proto` to regenerate our code	2019-09-04 07:33:29 +01:00
Jakob Borg	c2ea9d119d	lib/connections: Upgrade QUIC package, use contexts for timeout (#5972)	2019-08-23 10:15:52 +02:00
Jakob Borg	09b872cef4	build: go mod tidy	2019-08-21 07:47:05 +02:00
dependabot-preview[bot]	e910acdc17	build(deps): bump github.com/mattn/go-isatty from 0.0.7 to 0.0.9 (#5965) ... Bumps [github.com/mattn/go-isatty](https://github.com/mattn/go-isatty) from 0.0.7 to 0.0.9. - [Release notes](https://github.com/mattn/go-isatty/releases) - [Commits](https://github.com/mattn/go-isatty/compare/v0.0.7...v0.0.9) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>	2019-08-19 12:13:26 +01:00
Audrius Butkevicius	58ef5368f8	lib/connections: Validate device id before assuming success (fixes #5934) (#5935) ... * lib/connections: Validate device id before assuming success (fixes #5934) * Vet	2019-08-09 12:31:42 +01:00
dependabot-preview[bot]	d43b0a4395	build(deps): bump github.com/urfave/cli from 1.20.0 to 1.21.0 (#5928) ... Bumps [github.com/urfave/cli](https://github.com/urfave/cli) from 1.20.0 to 1.21.0. - [Release notes](https://github.com/urfave/cli/releases) - [Changelog](https://github.com/urfave/cli/blob/master/CHANGELOG.md) - [Commits](https://github.com/urfave/cli/compare/v1.20.0...v1.21.0) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>	2019-08-05 12:18:35 +02:00
Simon Frei	1d182e4631	lib/fs: Use gopsutils for disk usage (#5905)	2019-07-29 20:06:17 +02:00
Simon Frei	7a4c88d4e4	lib: Add mtime window when comparing files (#5852)	2019-07-23 21:48:53 +02:00
dependabot-preview[bot]	15c262184b	build(deps): bump github.com/maruel/panicparse from 1.2.1 to 1.3.0 (#5879) ... Bumps [github.com/maruel/panicparse](https://github.com/maruel/panicparse) from 1.2.1 to 1.3.0. - [Release notes](https://github.com/maruel/panicparse/releases) - [Commits](https://github.com/maruel/panicparse/compare/v1.2.1...v1.3.0) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>	2019-07-22 19:46:52 +01:00
dependabot-preview[bot]	484fa0592e	build(deps): bump github.com/lib/pq from 1.1.1 to 1.2.0 (#5878) ... Bumps [github.com/lib/pq](https://github.com/lib/pq) from 1.1.1 to 1.2.0. - [Release notes](https://github.com/lib/pq/releases) - [Commits](https://github.com/lib/pq/compare/v1.1.1...v1.2.0) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>	2019-07-22 08:07:21 +01:00
Simon Frei	05e9e0bfa9	build: Update notify dependency (#5838)	2019-07-09 21:33:22 +01:00
Jakob Borg	2b4df6b874	go.mod: Update AudriusButkevicius/pfilter (fixes #5820)	2019-06-28 07:38:52 +02:00
Jakob Borg	bff1a5f5e4	build: Upgrade github.com/syndtr/goleveldb	2019-06-14 06:56:52 +02:00
Audrius Butkevicius	b7c70a9817	lib/fs: Enhance mtimefs, use everywhere (fixes #5777) (#5776)	2019-06-11 08:27:12 +02:00
Jakob Borg	42ce6be9b9	lib/ur: Implement crash (panic) reporting (fixes #959) (#5702) ... * lib/ur: Implement crash (panic) reporting (fixes #959) This implements a simple crash reporting method. It piggybacks on the panic log files created by the monitor process, picking these up and uploading them from the usage reporting routine. A new config value points to the crash receiver base URL, which defaults to "https://crash.syncthing.net/newcrash" (following the pattern of "https://data.syncthing.net/newdata" for usage reports, but allowing us to separate the service as required).	2019-06-11 08:19:11 +02:00
dependabot-preview[bot]	93e57bd357	build(deps): bump github.com/prometheus/client_golang (#5775) ... Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 0.9.3 to 0.9.4. - [Release notes](https://github.com/prometheus/client_golang/releases) - [Changelog](https://github.com/prometheus/client_golang/blob/master/CHANGELOG.md) - [Commits](https://github.com/prometheus/client_golang/compare/v0.9.3...v0.9.4) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>	2019-06-10 19:06:25 +02:00
dependabot-preview[bot]	97057eb9de	build(deps): bump github.com/lucas-clemente/quic-go (#5761) ... Bumps [github.com/lucas-clemente/quic-go](https://github.com/lucas-clemente/quic-go) from 0.11.1 to 0.11.2. - [Release notes](https://github.com/lucas-clemente/quic-go/releases) - [Changelog](https://github.com/lucas-clemente/quic-go/blob/master/Changelog.md) - [Commits](https://github.com/lucas-clemente/quic-go/compare/v0.11.1...v0.11.2)	2019-06-03 12:42:23 +01:00
dependabot-preview[bot]	e2a647a6a4	build(deps): bump golang.org/x/text from 0.3.0 to 0.3.2 (#5751) ... Bumps [golang.org/x/text](https://github.com/golang/text) from 0.3.0 to 0.3.2. - [Release notes](https://github.com/golang/text/releases) - [Commits](https://github.com/golang/text/compare/v0.3.0...v0.3.2)	2019-05-29 11:42:07 +02:00
Audrius Butkevicius	e714df013f	lib/connections: Add QUIC protocol support (fixes #5377) (#5737)	2019-05-29 09:56:40 +02:00
Jakob Borg	eb8df7f632	cmd/ursrv: Lint fixes	2019-05-18 11:59:32 +02:00
dependabot[bot]	78d6eee74a	build(deps): bump github.com/prometheus/client_golang (#5729) ... Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 0.9.2 to 0.9.3. - [Release notes](https://github.com/prometheus/client_golang/releases) - [Changelog](https://github.com/prometheus/client_golang/blob/master/CHANGELOG.md) - [Commits](https://github.com/prometheus/client_golang/compare/v0.9.2...v0.9.3) Signed-off-by: dependabot[bot] <support@dependabot.com>	2019-05-18 09:57:35 +02:00
dependabot[bot]	ad44d77a21	build(deps): bump github.com/lib/pq from 1.0.0 to 1.1.1 (#5714) ... Bumps [github.com/lib/pq](https://github.com/lib/pq) from 1.0.0 to 1.1.1. - [Release notes](https://github.com/lib/pq/releases) - [Commits](https://github.com/lib/pq/compare/v1.0.0...v1.1.1) Signed-off-by: dependabot[bot] <support@dependabot.com>	2019-05-11 22:15:30 +02:00
dependabot[bot]	6bb5d140fa	build(deps): bump github.com/oschwald/geoip2-golang from 1.1.0 to 1.3.0 (#5713) ... Bumps [github.com/oschwald/geoip2-golang](https://github.com/oschwald/geoip2-golang) from 1.1.0 to 1.3.0. - [Release notes](https://github.com/oschwald/geoip2-golang/releases) - [Commits](https://github.com/oschwald/geoip2-golang/compare/v1.1.0...v1.3.0) Signed-off-by: dependabot[bot] <support@dependabot.com>	2019-05-11 21:02:49 +01:00
dependabot[bot]	d082b2ba4c	build(deps): bump github.com/mattn/go-isatty from 0.0.4 to 0.0.7 (#5712) ... Bumps [github.com/mattn/go-isatty](https://github.com/mattn/go-isatty) from 0.0.4 to 0.0.7. - [Release notes](https://github.com/mattn/go-isatty/releases) - [Commits](https://github.com/mattn/go-isatty/compare/v0.0.4...v0.0.7) Signed-off-by: dependabot[bot] <support@dependabot.com>	2019-05-11 21:00:00 +01:00
dependabot[bot]	638789899c	build(deps): bump github.com/gogo/protobuf from 1.2.0 to 1.2.1 (#5715) ... Bumps [github.com/gogo/protobuf](https://github.com/gogo/protobuf) from 1.2.0 to 1.2.1. - [Release notes](https://github.com/gogo/protobuf/releases) - [Commits](https://github.com/gogo/protobuf/compare/v1.2.0...v1.2.1) Signed-off-by: dependabot[bot] <support@dependabot.com>	2019-05-11 20:59:44 +01:00
Jakob Borg	73fa7a6e5b	go.mod: Update golang.org/x/sys ... There was a vulnerability in salsa20, which we don't use, but better safe than sorry.	2019-03-21 08:08:15 +01:00
Audrius Butkevicius	dc929946fe	all: Use new reflect based CLI (#5487)	2019-02-12 07:58:24 +01:00
Simon Frei	8815ef922b	mod: Update dependencies and tidy (fixes #5311) (#5486)	2019-01-28 22:45:56 +01:00
Audrius Butkevicius	a80c0fdae8	vendor: Update minio/sha256-simd	2019-01-18 20:30:40 +00:00
Jakob Borg	b01edca420	all: Update protobuf package 1.0.0 -> 1.2.0 (#5452) ... Also adds a few file global options to keep the generated code similar to what we already had.	2019-01-14 11:53:36 +01:00
Audrius Butkevicius	ad30192dca	vendor: Update minio/sha256-simd (#5433) ... * vendor: Update minio/sha256-simd * Add go module stuff	2019-01-05 10:21:42 +01:00
Jakob Borg	944ddcf768	all: Become a Go module (fixes #5148) (#5384) ... * go mod init; rm -rf vendor * tweak proto files and generation * go mod vendor * clean up build.go * protobuf literals in tests * downgrade gogo/protobuf	2018-12-18 12:36:38 +01:00