Commit Graph

7499 Commits

Author SHA1 Message Date
Simon Frei
1f4fde9525
chore(protocol): prioritize closing a connection (#9711)
The read/write loops may keep going for a while on a closing connection
with lots of read/write activity, as it's random which select case is
chosen. And if the connection is slow or even broken, a single
read/write
can take a long time/until timeout. Add initial non-blocking selects
with only the cases relevant to closing, to prioritize those.
2024-09-15 21:13:56 +02:00
Jakob Borg
2238a288d9
fix(model): shut down index sender faster (#9704) 2024-09-15 11:37:49 +02:00
Ross Smith II
1704827d04
chore(gui): update HumanDuration.js (#9710)
Relevant changes:

ko: Use correct names for month and hour in Korean (465eaed)
Hide unit count if 2 in Arabic (f90d847)
2024-09-15 10:21:18 +02:00
Jakob Borg
a156e88eef
Merge branch 'infrastructure'
* infrastructure:
  chore(stdiscosrv): hide internal/undocumented flags
  chore(stdiscosrv): remove legacy replication
  chore(stdiscosrv): clean up s3 handling
  chore(stdiscosrv): less garbage in statistics
  chore(stdiscosrv): improve expire, logging
  chore(stdiscosrv): sched in loop
  chore(stdiscosrv): database writing logging
  chore(stdiscosrv): use order-preserving expire
  chore(stdiscosrv): simplify sorting
  chore(stdiscosrv): reduce allocations in cert handling
  chore(stdiscosrv): reduce unnecessary allocations in merge
  feat(stdiscosrv): enable HTTP profiler
  feat(discosrv): in-memory storage with S3 backing
  feat(stdiscosrv): make compression optional (and faster)
2024-09-13 08:50:24 +02:00
Jakob Borg
94d0195b63
chore(stdiscosrv): hide internal/undocumented flags 2024-09-13 08:49:13 +02:00
Jakob Borg
1616edcee3
chore(stdiscosrv): remove legacy replication 2024-09-13 08:49:13 +02:00
Jakob Borg
6505e123bb
chore(stdiscosrv): clean up s3 handling 2024-09-13 08:48:04 +02:00
Jakob Borg
63e4659282
chore(stdiscosrv): less garbage in statistics 2024-09-13 08:48:04 +02:00
Jakob Borg
f3f5557c8e
chore(stdiscosrv): improve expire, logging 2024-09-13 08:48:04 +02:00
Jakob Borg
b794726e1f
chore(stdiscosrv): sched in loop 2024-09-13 08:48:04 +02:00
Jakob Borg
3d59740a0a
chore(stdiscosrv): database writing logging 2024-09-13 08:48:04 +02:00
Jakob Borg
66fb65b01f
chore(stdiscosrv): use order-preserving expire 2024-09-13 08:48:04 +02:00
Jakob Borg
5c2fcbfd19
chore(stdiscosrv): simplify sorting 2024-09-13 08:48:03 +02:00
Jakob Borg
f9b72330a8
chore(stdiscosrv): reduce allocations in cert handling 2024-09-13 08:48:03 +02:00
Jakob Borg
822b6ac36b
chore(stdiscosrv): reduce unnecessary allocations in merge 2024-09-13 08:48:03 +02:00
Jakob Borg
77f7778292
feat(stdiscosrv): enable HTTP profiler 2024-09-13 08:48:03 +02:00
Jakob Borg
aed2c66e52
feat(discosrv): in-memory storage with S3 backing 2024-09-13 08:48:03 +02:00
Jakob Borg
68a1fd010f
feat(stdiscosrv): make compression optional (and faster) 2024-09-13 08:33:03 +02:00
Simon Frei
ac8b3342ac
chore(fs): only cache the cache for case FS, not the entire FS (#9701)
This would have addressed a recent issue that arose when re-ordering our
"filesystem layers". Specifically moving the caseFilesystem to the
outermost layer. The previous cache included the filesystem, and as such
all the layers below. This isn't desirable (to put it mildly), as you
can create different variants of filesystems with different layers for
the same path and options. Concretely this did happen with the mtime
layer, which isn't always present. A test for the mtime related breakage
was added in #9687, and I intend to redo the caseFilesystem reordering
after this.

Ref: #9677
Followup to: #9687
2024-09-12 20:35:21 +02:00
Jakob Borg
0ea90dd932
build: add generating compat.json (#9700)
This is to add the generation of `compat.json` as a release artifact. It
describes the runtime requirements of the release in question. The next
step is to have the upgrade server use this information to filter
releases provided to clients. This is per the discussion in #9656

---------

Co-authored-by: Ross Smith II <ross@smithii.com>
2024-09-11 09:29:49 +02:00
Jakob Borg
718b1ce2b7
chore(discovery,upgrade): use regular TLS certificate verification (#9673)
This changes the two remaining instances where we use insecure HTTPS to
use standard HTTPS certificate verification.

When we introduced these things, almost a decade ago, HTTPS certificates
were expensive and annoying to get, much of the web was still HTTP, and
many devices seemed to not have up-to-date CA bundles.

Nowadays _all_ of the web is HTTPS and I'm skeptical that any device can
work well without understanding LetsEncrypt certificates in particular.

Our current discovery servers use hardcoded certificates which has
several issues:
- Not great for security if it leaks as there is no way to rotate it
- Not great for infrastructure flexibility as we can't use many load
balancer or TLS termination services
- The certificate is a very oddball ECDSA-SHA384 type certificate which
has higher CPU cost than a more regular certificate, which has real
effects on our infrastructure

Using normal TLS certificates here improves these things.

I expect there will be some very few devices out there for which this
doesn't work. For the foreseeable future they can simply change the
config to use the old URLs and parameters -- it'll be years before we
can retire those entirely.

For the upgrade client this simply seems like better hygiene. While our
releases are signed anyway, protecting the metadata exchange is _better_
and, again, I doubt many clients will fail this today.
2024-09-11 09:29:19 +02:00
Simon Frei
29f7510f5a
lib/fs: Add test reproducing missing mtimefs issue (ref #9677) (#9687)
The test is quite odd and specific, but it does reproduce the issue that
caused #9677, so I'd propose to add it to have a simple regression test
for the basic scenario. Also the option to the fakefs might come handy
for other scenarios where you want to quickly test some behaviour on a
filesystem without nanosecond precision, without actually needing access
to one.
2024-09-10 13:36:17 +02:00
Syncthing Release Automation
a7f9ed4a80 gui, man, authors: Update docs, translations, and contributors 2024-09-09 03:45:15 +00:00
tomasz1986
1baefea410
gui: Actually filter scope ID out of IPv6 address when using Remote GUI (ref #8084) (#9688)
gui: Actually filter scope ID out of IPv6 address when using Remote GUI
(ref #8084)

The current code does not work, because it uses a string in the
replace() method instead of regex. Thus, change it to a proper regex.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2024-09-08 12:42:32 +02:00
Jakob Borg
563cec8923
Merge branch 'release'
* release:
  Revert "lib/fs: Put the caseFS as the outermost layer (#9648)"
2024-09-06 09:39:09 +02:00
Jakob Borg
a3c340ece9
Revert "lib/fs: Put the caseFS as the outermost layer (#9648)"
This reverts commit 7517d18fbb.

Fixes #9677
2024-09-06 09:15:45 +02:00
André Colomb
cb24638ec9
lib/api: Correct ordering of Accept-Language codes by weight (fixes #9670) (#9671)
The preference for languages in the Accept-Language header field
should not be deduced from the listed order, but from the passed
"quality values", according to the HTTP specification:
https://httpwg.org/specs/rfc9110.html#field.accept-language

This implements the parsing of q=values and ordering within the API
backend, to not complicate things further in the GUI code.  Entries
with invalid (unparseable) quality values are discarded completely.

* gui: Fix API endpoint in comment.
2024-09-02 10:15:04 +02:00
Syncthing Release Automation
2fb24dc2cc gui, man, authors: Update docs, translations, and contributors 2024-09-02 03:45:15 +00:00
tomasz1986
9aa2d2c92f
gui: Fix incorrect UI language auto detection (fixes #9668) (#9669)
gui: Fix incorrect UI language auto detection (fixes #9668)

Currently, the code only checks whether the detected language partially
matches one of the available languages. This means that if the detected
language is "fi" but among the available languages there is only "fil"
and no "fi", then it will match "fi" with "fil", even though the two are
completely different languages.

With this change, the matching is only done when there is a hyphen in
the language code, e.g. "en" will match with "en-US".

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2024-08-31 22:47:10 +02:00
Jakob Borg
d1c5100c98
Merge branch 'release'
* release:
  lib/upgrade: Send OS version header to upgrade server (#9663)
2024-08-30 11:32:49 +02:00
Jakob Borg
42e677c055
lib/model, lib/protocol: Index sending/receiving debugging (#9657)
This adds guardrails to the index sending and receiving, to verify that
what we thinks is happening is what actually happens.
2024-08-28 15:00:19 +02:00
Jakob Borg
27bba2c0c2
lib/upgrade: Send OS version header to upgrade server (#9663)
This adds a header with the operating system version, verbatim in
whatever format the operating system reports it, to the upgrade check.
The intention is that the upgrade server can use this information to
filter out (or maybe just mark) potentially unsupported upgrades.
2024-08-28 08:32:03 +02:00
Jakob Borg
feff334547
lib/upgrade: Send OS version header to upgrade server (#9663)
This adds a header with the operating system version, verbatim in
whatever format the operating system reports it, to the upgrade check.
The intention is that the upgrade server can use this information to
filter out (or maybe just mark) potentially unsupported upgrades.
2024-08-28 08:31:10 +02:00
Syncthing Release Automation
713cf357ce gui, man, authors: Update docs, translations, and contributors 2024-08-26 03:45:23 +00:00
Jakob Borg
5342bec1b7
lib/protocol: Further interface refactor (#9396)
This is a symmetric change to #9375 -- where that PR changed the
protocol->model interface, this changes the model->protocol one.
2024-08-24 12:45:10 +02:00
Emil Lundberg
7df75e681d
gui: Replace global "Panel padding decrease" style with targeted class (#9659)
Transplanted from https://github.com/emlun/syncthing/pull/8 (meta-PR
into https://github.com/syncthing/syncthing/pull/9175) by request of
@acolomb (see:
https://github.com/emlun/syncthing/pull/8#discussion_r1724470574).

This padding decrease currently applies to _all_ collapsible panels, but
this padding decrease may not be appropriate for all collapsible panels.
In particular, it will not be appropriate for the collapsible panels
introduced in https://github.com/emlun/syncthing/pull/8.
2024-08-21 15:02:45 +02:00
Jakob Borg
8dc826b234
build: use Go 1.23, require minimum 1.22 (#9651)
🥳

---------

Co-authored-by: Ross Smith II <ross@smithii.com>
2024-08-19 20:26:08 +02:00
Syncthing Release Automation
9ef37e1485 gui, man, authors: Update docs, translations, and contributors 2024-08-19 03:45:18 +00:00
Simon Frei
7517d18fbb
lib/fs: Put the caseFS as the outermost layer (#9648)
Reasoning in comments. The main motivation is to avoid all the case
checks when walking the filesystem.
2024-08-13 10:59:31 +02:00
André Colomb
42d0fee536
gui: Add Irish (ga) translation template (#9646)
Based on user request from Weblate, user `@aindriu80`.

Looks promising based on the profile:
https://hosted.weblate.org/user/aindriu80/ Not sure whether almost
30.000 translations in about one month is realistic for a human though.
2024-08-12 13:32:29 +02:00
Syncthing Release Automation
2ca9d3b5c5 gui, man, authors: Update docs, translations, and contributors 2024-08-12 03:45:16 +00:00
Tommy van der Vorst
9cde068f2a
lib/syncthing: Add wrapper for access to model (#9627)
### Purpose

Wrap access to Model for users that use the syncthing Go package. See
discussion:
https://github.com/syncthing/syncthing/pull/9619#pullrequestreview-2212484910

### Testing

It works with the iOS app. Other than that, there are no current users
of this API (to my knowledge) as Model was only exposed recently form
the iOS app.
2024-08-11 20:20:43 +02:00
Gusted
1243083831
cli: Remove go-shlex dependency (#9644) 2024-08-11 11:37:18 +02:00
Gusted
356c5055ad
lib/sha256: Remove it (#9643)
### Purpose

Remove the `lib/sha256` package, because it's no longer necessary. Go's
standard library now has the same performance and is on par with
`sha256-simd` since [Since Go
1.21](1a64574f42).
Therefore using `sha256-simd` has no benefits anymore.

ARM already has optimized sha256 assembly code since
7b8a7f8272,
`sha256-simd` published their results before that optimized assembly was
implemented,
f941fedda8.
The assembly looks very similar and the benchmarks in the Go commit
match that of `sha256-simd`.

This patch removes all of the related code of `lib/sha256` and makes
`crypto/sha256` the 'default'.

Benchmark of `sha256-simd` and `crypto/sha256`:
<details>

```
cpu: AMD Ryzen 5 3600X 6-Core Processor
                │  simd.txt   │               go.txt                │
                │   sec/op    │    sec/op     vs base               │
Hash/8Bytes-12    63.25n ± 1%    73.38n ± 1%  +16.02% (p=0.002 n=6)
Hash/64Bytes-12   98.73n ± 1%   105.30n ± 1%   +6.65% (p=0.002 n=6)
Hash/1K-12        567.2n ± 1%    572.8n ± 1%   +0.99% (p=0.002 n=6)
Hash/8K-12        4.062µ ± 1%    4.062µ ± 1%        ~ (p=0.396 n=6)
Hash/1M-12        512.1µ ± 0%    510.6µ ± 1%        ~ (p=0.485 n=6)
Hash/5M-12        2.556m ± 1%    2.564m ± 0%        ~ (p=0.093 n=6)
Hash/10M-12       5.112m ± 0%    5.127m ± 0%        ~ (p=0.093 n=6)
geomean           13.82µ         14.27µ        +3.28%

                │   simd.txt   │               go.txt                │
                │     B/s      │     B/s       vs base               │
Hash/8Bytes-12    120.6Mi ± 1%   104.0Mi ± 1%  -13.81% (p=0.002 n=6)
Hash/64Bytes-12   618.2Mi ± 1%   579.8Mi ± 1%   -6.22% (p=0.002 n=6)
Hash/1K-12        1.682Gi ± 1%   1.665Gi ± 1%   -0.98% (p=0.002 n=6)
Hash/8K-12        1.878Gi ± 1%   1.878Gi ± 1%        ~ (p=0.310 n=6)
Hash/1M-12        1.907Gi ± 0%   1.913Gi ± 1%        ~ (p=0.485 n=6)
Hash/5M-12        1.911Gi ± 1%   1.904Gi ± 0%        ~ (p=0.093 n=6)
Hash/10M-12       1.910Gi ± 0%   1.905Gi ± 0%        ~ (p=0.093 n=6)
geomean           1.066Gi        1.032Gi        -3.18%
```

</details>


### Testing

Compiled and tested on Linux.

### Documentation

https://github.com/syncthing/docs/pull/874
2024-08-10 12:58:20 +01:00
Jakob Borg
19693734a3
build: Update dependencies (#9640) 2024-08-09 16:04:51 +02:00
Ross Smith II
17e60b9e0c
Chmod -x non-executable files (fixes #9629) (#9630)
Fixed via
```
git ls-files -s | grep 100755 | grep -E -v '(run|sh)$' | cut -f 2 | xargs git update-index --chmod -x
```
See #9629.
2024-08-05 18:32:23 +02:00
Syncthing Release Automation
ac22b2d00a gui, man, authors: Update docs, translations, and contributors 2024-08-05 03:45:25 +00:00
Tommy van der Vorst
de0b4270df
all: minimal set of changes for iOS app (#9619)
### Purpose

This PR contains the set of changes needed to make Syncthing work on iOS
for [my iOS app for
Syncthing](https://github.com/pixelspark/sushitrain).

Most changes originate from [the Mobius Sync
fork](http://github.com/MobiusSync/syncthing/tree/ios). I have removed
the changes from their fork that are not strictly needed for my app
(i.e. their changes to the GUI and command line utilities, for instance)
and squashed it all in a single commit.

In summary, the changes are:

* Resolve non-absolute paths to the 'Documents' folder (basically the
only one an app can/should write user data to by default on iOS)
* Tweaking of build flags/conditions for iOS (i.e. determine which
basicfs_watch, ignoreresult variant to build for iOS)
* Disable upgrade mechanism on iOS
* Make `RequestGlobal` and `PullerProgress` public symbols
* Expose syncthing.app's Model instance (app.M)
* Add no-op stub for SetLowPriority on iOS

I would very much appreciate these changes to be (eventually) merged to
mainline syncthing, as this would allow my iOS app to track the mainline
source code directly and removes the need (for me at least) for
maintaining a separate fork. Perhaps the Mobius folks can also benefit
from this (although as noted this branch does not contain their changes
to e.g. the GUI).

### Testing

This branch has been tested with the iOS app and appears to work fine.
The full set of MobiusSync changes has been used before with success.

### Screenshots

n/a

### Documentation

There should be no visible changes for users due to this set of changes.

---------

Co-authored-by: Simon Pickup <simon@pickupinfinity.com>
2024-07-31 07:31:14 +02:00
Syncthing Release Automation
e738af7c56 gui, man, authors: Update docs, translations, and contributors 2024-07-29 03:45:25 +00:00
Syncthing Release Automation
a28441a9bf gui, man, authors: Update docs, translations, and contributors 2024-07-22 03:45:28 +00:00