Release 2.0.0b2 - beta quality, for testing only

[ThomasWaldmann]

borgbackup 2.0.0b2 beta release - do not use this on production backup repositories.

Long changelog:
https://borgbackup.readthedocs.io/en/2.0.0b2/changes.html#version-2-0-0b2-2022-09-10

If you use pip to install this, use: pip install "borgbackup==2.0.0b2"

Docs: https://borgbackup.readthedocs.io/en/2.0.0b2/

---

This discussion was created from the release Release 2.0.0b2 - beta quality, for testing only.

[ThomasWaldmann]

There are not many entries in the changelog, but some of the changes required widespread changes in the source code.

So gives this a good bit of testing, please! Due to the repo format changes, you'll need to create fresh repos, it can't work with repos created by beta1 (or borg 1.2). You can borg transfer from borg 1.2.x repos though.

[qupfer]

not sure if a (already known) bug or just "not up-to-date" documentation,
but borg delete seems not working without "-a" switch.

root@borgtest:~# /root/borg-linux64 --version
borg-linux64 2.0.0b2

root@borgtest:~# /root/borg-linux64 rlist
opnsense-1                           Tue, 2022-09-20 08:46:33 +0200 [f8859c44cffe2da5d72c71fde1d304ae620b8fa1ef94f102bae9a23a5662a378]
opnsense-2                           Tue, 2022-09-20 09:12:19 +0200 [36c656c63e8977efe426231f2ef7f40bc15e792ad2b899668aacff02080010bd]

root@borgtest:~# /root/borg-linux64 delete -n --list  "opnsense-2"
usage: borg-linux64 [-V] [-h] [--critical] [--error] [--warning] [--info] [--debug] [--debug-topic TOPIC] [-p] [--iec] [--log-json] [--lock-wait SECONDS] [--bypass-lock] [--show-version] [--show-rc] [--umask M] [--remote-path PATH] [--upload-ratelimit RATE]
                    [--upload-buffer UPLOAD_BUFFER] [--consider-part-files] [--debug-profile FILE] [--rsh RSH] [-r REPO]
                    <command> ...
borg-linux64: error: unrecognized arguments: opnsense-2

root@borgtest:~# /root/borg-linux64 delete -n --list -a  "opnsense-2"
Would delete archive: opnsense-2                           Tue, 2022-09-20 09:12:19 +0200 [36c656c63e8977efe426231f2ef7f40bc15e792ad2b899668aacff02080010bd] (1/1)

[ThomasWaldmann]

I had the idea to use archive names as positional arguments only for commands which need a specific amount of them (like e.g. borg diff needs 2 archive names) and use -a for anything that is happy with an arbitrary amount ( like 0..N) archives, like borg delete.

As you have noticed, if you give a specific non-wildcard string to -a that can work as a replacement for a single archive name.

But guess it will be easier and more user friendly, if borg also accepts positional archive names for such commands. One kind of needs them also for the use case if one wants to delete multiple archives in one go and they can not be matched by a pattern.

[awgcooper]

Same issue with 'borg info' (i.e. for an individual archive). Positional arguments would be good, thanks :)

[awgcooper]

Borg 1.x provides stats at the end of a backup: 'original size', ' compressed size', 'depduplicated size'. Borg 2 shows original and deduplicated. Perhaps you'll be adding compressed to the release version but wanted to enquire. It's definitely useful to know actual benefit of the compression algo one's selected. Same applies to 'info' command. FYI, I'm using '--list --progress --stats'.

[ThomasWaldmann]

Had to remove csize (compressed size) at quite a lot of places to make the way free for some other features (like e.g. efficient repo-wide recompression), that's why it is also gone from the create stats.

Maybe some stats can come back later and get re-implemented in another way.

#6763
#7037

[awgcooper]

Fair enough, the trade off makes sense. If you can bring it back in a different way at some point, I certainly find it useful.

Am I correct in saying that:

• the deduplicated size is 'deduplicated, compressed data', i.e. you compress first and then deduplicate?
• if I created two repos which backed up an identical data set once, with identical settings except compression, the difference between the deduplicated sizes of the two archives would represent the difference in the amount of compression?

[ThomasWaldmann]

You get the repo size that way. If there is only one archive in the repo, it is also the deduped and compressed archive size. There is some overhead, e.g. for the repo index.

[real-yfprojects]

I also think borg should provide a way to reliably determine the compressed size of an archive.

[awgcooper]

Just curious, do you think you'll be able to add back in compression stats? I've been using version 2 for a while now and I do notice their absence.

[ThomasWaldmann]

I can think of these options, but there is no code for these yet:

• while creating an archive, maintain stats inside the compressor object and output it once with create stats - not much effort needed to do this.
• query the compressed sizes from the repo / chunk metadata in the repo - that could be done at any time, but is quite some effort, but possible at any time.

[awgcooper]

Can the first option be used to drop compressed stats into the summary after a backup, like in v1.0? If it's not too much effort, seems like it makes sense to go with this option first. I'm not looking to be able to query compression stats at any other point in time, just like I don't with the current version.

[awgcooper]

I was trying a new test archive using obfuscation. I received the following traceback:

Local Exception
Traceback (most recent call last):
  File "/usr/local/bin/build-borg2/lib/python3.10/site-packages/borg/archiver/__init__.py", line 591, in main
    exit_code = archiver.run(args)
  File "/usr/local/bin/build-borg2/lib/python3.10/site-packages/borg/archiver/__init__.py", line 519, in run
    return set_ec(func(args))
  File "/usr/local/bin/build-borg2/lib/python3.10/site-packages/borg/archiver/_common.py", line 157, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/local/bin/build-borg2/lib/python3.10/site-packages/borg/archiver/create_cmd.py", line 267, in do_create
    create_inner(archive, cache, fso)
  File "/usr/local/bin/build-borg2/lib/python3.10/site-packages/borg/archiver/create_cmd.py", line 162, in create_inner
    self._rec_walk(
  File "/usr/local/bin/build-borg2/lib/python3.10/site-packages/borg/archiver/create_cmd.py", line 449, in _rec_walk
    self._rec_walk(
  File "/usr/local/bin/build-borg2/lib/python3.10/site-packages/borg/archiver/create_cmd.py", line 449, in _rec_walk
    self._rec_walk(
  File "/usr/local/bin/build-borg2/lib/python3.10/site-packages/borg/archiver/create_cmd.py", line 396, in _rec_walk
    status = self._process_any(
  File "/usr/local/bin/build-borg2/lib/python3.10/site-packages/borg/archiver/create_cmd.py", line 280, in _process_any
    return fso.process_file(path=path, parent_fd=parent_fd, name=name, st=st, cache=cache)
  File "/usr/local/bin/build-borg2/lib/python3.10/site-packages/borg/archive.py", line 1442, in process_file
    self.process_file_chunks(
  File "/usr/local/bin/build-borg2/lib/python3.10/site-packages/borg/archive.py", line 1252, in process_file_chunks
    item.chunks.append(chunk_processor(chunk))
  File "/usr/local/bin/build-borg2/lib/python3.10/site-packages/borg/archive.py", line 1240, in chunk_processor
    chunk_entry = cache.add_chunk(chunk_id, {}, data, stats=stats, wait=False)
  File "/usr/local/bin/build-borg2/lib/python3.10/site-packages/borg/cache.py", line 958, in add_chunk
    cdata = self.repo_objs.format(id, meta, data, compress=compress, size=size, ctype=ctype, clevel=clevel)
  File "/usr/local/bin/build-borg2/lib/python3.10/site-packages/borg/repoobj.py", line 43, in format
    meta, data_compressed = self.compressor.compress(meta, data)
  File "src/borg/compress.pyx", line 569, in borg.compress.ObfuscateSize.compress
TypeError: unsupported operand type(s) for +: 'memoryview' and 'bytes'

Platform: Linux raspberrypi 5.15.0-1015-raspi #17-Ubuntu SMP PREEMPT Mon Sep 12 13:14:51 UTC 2022 aarch64
Linux: Unknown Linux
Borg: 2.0.0b2  Python: CPython 3.10.6 msgpack: 1.0.4 fuse: None [pyfuse3,llfuse]
PID: 15580  CWD: /borg/borg2
sys.argv: ['/usr/local/bin/borg2', 'create', '--compression', 'obfuscate,2,auto,lzma', '--list', '--stats', '--progress', '--patterns-from', 'patterns.txt', 'archive1', '/backups_borg2']
SSH_ORIGINAL_COMMAND: None

[awgcooper]

Done! Obfuscate now working for me.

If possible, can I ask: how should I evaluate whether to use obfuscation? I backup several TB of personal data to a VPS. My security 'model' is 'make it hard for opportunists' (i.e. I'm not looking to secure sensitive corporate data or protect against a dedicated actor). I don't really understand the fingerprinting attack that obfuscation is designed to add protection against. I'm basically look to answer the question: is obfuscation worth the extra storage overhead? Any comments aimed at the layperson would be appreciated (I've read the documentation).

P.S. could I also ask: how does one evaluate the security margin between the different obfuscation levels? It seems the tradeoff is size vs security. Level 1 doesn't appear to increase the size of an archive too much. What extra security does one get by moving to level 2 and the larger archive size? I get that more obfuscation=harder to spot patterns. I'm just wondering if there's a more precise way to measure it.

[ThomasWaldmann]

I'm not a cryptographer, not an adviser for endangered persons and borg did not yet receive any formal/professional security review, so take this with a grain of salt and on your own risk:

For normal persons living in free/liberal states that are mainly concerned about privacy and confidentiality, my gut feeling is that they do not need the additional obfuscating and the AEAD encryption / authentication protects them / their data well enough. Thus, the additional space which is needed esp. for the higher obfuscation levels maybe isn't worth it.

I added the obfuscation feature after becoming aware that a size fingerprinting attack against a borg repo might actually work for an attacker, e.g. if the attacker wants to prove that you own a specific collection of relatively small files. It's not very easy to do because the bigger files are chunked, all chunks are potentially compressed and everything is encrypted afterwards but the size of objects in the repository is visible to an attacker with repo access. Thus, a powerful attacker (like a nation state, secret service or otherwise powerful/motivated attacker) could try it.

Adding the obfuscation feature was relatively easy, so I thought "just add it for whoever might want to use it". It is just making the chunks randomly longer within specific, configurable bounds (level 1+ is a different algorithm than level 110+). Level 1 and 110 only add a very small size variation, the higher levels add more variation and need more space, so a user can decide how much repo space the obfuscation is worth to them. Of course, the more variation you add, the harder it gets for an attacker who is willing to also try fuzzy size matches (not just exact matches).

The exact algorithm / size formula can be seen in compress.pyx in class ObfuscateSize.

https://github.com/borgbackup/borg/blob/2.0.0b2/src/borg/compress.pyx#L517

[ThomasWaldmann]

Maybe someone who is more into math, statistics and security than me can have a look and tell something about how much protection it actually adds.

[awgcooper]

Very helpful reply and perfectly pitched at my level! I think I'll probably add in level 1 or 110 and if I'm feeling articulate enough, also ask the folks at crypto stack exchange about the subject.

Once the 2.0 release is done and perhaps at >=2.1, would you be interested in exploring the possibility of a security audit? I know they are expensive - my guess would be $5-10k - but it seems like it would be a good juncture to do it. I'd certainly make a financial contribution and I reckon enough people use and respect the software to provide a decent base from which to ask for donations. I'm happy to discuss further offline if you want.

[ThomasWaldmann]

#2481 see there for the security audit topic.