Skip to content

Commit

Permalink
Zstandard multithreaded compression, exporting & importing of records…
Browse files Browse the repository at this point in the history
…, file format optimisation, info about crc, incremental load, refactoring
  • Loading branch information
piotrj committed Nov 24, 2023
1 parent 7256ab1 commit 705e955
Show file tree
Hide file tree
Showing 9 changed files with 316 additions and 266 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ Custom data extractor is a command that can be invoked with a single parameter -


## Technical information
Record in librer is the result of a single scan operation and is shown as one of many top nodes in the main tree window. Contains a directory tree with collected custom data and CRC data. It is stored as a single .dat file in librer database directory. Its internal format is optimized for security, fast initial access and maximum compression (just check :)) Every section is a python data structure serialized by [pickle](https://docs.python.org/3/library/pickle.html) and compressed separately by [bzip2](https://docs.python.org/3/library/bz2.html#module-bz2) algorithm. The record file, once saved, is never modified afterward. It can only be deleted upon request. All record files are independent of each other.
Record in librer is the result of a single scan operation and is shown as one of many top nodes in the main tree window. Contains a directory tree with collected custom data and CRC data. It is stored as a single .dat file in librer database directory. Its internal format is optimized for security, fast initial access and maximum compression (just check :)) Every section is a python data structure serialized by [pickle](https://docs.python.org/3/library/pickle.html) and compressed separately by [Zstandard](https://pypi.org/project/zstandard/) algorithm. The record file, once saved, is never modified afterward. It can only be deleted upon request or exported. All record files are independent of each other.

###### Manual build (linux):
```
Expand Down
2 changes: 1 addition & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
pyinstaller==6.2.0
nuitka==1.8.6
nuitka==1.9.1
appdirs==1.4.4
send2trash==1.8.2
zstandard==0.22.0
Expand Down
Loading

0 comments on commit 705e955

Please sign in to comment.