Papermerge DMS or simply Papermerge is a open source document management system designed to work with scanned documents (also called digital archives). It extracts text from your scans using OCR, indexes them, and prepares them for full text search. Papermerge provides look and feel of modern desktop file browsers. It has features like dual panel document browser, drag and drop, tags, hierarchical folders and full text search so that you can efficiently store and organize your documents.
It supports PDF, TIFF, JPEG and PNG document file formats. Papermerge is perfect tool for long term storage of your documents.
"},{"location":"#what-it-does","title":"What It Does","text":"
It extracts (using OCR) and indexes text from your documents
Provides modern, desktop like user interface to easily find your documents
Helps you instantly find your documents
Helps you fix scanned documents issues like:
delete blank, semi-blank or just irrelevant pages
rotate pages
move strayed pages between documents
change page order within the document
"},{"location":"#what-it-doesnt-do","title":"What It Doesn't Do","text":"
It does not take control of your documents. Documents are stored on filesystem in a simple and intuitive manner so that you can take snapshot of your data at any time
It does not overwrite your original documents
"},{"location":"#what-is-a-document","title":"What is a Document?","text":"
For Papermerge a document is anything which is a good candidate for archiving - some piece of information which is not editable but you need to store it for future reference. For example receipts are good examples - you don't need to read receipts everyday, but eventually you will need them for your tax declaration. In this sense - scanned documents, which are usually in PDF or TIFF format, are perfect match.
PDF (Portable Document Format) is de facto standard for storing archived documents. In correct technical terms - it is PDF/A subset. PDF/A differs from PDF by prohibiting features unsuitable for long-term archiving, such as font linking and encryption.
Most of the modern office scanners will output scanned files in PDF/A format. This is why, PDF is practically synonymous for document in context of Papermerge.
A picture made with smart phone of A4 paper document is regarded by Papermerge as document.
Papermerge docker image is shipped with backup and restore utilities. Shipped utility will backup all your folders, documents with their associated versions and OCR data, tags and users. Search engine index is not included in backup though.
Note
User passwords are included in backup file as well. Passwords are stored as digests.
where <optional-location> is the path to file or folder inside container where to save backup file. If location is not provided, backup file will be saved in /core_app/ folder - the papermerge core application's current folder.
In above example the Papermerge has 5 containers: app server (the core or web or http or REST API server, pick the name you like :P), solr search engine, redis, database and finally one paper worker.
To create a backup in root folder of the app container just run:
$ docker exec 914dda21dd3d backup.sh /\n
When above command is ready, check that backup file was created:
$ docker exec 914dda21dd3d ls /\n\nauth_server_app\nbackup_10_12_2023-11_30_37.tar.gz\nbin\nboot\n...\ncore_app\ncore_ui\ndb\n...\nusr\nvar\n
Backup file is backup_10_12_2023-06_30_37.tar.gz. Now you can copy backup file to your local filesystem:
When you plan to restore previous backup, we suggest to start with new Papermerge instance, with only one superuser (which is created by default anyway). Make sure there are no documents in the new instance.
For sake of example, let's say the superuser's username is \"admin\". For restoring use restore.sh command:
The backup file is a gzipped tar archive with following content:
backup.json file
ocr/ folder
docvers/ folder
username1/, username2, ... i.e. one folder per user with folder title being user's username
backup.json file contains all necessary info to restore the database i.e. all users, their nodes, tags etc.
docvers/ contains actually document versions files. Your documents are here.
ocr/ contains OCR data of each individual page in the document.
User folders mentioned in point 4. are provided for convenience, so that you may quickly get an understanding of the folder structure and their content. Each file in user folder is actually a symbolic link pointing to the last version of the document (from docvers).
Warning
Each user has two special folders: .home and .inbox; special folder's title start with dot. If you open backup archive in file browsers which hides dot files (file starting with dot character) - the content of user folder may appear empty! When opening backup archive make sure you set 'show hidden files flag' on.
Command line utility which uses REST API to interact with your Papermerge instance. It can be used to upload documents from local filesystem to yout Papermerge instance.
Papermerge Cli is configured via environment variables:
PAPERMERGE_CLI__HOST
PAPERMERGE_CLI__TOKEN
as the name suggests, the first one is the host of the REST API server and second value is the REST API token.
REST API server should be specified with http:// or https:// prefix, but without the /api suffix. Valid values examples: http://papermege.local, https://my-dms.papermerge.de.
Note
The host may or may not contain the /. E.g. http://papermege.local and http://papermerge.local/ are both valid values and point to the same host
In order to learn UUID of the folder you want to import to use papermerge-cli ls command. To get UUIDs of .home and .inbox folders, use papermerge-cli me command.
If you want the local copy the uploaded documents to be deleted after successful import - use --delete flag:
papermerge-cli import --delete /path/to/folder/\n
Danger
Be careful with --delete flag! When present,
papermerge-cli will irreversible delete the local copy of all documents and folders in the /path/to/folder/!
Danger
Always, before using this flag make safe backup of the documents to be uploaded!
!!! note: --delete flag deletes the local copy of the documents/path to import after successful upload - this means that even if though you local copy of the documents vanished - the originals are still available in Papermerge!
There is docker image for development mode. Docker image is tagged with 3.0devX. With dev image, you can get feedback of your source code changes without needing to install any dependency or any deveopment environment configurations.
All examples described below assume that you got Papermerge source code and you are in root repository of source code folder:
$ git clone git@github.com:papermerge/papermerge-core.git PapermergeSourceCode\n$ cd PapermergeSourceCode\n
This is the simplest local dev scenario, you start docker compose file only with web app i.e. REST API server + ui.
The go to the folder where source was cloned and create following docker compose file:
version: \"3.9\"\n\nservices:\n backend:\n image: papermerge/papermege:3.0dev # check the latest dev image number in dockerhub!\n volumes:\n - ./papermerge:/core_app/papermerge/\n - ./ui:/core_ui/\n environment:\n PAPERMERGE__SECURITY__SECRET_KEY: 1234 # top secret\n PAPERMERGE__AUTH__USERNAME: admin\n PAPERMERGE__AUTH__PASSWORD: 1234\n ports:\n - \"11000:80\"\n
Assuming you are in root folder of the source code, above docker compose will mount the source code to the correct location in docker image. The appliction will be accessible to on local port 11000.
Here is docker compose file for the case when you want to build dev docker image yourself:
Both worker and web app read their logging configurations from file pointed by PAPERMERGE__MAIN__LOGGING_CFG environment variable. An example of custom logging config would be:
Notice that Solr is started with solr-precreate pmg-index command, which means that Solr service will be started with pre-created index named pmg-index.
In above example the Papermerge has four containers: app server, redis, database and one worker. For our purpose we need app container (in example above - fordoc-web-1). Let's list all users first:
Applies only for Tivoli. Set database connections pool size. Defaults to 5.
Note
This configuration option applies only to Tivoli, which is internal JWT token validator component. Tivoli uses SQLAlchemy as ORM. SQLAlchemy has built-in database connections pooling. The core app though, uses Django ORM - which does not have built-in pooling capabilities.
Absolute filesystem path to the yaml file that will hold logging detailed configuration. Content of logging configuration file is expected to be in yaml format and it is very python specific.
By default Papermerge will use language specified with this option to perform OCR. Change this value for language used by majority of your documents. For detailed list of three letter codes see 639-2/T column from ISO 639 2.
Papermerge loads its settings from environment variables.
Environment variables have following format:
PAPERMERGE__<section>__<name>\n
double underscores are used as delimiter; environment variables names must be all upper case.
Only required environment variables are:
PAPERMERGE__SECURITY__SECRET_KEY
PAPERMERGE__AUTH__PASSWORD
PAPERMERGE__SECURITY__SECRET_KEY is the key to securing signed data \u2013 it is vital you keep this secure, or attackers could use it to generate their own signed values.
PAPERMERGE__AUTH__PASSWORD is the password for super user (administrative user or admin user). Super user is created automatically for you when Papermerge starts for the first time.
Note
In the documentation, for brevity sake, PAPERMERGE__ prefix may be omitted. For example docs may say: default value for DATABASE__URL is \"sqlite:////db/db.sqlite3\"; what is meant actually is: default value for PAPERMERGE__DATABASE_URL is \"sqlite:////db/db.sqlite3\".
Required . Unique secret key. The secret key must be a large random value and it must be kept secret. This option does not have default value, you always need to supply a value for it. Secret key is used to sign JWT tokens.
Ansible playbook is available at papermerge/ansible.
Playbook will install web app, one worker, database, Redis and Solr search engine on target host. All services will be started via docker containers. All services will be placed behind traefik, reverse proxy which will take care of TLS certficates.
Choose one of following options:
option 1: install Papermerge with PostgreSQL database
option 2: install Papermerge with PostgreSQL + PgBouncer
option 3: install Papermerge with MariaDB as database
Make sure database_url in your secrets files matches database related options in group_vars/all (db_user, db_name). Also port number database_url should match the one in db_postgres/vars/main.yml.
Application will be accessible via https:// acme_domain is variable you set in group_vars/all e.g. trusel.net"},{"location":"setup/ansible/#option-2-postgresql-pgbouncer","title":"Option 2 / PostgreSQL + PgBouncer","text":"
In this setup application will connect to the database via pgbouncer, this means that database_url should point to pgbouncer.
Application will be accessible via https:// acme_domain is variable you set in group_vars/all e.g. trusel.net"},{"location":"setup/ansible/#option-3-mariadb","title":"Option 3 / MariaDB","text":"
For Mysql/MariaDB database_url should have following format:
You can access Papermerge user interface using any modern web browser (e.g. Firefox, Chrome). Open your web browser and point it to http://localhost:12000.
The only two required environment variables are PAPERMERGE__SECURITY__SECRET_KEY and PAPERMERGE__AUTH__PASSWORD. To start web ui part use following command:
"},{"location":"setup/overview/","title":"Overview","text":""},{"location":"setup/overview/#web-app-and-worker","title":"Web App and Worker","text":"
Papermerge consists of one web app and multiple workers (at least one). The web app is the one which you see in your browsers or interact via REST API. The worker (one or multiple) is the part which performs background tasks like OCR, updating search engine index etc.
Hardware specification for Papermerge depends on number of documents and users.
For one user with 1000-2000 pages a system spec with:
2 CPU at 2.8GHz clock
2 GB RAM
25 GB Storage
will do just fine.
For OCR, Papermerge uses Tesseract. OCR is very CPU intensive operation, thus more CPUs and RAM memory your system has - better. More CPU cores and more powerful the CPUs means OCR will be performed faster.
Note
GPU is not required as Tesseract runs OCR entirely on your CPU.
Testing system for Papermerge has following specs:
From the very beginning of its existence, Papermerge started to receive features requests. Many of those requests made perfect sense and were implemented (or will be implemented). It is thanks to the community's feedback and contributions that Papermerge evolves naturally into a mature archives management system.
However, a great number of features requests were subject to debate, or just not suitable for majority of users. A very good example of such feature is duplicates removal. First of all, how to define that a specific document is a duplicate (of other document)?
There are at least three possibilities:
duplicate are documents with same file name
duplicates are documents with same digest value
duplicates are those documents where extracted text is >= 95% similar (i.e. very similar)
Each of points above has pros and cons. Each of them may be right under specific circumstances and for specific user or company requirements. By default, Papermerge does not implement any of those points. Event more, if you upload same files (with same content and even same filename) it will gladly accept them. In the end, whole this document duplicates thingy is just a matter of preference.
Apps were introduced to address such issues. App (or plugin if you like) is a way extend base Papermerge application. Continuing with above example with document duplicates, by adding/removing specific Papermerge app you can add/remove specific document duplicate detection criteria. This way, Papermerge base application, which is called Papermerge Core will contain only essential features.
On the other hand, document management system are complex applications, with rich set of requirements such as LDAP authentication, document electronic signatures and even such fancy things as data retention policies. Not everybody needs LDAP authentication though. Also, not everybody uses document electronic signatures and data retention policies. These type of features are very good candidates for Papermerge apps. All non essential features will be implemented as apps which can be easily plugged or removed.
Papermerge app is basically a Django app with few extras. You may wonder what is Django app anyway? First of all, Papermerge is build with help of web framework called Django. Secondly, Papermerge project is a Django project which in turn is a loose collection of Django apps plus (project) settings. Figure 1 illustrates this idea.
Notice that Papermerge Core app is part of every Papermerge project.
The gory details of how to develop apps and include them into papermerge project are explained in contributing guide. Here, in user manual, only high level concepts are explained.
Let's consider an app which will detect file name duplicates i.e. an app which if used, will issue an error when user tries to upload twice documents with same file names. Keep in mind that here the goal is to familiarize with general concepts.
Inside Papermerge project create a new django app:
./manage.py startapp filebased_unique\n
To highlight that filebased_unique app is part of papermerge ecosystem, it will be moved into a namespace papermerge (which is just an extra folder):
Those extra fields added by apps to core document model - are called document parts. Papermerge application treats core document fields and those extra fields added by apps as one whole:
This way, any user or company, can extend core document model with whatever number of custom fields they wish without interfering with core application code.
Each document part (i.e all those extra fields) will be visualized as widgets on the right side widget panel along with metadata widget and basic info widget.
For Papermerge a document is anything which is a good candidate for archiving - some piece of information which is not editable but you need to store it for future reference. For example receipts - you don't need to edit receipts or read them everyday, but eventually you will need them for your tax declaration. In this sense - scanned documents, which are usually in PDF, JPEG or TIFF format, are perfect match.
If you take a picture of a paper document with your mobile phone - you'll have a file in jpeg format (or maybe png file format). In context of Papermerge that picture of a document (though just a single jpeg file) is a valid one page document.
On the other hand, if you take a picture of a flower and upload that jpeg image to Papermerge - the 'document' will be processed. However, that jpeg format flower image is not a document in Papermerge sense.
Usually office formats with .docx (Microsoft Word), .odt (Libre Office), .txt (plain text) are not good candidates for archiving - as by their nature they are meant to be changed/edit regularly. However, once converted to PDF format (for instance Contract_C2.docx to Contract_C2.pdf) they are full fledged documents in Papermerge sense.
Info
Papermerge works with four file formats: PDF, TIFF, JPEG and PNG.
One document has one or multiple versions. The original document version - is version number 1. For every change applied to the document - a new document version is created with that change applied.
When we say \"change applied to a document\" - we mean things like rotate pages, reorder pages or merge two documents.
The point of document versions is to keep track of changes applied to the document.
Folder in Papermerge is counterpart of \"folder\" concept used in major computer file manager applications (e.g. Finder in macOS). Folders in Papermerge are, intuitively enough, hierarchical - in other words one folder may contain other folders and/or documents.
Node is an abstraction of two concepts: document and folder. Every time you read node, you can mentally replace that term with either document or folder and the statement will still hold.
Bellow is graphical example of Folder, Document, Document Version relationship:
Each user in Papermerge has two special folders: Inbox and Home.
Inbox folder is where all incoming documents land first. Home folder is where all user documents are.
Special folders are top level folders (they don't have parent folder).
Note
Both Inbox and Home folders are special only by convention; structurally they are just normal folders. Internally their title is actually \".inbox\" and \".home\". By convention special folders start with dot character.
OCR (Optical Character Recognition) is a technique to extract text information from binary image formats. This technique enables users to:
copy/paste text from the document's content
search documents by document's actual text content
OCR is essential tool (or technique if you will) which helps basically to extract textual information and thus derive useful work-flows (based on document's actual content) with the documents. Papermerge relies on external open source specialized tools like Google's Tesseract OCR
Organizing documents in folders is very common. Thus the idea of keeping your documents in folders doesn't need further introduction. The idea of using tags to organize your documents may be new for you though. Tags are kind of labels. You can associate tags to a document or to a folder. Tags have a color and a name.
Once tagged, documents can be searched by their tags. Conversely, is it also possible to show all the documents tagged with a particular tag(s).
Both tags and folders complement each other and provide you with powerful means to stay organized.
Many times scanning documents in bulk yields documents with blank pages; some pages my be out of order, rotated, maybe part of totally different document. Even if you notices these flaws immediately it is time consuming and frustrating to redo scanning process. Papermerge helps you with your scanned documents like no other tool. With Papermerge you can reorder, rotate or even delete pages in case you need to do so.
There is a separate chapter about page management where you can learn details about this feature.
Let's first clarify what is meant by documents merging. Merging is the process of combining two documents into one: all pages from the source document are transferred into destination document and then source document is deleted.
On the target document, transferred pages can:
entirely replace target pages
be appended to the target pages
The rest of this documentation chapter describes how to use Papermerge in case 1. For how to use Papermerge in case 2, see :ref:Page Moving <page_move>.
Figure 1 illustrate this case. Both source (better_scan.pdf) and target (scan_d.pdf) documents have only one version (v1). Both source and target have two pages.
In this case merge result is that in scan_d.pdf document there is a new version created (v2) and new version contains only source pages (BS1 and BS2). Previous pages of scan_d.pdf document (D1, D2) are still available in version 1 (v1 in figure) of the document.
This use case is useful when you scan same document twice and for some reason you want to keep both copies around. Because both copies contain slightly different versions of the same document, it is more practical to keep them as two document versions in one single file. In such case you will avoid duplicate results in search results.
"},{"location":"user/merge-documents/#2-source-pages-are-appended-to-the-target-pages","title":"2. Source pages are appended to the target pages","text":"
Figure 2 illustrate this case. Both source (better_scan.pdf) and target (scan_d.pdf) documents have only one version (v1). Also, both source and target have two pages.
In this case result is that in scan_d.pdf document there is a new version created (v2) and new version contains now four pages: BS1, BS2, D1, D2. Previous version of scan_d.pdf document (v1) has contains two pages: D1 and D2.
This scenario is special case of 'moving pages' between documents with all pages selected on the source. How to use Papermerge in this scenario is described in detail in Moving Pages section.
Important
When merging two documents, one of them (source) is deleted. That's why, it is very important that when you merge two documents, you correctly choose which one is the source and which one is the target.
Now, that you understand what is meant exactly by \"document merging\", let's see how you can merge document with Papermerge.
In order to merge two documents in Papermerge you need to open each of them in two panels:
In one of the panels, the one which you want to be the source, right click the mouse button to open the context menu.
Important
Merge Documents context menu item will be displayed only if there are no selected pages.
In Figure 3, notice the direction of the arrow icon just before \"Merge Document\". The arrow icon points from source to the target. In Figure 3, context menu was opened in left panel, this means that document opened in left panel (better_scan.pdf) is the source. On the other hand if we would open context menu in right panel, then the arrow will point from right to left - which also implies that in such case document opened in the right panel would be the source.
Click the \"Merge Document\" context menu item. After you confirm the operation, the source document (better_scan.pdf) will be merged with scan_d.pdf.
"},{"location":"user/ocr/","title":"OCR","text":"
OCR is the process which extracts text information from the scanned document and makes them searchable.
By default, ocr process is triggered automatically on document file upload. The OCR process status is indicated by little circle next to document's title. When OCR process is completed new document version is created and document becomes searchable.
By default OCR is triggered automatically when document is uploaded. However, you can disable automatic OCR triggering, in such case you can start OCR only when you consider necessary.
Important
Documents for which OCR was skipped - are not searchable!
In order to disable automatic OCR, go to User Menu -> Preferences -> OCR -> Trigger -> Manual
In order to perform OCR on the document you need to indicate beforehand the language of respective document. Choosing ocr language for each and every document uploaded is tedious - instead, in preferences a default OCR Language is set - and that language is applied for each uploaded document.
In order to set default OCR language, go to User Menu -> Preferences -> OCR -> Language
Papermerge features real time OCR status indicator - this means that you can see document's OCR status updates as they happen (i.e. in real time). The OCR status is displayed by a small circle next to the document's title. The status indicates has following meanings:
gray circle - status is unknown (figure 1)
orange still circle - document was scheduled for OCR (figure 2)
orange rotating circle - document's OCR process is in progress (figure 3)
green check - document's OCR process completed successfully and document is now searchable (figure 4)
red cross - document's OCR process failed.
"},{"location":"user/ocr/#ocred-text-layer","title":"OCRed Text Layer","text":"
Once OCR process completed successfully a new document version is created - version with OCRed text layer. This version is available for download from the Download dropdown in document view.
Note
Under the hood Papermerge uses awesome OCRmyPDF utility to create OCRed text layer. Thus, in respect of OCRed text layer, Papermerge acts like a graphical user interface for OCRmyPDF.
You can view OCRed text of the entire document either from commander or from viewer, in both cases choose \"OCRed Text\" from context menu:
If you want to see OCRed text of entire document (to be exact - all pages of the last document version) from the viewer - just make sure that no pages are selected:
In case document has many pages and you are interested in OCRed text of one (or multiple) very specific pages, then select pages first and then from context menu choose \"OCRed Text\" item:
Note
In case there are selected pages, OCRed Text menu item will show you OCRed text ONLY of the selected pages.
"},{"location":"user/ocr/#ocr-languages-support","title":"OCR Languages Support","text":"
Papermerge uses Tesseract to extract text from scanned documents. Tesseract supports over 130 languages - thus with Papermerge you can have documents in any of those languages.
Many times scanning documents in bulk results in documents with blank pages; some pages maybe out of order or maybe part of totally different document. Even if you notice these problems immediately it is time consuming to redo scanning process. Wouldn't it be nice to fix out of order pages without scanning all docs again?
Page management is set of features which helps to fix scanning process errors. In other words you can delete, reorder, rotate, and extract pages within document(s).
Every time one of the operations described in this section is applied - a new document version is created. Because of this, the changes you apply on the document like rotate, delete, extract, reorder, do not destroy the document, in other words page management is non-destructive process.
Note
In order perform any of operations described below (delete, reorder, rotate or extract) you need to have Change Permission on respective document. You have automatically granted Change Permission on the documents you uploaded (because you own the documents uploaded by you).
You can delete specific pages (for instance blank pages) from the document. Although many scanners have automatic \"remove blank pages\" feature, many times they get confused of what a blank page is. In case your scans end up with undesired blank pages you can easily remove those pages.
In order to delete a page, you need to select desired page by clicking on it, then Right Click--> Delete Page.
Every time you delete one or several pages, document version is incremented by one. For instance if document Invoice-X56.pdf currently has four pages and the document latest version is version 1, then, after deleting one page - document latest version will be 2. Thus document's version 1 has all four pages and document version 2 has three pages:
Out of order pages occur very often during scanning process. Papermerge empowers users to change pages order within the document.
For instance, in figure below you can see that pages 2 and 4 are out of place. To correct pages' order use drag 'n drop. For example grab page 2 and drop it in correct position, and then do same thing with page 3:
For these changes to take effect you need to click 'Apply Changes' button.
Warning
Document pages reorder will only be saved when you click 'Apply Changes'
Similarly to deleting pages, every time you save new pages order, document version will be incremented (i.e. advanced by one).
Often scanned pages are upside down or maybe rotated 90\u00b0 (degrees). In order to quickly fix that, select one or multiple pages you want to rotate and then Right Click --> Rotate --> 180\u00b0 CCW (or 90\u00b0 CW, 90\u00b0 CCW depending on your specific case):
Note
CW stand for clockwise. CCW stands for counter-clockwise.
Similarly to page deletion and page ordering, every time you rotate a page, document version will be incremented (i.e. advanced by one).
Warning
After page rotation you have to re-run OCR for the document. It is because if page was upside down when ingested, the OCR operation won't make sense of it and thus won't be able to extract text (and then index) from that page. After you have manually fixed the page (by correctly rotating it) - OCR will be able to extract and index page's contented.
"},{"location":"user/page-management/#move-document-to-document","title":"Move (Document to Document)","text":"
You can move one, multiple or even all strayed pages from one document (source) to another (target). If you choose to move all documents from the source - the source will be deleted, because it does not make sense to have \"document with zero pages\".
When moving pages between documents you will be prompted to choose between two different move strategies:
Replace target
Append to target
The outcome between replace vs append strategies is illustrated below:
The difference is outcome of the B.pdf (target). With replace strategy, the document B.pdf ended up having two pages (which replaced previous ones), while with append strategy the document B.pdf ended up with four pages as source pages were appened to the existing ones.
Note
What happens if you select all source pages, i.e. when you
select A1, B1, B2, A2? In such case - source document (A.pdf) will be deleted, because it does not make any sense to have a document with zero pages. For the target document (B.pdf) this case does not make any difference, as the outcome is always the same.
Note
Use case when you select all pages and chose \"replace strategy\" has same outcome as merging documents.
Now, that \"theory\" is clear, let's move on to the practical part and see Papermerge in action. First of all, note that in Papermerge you can move pages between documents either using context menu or by using drag 'n drop.
Tip
You can also move pages between documents with REST API as well
In order to move pages between documents, using context menu:
Make sure both documents are open in left and right panels
Select pages you want to move in :ref:thumbnails panel <Thumbnails_Panel> of the source document viewer
Use right click to open context menu in source document viewer
Choose \"Move\" context menu item
Context menu is dynamic - which means it renders only relevant menu items. If for example you have in one panel opened :ref:document viewer <Viewer> while other panel is in :ref:commander mode <Commander>, then \"there will be \"extract\" menu item instead of \"move\". In other words, \"move\" menu item will be visible only if:
both panels are in :ref:viewer mode <Viewer>
there is at least one page selected
Important
The arrow next to the \"Move\" menu item changes direction depending in which panel you invoke context menu - it hints direction of the pages transfer. Arrow icon of the \"Move\" item always points from source to target.
"},{"location":"user/page-management/#drag-n-drop","title":"Drag 'n Drop","text":"
In example illustrated in pictures below there are two documents:
document A - with pages marked as A1, A2, B1, A3
document B - with pages B2, B3
During scan page B1 wrongly ended up in document A, although it belongs to document B.
Note
A page that during the scan ended up in wrong document is called strayed page. In example above, page B1 is strayed page.
In order to fix this scanning issue, you need open documents in two panels and then drag 'n drop page B1 from document A (source) to document B (target):
Note
Pages are moved immediately after 'mouse drop' i.e. there is no need to 'click apply button' as in re-order operation
Note
Both documents' (source and target) version will be incremented by one
"},{"location":"user/page-management/#extract-document-to-folder","title":"Extract (Document to Folder)","text":"
Page extraction is moving page out of the document as completely new document. It differs from page moving <page_move> because the destination is a folder, not a document.
You can extract one or multiple pages at once. Pages can be extracted:
as one document; in this case one document will be created in the target folder, new document will contain all extracted pages
as multiple documents; in this case multiple documents will be created in the target folder, each new document will have one page
Note that in Papermerge you can extract pages either using context menu or by using drag 'n drop.
In order to extract pages from the document, using context menu:
Open document in one panel and open target folder in another panel
Select pages you want to extract in thumbnails panel of the source document viewer
Use right click to open context menu in source document viewer
Choose \"Extract\" context menu item
Because of the dynamic nature of the context menu, \"Extract\" menu item will be visible only if all of the following conditions are true:
Two panels are opened
One of the panels is in viewer mode while another is in commander mode
There is at least one page selected
After you've clicked \"Extract\", the \"Extract Pages\" modal dialog will prompt you for additional details like title of the newly created document(s) and if you want to extract all pages as you or multiple documents:
A couple of notes here. First, newly created document will have extension \".pdf\", you cannot change that. Second, if \"Extract each page into separate document\" is checked, each pages will be, obviously enough, extracted as separate one page documents, otherwise all extracted pages will be placed into a single document in the target folder.
Note
Papermerge will try to make sure that newly created documents feature unique name. Thus if you choose to extract, say, two pages as separate documents, Papermerge will append to the title an UUID number. In case you choose to extract two pages into a single document - no UUID number will be appended. In case you leave \"Title format\" field empty, Papermerge will generate an unique title for you.
"},{"location":"user/page-management/#using-drag-n-drop","title":"Using Drag 'n Drop","text":"
Let's show how page extraction works by example. Say we have one document - document A - with following pages: A1, A2, B1, B2, A3. What we want to do is to extract pages B1 and B2 into a new document. As mentioned above there are two cases:
Both pages B1 and B2 are extracted into one single new document (two page document)
Both pages B1 and B2 are extracted into multiple (new) single page documents.
In order to extract pages B1 and B2 into one single new document you need to uncheck 'Extract each page into separate document' checkbox in modal dialog:
Similarly to other operations document A's (source document) version is incremented by one.
Do you need re-run OCR after document's page was moved/rotated/extracted/deleted ?
In short - no, you don't need to re-run OCR. The only exception is page rotation. Every time you rotate a page in the document, you need to re-run OCR for that document. It actually makes sense, because if page was upside down when document was ingested, the OCR operation won't make much sense of it and thus won't be able to extract any text data from the page. Once you correct that part manually (rotate page), you re-run OCR so that correct text will be extracted and then indexed.
Note
Generally speaking you don't need to re-run OCR after performing page management operations. The only exception from this rule is page rotation.
For longer answer, let's clarify first what OCR data is. OCR data is: text information extracted from the document by OCR and associated with that document. That text information is stored in both database and on filesystem.
When one page is moved from one document into another (or when page is deleted), the text associated with source (or target) document changes as well. For example, say document fruits.pdf has three pages: apples, oranges and bananas, i.e each page has only one word page 1 has work apples etc. You can find document fruits.pdf by searching 'apples' (will match first page), 'orages' (will match second page) or bananas (will match last page).
After you extract first page (apples) from document fruits.pdf into another document, searching by term 'apples' should not reveal document 'fruits.pdf' - because term/page 'apples' is not part of it anymore.
In order to keep text information associated with document fruits.pdf up to date, there are at least two possibilities:
re-run OCR after each extract/delete/move/rotate operation
re-use existing OCR and move it/delete it according to the operation
From technical point of view 1. is very easy to implement but very inefficient in terms of computing power. Think that you have 100 pages document and you delete one blank page - what a waste of CPU resources to re-OCR entire document when OCR data is already available!
The second possibility (point 2.) is very challenging to implement, but extremely efficient - you need to run OCR on the document only once (maybe twice, in case you decide to fix couple of pages by rotating them).
Papermerge decided on 2. in other words, Papermerge reuses already extracted OCR data and updates it accordingly every time you re-order/move/extract/delete pages.
The result is that whatever page management operation you perform the search results are always up-to-date without the need to re-OCR the document! As mentioned above the only exception are page rotations.
Below is illustrated the case of three page fruits.pdf document with apples/oranges/bananas content. Initially search term 'apples' will reveal fruits.pdf document (from Inbox). After 'apples' page was extracted into separate document (found in Home/My Documents folder) search term 'apples' correctly reveals new document! Notice here that search index is updated instantaneously:
Tags are sort of labels. You can associate tags to a document or to a folder. Tags have a color and a name. Once tagged, documents can be searched by their tags.
There are multiple ways to upload documents to your Papermerge instance: web UI, command line utilities, REST API.
The obvious way is via web UI. After briefly explaining how to upload documents from web user interface, this page will dive into more interesting parts: command line utilities and REST API.
Uploading documents via user interface is the most straightforward method, just click upload button:
Documents will be uploaded into your current folder. Current folder is considered the one which you currently see as opened in web UI:
Also, instead of using upload button, you can drag'n drop documents from your desktop file manager into Papermerge's web ui.
Warning
Currently drag 'n drop feature does not work for folders, in other words you can drag 'n drop only documents. If you want to import an folders with entire content preserved - use papermerge-cli described in next paragraph.
You can upload documents and folders from your local filesystem using papermerge-cli command line utility:
papermerge-cli import /path/to/local/folder/\n
Note that papermerge-cli will import all content of /path/to/local/folder/ directory recursively i.e. it will preserve the structure of local folder in Papermerge as well.
You can upload one document by providing path to the document:
papermerge-cli import /path/to/document.pdf\n
Note
By default all imported documents and folders will end up inside user's Inbox folder.
For more information about papermerge-cli check papermerge-cli section.
For uploading documents you can use directly REST API. You can access REST API swagger schema definition from user menu (upper right corner of the web UI). In order to upload a document there are two steps:
Create document node without file
Upload the file for the created node in previous step
For step 1. use POST /nodes/ REST API endpoint. For step 2 use POST /documents/<doc-uuid>/upload REST API endpoint, where <doc-uuid> is the ID of the node created in first step.
Commander (in figure 1. marked with number 4.) is designed to have similar look and feel of modern desktop file browsers. This is the place where you browse your documents and folders.
In order to assist you to quickly move around documents, folders and pages - there is a special mode - dual panel model. In dual panel mode there are two panels displayed side by side. Between two panels documents (as well as folders and pages) can be moved with one simple drag'n drop. Figure below shows how dual panel mode looks like:
In order to switch to dual panel mode, use Commander's upper right button:
To switch back to single panel mode, use close button - which is in the upper right corner of one of the panels:
Note
Close button will be displayed only on one of the panels. Although both panels look and feel exactly the same, internally application still distinguishes them as main and secondary one. Main panel is the one which is always visible and secondary panel is the on which opens and closes i.e. the one with \"close button\" in upper right corner.
Commander or Commander Panel is one of the two available panels. Commander is the panel which shows documents and folders - modern web based file browser if you will.
Viewer or Viewer Panel or Document Viewer is one of the two available panels. Viewer is the panel in which document is opened.
There can be two Viewers opened side by side. This mode (i.e. dual panel mode with a Viewer in each panel) is very handy when it comes to moving pages between documents.
Document viewer features a thumbnails panel which can be toggles on and off. Pages can be selected only inside thumbnails panel; also pages can be dragged/dropped only from thumbnails panel.
The most privileged user (which has all permissions) is called superuser. Brand new Papermerge instance ships with one default user - which happens to be superuser. With default user, you can add as many users as you wish.
Papermerge is a non-destructive DMS, which means you always have available original document regardless how many transformations (page rotations, deletion, document merges) you apply on the document.
Retention of the original is ensured because of document versioning feature. With each extra transformation you apply - a new document version is created.
Version 1 (one) of the uploaded document is the original file i.e. document without any changes applied. Original document version is always available, regardless what operation(s) you apply to the document (except deletion of the document itself).
Any page management OCR operation on the document will increment (increase by one) its version.
"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Papermerge DMS","text":"
Papermerge DMS or simply Papermerge is a open source document management system designed to work with scanned documents (also called digital archives). It extracts text from your scans using OCR, indexes them, and prepares them for full text search. Papermerge provides look and feel of modern desktop file browsers. It has features like dual panel document browser, drag and drop, tags, hierarchical folders and full text search so that you can efficiently store and organize your documents.
It supports PDF, TIFF, JPEG and PNG document file formats. Papermerge is perfect tool for long term storage of your documents.
"},{"location":"#what-it-does","title":"What It Does","text":"
It extracts (using OCR) and indexes text from your documents
Provides modern, desktop like user interface to easily find your documents
Helps you instantly find your documents
Helps you fix scanned documents issues like:
delete blank, semi-blank or just irrelevant pages
rotate pages
move strayed pages between documents
change page order within the document
"},{"location":"#what-it-doesnt-do","title":"What It Doesn't Do","text":"
It does not take control of your documents. Documents are stored on filesystem in a simple and intuitive manner so that you can take snapshot of your data at any time
It does not overwrite your original documents
"},{"location":"#what-is-a-document","title":"What is a Document?","text":"
For Papermerge a document is anything which is a good candidate for archiving - some piece of information which is not editable but you need to store it for future reference. For example receipts are good examples - you don't need to read receipts everyday, but eventually you will need them for your tax declaration. In this sense - scanned documents, which are usually in PDF or TIFF format, are perfect match.
PDF (Portable Document Format) is de facto standard for storing archived documents. In correct technical terms - it is PDF/A subset. PDF/A differs from PDF by prohibiting features unsuitable for long-term archiving, such as font linking and encryption.
Most of the modern office scanners will output scanned files in PDF/A format. This is why, PDF is practically synonymous for document in context of Papermerge.
A picture made with smart phone of A4 paper document is regarded by Papermerge as document.
Papermerge docker image is shipped with backup and restore utilities. Shipped utility will backup all your folders, documents with their associated versions and OCR data, tags and users. Search engine index is not included in backup though.
Note
User passwords are included in backup file as well. Passwords are stored as digests.
where <optional-location> is the path to file or folder inside container where to save backup file. If location is not provided, backup file will be saved in /core_app/ folder - the papermerge core application's current folder.
In above example the Papermerge has 5 containers: app server (the core or web or http or REST API server, pick the name you like :P), solr search engine, redis, database and finally one paper worker.
To create a backup in root folder of the app container just run:
$ docker exec 914dda21dd3d backup.sh /\n
When above command is ready, check that backup file was created:
$ docker exec 914dda21dd3d ls /\n\nauth_server_app\nbackup_10_12_2023-11_30_37.tar.gz\nbin\nboot\n...\ncore_app\ncore_ui\ndb\n...\nusr\nvar\n
Backup file is backup_10_12_2023-06_30_37.tar.gz. Now you can copy backup file to your local filesystem:
When you plan to restore previous backup, we suggest to start with new Papermerge instance, with only one superuser (which is created by default anyway). Make sure there are no documents in the new instance.
For sake of example, let's say the superuser's username is \"admin\". For restoring use restore.sh command:
The backup file is a gzipped tar archive with following content:
backup.json file
ocr/ folder
docvers/ folder
username1/, username2, ... i.e. one folder per user with folder title being user's username
backup.json file contains all necessary info to restore the database i.e. all users, their nodes, tags etc.
docvers/ contains actually document versions files. Your documents are here.
ocr/ contains OCR data of each individual page in the document.
User folders mentioned in point 4. are provided for convenience, so that you may quickly get an understanding of the folder structure and their content. Each file in user folder is actually a symbolic link pointing to the last version of the document (from docvers).
Warning
Each user has two special folders: .home and .inbox; special folder's title start with dot. If you open backup archive in file browsers which hides dot files (file starting with dot character) - the content of user folder may appear empty! When opening backup archive make sure you set 'show hidden files flag' on.
Command line utility which uses REST API to interact with your Papermerge instance. It can be used to upload documents from local filesystem to yout Papermerge instance.
Papermerge Cli is configured via environment variables:
PAPERMERGE_CLI__HOST
PAPERMERGE_CLI__TOKEN
as the name suggests, the first one is the host of the REST API server and second value is the REST API token.
REST API server should be specified with http:// or https:// prefix, but without the /api suffix. Valid values examples: http://papermege.local, https://my-dms.papermerge.de.
Note
The host may or may not contain the /. E.g. http://papermege.local and http://papermerge.local/ are both valid values and point to the same host
In order to learn UUID of the folder you want to import to use papermerge-cli ls command. To get UUIDs of .home and .inbox folders, use papermerge-cli me command.
If you want the local copy the uploaded documents to be deleted after successful import - use --delete flag:
papermerge-cli import --delete /path/to/folder/\n
Danger
Be careful with --delete flag! When present,
papermerge-cli will irreversible delete the local copy of all documents and folders in the /path/to/folder/!
Danger
Always, before using this flag make safe backup of the documents to be uploaded!
!!! note: --delete flag deletes the local copy of the documents/path to import after successful upload - this means that even if though you local copy of the documents vanished - the originals are still available in Papermerge!
There is docker image for development mode. Docker image is tagged with 3.0devX. With dev image, you can get feedback of your source code changes without needing to install any dependency or any deveopment environment configurations.
All examples described below assume that you got Papermerge source code and you are in root repository of source code folder:
$ git clone git@github.com:papermerge/papermerge-core.git PapermergeSourceCode\n$ cd PapermergeSourceCode\n
This is the simplest local dev scenario, you start docker compose file only with web app i.e. REST API server + ui.
The go to the folder where source was cloned and create following docker compose file:
version: \"3.9\"\n\nservices:\n backend:\n image: papermerge/papermege:3.0dev # check the latest dev image number in dockerhub!\n volumes:\n - ./papermerge:/core_app/papermerge/\n - ./ui:/core_ui/\n environment:\n PAPERMERGE__SECURITY__SECRET_KEY: 1234 # top secret\n PAPERMERGE__AUTH__USERNAME: admin\n PAPERMERGE__AUTH__PASSWORD: 1234\n ports:\n - \"11000:80\"\n
Assuming you are in root folder of the source code, above docker compose will mount the source code to the correct location in docker image. The appliction will be accessible to on local port 11000.
Here is docker compose file for the case when you want to build dev docker image yourself:
Both worker and web app read their logging configurations from file pointed by PAPERMERGE__MAIN__LOGGING_CFG environment variable. An example of custom logging config would be:
Notice that Solr is started with solr-precreate pmg-index command, which means that Solr service will be started with pre-created index named pmg-index.
In above example the Papermerge has four containers: app server, redis, database and one worker. For our purpose we need app container (in example above - fordoc-web-1). Let's list all users first:
Applies only for Tivoli. Set database connections pool size. Defaults to 5.
Note
This configuration option applies only to Tivoli, which is internal JWT token validator component. Tivoli uses SQLAlchemy as ORM. SQLAlchemy has built-in database connections pooling. The core app though, uses Django ORM - which does not have built-in pooling capabilities.
Absolute filesystem path to the yaml file that will hold logging detailed configuration. Content of logging configuration file is expected to be in yaml format and it is very python specific.
By default Papermerge will use language specified with this option to perform OCR. Change this value for language used by majority of your documents. For detailed list of three letter codes see 639-2/T column from ISO 639 2.
Papermerge loads its settings from environment variables.
Environment variables have following format:
PAPERMERGE__<section>__<name>\n
double underscores are used as delimiter; environment variables names must be all upper case.
Only required environment variables are:
PAPERMERGE__SECURITY__SECRET_KEY
PAPERMERGE__AUTH__PASSWORD
PAPERMERGE__SECURITY__SECRET_KEY is the key to securing signed data \u2013 it is vital you keep this secure, or attackers could use it to generate their own signed values.
PAPERMERGE__AUTH__PASSWORD is the password for super user (administrative user or admin user). Super user is created automatically for you when Papermerge starts for the first time.
Note
In the documentation, for brevity sake, PAPERMERGE__ prefix may be omitted. For example docs may say: default value for DATABASE__URL is \"sqlite:////db/db.sqlite3\"; what is meant actually is: default value for PAPERMERGE__DATABASE_URL is \"sqlite:////db/db.sqlite3\".
Required . Unique secret key. The secret key must be a large random value and it must be kept secret. This option does not have default value, you always need to supply a value for it. Secret key is used to sign JWT tokens.
Ansible playbook is available at papermerge/ansible.
Playbook will install web app, two workers, database, Redis and Solr search engine on target host. All services will be deployed as docker containers. All services will be placed behind traefik, reverse proxy which will take care of TLS certficates.
Choose one of following options:
option 1: install Papermerge with PostgreSQL database
option 2: install Papermerge with PostgreSQL + PgBouncer
option 3: install Papermerge with MariaDB as database
Make sure database_url in your secrets files matches database related options in group_vars/all (db_user, db_name). Also port number database_url should match the one in db_postgres/vars/main.yml.
Application will be accessible via https:// acme_domain is variable you set in group_vars/all e.g. trusel.net"},{"location":"setup/ansible/#option-2-postgresql-pgbouncer","title":"Option 2 / PostgreSQL + PgBouncer","text":"
In this setup application will connect to the database via pgbouncer, this means that database_url should point to pgbouncer.
Application will be accessible via https:// acme_domain is variable you set in group_vars/all e.g. trusel.net"},{"location":"setup/ansible/#option-3-mariadb","title":"Option 3 / MariaDB","text":"
For Mysql/MariaDB database_url should have following format:
You can access Papermerge user interface using any modern web browser (e.g. Firefox, Chrome). Open your web browser and point it to http://localhost:12000.
The only two required environment variables are PAPERMERGE__SECURITY__SECRET_KEY and PAPERMERGE__AUTH__PASSWORD. To start web ui part use following command:
"},{"location":"setup/overview/","title":"Overview","text":""},{"location":"setup/overview/#web-app-and-worker","title":"Web App and Worker","text":"
Papermerge consists of one web app and multiple workers (at least one). The web app is the one which you see in your browsers or interact via REST API. The worker (one or multiple) is the part which performs background tasks like OCR, updating search engine index etc.
Hardware specification for Papermerge depends on number of documents and users.
For one user with 1000-2000 pages a system spec with:
2 CPU at 2.8GHz clock
2 GB RAM
25 GB Storage
will do just fine.
For OCR, Papermerge uses Tesseract. OCR is very CPU intensive operation, thus more CPUs and RAM memory your system has - better. More CPU cores and more powerful the CPUs means OCR will be performed faster.
Note
GPU is not required as Tesseract runs OCR entirely on your CPU.
Testing system for Papermerge has following specs:
For Papermerge a document is anything which is a good candidate for archiving - some piece of information which is not editable but you need to store it for future reference. For example receipts - you don't need to edit receipts or read them everyday, but eventually you will need them for your tax declaration. In this sense - scanned documents, which are usually in PDF, JPEG or TIFF format, are perfect match.
If you take a picture of a paper document with your mobile phone - you'll have a file in jpeg format (or maybe png file format). In context of Papermerge that picture of a document (though just a single jpeg file) is a valid one page document.
On the other hand, if you take a picture of a flower and upload that jpeg image to Papermerge - the 'document' will be processed. However, that jpeg format flower image is not a document in Papermerge sense.
Usually office formats with .docx (Microsoft Word), .odt (Libre Office), .txt (plain text) are not good candidates for archiving - as by their nature they are meant to be changed/edit regularly. However, once converted to PDF format (for instance Contract_C2.docx to Contract_C2.pdf) they are full fledged documents in Papermerge sense.
Info
Papermerge works with four file formats: PDF, TIFF, JPEG and PNG.
One document has one or multiple versions. The original document version - is version number 1. For every change applied to the document - a new document version is created with that change applied.
When we say \"change applied to a document\" - we mean things like rotate pages, reorder pages or merge two documents.
The point of document versions is to keep track of changes applied to the document.
Folder in Papermerge is counterpart of \"folder\" concept used in major computer file manager applications (e.g. Finder in macOS). Folders in Papermerge are, intuitively enough, hierarchical - in other words one folder may contain other folders and/or documents.
Node is an abstraction of two concepts: document and folder. Every time you read node, you can mentally replace that term with either document or folder and the statement will still hold.
Bellow is graphical example of Folder, Document, Document Version relationship:
Each user in Papermerge has two special folders: Inbox and Home.
Inbox folder is where all incoming documents land first. Home folder is where all user documents are.
Special folders are top level folders (they don't have parent folder).
Note
Both Inbox and Home folders are special only by convention; structurally they are just normal folders. Internally their title is actually \".inbox\" and \".home\". By convention special folders start with dot character.
OCR (Optical Character Recognition) is a technique to extract text information from binary image formats. This technique enables users to:
copy/paste text from the document's content
search documents by document's actual text content
OCR is essential tool (or technique if you will) which helps basically to extract textual information and thus derive useful work-flows (based on document's actual content) with the documents. Papermerge relies on external open source specialized tools like Google's Tesseract OCR
Organizing documents in folders is very common. Thus the idea of keeping your documents in folders doesn't need further introduction. The idea of using tags to organize your documents may be new for you though. Tags are kind of labels. You can associate tags to a document or to a folder. Tags have a color and a name.
Once tagged, documents can be searched by their tags. Conversely, is it also possible to show all the documents tagged with a particular tag(s).
Both tags and folders complement each other and provide you with powerful means to stay organized.
Many times scanning documents in bulk yields documents with blank pages; some pages my be out of order, rotated, maybe part of totally different document. Even if you notices these flaws immediately it is time consuming and frustrating to redo scanning process. Papermerge helps you with your scanned documents like no other tool. With Papermerge you can reorder, rotate or even delete pages in case you need to do so.
There is a separate chapter about page management where you can learn details about this feature.
Let's first clarify what is meant by documents merging. Merging is the process of combining two documents into one: all pages from the source document are transferred into destination document and then source document is deleted.
On the target document, transferred pages can:
entirely replace target pages
be appended to the target pages
The rest of this documentation chapter describes how to use Papermerge in case 1. For how to use Papermerge in case 2, see :ref:Page Moving <page_move>.
Figure 1 illustrate this case. Both source (better_scan.pdf) and target (scan_d.pdf) documents have only one version (v1). Both source and target have two pages.
In this case merge result is that in scan_d.pdf document there is a new version created (v2) and new version contains only source pages (BS1 and BS2). Previous pages of scan_d.pdf document (D1, D2) are still available in version 1 (v1 in figure) of the document.
This use case is useful when you scan same document twice and for some reason you want to keep both copies around. Because both copies contain slightly different versions of the same document, it is more practical to keep them as two document versions in one single file. In such case you will avoid duplicate results in search results.
"},{"location":"user/merge-documents/#2-source-pages-are-appended-to-the-target-pages","title":"2. Source pages are appended to the target pages","text":"
Figure 2 illustrate this case. Both source (better_scan.pdf) and target (scan_d.pdf) documents have only one version (v1). Also, both source and target have two pages.
In this case result is that in scan_d.pdf document there is a new version created (v2) and new version contains now four pages: BS1, BS2, D1, D2. Previous version of scan_d.pdf document (v1) has contains two pages: D1 and D2.
This scenario is special case of 'moving pages' between documents with all pages selected on the source. How to use Papermerge in this scenario is described in detail in Moving Pages section.
Important
When merging two documents, one of them (source) is deleted. That's why, it is very important that when you merge two documents, you correctly choose which one is the source and which one is the target.
Now, that you understand what is meant exactly by \"document merging\", let's see how you can merge document with Papermerge.
In order to merge two documents in Papermerge you need to open each of them in two panels:
In one of the panels, the one which you want to be the source, right click the mouse button to open the context menu.
Important
Merge Documents context menu item will be displayed only if there are no selected pages.
In Figure 3, notice the direction of the arrow icon just before \"Merge Document\". The arrow icon points from source to the target. In Figure 3, context menu was opened in left panel, this means that document opened in left panel (better_scan.pdf) is the source. On the other hand if we would open context menu in right panel, then the arrow will point from right to left - which also implies that in such case document opened in the right panel would be the source.
Click the \"Merge Document\" context menu item. After you confirm the operation, the source document (better_scan.pdf) will be merged with scan_d.pdf.
"},{"location":"user/ocr/","title":"OCR","text":"
OCR is the process which extracts text information from the scanned document and makes them searchable.
By default, ocr process is triggered automatically on document file upload. The OCR process status is indicated by little circle next to document's title. When OCR process is completed new document version is created and document becomes searchable.
By default OCR is triggered automatically when document is uploaded. However, you can disable automatic OCR triggering, in such case you can start OCR only when you consider necessary.
Important
Documents for which OCR was skipped - are not searchable!
In order to disable automatic OCR, go to User Menu -> Preferences -> OCR -> Trigger -> Manual
In order to perform OCR on the document you need to indicate beforehand the language of respective document. Choosing ocr language for each and every document uploaded is tedious - instead, in preferences a default OCR Language is set - and that language is applied for each uploaded document.
In order to set default OCR language, go to User Menu -> Preferences -> OCR -> Language
Papermerge features real time OCR status indicator - this means that you can see document's OCR status updates as they happen (i.e. in real time). The OCR status is displayed by a small circle next to the document's title. The status indicates has following meanings:
gray circle - status is unknown (figure 1)
orange still circle - document was scheduled for OCR (figure 2)
orange rotating circle - document's OCR process is in progress (figure 3)
green check - document's OCR process completed successfully and document is now searchable (figure 4)
red cross - document's OCR process failed.
"},{"location":"user/ocr/#ocred-text-layer","title":"OCRed Text Layer","text":"
Once OCR process completed successfully a new document version is created - version with OCRed text layer. This version is available for download from the Download dropdown in document view.
Note
Under the hood Papermerge uses awesome OCRmyPDF utility to create OCRed text layer. Thus, in respect of OCRed text layer, Papermerge acts like a graphical user interface for OCRmyPDF.
You can view OCRed text of the entire document either from commander or from viewer, in both cases choose \"OCRed Text\" from context menu:
If you want to see OCRed text of entire document (to be exact - all pages of the last document version) from the viewer - just make sure that no pages are selected:
In case document has many pages and you are interested in OCRed text of one (or multiple) very specific pages, then select pages first and then from context menu choose \"OCRed Text\" item:
Note
In case there are selected pages, OCRed Text menu item will show you OCRed text ONLY of the selected pages.
"},{"location":"user/ocr/#ocr-languages-support","title":"OCR Languages Support","text":"
Papermerge uses Tesseract to extract text from scanned documents. Tesseract supports over 130 languages - thus with Papermerge you can have documents in any of those languages.
Many times scanning documents in bulk results in documents with blank pages; some pages maybe out of order or maybe part of totally different document. Even if you notice these problems immediately it is time consuming to redo scanning process. Wouldn't it be nice to fix out of order pages without scanning all docs again?
Page management is set of features which helps to fix scanning process errors. In other words you can delete, reorder, rotate, and extract pages within document(s).
Every time one of the operations described in this section is applied - a new document version is created. Because of this, the changes you apply on the document like rotate, delete, extract, reorder, do not destroy the document, in other words page management is non-destructive process.
Note
In order perform any of operations described below (delete, reorder, rotate or extract) you need to have Change Permission on respective document. You have automatically granted Change Permission on the documents you uploaded (because you own the documents uploaded by you).
You can delete specific pages (for instance blank pages) from the document. Although many scanners have automatic \"remove blank pages\" feature, many times they get confused of what a blank page is. In case your scans end up with undesired blank pages you can easily remove those pages.
In order to delete a page, you need to select desired page by clicking on it, then Right Click--> Delete Page.
Every time you delete one or several pages, document version is incremented by one. For instance if document Invoice-X56.pdf currently has four pages and the document latest version is version 1, then, after deleting one page - document latest version will be 2. Thus document's version 1 has all four pages and document version 2 has three pages:
Out of order pages occur very often during scanning process. Papermerge empowers users to change pages order within the document.
For instance, in figure below you can see that pages 2 and 4 are out of place. To correct pages' order use drag 'n drop. For example grab page 2 and drop it in correct position, and then do same thing with page 3:
For these changes to take effect you need to click 'Apply Changes' button.
Warning
Document pages reorder will only be saved when you click 'Apply Changes'
Similarly to deleting pages, every time you save new pages order, document version will be incremented (i.e. advanced by one).
Often scanned pages are upside down or maybe rotated 90\u00b0 (degrees). In order to quickly fix that, select one or multiple pages you want to rotate and then Right Click --> Rotate --> 180\u00b0 CCW (or 90\u00b0 CW, 90\u00b0 CCW depending on your specific case):
Note
CW stand for clockwise. CCW stands for counter-clockwise.
Similarly to page deletion and page ordering, every time you rotate a page, document version will be incremented (i.e. advanced by one).
Warning
After page rotation you have to re-run OCR for the document. It is because if page was upside down when ingested, the OCR operation won't make sense of it and thus won't be able to extract text (and then index) from that page. After you have manually fixed the page (by correctly rotating it) - OCR will be able to extract and index page's contented.
"},{"location":"user/page-management/#move-document-to-document","title":"Move (Document to Document)","text":"
You can move one, multiple or even all strayed pages from one document (source) to another (target). If you choose to move all documents from the source - the source will be deleted, because it does not make sense to have \"document with zero pages\".
When moving pages between documents you will be prompted to choose between two different move strategies:
Replace target
Append to target
The outcome between replace vs append strategies is illustrated below:
The difference is outcome of the B.pdf (target). With replace strategy, the document B.pdf ended up having two pages (which replaced previous ones), while with append strategy the document B.pdf ended up with four pages as source pages were appened to the existing ones.
Note
What happens if you select all source pages, i.e. when you
select A1, B1, B2, A2? In such case - source document (A.pdf) will be deleted, because it does not make any sense to have a document with zero pages. For the target document (B.pdf) this case does not make any difference, as the outcome is always the same.
Note
Use case when you select all pages and chose \"replace strategy\" has same outcome as merging documents.
Now, that \"theory\" is clear, let's move on to the practical part and see Papermerge in action. First of all, note that in Papermerge you can move pages between documents either using context menu or by using drag 'n drop.
Tip
You can also move pages between documents with REST API as well
In order to move pages between documents, using context menu:
Make sure both documents are open in left and right panels
Select pages you want to move in :ref:thumbnails panel <Thumbnails_Panel> of the source document viewer
Use right click to open context menu in source document viewer
Choose \"Move\" context menu item
Context menu is dynamic - which means it renders only relevant menu items. If for example you have in one panel opened :ref:document viewer <Viewer> while other panel is in :ref:commander mode <Commander>, then \"there will be \"extract\" menu item instead of \"move\". In other words, \"move\" menu item will be visible only if:
both panels are in :ref:viewer mode <Viewer>
there is at least one page selected
Important
The arrow next to the \"Move\" menu item changes direction depending in which panel you invoke context menu - it hints direction of the pages transfer. Arrow icon of the \"Move\" item always points from source to target.
"},{"location":"user/page-management/#drag-n-drop","title":"Drag 'n Drop","text":"
In example illustrated in pictures below there are two documents:
document A - with pages marked as A1, A2, B1, A3
document B - with pages B2, B3
During scan page B1 wrongly ended up in document A, although it belongs to document B.
Note
A page that during the scan ended up in wrong document is called strayed page. In example above, page B1 is strayed page.
In order to fix this scanning issue, you need open documents in two panels and then drag 'n drop page B1 from document A (source) to document B (target):
Note
Pages are moved immediately after 'mouse drop' i.e. there is no need to 'click apply button' as in re-order operation
Note
Both documents' (source and target) version will be incremented by one
"},{"location":"user/page-management/#extract-document-to-folder","title":"Extract (Document to Folder)","text":"
Page extraction is moving page out of the document as completely new document. It differs from page moving <page_move> because the destination is a folder, not a document.
You can extract one or multiple pages at once. Pages can be extracted:
as one document; in this case one document will be created in the target folder, new document will contain all extracted pages
as multiple documents; in this case multiple documents will be created in the target folder, each new document will have one page
Note that in Papermerge you can extract pages either using context menu or by using drag 'n drop.
In order to extract pages from the document, using context menu:
Open document in one panel and open target folder in another panel
Select pages you want to extract in thumbnails panel of the source document viewer
Use right click to open context menu in source document viewer
Choose \"Extract\" context menu item
Because of the dynamic nature of the context menu, \"Extract\" menu item will be visible only if all of the following conditions are true:
Two panels are opened
One of the panels is in viewer mode while another is in commander mode
There is at least one page selected
After you've clicked \"Extract\", the \"Extract Pages\" modal dialog will prompt you for additional details like title of the newly created document(s) and if you want to extract all pages as you or multiple documents:
A couple of notes here. First, newly created document will have extension \".pdf\", you cannot change that. Second, if \"Extract each page into separate document\" is checked, each pages will be, obviously enough, extracted as separate one page documents, otherwise all extracted pages will be placed into a single document in the target folder.
Note
Papermerge will try to make sure that newly created documents feature unique name. Thus if you choose to extract, say, two pages as separate documents, Papermerge will append to the title an UUID number. In case you choose to extract two pages into a single document - no UUID number will be appended. In case you leave \"Title format\" field empty, Papermerge will generate an unique title for you.
"},{"location":"user/page-management/#using-drag-n-drop","title":"Using Drag 'n Drop","text":"
Let's show how page extraction works by example. Say we have one document - document A - with following pages: A1, A2, B1, B2, A3. What we want to do is to extract pages B1 and B2 into a new document. As mentioned above there are two cases:
Both pages B1 and B2 are extracted into one single new document (two page document)
Both pages B1 and B2 are extracted into multiple (new) single page documents.
In order to extract pages B1 and B2 into one single new document you need to uncheck 'Extract each page into separate document' checkbox in modal dialog:
Similarly to other operations document A's (source document) version is incremented by one.
Do you need re-run OCR after document's page was moved/rotated/extracted/deleted ?
In short - no, you don't need to re-run OCR. The only exception is page rotation. Every time you rotate a page in the document, you need to re-run OCR for that document. It actually makes sense, because if page was upside down when document was ingested, the OCR operation won't make much sense of it and thus won't be able to extract any text data from the page. Once you correct that part manually (rotate page), you re-run OCR so that correct text will be extracted and then indexed.
Note
Generally speaking you don't need to re-run OCR after performing page management operations. The only exception from this rule is page rotation.
For longer answer, let's clarify first what OCR data is. OCR data is: text information extracted from the document by OCR and associated with that document. That text information is stored in both database and on filesystem.
When one page is moved from one document into another (or when page is deleted), the text associated with source (or target) document changes as well. For example, say document fruits.pdf has three pages: apples, oranges and bananas, i.e each page has only one word page 1 has work apples etc. You can find document fruits.pdf by searching 'apples' (will match first page), 'orages' (will match second page) or bananas (will match last page).
After you extract first page (apples) from document fruits.pdf into another document, searching by term 'apples' should not reveal document 'fruits.pdf' - because term/page 'apples' is not part of it anymore.
In order to keep text information associated with document fruits.pdf up to date, there are at least two possibilities:
re-run OCR after each extract/delete/move/rotate operation
re-use existing OCR and move it/delete it according to the operation
From technical point of view 1. is very easy to implement but very inefficient in terms of computing power. Think that you have 100 pages document and you delete one blank page - what a waste of CPU resources to re-OCR entire document when OCR data is already available!
The second possibility (point 2.) is very challenging to implement, but extremely efficient - you need to run OCR on the document only once (maybe twice, in case you decide to fix couple of pages by rotating them).
Papermerge decided on 2. in other words, Papermerge reuses already extracted OCR data and updates it accordingly every time you re-order/move/extract/delete pages.
The result is that whatever page management operation you perform the search results are always up-to-date without the need to re-OCR the document! As mentioned above the only exception are page rotations.
Below is illustrated the case of three page fruits.pdf document with apples/oranges/bananas content. Initially search term 'apples' will reveal fruits.pdf document (from Inbox). After 'apples' page was extracted into separate document (found in Home/My Documents folder) search term 'apples' correctly reveals new document! Notice here that search index is updated instantaneously:
Tags are sort of labels. You can associate tags to a document or to a folder. Tags have a color and a name. Once tagged, documents can be searched by their tags.
There are multiple ways to upload documents to your Papermerge instance: web UI, command line utilities, REST API.
The obvious way is via web UI. After briefly explaining how to upload documents from web user interface, this page will dive into more interesting parts: command line utilities and REST API.
Uploading documents via user interface is the most straightforward method, just click upload button:
Documents will be uploaded into your current folder. Current folder is considered the one which you currently see as opened in web UI:
Also, instead of using upload button, you can drag'n drop documents from your desktop file manager into Papermerge's web ui.
Warning
Currently drag 'n drop feature does not work for folders, in other words you can drag 'n drop only documents. If you want to import an folders with entire content preserved - use papermerge-cli described in next paragraph.
You can upload documents and folders from your local filesystem using papermerge-cli command line utility:
papermerge-cli import /path/to/local/folder/\n
Note that papermerge-cli will import all content of /path/to/local/folder/ directory recursively i.e. it will preserve the structure of local folder in Papermerge as well.
You can upload one document by providing path to the document:
papermerge-cli import /path/to/document.pdf\n
Note
By default all imported documents and folders will end up inside user's Inbox folder.
For more information about papermerge-cli check papermerge-cli section.
For uploading documents you can use directly REST API. You can access REST API swagger schema definition from user menu (upper right corner of the web UI). In order to upload a document there are two steps:
Create document node without file
Upload the file for the created node in previous step
For step 1. use POST /nodes/ REST API endpoint. For step 2 use POST /documents/<doc-uuid>/upload REST API endpoint, where <doc-uuid> is the ID of the node created in first step.
Commander (in figure 1. marked with number 4.) is designed to have similar look and feel of modern desktop file browsers. This is the place where you browse your documents and folders.
In order to assist you to quickly move around documents, folders and pages - there is a special mode - dual panel model. In dual panel mode there are two panels displayed side by side. Between two panels documents (as well as folders and pages) can be moved with one simple drag'n drop. Figure below shows how dual panel mode looks like:
In order to switch to dual panel mode, use Commander's upper right button:
To switch back to single panel mode, use close button - which is in the upper right corner of one of the panels:
Note
Close button will be displayed only on one of the panels. Although both panels look and feel exactly the same, internally application still distinguishes them as main and secondary one. Main panel is the one which is always visible and secondary panel is the on which opens and closes i.e. the one with \"close button\" in upper right corner.
Commander or Commander Panel is one of the two available panels. Commander is the panel which shows documents and folders - modern web based file browser if you will.
Viewer or Viewer Panel or Document Viewer is one of the two available panels. Viewer is the panel in which document is opened.
There can be two Viewers opened side by side. This mode (i.e. dual panel mode with a Viewer in each panel) is very handy when it comes to moving pages between documents.
Document viewer features a thumbnails panel which can be toggles on and off. Pages can be selected only inside thumbnails panel; also pages can be dragged/dropped only from thumbnails panel.
The most privileged user (which has all permissions) is called superuser. Brand new Papermerge instance ships with one default user - which happens to be superuser. With default user, you can add as many users as you wish.
Papermerge is a non-destructive DMS, which means you always have available original document regardless how many transformations (page rotations, deletion, document merges) you apply on the document.
Retention of the original is ensured because of document versioning feature. With each extra transformation you apply - a new document version is created.
Version 1 (one) of the uploaded document is the original file i.e. document without any changes applied. Original document version is always available, regardless what operation(s) you apply to the document (except deletion of the document itself).
Any page management OCR operation on the document will increment (increase by one) its version.
"}]}
\ No newline at end of file
diff --git a/3.0/setup/ansible/index.html b/3.0/setup/ansible/index.html
index 7e8ba253..c8489c0f 100644
--- a/3.0/setup/ansible/index.html
+++ b/3.0/setup/ansible/index.html
@@ -1342,10 +1342,11 @@
Playbook will install web app, one worker, database, Redis and Solr search
-engine on target host. All services will be started via docker containers.
+
Playbook will install web app, two workers, database, Redis and Solr search
+engine on target host. All services will be deployed as docker containers.
All services will be placed behind traefik, reverse proxy
which will take care of TLS certficates.
+
Choose one of following options:
option 1: install Papermerge with PostgreSQL database
From the very beginning of its existence, Papermerge started to receive
-features requests. Many of those requests made perfect sense and were
-implemented (or will be implemented). It is thanks to the community's
-feedback and contributions that Papermerge evolves naturally into a mature
-archives management system.
-
However, a great number of features requests were subject to debate, or just
-not suitable for majority of users. A very good example of such feature is
-duplicates removal. First of all, how to define that a specific document is a
-duplicate (of other document)?
-
There are at least three possibilities:
-
-
duplicate are documents with same file name
-
duplicates are documents with same digest value
-
duplicates are those documents where extracted text is >= 95% similar (i.e. very similar)
-
-
Each of points above has pros and cons. Each of them may be right under
-specific circumstances and for specific user or company requirements. By
-default, Papermerge does not implement any of those points. Event more, if you
-upload same files (with same content and even same filename) it will gladly
-accept them. In the end, whole this document duplicates thingy is just a
-matter of preference.
-
Apps were introduced to address such issues. App (or plugin if you like) is a
-way extend base Papermerge application. Continuing with above example with
-document duplicates, by adding/removing specific Papermerge app you can
-add/remove specific document duplicate detection criteria. This way, Papermerge
-base application, which is called Papermerge Core will contain only essential
-features.
-
Papermerge Core
-
On the other hand, document management system are complex applications, with
-rich set of requirements such as LDAP authentication, document electronic
-signatures and even such fancy things as data retention policies. Not
-everybody needs LDAP authentication though. Also, not everybody uses document
-electronic signatures and data retention policies. These type of features are
-very good candidates for Papermerge apps. All non essential features will be
-implemented as apps which can be easily plugged or removed.
-
Apps
-
Papermerge app is basically a Django app with few extras. You may wonder what
-is Django app anyway? First of all, Papermerge is build with help of web
-framework called Django. Secondly,
-Papermerge project is a Django project which in turn is a loose collection of
-Django apps plus (project) settings. Figure 1 illustrates this idea.
-
-
Notice that Papermerge Core app is part of every Papermerge project.
-
Writing Basic App
-
The gory details of how to develop apps and include them into papermerge
-project are explained in contributing guide. Here, in user manual, only high
-level concepts are explained.
-
Let's consider an app which will detect file name duplicates i.e. an app which
-if used, will issue an error when user tries to upload twice documents with same
-file names. Keep in mind that here the goal is to familiarize with general
-concepts.
-
Inside Papermerge project create a new django app:
-
./manage.py startapp filebased_unique
-
-
To highlight that filebased_unique app is part of papermerge ecosystem, it will be moved into a namespace papermerge (which is just an extra folder):
Those extra fields added by apps to core document model - are called document
-parts. Papermerge application treats core document fields and those extra
-fields added by apps as one whole:
-
-
This way, any user or company, can extend core document model with whatever
-number of custom fields they wish without interfering with core application
-code.
-
Document Widgets
-
-
Note
-
This feature is experimental.
-
-
Each document part (i.e all those extra fields) will be visualized as widgets on
-the right side widget panel along with metadata widget and basic info widget.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
\ No newline at end of file
diff --git a/latest/user/apps/index.html b/latest/user/apps/index.html
deleted file mode 100644
index 63bb1c5e..00000000
--- a/latest/user/apps/index.html
+++ /dev/null
@@ -1,16 +0,0 @@
-
-
-
-
- Redirecting
-
-
-
-
- Redirecting to ../../../3.0/user/apps/...
-
-
\ No newline at end of file
