Merge pull request #70 from catalyst/libre-lambda-out

WR408408: Remove libreoffice binaries and AWS lambda stack infrastruc…

.github/workflows/ci.yml

@@ -8,3 +8,4 @@ jobs:
     uses: catalyst/catalyst-moodle-workflows/.github/workflows/ci.yml@main
     with:
       extra_plugin_runners: 'moodle-plugin-ci add-plugin catalyst/moodle-local_aws'
+      disable_phpcpd: true

README.md

@@ -80,6 +80,8 @@ The Libre Lambda document converter must be enabled in Moodle before it can be u
 Before the converter can be used the required AWS infrastructure needs to be setup. This is covered in the next section.
 
 ## AWS Stack Setup
+Binaries and scripts required for the stack are kept in the separate repository - https://github.com/catalyst/moodle-fileconverter_librelambda-aws_stack . The provision script will try to check it out.
+
 The following steps will setup the Amazon Web Services (AWS) infrastructure. The AWS infrastructure is required to do the actual conversion of documents into PDF. While setting up the AWS infrastructure is largely automated by scripts included in this plugin, a working knowledge of AWS is highly recommended.
 
 For more information on how the submitted files are processed in AWS please refer to the topic: [Conversion Architecture](#conversion-architecture)
@@ -354,98 +356,6 @@ This plugin includes precompiled LibreOffice archives as a compressed archive in
 
 The precompiled binary archive for LibreOffice is provided as a convienence to make setting everything up easier. However, you can obtain the LibreOffice source code and compile it yourself. See the section: *Compiling Libre Office* for instructions on how to do this.
 
-#### Compiling Libre Office
-This section will outline how to compile LibreOffice for yourself to be used by AWS Lambda to convert files.
-
-There are two main reasons why we need to custom compile LibreOffice to work with AWS Lambda. The first is we need to compile LibreOffice so it works in the same runtime environment as Lambda. The second is that Lambda has very limited disk space (512MB) which we can use to store and execute binaries. So we need to create a very minimal version of LibreOffice to stay under the disk space limits.
-
-Knowledge of Docker as well as command line Linux administration is required to compile your own LibreOffice installation.
-
-The process to create your own compiled LibreOffice binary archive is:
-* Get the LibreOffice code from the LibreOffice project
-* Compile LibreOffice binaries in a container based on one used for AWS Lambda
-* Create the LibreOffice archive
-
-Following steps are done in the `librelambda/build` directory.
-
-##### Get the LibreOffice code from the LibreOffice project
-
-    wget https://download.documentfoundation.org/libreoffice/src/6.4.7/libreoffice-6.4.7.2.tar.xz
-
-This is the latest version in the 6 family at the time of writing. Amazon 2 image is based on Centos 7,
-which proved challenging enough to make us not even consider trying with LibreOffice 7.
-
-In (unlikely) case that another minor version is requested, `docker build` observes `lo_ver` env var, eg
-    lo_ver=6.4.7.5 docker build ...
-
-##### Compile LibreOffice binaries
-Build the image and launch a container. Depending on your circumnstances/preferences you may use
-slightly different arguments with `docker` commands:
-
-    docker build -t lo-build .
-    # OR for arm64/aarch64 (graviton):
-    docker build -t lo-build --build-arg ARCH=aarch64 .
-    docker run -it --rm lo-build bash
-
-This should give you a shell in a running container. In the container shell:
-
-    make
-    # OR for arm64/aarch64 (graviton):
-    CPPFLAGS="-DPNG_ARM_NEON_OPT=0" make
-    strip instdir/program/*
-
-Once the binaries are compiled (and stripped) you can run the following commands (still in the container shell)
-to test the conversion. You will probably get a fontconfig warning, ignore:
-
-    echo "hello world" > /tmp/a.txt
-    ./instdir/program/soffice.bin --headless --invisible --nodefault --nofirststartwizard \
-        --nolockcheck --nologo --norestore --convert-to pdf --outdir /tmp /tmp/a.txt
-    ls -l /tmp/a.pdf
-
-**Note:** For some reason conversion may silently do nothing. In that case just re-run it.
-
-If everything seems fine, pack it up:
-
-    tar -cf /lo.tar instdir
-
-Now you have lo.tar in the running container. Do not exit it yet. In another terminal on your computer:
-
-    docker ps
-    docker cp <container id>:/lo.tar .
-`docker ps` should give you the id of the `lo-build` running container that you will use for copying.
-
-After checking that you have `lo.tar` you can leave the container.
-
-**NOTE:** Compiling LibreOffice will take time.
-
-**If something goes wrong:**
-
-    docker run -it --rm --cap-add=SYS_PTRACE --security-opt seccomp=unconfined lo-build bash
-
-If you are rebuilding the `lo-build` image, you can copy tarballs from the running container's
-`<build dir>/external/tarballs/` to the `tarballs` directory. That will save you downloading each time you start new container.
-
-After the above steps are completed follow the instructions in the next section, to create the LibreOffice archive.
-
-##### Create the LibreOffice archive
-Now is time to remove unneeded things from lo.tar if you wish
-(share/gallery,template,fonts/truetype/EmojiOneColor-SVGinOT.ttf...). Either untar/remove/tar back, or
-
-    tar -f lo.tar --delete xyz...
-
-Once you are happy with the content of `lo.tar`:
-
-    xz -e9 lo.tar
-
-And replace the existing `lo.tar.xz`:
-
-    mv lo.tar.xz ..
-
-Next time you run the provisioning script to setup the environment it will use the newly created LibreLambda archive for Lambda.
-
-### Lambda Function
-TODO: this
-
 ## FAQs
 
 ### Why make this plugin?

cli/provision.php

@@ -74,6 +74,16 @@ function abort(string $msg) {
     die;
 }
 
+/** exec() with return value check
+ * @param string $command
+ */
+function os_exec(string $command) {
+    exec($command, $output, $retval);
+    if ($retval != 0) {
+        abort("$command failed:\n" . implode("\n", $output));
+    }
+}
+
 echo PHP_EOL;
 
 // Get cli options.
@@ -141,16 +151,24 @@ function abort(string $msg) {
     abort("$stackexistsmsg\n$help");
 }
 
+// check out latest moodle-fileconverter_librelambda-aws_stack
+$repo = "https://github.com/catalyst/moodle-fileconverter_librelambda-aws_stack.git";
+$stackdir = sys_get_temp_dir() . '/fileconverter_librelambda-aws_stack';
+if (!is_dir($stackdir)) {
+    os_exec("git clone $repo $stackdir");
+}
+os_exec("cd $stackdir && git pull");
+
 // First we make the Libre archive a zip file so it can be a Lambda layer.
-$librepath = $CFG->dirroot . '/files/converter/librelambda/libre/lo.tar.xz';
+$librepath = "$stackdir/libre/lo.tar.xz";
 $tmpfname = sys_get_temp_dir() . '/lo.zip';
 $zip = new ZipArchive();
 $zip->open($tmpfname, ZipArchive::CREATE);
 $zip->addFile($librepath, 'lo.tar.xz');
 $zip->close();
 
-$cloudformationpath = $CFG->dirroot . '/files/converter/librelambda/lambda/stack.template';
-$lambdapath = $CFG->dirroot . '/files/converter/librelambda/lambda/lambdaconvert.zip';
+$cloudformationpath = "$stackdir/lambda/stack.template";
+$lambdapath = "$stackdir/lambda/lambdaconvert.zip";
 
 // Create Lambda function, IAM roles and the rest of the stack.
 cli_heading(get_string('provision:stack', 'fileconverter_librelambda'));