Software for the reMarkable tablet

rm2svg.py

incorporated changes from Alan Schmitt which ensure that pages with annotations have the correct geometry (width & height).

Version 41 (2022-09-28T07:58:43Z).

The REMARKABLE environment variable will be used to determine where to copy the files for the PDF document the user wishes to upload. This scripts assumes that the tablet's full file system (root, aka /) has been mounted at this point.

REMARKABLE=${HOME}/remarkable

The tablet documents need to know how many pages there are in any PDF document. We use the pdfinfo tool, from the poppler-utils Debian package, to extract this information. The pdfseparate tool from this package will also be used later on to extract the individual pages from the PDF document.

for pdf in $*
do
    if [ -f ${pdf} ]; then
        n=$(pdfinfo ${pdf} | grep '^Pages' | awk '{print $2}')
        name=$(basename ${pdf} .pdf)
        date="$(date +%s)000"
        echo Uploading PDF document ${name} with ${n} pages, date ${date}
        temp=$(mktemp --directory rm.XXXXX)
        echo Resulting documents in ${temp}

All files in the xochitl directory of the tablet are based on UUID (universally unique identifier) names. We create one such uuid for this new document and, later, we will also create uuids for each of the pages in the document.

The script also creates a temporary directory on the host system. All the files necessary for the tablet to define a document will be placed here and copied to the tablet at the end of the script. This part of the script prepares the actual PDF document for copying by placing it in the temporary directory with the correct name.

uuid=$(uuidgen)
echo Creating $uuid document directories.
dest=${temp}/${uuid}
mkdir ${dest}
mkdir ${dest}.{cache,highlights,textconversion,thumbnails}
cp ${pdf} ${dest}.pdf

The content file specifies the uuids for each of the pages in the document. We create these uuids here. The values for all the other parameters have been taken from an example document I uploaded using the traditional method (i.e. the Android application, in my case). It could be that some of these parameter values should be different in some cases but I have not yet run into any problems. I would welcome any feedback that says otherwise, of course!

cat > ${dest}.content <<EOF
{
    "extraMetadata": {
    },
    "fileType": "pdf",
    "fontName": "",
    "lastOpenedPage": 0,
    "lineHeight": -1,
    "margins": 100,
    "orientation": "portrait",
EOF
echo '    "pageCount":' ${n}',' >> ${dest}.content
echo '    "pages": [' >> ${dest}.content
page=1
while [ ${page} -le ${n} ]
do
    pageuuid=$(uuidgen)
    if [ ${page} -ne ${n} ]
    then
        echo '        "'${pageuuid}'",' >> ${dest}.content
    else
        echo '        "'${pageuuid}'"' >> ${dest}.content
    fi
    ((page++))
done
cat >> ${dest}.content <<EOF
],
    "textScale": 1,
    "transform": {
        "m11": 1,
        "m12": 0,
        "m13": 0,
        "m21": 0,
        "m22": 1,
        "m23": 0,
        "m31": 0,
        "m32": 0,
        "m33": 1
    }
}
EOF

The important bits of information in this file, as near as I can tell, are the last modification time and the actual name of the document. I am not entirely sure what the other parameters mean except for synced which we specify as false.

cat > ${dest}.metadata <<EOF
{
    "deleted": false,
EOF
echo '    "lastModified": "'${date}'",' >> ${dest}.metadata 
cat >> ${dest}.metadata <<EOF
    "metadatamodified": false,
    "modified": false,
    "parent": "",
    "pinned": false,
    "synced": false,
    "type": "DocumentType",
    "version": 1,
EOF
echo '    "visibleName": "'${name}'"' >> ${dest}.metadata 
echo '}' >> ${dest}.metadata 

The pagedata file contains one line per page. On each line is the template used. By default, a freshly uploaded PDF document would always (I assume) have a Blank template. It would be interesting to make the template used optional as I can imagine cases where, for instance, a grid template would be good to impose.

page=1
touch ${dest}.pagedata
while [ ${page} -le ${n} ]
do
    echo Blank >> ${dest}.pagedata
    ((page++))
done

Each page in the document has a corresponding thumbnail, a small jpeg image. I do not know if these would be generated automatically by the tablet but it is relatively straightforward to generate these at this point. I have observed that some thumbnails get regenerated on the tablet in any case. Note that the pages are numbered from 0 upwards but the pdfseparate tool used to extract the individual pages numbers from 1. The image creation is done using the convert tool from the ImageMagick (imagemagick-6 to be precise) Debian package.

pdfseparate ${pdf} ${dest}.thumbnails/%d.pdf
page=0
while [ $page -lt $n ]
do
    convert ${dest}.thumbnails/$((page+1)).pdf -monochrome -scale 362x512 ${dest}.thumbnails/${page}.jpg
    rm ${dest}.thumbnails/$((page+1)).pdf
    ((page++))
done

At this point, all the necessary files and directories have been created within the ${temp} directory. We now copy these to the tablet. The assumption is that the tablet's file system has been mounted onto this system, probably using sshfs. The location is defined above in the environment variable $REMARKABLE.

( cd ${temp} ; tar cf - . | \
      (cd ${REMARKABLE}/home/root/.local/share/remarkable/xochitl/ ; tar xvf - ))

Remove the temporary files created.

rm -rf ${temp}

Once all the files have been created and copied over, we ask the reMarkable tablet to restart the graphical interface, i.e. xochitl. There may be a faster, more efficient way of telling the tablet that there is a new file there but I have not found it yet. Any suggestions would be more than welcome. The script does allow uploading more than one file and only when all have been uploaded is the graphical interface restarted.

This part of the script requires that we log in directly, as root, on the tablet to be able to restart the graphical interface.

ssh root@remarkablewifi systemctl restart xochitl

List of revisions to the rmconvert.sh shell script:

11. 2020-06-23T16:01:13Z  fixed script due to changes in inkscape's arguments for PDF export
10. 2019-06-08T22:43:08Z  clean up after finishing the conversion
 9. 2019-06-07T23:05:57Z  added link for PDF compression web site
 8. 2019-06-07T22:50:10Z  use gs to compress the PDF pages
 7. 2019-06-07T22:34:07Z  placed annotated PDF document in /tmp directly
 6. 2019-06-07T22:07:09Z  annotated document must include non-annotated pages
 5. 2019-06-07T21:50:28Z  conversion works for notebooks and annotated documents
 4. 2019-06-07T21:38:48Z  overlaying annotations on PDF document works
 3. 2019-06-07T13:23:42Z  conversion makes use of templates
 2. 2019-06-07T13:12:44Z  pages successfully converted to SVG
 1. 2019-05-29T10:57:25Z  started writing the script

This function extracts the document's name from the metadata file. The argument is the uuid document metadata file.

function getmetadataname {
    # echo "Looking for document name in $1"
    grep visibleName $1 | sed -e 's/.*": "\?//' -e 's/"\?,*$//'
    ## grep visibleName $1 | sed -e 's/.*": "//' -e 's/",*$//' -e 's/ /./g'
}

Recursively traverse the required uuid files to build up the full name of the document whose uuid has been passed to this function.

function getfullname {
    xochitl=${1}
    uuid=${2}
    meta="${xochitl}/${uuid}.metadata"
    documentname=$(getmetadataname ${meta})
    # echo Document name is $documentname
    # the document may be in a folder.  Use the recursive function defined
    # above to extra a full path to the destination folder
    folders=""
    parentuuid=$(grep parent $meta \
                     | sed -e 's/.*": "\?//' \
                           -e 's/"\?,*$//' \
                           -e 's/null,*$// ')
    ## | sed -e 's/.*": "\?//' -e 's/"\?,*$//' -e 's/ /./g' )
    while [[ "$parentuuid" != "" && "$parentuuid" != "null" ]] 
    do
        if [ "$parentuuid" != "trash" ]
        then
            # echo ".. found folder uuid = $parentuuid"
            directoryname=$(getmetadataname ${xochitl}/${parentuuid}.metadata)
            # echo ".. with name $documentname"
            folders="${directoryname}/${folders}"
            # move up a folder
            parentuuid=$(grep parent "${xochitl}/${parentuuid}.metadata" \
                             | sed -e 's/.*": "\?//' \
                                   -e 's/"\?,*$//'  \
                                   -e 's/null,*$// ')
            ## | sed -e 's/.*": "//' -e 's/",*$//' -e 's/ /./g' )
        else
            folders="trash/${folders}"
            parentuuid=""
        fi
    done  
    # echo "back in main, folders = $folders"
    destination="${destdir}/${folders}"
    echo ${destination}${documentname}
}

For each document in the xochitl directory, output the uuid and the full name of the document.

xochitl=${1}
for metadata in $(find ${xochitl} -name '*.metadata')
do
    uuid=$(basename ${metadata} .metadata)
    # check to see if this is a file or a directory (Collection)
    type=$(grep type ${metadata} | sed -e 's/^.*": "//' -e 's/",.*$//' )
    # echo 'Checking type of file: ' ${metadata} 'is of type' ${type}
    if [ "${type}" == "DocumentType" ]
    then
        fullname=$(getfullname ${xochitl} ${uuid})
        lastchange=$(grep lastModified ${metadata} | sed  's/^.*"\([0-9]\{10\}\).*$/\1/')
        date=$(date --date="@${lastchange}" +%Y%m%d-%H:%M:%S)
        echo ${uuid} ${date} ${fullname}
    fi
done

./rmlist.sh ~/xochitl | sort -k 2 -r

Testing code which will be used to create the reMarkable Emacs mode:

(split-string
 (shell-command-to-string "./rmlist.sh ~/xochitl | sort -k 2 -r")
 "\n")

This section implements a major mode for the Emacs system which provides an interface to the documents on the tablet. The mode presents all the documents (both notebooks and PDF) in a human friendly way (I hope) and allows the user to select any to convert to a screen friendly PDF using the rmconvert.sh scripts defined above. The mode depends on the rmlist.sh script defined above and available here.

The inspiration for this mode was this presentation.

The mode is defined in remarkable.el.

The typical usage is:

1. copy the xochitl directory from the tablet to your local system. I use scp for this. You can, however, direct the mode to work with the tablet directory if you have mounted the tablet's file system on your own but this is significantlly slower. Customize the remarkable-xochitl-directory Emacs variable with the actual location on your system.

2. load the Emacs file. This is what I do:

   (push "where you downloaded remarkable.el" 'load-path)
   (require 'remarkable)
   

3. invoke the command provided in Emacs: M-x remarkable RET and the table view of all your documents in the xochitl directory will be presented:

   

   where some entries have been redacted. The entries have been sorted by date with most recent first; the S key allows you to sort on either column and in both ascending and descending order.

4. The tabulated-list-mode provides a number of key bindings; type ? to see these. The remarkable-mode adds 2 key bindings to the defaults ones:

   i

   display the uuid of the entry in the mini-buffer below the window.

   v

   convert, if necessary, the document to PDF and view the PDF within Emacs. This works for both reMarkable notebooks and annotated PDF documents.

The argument to the method is the directory where all the files on the tablet are found. In practice, for me, this will be a directory on the local system which has been created by copying all the contents of the /home/root/.local/share/remarkable/xochitl directory from the tablet.

(defun remarkable--get-documents (dir)
  (mapcar (lambda (x) (split-string x " "))
          (split-string
           (shell-command-to-string (concat remarkable-rmlist-script " " dir " | sort -k 2 -r")) "\n")))

The mode derives from the tabulated-list-mode which presents a table view into the tablet:

(define-derived-mode remarkable-mode tabulated-list-mode "reMarkable"
  "A mode for viewing documents on the reMarkable tablet."
  (with-demoted-errors "reMarkable mode error: %s"
    (let ((columns [("Date" 18 t) ("Document name" 61 t)])
          (rows (mapcar (lambda (x)
                          (list (car x) (vconcat (cdr x))))
                        (remarkable--get-documents remarkable-xochitl-directory))))
      (setq tabulated-list-format columns)
      (setq tabulated-list-entries rows)
      (tabulated-list-init-header)
      (tabulated-list-print))))

(provide 'remarkable)

Create a buffer (or switch to an existing one) and start up the mode defined above:

(defun remarkable ()
  (interactive)
  (switch-to-buffer "*reMarkable*")
  (remarkable-mode)
  (goto-char (point-min)))