wpu.nu

VariableNaming

Från wpu.nu

DOCUMENT STANDARD

NAMING

Documents are stored as PDF with a filename of the format: [document_md5sum].pdf

Document metadata is saved as a json-file with a filename of the format: [document_md5sum].json in the same folder as it's corresponding pdf-file.

Exceptions: On wiki import the file is copied to the import folder with a descriptive filename, as that is what the corresponding wiki pages will be named.

On download the file is named to whatever name the file is given upon download, as a checksum cannot be calculated until the file is on disk. It is then imediately renamed according to the standard

On generation the file is named with a unique name (like parent doc + pages to include) When the file has been generated, a checksum is calculated and the file is renamed.

ATTRIBUTES

Some document attributes are reserved and given a defined meaning here. Other attributes can exists, but should not be passed to other programs

CamelCase is used due to it being used in the wiki

GENERAL

  • Type [interrogation, promemoria, confrontation, letter, etc]
  • Title A well formatted and descriptive title that will be the base name on the wiki
  • WikiTitle Same as above but after wiki name formatting and folding
  • TitleBeforeChange If the title has been changed manually this variable contains whatever it was before, else the empty string
  • OriginalFilename Filename of the downloaded or imported file as first seen
  • Checksum MD5 checksum of the corresponding PDF-file
  • SectionCode A code refering to the ledger
  • InterrogationStart Date and/or time of interrogation start or generation of document
  • InterrogationEnd Date and/or time of interrogation end
  • Language sv
  • Interrogator Name of interrogator
  • Interrogated Name of interrogated
  • ShortSummary A short summary displayed on the wiki
  • Author Name of person generating a document if not the interrogator
  • Progress Stage of document in wiki-terms (e.g. proofread, waiting for OCR)
  • Notes Notes on the document, also used for OCR errors etc
  • Stage The stage in the import process [OCR, waiting, downloaded, etc]
  • Release True if doc should be deleted from a list in the next iteration

SPIDER

  • FoundAt Name of the page where the download link was found
  • FoundAtURL The URL from which the download link was found (if any)
  • DownloadURL The URL from which the pdf was downloaded
  • DownloadURLText Text tag from the above URL
  • DownloadDescription A description found in association with the download URL
  • DownloadFileName The name of the file as downloaded

IMPORT

  • NumberOfPages Number of pages contained in the corresponding PDF
  • SizeHuman PDF file size in human readable form incl units (str)
  • SizeBytes PDF file size in bytes (int)
  • SlackDocShouldImportMsgTS ID of slack-message asking about import
  • SlackDocShouldImportResponse slack-response asking about import

DERIVED

  • PDFFilename <checksum>.pdf
  • JSONFilename <checksum>.json
  • PDFFilePath <directory/<checksum>.pdf
  • JSONFilePath <directory/<checksum>.json
  • Directory <base_directory>/<Stage>/
  • OCRCacheTestFile Filename which if present should cause use of cached OCR results
  • WikiIndex Index:<WikiTitle>

LOG

list of dict log[<date>] = {text:str,...}


STAGES

downloading -> downloaded -> ocr_queue -> ocr -> ask_slack -> import
                                                    \------> waiting -> import

PROCESSES

MOPOCR2

Opens a checksum-named file with pdf and json, OCRs it, adds info in json and saves it back to output folder


wpu-spindeln

Crawls pages (incl the wiki), downloads and renames files. New files are shown to slack. If manual import is needet wpu-spindeln can split and import documents. Adds metadata to json