wpu.nu

Skillnad mellan versioner av "VariableNaming"

Från wpu.nu

m
m
 
Rad 31: Rad 31:
  
 
* <code>Type [interrogation, promemoria, confrontation, letter, etc]</code>
 
* <code>Type [interrogation, promemoria, confrontation, letter, etc]</code>
* <code>Name</code> A well formatted and descriptive name that will be the base name of the wiki page. If not overriden with NameeOverride, this is re-generated from document meta data on each access  
+
* <code>Name</code> A well formatted and descriptive name that will be the base name of the wiki page. If not overriden with NameeOverride, this is re-generated from document meta data on each access. Includes file extension (.pdf)
 
* <code>WikiName</code> Same as above but after wiki name formatting and folding
 
* <code>WikiName</code> Same as above but after wiki name formatting and folding
 
* <code>NameeOverride</code> If the name has been changed manually this variable contains the new name, else the empty string
 
* <code>NameeOverride</code> If the name has been changed manually this variable contains the new name, else the empty string

Nuvarande version från 18 juni 2021 kl. 16.05

DOCUMENT STANDARD

NAMING

Documents are stored as PDF with a filename of the format: [document_md5sum].pdf

Document metadata is saved as a json-file with a filename of the format: [document_md5sum].json in the same folder as it's corresponding pdf-file.

Exceptions: On wiki import the file is copied to the import folder with a descriptive filename, as that is what the corresponding wiki pages will be named.

On download the file is named to whatever name the file is given upon download, as a checksum cannot be calculated until the file is on disk. It is then imediately renamed according to the standard

On generation the file is named with a unique name (like parent doc + pages to include) When the file has been generated, a checksum is calculated and the file is renamed.

ATTRIBUTES

Some document attributes are reserved and given a defined meaning here. Other attributes can exists, but should not be passed to other programs

CamelCase is used due to it being used in the wiki

GENERAL

  • Type [interrogation, promemoria, confrontation, letter, etc]
  • Name A well formatted and descriptive name that will be the base name of the wiki page. If not overriden with NameeOverride, this is re-generated from document meta data on each access. Includes file extension (.pdf)
  • WikiName Same as above but after wiki name formatting and folding
  • NameeOverride If the name has been changed manually this variable contains the new name, else the empty string
  • Title A short sub-title
  • OriginalFilename Filename of the downloaded or imported file as first seen
  • Checksum MD5 checksum of the corresponding PDF-file
  • SectionCode A code refering to the ledger
  • InterrogationStart Date and/or time of interrogation start or generation of document
  • InterrogationEnd Date and/or time of interrogation end
  • PrintoutDate Date of printout of document (used in some templates)
  • Language sv
  • Interrogator Name of interrogator
  • Interrogated Name of interrogated
  • ShortSummary A short summary displayed on the wiki
  • Author Name of person generating a document if not the interrogator
  • Progress Stage of document in wiki-terms (e.g. proofread, waiting for OCR)
  • Notes Notes on the document, also used for OCR errors etc
  • Stage The stage in the import process [OCR, waiting, downloaded, etc]
  • Release True if doc should be deleted from a list in the next iteration

SPIDER

  • FoundAt Name of the page where the download link was found
  • FoundAtURL The URL from which the download link was found (if any)
  • DownloadURL The URL from which the pdf was downloaded
  • DownloadURLText Text tag from the above URL
  • DownloadDescription A description found in association with the download URL

IMPORT

  • NumberOfPages Number of pages contained in the corresponding PDF
  • SizeBytes PDF file size in bytes (int)
  • SlackDocShouldImportMsgTS ID of slack-message asking about import
  • SlackDocShouldImportResponse slack-response asking about import

DERIVED

  • PDFFilename <checksum>.pdf
  • JSONFilename <checksum>.json
  • PDFFilePath <directory/<checksum>.pdf
  • JSONFilePath <directory/<checksum>.json
  • SizeHuman PDF file size in human readable form incl units (str)
  • Directory <base_directory>/<Stage>/
  • OCRCacheTestFile Filename which if present should cause use of cached OCR results
  • WikiIndex Index:<WikiTitle>


LOG

list of dict log[<date>] = {text:str,...}


STAGES

downloading -> downloaded -> ocr_queue -> ocr -> ask_slack -> import
                                                    \------> waiting -> import

PROCESSES

MOPOCR2

Opens a checksum-named file with pdf and json, OCRs it, adds info in json and saves it back to output folder


wpu-spindeln

Crawls pages (incl the wiki), downloads and renames files. New files are shown to slack. If manual import is needet wpu-spindeln can split and import documents. Adds metadata to json