wpu.nu

Skillnad mellan versioner av "VariableNaming"

Från wpu.nu

m
m
 
(21 mellanliggande versioner av samma användare visas inte)
Rad 1: Rad 1:
 
__NOTOC__
 
__NOTOC__
 +
===DOCUMENT STANDARD===
 +
====NAMING====
 +
Documents are stored as PDF with a filename of the format:
 +
[document_md5sum].pdf
 +
 +
Document metadata is saved as a json-file with a filename of the format:
 +
[document_md5sum].json
 +
in the same folder as it's corresponding pdf-file.
 +
 +
Exceptions:
 +
On wiki import the file is copied to the import folder with
 +
a descriptive filename, as that is what the corresponding
 +
wiki pages will be named.
  
==DOCUMENT STANDARD==
+
On download the file is named to whatever name the file is given upon
   
+
download, as a checksum cannot be calculated until the file is on disk.
 +
It is then imediately renamed according to the standard
  
===NAMING===
+
On generation the file is named with a unique name (like parent doc + pages to include)
 +
When the file has been generated, a checksum is calculated and
 +
the file is renamed.
  
Documents are stored as PDF with a filename of the format:
+
===ATTRIBUTES===
      [document_md5sum].pdf
+
Some document attributes are reserved and given a defined meaning here.
 +
Other attributes can exists, but should not be passed to other programs
  
Document metadata is saved as a json-file with a filename of the format:
+
CamelCase is used due to it being used in the wiki
      [document_md5sum].json
 
  in the same folder as it's corresponding pdf-file.
 
  
Exceptions:
+
====GENERAL====
  On wiki import the file is copied to the import folder with
 
  a descriptive filename, as that is what the corresponding
 
  wiki pages will be named.
 
  
  On download the file is named to whatever name the file is given upon
+
* <code>Type [interrogation, promemoria, confrontation, letter, etc]</code>
  download, as a checksum cannot be calculated until the file is on disk.
+
* <code>Name</code> A well formatted and descriptive name that will be the base name of the wiki page. If not overriden with NameeOverride, this is re-generated from document meta data on each access. Includes file extension (.pdf)
  It is then imediately renamed according to the standard
+
* <code>WikiName</code> Same as above but after wiki name formatting and folding
 +
* <code>NameeOverride</code> If the name has been changed manually this variable contains the new name, else the empty string
 +
* <code>Title</code> A short sub-title
 +
* <code>OriginalFilename</code> Filename of the downloaded or imported file as first seen
 +
* <code>Checksum</code> MD5 checksum of the corresponding PDF-file
 +
* <code>SectionCode</code> A code refering to the ledger
 +
* <code>InterrogationStart</code> Date and/or time of interrogation start or generation of document
 +
* <code>InterrogationEnd</code> Date and/or time of interrogation end
 +
* <code>PrintoutDate</code> Date of printout of document (used in some templates)
 +
* <code>Language</code> sv
 +
* <code>Interrogator</code> Name of interrogator
 +
* <code>Interrogated</code> Name of interrogated
 +
* <code>ShortSummary</code> A short summary displayed on the wiki
 +
* <code>Author</code> Name of person generating a document if not the interrogator
 +
* <code>Progress</code> Stage of document in wiki-terms (e.g. proofread, waiting for OCR)
 +
* <code>Notes</code> Notes on the document, also used for OCR errors etc
 +
* <code>Stage</code> The stage in the import process [OCR, waiting, downloaded, etc]
 +
* <code>Release</code> True if doc should be deleted from a list in the next iteration
  
  On generation the file is named with a unique name (like parent doc + pages to include)
+
====SPIDER====
  When the file has been generated, a checksum is calculated and
 
  the file is renamed.
 
  
 +
* <code>FoundAt</code> Name of the page where the download link was found
 +
* <code>FoundAtURL</code> The URL from which the download link was found (if any)
 +
* <code>DownloadURL</code> The URL from which the pdf was downloaded
 +
* <code>DownloadURLText</code> Text tag from the above URL
 +
* <code>DownloadDescription</code> A description found in association with the download URL
  
 +
====IMPORT====
  
==ATTRIBUTES==
+
* <code>NumberOfPages</code> Number of pages contained in the corresponding PDF
 
+
* <code>SizeBytes</code> PDF file size in bytes (int)
  Some document attributes are reserved and given a defined meaning here.
+
* <code>SlackDocShouldImportMsgTS</code> ID of slack-message asking about import
  Other attributes can exists, but should not be passed to other programs
+
* <code>SlackDocShouldImportResponse</code> slack-response asking about import
  
  CamelCase is used due to it being used in the wiki
+
====DERIVED====
  
  ===GENERAL===
+
* <code>PDFFilename</code> <checksum>.pdf
 +
* <code>JSONFilename</code> <checksum>.json
 +
* <code>PDFFilePath</code> <directory/<checksum>.pdf
 +
* <code>JSONFilePath</code> <directory/<checksum>.json
 +
* <code>SizeHuman</code> PDF file size in human readable form incl units (str)
 +
* <code>Directory</code> <base_directory>/<Stage>/
 +
* <code>OCRCacheTestFile</code> Filename which if present should cause use of cached OCR results
 +
* <code>WikiIndex</code> Index:<WikiTitle>
  
  Type: [interrogation, promemoria, confrontation, letter, etc]'
 
  Title: A well formatted and descriptive title that will be the base name on the wiki
 
  WikiTitle: Same as above but after wiki name formatting and folding
 
  Checksum: MD5 checksum of the corresponding PDF-file
 
  SectionCode: A code refering to the ledger
 
  InterrogationStart: Date and/or time of interrogation start or generation of document
 
  InterrogationEnd: Date and/or time of interrogation end
 
  Language: sv
 
  Interrogator: Name of interrogator
 
  Interrogated: Name of interrogated
 
  ShortSummary: A short summary displayed on the wiki
 
  Author: Name of person generating a document if not the interrogator
 
  Progress: Stage of document in wiki-terms (e.g. proofread, waiting for OCR)
 
  Notes: Notes on the document, also used for OCR errors etc
 
  Stage: The stage in the import process [OCR, waiting, downloaded, etc]
 
  Release: True if doc should be deleted in the next iteration
 
  
  ===SPIDER===
+
===LOG===
 +
list of dict
 +
log[<date>] = {text:str,...}
  
  FoundAt: Name of the page where the download link was found
 
  FoundAtURL: The URL from which the download link was found (if any)
 
  DownloadURL: The URL from which the pdf was downloaded
 
  DownloadURLText: Text tag from the above URL
 
  DownloadDescription: A description found in association with the download URL
 
  DownloadFileName: The name of the file as downloaded
 
  
  ===IMPORT===
+
===STAGES===
  
  NumberOfPages: Number of pages contained in the corresponding PDF
+
downloading -> downloaded -> ocr_queue -> ocr -> ask_slack -> import
  SizeHuman: PDF file size in human readable form incl units (str)
+
                                                    \------> waiting -> import
  SizeBytes: PDF file size in bytes (int)
+
==PROCESSES==
  SlackDocShouldImportMsgTS: ID of slack-message asking about import
 
  SlackDocShouldImportResponseTS: ID of slack-response asking about import
 
  
  ===DERIVED===
+
===MOPOCR2===
 +
Opens a checksum-named file with pdf and json, OCRs it, adds info in json and saves it back to output folder
  
  PDFFilename: <checksum>.pdf
 
  JSONFilename: <checksum>.json
 
  Directory: <base_directory>/<Stage>/
 
  OCRCacheTestFile: Filename which if present should cause use of cached OCR results
 
  WikiIndex: Index:<WikiTitle>
 
  
==LOG==
+
===wpu-spindeln===
  list of dict
+
Crawls pages (incl the wiki), downloads and renames files. New files are shown to slack. If manual import is needet wpu-spindeln can split and import documents. Adds metadata to json
  log[<date>] = {text:str,...}
 

Nuvarande version från 18 juni 2021 kl. 17.05

DOCUMENT STANDARD

NAMING

Documents are stored as PDF with a filename of the format: [document_md5sum].pdf

Document metadata is saved as a json-file with a filename of the format: [document_md5sum].json in the same folder as it's corresponding pdf-file.

Exceptions: On wiki import the file is copied to the import folder with a descriptive filename, as that is what the corresponding wiki pages will be named.

On download the file is named to whatever name the file is given upon download, as a checksum cannot be calculated until the file is on disk. It is then imediately renamed according to the standard

On generation the file is named with a unique name (like parent doc + pages to include) When the file has been generated, a checksum is calculated and the file is renamed.

ATTRIBUTES

Some document attributes are reserved and given a defined meaning here. Other attributes can exists, but should not be passed to other programs

CamelCase is used due to it being used in the wiki

GENERAL

  • Type [interrogation, promemoria, confrontation, letter, etc]
  • Name A well formatted and descriptive name that will be the base name of the wiki page. If not overriden with NameeOverride, this is re-generated from document meta data on each access. Includes file extension (.pdf)
  • WikiName Same as above but after wiki name formatting and folding
  • NameeOverride If the name has been changed manually this variable contains the new name, else the empty string
  • Title A short sub-title
  • OriginalFilename Filename of the downloaded or imported file as first seen
  • Checksum MD5 checksum of the corresponding PDF-file
  • SectionCode A code refering to the ledger
  • InterrogationStart Date and/or time of interrogation start or generation of document
  • InterrogationEnd Date and/or time of interrogation end
  • PrintoutDate Date of printout of document (used in some templates)
  • Language sv
  • Interrogator Name of interrogator
  • Interrogated Name of interrogated
  • ShortSummary A short summary displayed on the wiki
  • Author Name of person generating a document if not the interrogator
  • Progress Stage of document in wiki-terms (e.g. proofread, waiting for OCR)
  • Notes Notes on the document, also used for OCR errors etc
  • Stage The stage in the import process [OCR, waiting, downloaded, etc]
  • Release True if doc should be deleted from a list in the next iteration

SPIDER

  • FoundAt Name of the page where the download link was found
  • FoundAtURL The URL from which the download link was found (if any)
  • DownloadURL The URL from which the pdf was downloaded
  • DownloadURLText Text tag from the above URL
  • DownloadDescription A description found in association with the download URL

IMPORT

  • NumberOfPages Number of pages contained in the corresponding PDF
  • SizeBytes PDF file size in bytes (int)
  • SlackDocShouldImportMsgTS ID of slack-message asking about import
  • SlackDocShouldImportResponse slack-response asking about import

DERIVED

  • PDFFilename <checksum>.pdf
  • JSONFilename <checksum>.json
  • PDFFilePath <directory/<checksum>.pdf
  • JSONFilePath <directory/<checksum>.json
  • SizeHuman PDF file size in human readable form incl units (str)
  • Directory <base_directory>/<Stage>/
  • OCRCacheTestFile Filename which if present should cause use of cached OCR results
  • WikiIndex Index:<WikiTitle>


LOG

list of dict log[<date>] = {text:str,...}


STAGES

downloading -> downloaded -> ocr_queue -> ocr -> ask_slack -> import
                                                    \------> waiting -> import

PROCESSES

MOPOCR2

Opens a checksum-named file with pdf and json, OCRs it, adds info in json and saves it back to output folder


wpu-spindeln

Crawls pages (incl the wiki), downloads and renames files. New files are shown to slack. If manual import is needet wpu-spindeln can split and import documents. Adds metadata to json