Skillnad mellan versioner av "VariableNaming"
Från wpu.nu
m |
m |
||
| Rad 1: | Rad 1: | ||
__NOTOC__ | __NOTOC__ | ||
| − | |||
===DOCUMENT STANDARD=== | ===DOCUMENT STANDARD=== | ||
| − | |||
| − | |||
====NAMING==== | ====NAMING==== | ||
| − | |||
Documents are stored as PDF with a filename of the format: | Documents are stored as PDF with a filename of the format: | ||
[document_md5sum].pdf | [document_md5sum].pdf | ||
| Rad 25: | Rad 21: | ||
When the file has been generated, a checksum is calculated and | When the file has been generated, a checksum is calculated and | ||
the file is renamed. | the file is renamed. | ||
| − | |||
| − | |||
===ATTRIBUTES=== | ===ATTRIBUTES=== | ||
| Rad 83: | Rad 77: | ||
list of dict | list of dict | ||
log[<date>] = {text:str,...} | log[<date>] = {text:str,...} | ||
| + | |||
| + | |||
| + | ==PROCESSES== | ||
| + | |||
| + | ===MOPOCR2=== | ||
| + | Opens a checksum-named file with pdf and json, OCRs it, adds info in json and saves it back to output folder | ||
| + | |||
| + | |||
| + | ===wpu-spindeln=== | ||
| + | Crawls pages (incl the wiki), downloads and renames files. New files are shown to slack. If manual import is needet wpu-spindeln can split and import documents. Adds metadata to json | ||
Versionen från 7 juni 2021 kl. 19.42
DOCUMENT STANDARD
NAMING
Documents are stored as PDF with a filename of the format: [document_md5sum].pdf
Document metadata is saved as a json-file with a filename of the format: [document_md5sum].json in the same folder as it's corresponding pdf-file.
Exceptions: On wiki import the file is copied to the import folder with a descriptive filename, as that is what the corresponding wiki pages will be named.
On download the file is named to whatever name the file is given upon download, as a checksum cannot be calculated until the file is on disk. It is then imediately renamed according to the standard
On generation the file is named with a unique name (like parent doc + pages to include) When the file has been generated, a checksum is calculated and the file is renamed.
ATTRIBUTES
Some document attributes are reserved and given a defined meaning here. Other attributes can exists, but should not be passed to other programs
CamelCase is used due to it being used in the wiki
GENERAL
Type [interrogation, promemoria, confrontation, letter, etc]TitleA well formatted and descriptive title that will be the base name on the wikiWikiTitleSame as above but after wiki name formatting and foldingTitleBeforeChangeIf the title has been changed manually this variable contains whatever it was before, else the empty stringOriginalFilenameFilename of the downloaded or imported file as first seenChecksumMD5 checksum of the corresponding PDF-fileSectionCodeA code refering to the ledgerInterrogationStartDate and/or time of interrogation start or generation of documentInterrogationEndDate and/or time of interrogation endLanguagesvInterrogatorName of interrogatorInterrogatedName of interrogatedShortSummaryA short summary displayed on the wikiAuthorName of person generating a document if not the interrogatorProgressStage of document in wiki-terms (e.g. proofread, waiting for OCR)NotesNotes on the document, also used for OCR errors etcStageThe stage in the import process [OCR, waiting, downloaded, etc]ReleaseTrue if doc should be deleted from a list in the next iteration
SPIDER
FoundAtName of the page where the download link was foundFoundAtURLThe URL from which the download link was found (if any)DownloadURLThe URL from which the pdf was downloadedDownloadURLTextText tag from the above URLDownloadDescriptionA description found in association with the download URLDownloadFileNameThe name of the file as downloaded
IMPORT
NumberOfPagesNumber of pages contained in the corresponding PDFSizeHumanPDF file size in human readable form incl units (str)SizeBytesPDF file size in bytes (int)SlackDocShouldImportMsgTSID of slack-message asking about importSlackDocShouldImportResponseTSID of slack-response asking about import
DERIVED
PDFFilename<checksum>.pdfJSONFilename<checksum>.jsonDirectory<base_directory>/<Stage>/OCRCacheTestFileFilename which if present should cause use of cached OCR resultsWikiIndexIndex:<WikiTitle>
LOG
list of dict log[<date>] = {text:str,...}
PROCESSES
MOPOCR2
Opens a checksum-named file with pdf and json, OCRs it, adds info in json and saves it back to output folder
wpu-spindeln
Crawls pages (incl the wiki), downloads and renames files. New files are shown to slack. If manual import is needet wpu-spindeln can split and import documents. Adds metadata to json