--- name: "Catmandu::MARC::Tutorial" section: "3pm" platform: "linux" package: "libcatmandu-marc-perl" version: "1.320-1" release: "Debian 13 (Trixie)" platforms: ["linux"] url: "https://new.carta.tech/Catmandu::MARC::Tutorial" --- # Catmandu::MARC::Tutorial(3pm) A documentation-only module for new users of Catmandu::MARC ## NAME Catmandu::MARC::Tutorial - A documentation-only module for new users of Catmandu::MARC ## SYNOPSIS ``` perldoc Catmandu::MARC::Tutorial ``` ## UTF-8 ### MARC8 and UTF-8 The current Catmandu MARC tools are targetted for processing UTF-8 encoded files. When you have MARC8 encoded data tools like MarcEdit \ or `"yaz-marcdump"` \ can be used to create a UTF-8 encoded file: ``` $ yaz-marcdump -f MARC-8 -t UTF-8 -o marc -l 9=97 marc21.raw > marc21.utf8.raw ``` ### Unicode errors If you process UTF-8 encoded files which contain faulty characters, you will get a fatal error message like: ``` utf8 "\xD8" does not map to Unicode at ... ``` Use the iconv (libc6-dev Linux package) tool, to preprocess the data and discard faulty characters: ``` $ iconv -c -f UTF-8 -t UTF-8 marc21.utf8.raw | catmandu convert MARC --type RAW to JSON ``` ### Convert a decomposed UTF-8 file to a combined UTF-8 file and vice versa For example, the character ä can be represented as "ä", that is the codepoint U+00E4 (two bytes c3 a4 in UTF-8 encoding), or as "ä", that is the two codepoints U+0061 U+0308 (three bytes 61 cc 88 in UTF-8). The uconv tool (from the libicu-dev Linux package) can be used to convert these types of files: ``` $ uconv -x any-nfc < decomposed.txt > combined.txt $ uconv -x any-nfd < combined.txt > decomposed.txt ``` ## READING ### Convert MARC21 records into JSON The command below converts file data.mrc into JSON: ``` $ catmandu convert MARC to JSON < data.mrc ``` ### Convert MARC21 records into MARC-XML ``` $ catmandu convert MARC to MARC --type XML < data.mrc ``` ### Convert UNIMARC records into JSON, XML, ... To read UNIMARC records use the RAW parser to get the correct character encoding. ``` $ catmandu convert MARC --type RAW to JSON < data.mrc $ catmandu convert MARC --type RAW to MARC --type XML < data.mrc ``` ### Create a CSV file containing all the titles To extract data from a MARC record on needs a Fix routine. This is a small language to manipulate data. In the example below we extract all 245 fields from MARC: ``` $ catmandu convert MARC to CSV --fix 'marc_map(245,title); retain(title)' < data.mrc ``` The Fix `"marc_map"` puts the MARC 245 field in the `"title"` field. The Fix `"retain"` makes sure only the title field ends up in the CSV file. ### Create a CSV file containing only the 245$a and 245$c subfields The `"marc_map"` Fix can get one or more subfields to extract from MARC: ``` $ catmandu convert MARC to CSV --fix 'marc_map(245ac,title); retain(title)' < data.mrc ``` ### Create a CSV file which contains a repeated field In the example below the 650a field can be repeated in some MARC records. We will join all the repetitions in a comma delimited list for each record. ``` $ catmandu convert MARC to CSV --fix 'marc_map(650a,subject,join:","); retain(subject)' < data.mrc ``` ### Create a list of all ISBN numbers in the data In the previous example we saw how all subjects can be printed using a few Fix commands. When a subject is repeated in a record, it will be written on one line joined by a comma: ``` subject1 subject2,subject3 subject4 ``` In this example, record 1 contained 'subject1', record 2 'subject2' and 'subject3' and record 3 'subject4'. What should we use when we want a list of all values in a single long list? In the example below we'll print all ISBN numbers in a batch of MARC records in one long list using the Text exporter: ``` $ catmandu convert MARC to Text --field_sep "\n" --fix 'marc_map(020a,isbn.$append); retain(isbn)' < data.mrc ``` The first new thing is `$append` in the marc\_map. This will create in `"isbn"` a list of all ISBN numbers found in the `"020a"` field. The `"Text"` exporter with the `"field_sep"` option will use all list values in the `"isbn"` field and writ them using new line as separator. ### Create a list of all unique ISBN numbers in the data Given the result of the previous command, it is now easy to create a unique list of ISBN numbers with the UNIX `"uniq"` command: ``` $ catmandu convert MARC to Text --field_sep "\n" --fix 'marc_map(020a,isbn.$append); retain(isbn)' < data.mrc | sort | uniq ``` ### Create a list of the number of subjects per record We will create a list of subjects (650a) and count the number of items in this list for each record. The CSV file will contain the `"_id"` (record identifier) and `"subject"` the number of 650a fields. Writing all Fixes on the command line can become tedious. In Catmandu it is possible to create a Fix script that contains all the Fix commands. Open a text editor and create the `"myfix.fix"` file with content: ``` marc_map(650a,subject.$append) count(subject) retain(_id, subject) ``` And execute the command: ``` $ catmandu convert MARC to CSV --fix myfix.fix < data.mrc ``` ### Create a list of all ISBN numbers for records with type 920a == book In the example we need an extra condition for match the content of the 920a field against the string `"book"`. Open a text editor and create the `"myfix.fix"` file with content: ``` marc_map(020a,isbn.$append) marc_map(920a,type) select all_match(type,"book") # select only the books select exists(isbn) # select only the records with ISBN numbers retain(isbn) # only keep this field ``` Text after the `"#"` sign are inline code comments. And run the command: ``` $ catmandu convert MARC to Text --field_sep "\n" --fix myfix.fix < data.mrc ``` ### Show which MARC records don't contain a 900a field matching some list of values First we need to create a list of keys that need to be matched against our MARC records. In the example below we create a CSV file with a `"key"` , `"value"` header and all the keys that are OK: ``` $ cat mylist.txt key,value book,OK article,OK journal,OK ``` Next we create a Fix script that maps the MARC 900a field to a field called `"type"`. This `"type"` field we lookup in the `"mylist.txt"` file. If a match is found, then the `"type"` field will contain the value in the list (OK). When no match is found then the `"type"` will contain the original value. We reject all records that have OK as `"type"` and keep only the ones that weren't matched in the file. Open a text editor and create the `"myfix.fix"` file with content: ``` marc_map(900a,type) lookup(type,'/tmp/mylist.txt') reject all_match(type,OK) retain(_id,type) ``` And now run the command: ``` $ catmandu convert MARC to CSV --fix myfix.fix < data.mrc ``` ## Create a CSV file of all ISSN numbers found at any MARC field To process this information we need to create a Fix script like the one below (line numbers are added here to explain the working of this script but should not be included in the script): ``` 01: marc_map('***',text.$append) 02: 03: filter(text,'(\b\d{4}-?\d{3}[\dxX]\b)') 04: replace_all(text.*,'.*(\b\d{4}-?\d{3}[\dxX]\b).*',$1) 05: 06: do list(path:text) 07: unless is_valid_issn(.) 08: reject() 09: end 10: end 11: 12: vacuum() 13: 14: select exists(text) 15: 16: join_field(text,' ; ') 17: 18: retain(_id,text) ``` On line 01 all the text in the MARC record is mapped into a `"text"` array. On line 03 we filter out this array all the lines that contain an ISSN string using a regular expression. On line 04 the `"replace_all"` is used to delete everything in the `"text"` array that isn't an ISSN number. On line 06-10 we go over every ISSN string and check if it has a valid checksum and erase it when not. On line 12 we use the `"vacuum"` function to remove any remaining empty fields On line 14 we select only the records that contain a valid ISSN number On line 16 the ISSN get joined by a semicolon ';' into a long string On line 18 we keep only the record id and the ISSNs in for the report. Run this Fix script (without the line number) using this command ``` $ catmandu convert MARC to CSV --fix myfix.fix < data.mrc ``` ### Create a MARC validator For this example we need a Fix script that contains validation rules we need to check. For instance, we require to have a 245 field and at least a 008 control field with a date filled in. This can be coded as in: ``` # Check if a 245 field is present unless marc_has('245') log("no 245 field",level:ERROR) end # Check if there is more than one 245 field if marc_has_many('245') log("more than one 245 field?",level:ERROR) end # Check if in 008 position 7 to 10 contains a 4 digit number ('\d' means digit) unless marc_match('008/07-10','\d{4}') log("no 4-digit year in 008 position 7 -> 10",level:ERROR) end ``` Put this Fix script in a file `"myfix.fix"` and execute the Catmandu command with the "-D" option for logging and the Null exporter to discard the normal output ``` $ catmandu -D convert MARC to Null --fix myfix.fix < data.mrc ``` ## TRANSFORMING ### Add a new MARC field In the example bellow we add new 856 field to the record with a `$u` subfield containing the Google homepage: ``` marc_add(856,u,"http://www.google.com") ``` A control field can be added by using the '\_' subfield ``` marc_add(009,_,0123456789) ``` Maybe you want to copy the data from one subfield to another. Use the marc\_map to store the data first in a temporary field and add it later to the new field: ``` # copy a subfield marc_map(001,tmp) # maybe process the data a bit append(tmp,"-mytest") # add the contents of the tmp field to the new 009 field marc_add(009,_,$.tmp) ``` ### Set a MARC subfield Set the `$h` subfield to a new value (or create it when it doesn't exist yet): ``` marc_set(100h, test123) ``` Only set the 100 field if the first indicator is 3 ``` marc_set(100[3]h, test123) ``` ### Remove a MARC (sub)field Remove all fields 500 , 501 , 5\*\* : ``` marc_remove(5**) ``` Remove all 245h fields: ``` marc_remove(245h) ``` ### Append text to a MARC field Append a period to the 500 field is there isn't already there: ``` do marc_each() unless marc_match(500, "\.$") # Only if the current field 500 doesn't end with a period marc_append(500,".") # Add to the current 500 field a period end end ``` Use the Catmandu::Fix::Bind::marc\_each Bind to loop over all MARC fields. In the context of the `"do -- end"` only one MARC field at a time is visible for the `"marc_*"` fixes. ### The marc\_each binder All `"marc_*"` fixes will operate on all MARC fields matching a MARC path. For example, ``` marc_remove(856) ``` will remove all 856 MARC fields. In some cases you may want to change only some of the fields in a record. You could write: ``` if marc_match(856u,"google") marc_remove(856) end ``` in the hope it would remove the 856 fields that contain the text "google" in the `$u` subfield. Alas, this is not what will happen. The `"if"` condition will match when the record contains one or more 856u fields containing "google". The `"marc_remove"` Fix will delete **all** 856 fields. To correctly remove only the 856 fields in the context of the `"if"` statement the `"marc_each"` binder is required: ``` do marc_each() if marc_match(856u,"google") marc_remove(856) end end ``` The `"marc_each"` will loop over all MARC fields one at a time. The if statement will only match when the current MARC field is 856 and the `$u` field contains "google". The `marc_remove(856)` will only delete the current 856 field. In `"marc_each"` binder, it seems for all Fixes as if there is only one field at a time visible in the record. This Fix will not work: ``` do marc_each() if marc_match(856u,"google") marc_remove(900) # <-- there is only a 856 field in the current context end end ``` ### marc\_copy, marc\_cut and marc\_paste The Catmandu::Fix::marc\_copy, Catmandu::Fix::marc\_cut, Catmandu::Fix::marc\_paste Fixes are needed when complicated edits are needed in MARC record. The `"marc_copy"` fill copy parts of a MARC record matching a MARC\_PATH to a temporary variable. This tempoarary variable will contain an ARRAY of HASHes containing the content of the MARC field. For instance, ``` marc_copy(650, tmp) ``` The `"tmp"` will contain something like: ``` tmp:[ { "subfields" : [ { "a" : "Perl (Computer program language)" } ], "ind1" : " ", "ind2" : "0", "tag" : "650" }, { "ind1" : " ", "subfields" : [ { "a" : "Web servers." } ], "tag" : "650", "ind2" : "0" } ] ``` This structure can be edited with all the Catmandu fixes. For instance you can set the first indicator to '1': ``` set_field(tmp.*.ind1 , 1) ``` The JSON path `"tmp.*.ind1"` will match all the first indicators. The JSON path `"tmp.*.tag"` will match all the MARC tags. The JSON path `"tmp.*.subfields.*.a"` will match all the `$a` subfields. For instance, to change all 'Perl' into 'Python' in the `$a` subfield use this Fix: ``` replace_all(tmp.*.subfields.*.a,"Perl","Python") ``` When the fields need to be places back into the record the `"marc_paste"` command can be used: ``` marc_paste(subjects) ``` This will add all 650 fields in the `"tmp"` temporary variable at the **end** of the record. You can change the MARC fields in place using the `"march_each"` binder: ``` do marc_each() # Select only the 650 fields if marc_has(650) # Create a working copy marc_copy(650,tmp) # Change some fields set_field(tmp.*.ind1 , 1) # Paste the result back marc_paste(tmp) end end ``` The `"marc_cut"` Fix works like `"marc_copy"` but will delete the matching MARC field from the record. ### Rename MARC subfields In the example below we rename each `$1` subfield in the MARC record to `$0` using the Catmandu::Fix::marc\_cut, Catmandu::Fix::marc\_paste and Catmandu::Fix::rename fixes: ``` # For each marc field... do marc_each() # Cut the field into tmp.. marc_cut(***,tmp) # Rename every 1 subfield to 0 rename(tmp.*.subfields.*,1,0) # And paste it back marc_paste(tmp) end ``` The `"marc_each"` bind will loop over all the MARC fields. With `"marc_cut"` we store any field (`"***"` matches every field) into a `"tmp"` field. The `"marc_cut"` creates an array structure in `"tmp"` which is easy to process using the Fix language. Using the `"rename"` function we search for all the subfields, and replace the field matching the regular expression `1` with `0`. At the end, we paste back the `"tmp"` field into the record. ### Setting and remove MARC indicators In the example below we set every indicator1 of the 500 field to the value "0". We will use the Catmandu::Fix::Bind::marc\_each bind with a loop variable: ``` # For each marc field... do marc_each(var:this) # If the marc field is a 500 field if marc_has(500) # Set the indicator1 to value "0" set_field(this.ind1,0) # Store the result back into the MARC record marc_remove(500) marc_paste(this) end end ``` Using the same method indicators can also be deleted by setting their value to a space " ". ### Adding a new MARC subfield In the example below we append a new MARC subfield `$z` to the 500 field with value test. We will use the Catmandu::Fix::Bind::marc\_each bind with a loop variable: ``` # For each marc field... do marc_each(var:this) # If the marc field is a 500 field if marc_has(500) # add a new subfield z add_field(this.subfields.$append.z,Test) # Store the result back into the MARC record marc_remove(500) marc_paste(this) end end ``` ### Remove all non-numeric fields from the MARC record ``` # For each marc field... do marc_each(var:this) # If we have a non-numeric fields unless all_match(this.tag,"\d{3}") # Remove this tag marc_remove(***) end end ``` ## WRITING ### Convert a MARC record into a MARC record (do nothing) ``` $ catmandu convert MARC to MARC < data.mrc > output.mrc ``` ### Add a 920a field with value 'checked' to all records ``` $ catmandu convert MARC to MARC --fix 'marc_add("900",a,"checked")' < data.mrc > output.mrc ``` ### Delete the 024 fields from all MARC records ``` $ catmandu convert MARC to MARC --fix 'marc_remove("024")' < data.mrc > output.mrc ``` ### Set the 650p field to 'test' for all records ``` $ catmandu convert MARC to MARC --fix 'marc_add("650p","test")' < data.mrc > output.mrc ``` ### Select only the records with 900a == book ``` $ catmandu convert MARC to MARC --fix 'marc_map(900a,type); select all_match(type,book)' < data.mrc > output.mrc ``` The `"all_match"` also allows a regular expressions: ``` $ catmandu convert MARC to MARC --fix 'marc_map(900a,type); select all_match(type,"[Bb]ook")' < data.mrc > output.mrc ``` ### Select only the records with 900a values in a given CSV file Create a CSV file with name,value pairs (need two columns): ``` $ cat values.csv name,values book,1 journal,1 movie,1 $ catmandu convert MARC to MARC --fix myfixes.txt < data.mrc > output.mrc with myfixes.txt like: do marc_each() marc_map(900a,test) lookup(test,values.csv,default:0) select all_match(test,1) remove_field(test) end ``` We use a "do **marc\_each()** ... end" loop because 900a fields can be repeated. If a MARC tag isn't repeatable this loop not isn't needed. With marc\_map we copy first the value of a marc subfield to a 'test' field. This test we lookup against the CSV file. Then, we select only the records that are found in the CSV file (and return the correct value). ## DEDUPLICATION ### Check for duplicate ISBN numbers in a MARC file In this example we extract from a MARC file all the ISBN numbers from the 020 and do a little bit of data cleaning using the Catmandu::Identifier project. To install this package, we run this command: ``` $ cpanm Catmandu::Identifier ``` To extract all the ISBN numbers we use this Fix script 'dedup.fix': ``` marc_map(020a, identifier.$append) replace_all(identifier.*,"\s+.*","") do list(path:identifier) isbn13(.) end do hashmap(exporter:YAML) copy_field(identifier,key) copy_field(_id,value) end ``` The first `"marc_map"` fix maps every 020 field to an identifier array. The `"replace_all"` cleans the data a bit and deletes some unwanted text. The `"do list"` will transform all the ISBN numbers to ISBN13. The `"do hashmap"` will create an internal mapping table of identifier,\_id key value pairs. For very identifier, one or more \_id can be stored. At the end of all MARC processing this mapping table is dumped from memory as a YAML document. Run this fix as: ``` $ catmandu convert MARC to Null --fix dedup.fix < marc.mrc > output.yml ``` The output YAML file will contain the ISBN to document ID mapping. We only need the ISBN numbers with more than one hit. We need a little bit of cleanup on this YAML file to reach our final result. Use the following 'cleanup.fix' script: ``` select exists(value.1) join_field(value,",") ``` This first `"select"` fix selects only the records with more than one hit. The `"join_field"` will turn the array of results into a string. Execute this Fix like: ``` $ catmandu convert YAML to TSV --fix cleanup.fix < output.yml > result.csv ``` This will provide a tab delimited file of double isbn numbers in the MARC input file.