Music Library Cleanup and Merging

Process

File Name issues

• Find all files with missing extensions
  • find <folder> ! -type d -regex "[^.]+"
    replace "<folder>" with directory to search. It cannot contain or be "." due to the regex used. This will find files with no "." in their name.

• Find all files with MP3 and make them mp3 (confirm options for "rename" they change from OS to OS)
  • find <folder> -name "*.MP3" -exec rename MP3 mp3 "{}" \;

• Find files that are DRM protected and remove them or the protection

#!/bin/bash
if /tmp/mplayer/mplayer.exe -ao dummy -identify "$1" 2>&1 | egrep -i -q -s "encumbered with drm"
then
        echo $1;
fi

• Find files with special or unicode characters and fix

find . -name "*[$(/bin/echo -e '\201')-$(/bin/echo -e '\377')]*" | tee /tmp/filelist.log

• Examples
  • \302\264 becomes apostrophe
  • \342\200\231
  • \241\257
  • \242\245
  • \222

Find Duplicates

• Use Jaikoz to add AcoustID Fingerprint and, if it exists, ID on all files.

Even if you do not purchase Jaikoz, the 20 file per session limit does not apply for adding the AcoustID value.
Having added this value you will be able to search for potential duplicates.

• Create csv/text file with AcoustID ID and full filename to be able to search for duplicates
  • Open Mp3Tag
  • Open your main library folder (File:Change Directory...)
  • For each folder you are merging with your library do File:Add Directory...
    When complete Mp3Tag will have a list of all files in your current library and any new files you are adding
  • Export a CSV file with AcoustID and Full path. (File:Export...)
    The following script should be used for the export.

$filename(csv,utf-8)AcoustID ID	Filename	
$loop(%_filename_ext%)%AcoustID ID%	%_path%
$loopend()

• • The resulting CSV (actually Tab not Comma)
• Using the following command line or similar create a playlist of duplicates

infile=acoust_id.csv ; egrep -f <(cut -f1 "$infile" | egrep -v '^$' | sort | uniq -d) "$infile"  | sed -e 's/.*Media\\Music/M:/' > "${infile}-dupes.m3u"

• Load the m3u playlist into Mp3Tag.
  This may take some time possibly even longer than loading the original folders. Not sure why but just be patient. My very first time was a greater than 8k list that took several minutes.

General Strategy to remove duplicates

• Sort by Artist (not Album Artist) and make sure you have an artist for most rows. Not absolutely necessary but very helpful for filtering.
• Use Filter (F3 or View:Filter) to pare off a group of files. e.g. %artist% MATCHES ZZ Top
• Look for whole album duplicates first by sorting on path name, selecting an album from the "import" folder then sorting by either album name (if it is already set) or by AcoustID which will put the two album copies next to each other.

Tagging

• Save "contextual" meta data like folders
  

Folders may represent compilations or playlists which were created artificially due to a lack of support by the library/player/etc. This contextual information should be saved prior to any file renaming and possibly before any tag corrections.

• Find full and mostly complete albums and verify tags
• Rename these tracks to new folder structure
  • OST/Various Artists
    • X:\_Stage\pass1\V\Various Artists\%album%\$num(%track%,2) - %artist% - %title%
  • Single Artist Albums
    • X:\_Stage\pass1\$left(%artist%,1)\%artist%\%album%\$num(%track%,2) - %title%
• Incomplete Albums/"singles"
  • • X:\_Stage\pass1\$left(%artist%,1)\%artist%\_Singles\%title%

File Format/codec/bitrate Issues

• Convert wma to MP3 with same quality and transfer tags
  • Load all tracks into MP3TAG and sort first by bitrate and then by encoding.
  • Go to the bottom and select all wma with bitrate set (missing bitrate most likely means a damaged file)
  • Open Xrecode
  • Drag from MP3TAG to Xrecode all selected tracks
  • Select mp3 output
  • Select encoding rate
  • (optional) Select tracks in MP3TAG by bitrate and choose encoding rate in Xrecode based on source
• Use MP3Tag to normalize track numbers to 2 digits.
  This will help in fixing tags using Jaikoz in a later step
• Open Folder in Jaikoz
  • sort by track number then subfolder
  • open

Tools

• MP3Tag is for bulk changing of tags and file renaming. It can look up tags online but Picard tends to be better.
• MusicBrainz picard for tagging based on audio fingerprint.
• Replay Gain is a technology for audio level normalization.
• xrecode$$ is a tool for recoding to different formats that supports moving of tags.
• AudioConverter is a free tool for recoding to different formats.
• Lyrics Finder is a free tool for adding Lyrics tags.