kurtmckee, posts by tag: bash - LiveJournal

I just love when the functionality I need is already baked into the software I use! Today I needed to create a diff file of some changes I'd made in something like 24 files. The problem was that the files were contained in subdirectories of two directories, and I needed to exclude two types of files that were new and I didn't want to include in the diff file. My first attempt was simplistic:

$ ls feedparser/tests/*/itunes/*explicit*

This got me a listing like:

feedparser/tests/illformed/itunes/itunes_channel_explicit_clean.xml
feedparser/tests/illformed/itunes/itunes_channel_explicit_false.xml
feedparser/tests/illformed/itunes/itunes_channel_explicit_no.xml
feedparser/tests/wellformed/itunes/itunes_channel_explicit_clean.xml
feedparser/tests/wellformed/itunes/itunes_channel_explicit_false.xml
feedparser/tests/wellformed/itunes/itunes_channel_explicit_no.xml

You can see that there are illformed and wellformed versions of these files. The problem was that I needed a way to exclude the 'clean' versions, as these were new files that had no business being in the diff file I was creating! A quick search for "exclude bash glob" got me to Stack Overflow, where someone had already asked this exact question. The solution is the extglob option. With it, far richer glob patterns are available:

$ ls feedparser/tests/*/itunes/*explicit!(_clean*)

This got me the listing I wanted:

feedparser/tests/illformed/itunes/itunes_channel_explicit_false.xml
feedparser/tests/illformed/itunes/itunes_channel_explicit_no.xml
feedparser/tests/wellformed/itunes/itunes_channel_explicit_false.xml
feedparser/tests/wellformed/itunes/itunes_channel_explicit_no.xml

extglob was already turned on in my shell, but if it's not turned on for you just use the shopt command:

$ shopt -s extglob

A long time ago I read a blog entry about a shell called zsh that supported command line completion like I had never seen before. Don't get me wrong, I've been using filename completion for years, but I had never seen a shell perform filename completion across the network over an SSH connection before, or complete a manpage name, or complete a long option to a program (like mplayer's obscure -noflip-hebrew-commas option). The powers of zsh impressed me, but I never got around to switching to it. Years later, now, I've discovered that my current shell, bash, actually does support programmable command line completion, and I've become so impressed that I wanted to share some of the things that I've found.

Directory-only completion

First, you need to know what command to use to actually set up programmable completion in bash. It's called complete, surprise, and I'll give you a simple example of how to use it. Let's say that you've just downloaded that totally rockin' piece of software in a zipfile. It's named "baloo-kit-latest.zip", and when it's extracted it goes into a single directory named "baloo-kit-5.0". Now, bash isn't too bright, so when you try to complete the directory name, bash unhelpfully fails to match only the directory:

$ cd b <tab>
$ cd baloo-kit- <tab> <tab>
baloo-kit-5.0/        baloo-kit-latest.zip

Thanks for nothing, bash! But just wait a second, it gets awesome:

$ complete -d cd
$ cd b <tab>
$ cd baloo-kit-5.0/

complete's -d option restricts filename completion to directories.

Filtering filenames

Suppose you're dealing with a command that deals with documents with certain extensions, such as unzip. complete has a switch, -X, with which you can specify filtering. The -X switch will exclude whatever you specify, but you can invert the matching (so that instead of excluding you're including) by prefacing the pattern with "!".

$ complete -f -X '!*.zip' unzip
$ unzip <tab>
$ unzip baloo-kit-latest.zip

The -f switch specifies filename completion, and -X excludes everything from the results that doesn't match the pattern specified. Be careful to quote everything properly!

Matching usernames

complete has many built-in completion possibility sources, all of which can be specified using the -A switch. The -f and -d switches are actually convenient shorthand equivalents for -A file and -A directory, respectively. As another simple example, I know that the usermod command deals not with files but users, which is something bash can provide completion for:

$ complete -A user usermod
$ usermod <tab>
alpha     bravo     charlie   root

Limitless possibilities

While bash provides a lot of built-in possibility sources, it can't possibly cover everything, which is why complete also allows you to run a command (the -C switch) or a shell function (the -F switch). With these, it's possible to complete hostnames from the SSH known_hosts file, to complete remote pathnames over an SSH connection, to complete git branch and tag names, or anything else. You can find a bunch of smart completions already written over at the Bash-Completion project, and git has a thorough set of completions included in its distribution.

Smart command line completion can make you faster and more effective, so get after it!

While reading a book about my Linux shell, bash, I've been working to incorporate the incredibly powerful tools it offers into my normal workflow. In addition to memorizing useful hotkeys (like ^r to search backward through the command history), I'm starting to use history expansion, and I've more thoroughly customized my initialization file. As an example, I've set the bash command history to store 10,000 entries (and to always append to, rather than overwrite, the history file). I'm already looking through my command history and identifying areas where I'm wasting typing energy.

Here's to knowing how to use your tools as efficiently as possible!

I have finally finished backing up all of my data. It took four DVDs, but I finally did it. It would have required several more had I not filtered my music files for backup, however. In particular, the only music that I truly had to save were the songs I've purchased on iTunes and eMusic...but I've got many, many ripped CDs as well, so how can I separate my .m4a's and .mp3's (the proprietary formats that my purchases are in) from my .ogg's and .flac's (the open formats I chose to rip my CDs to)?

Simple! I'll just whip out a shell script that takes all of the files that aren't ripped in open formats, and *BZZT* WRONG. You see, I usually write for loops at the command line. They're fast and easy, but when looping through something that has spaces in it (such as my directories and filenames), there are going to be errors. Lots of errors. Given a directory with two files, named "a 1.mp3" and "b 2.mp3", the following happens:

$ for i in $(ls); do echo $i; done
a
1.mp3
b
2.mp3

What I really need is a way to get the following output:

$ <some awesome command>
a 1.mp3
b 2.mp3

A long time ago I consulted with Allan, and we lit upon the solution: "while read i". As long as data is being piped in, "read" will take in another line, "while" will evaluate true, and our loop will continue churning. Unfortunately, I always forget about "read", which is what prompted this entry. So the "some awesome command" above is actually just:

$ ls | while read i; do echo "$i"; done

In truth, though, the full command was similar to the following (but without all the formatting):

$ find /opt/Music/ | grep -ve "\.ogg$" | while read i; do
    if [ ! -d "$i" ]; then
        echo "$i"
    fi
done

In plain English, that says "List everything in the Music directory, strip out all of the Ogg files and, if it's not a directory, print it out".

And now may I never forget "read" again.