LinuxCommand.org: Tips, News And Rants

We looked at the source builtin a little in TLCL when we were examining how the shell's environment is established with the .profile and .bashrc files. In this adventure, we will delve further into how to use this feature to support configuration files and shareable function libraries for our bash scripts.

http://linuxcommand.org/lc3_adv_source.php

A few months ago I wrote an adventure but I forgot to announce it. In it, we advance our skill with the vim text editor from the beginner level to an intermediate level. This adventure is one of my favorites so far. I learned a lot while writing it.

Vim is a very capable and configurable program. For example, it's easy to configure vim to behave differently according to the type of file it is editing and in this adventure we will configure vim to be optimized for writing shell scripts and plain text documentation files making vim a useful partner when working on the command line. We also introduce a number of powerful editing tricks that make using vim a lot more fun.


You can find the Vim, with Vigor adventure here.

Two new Adventures! First, we'll look at some of the other shell programs available to Linux users. Most are of historical interest, but one attempts to do bash one better. Learn more about Other Shells.

Second, we'll explore some of the often overlooked features of our most frequently used command line tool-- our terminal emulator. Explore Power Terminals.


Enjoy!

Another new Adventure! The AWK programming language is one of the truly classic Unix tools still in wide use today. Often embedded in shell scripts or employed directly at the command line, this powerful and elegant text processing and pattern matching language is a must-have for every Linux user's toolbox. In this adventure, we'll try it out.

Another new Adventure! dialog is a program that, as the name might suggest, creates dialog boxes in text mode. We can use it to give our scripts a friendly face. In this adventure, we will look at what it does, and how to use it.

Another new Adventure! Fingers getting tired? Making more mistakes than you should? You should learn to do more with Less Typing!

I have just posted another Adventure! This one explores terminal multiplexers; programs that allow your terminal to perform clever tricks. Enjoy!

I have just posted the first unit of a new series on LinuxCommand.org called Adventures. These are tutorials that supplement my book, The Linux Command Line.

The first tutorial in the Adventures series is Midnight Commander. Midnight Commander is a text-based directory browser and file manager. A very powerful and useful program.


Look for more Adventures in the coming weeks. Enjoy!

I was going to write the next installment in my New Features In Bash Version 4.x series today, but in thinking about the examples I want to use, I thought I should talk about the stat command first.

We're all familiar with ls.  It's the first command that most people learn.  Using ls you can get a lot of information about a file:

bshotts@twin7:~$ ls -l .bashrc
-rw-r--r-- 1 bshotts bshotts 3800 2010-03-25 13:18 .bashrc

Very handy.  But there is one problem with ls; it's output is not very script friendly.  Commands like cut cannot easily separate the fields (though awk can, but we're not talking about that yet).  Wouldn't it be great if there was a command that let you get file information in a more flexible way?

Fortunately there is such a command.  It's called stat.  The name "stat" derives from the word status.  The stat command shows the status of a file or file system.  In it's basic form, it works like this:

bshotts@twin7:~$ stat .bashrc
  File: `.bashrc'
  Size: 3800          Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d    Inode: 524890      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/ bshotts)   Gid: ( 1000/ bshotts)
Access: 2010-04-15 08:46:22.292601436 -0400
Modify: 2010-03-25 13:18:09.621972000 -0400
Change: 2010-03-27 08:41:31.024116233 -0400

As we can see, when given the name of a file (more than one may be specified), stat displays everything the system knows about the file short of examining its contents.  We see the file name, its size including the number of blocks it's using and the size of the blocks used on the device.  The attribute information includes the owner and group IDs, and the permission attributes in both symbolic and octal format.  Finally we see the access (when the file was last read), modify (when the file was last written), and change (when the file attributes were last changed) times for the file.

Using the -f option, we can examine file systems as well:

bshotts@twin7:~$ stat -f /
  File: "/"
    ID: 9e38fe0b56e0096d Namelen: 255     Type: ext2/ext3
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 18429754   Free: 10441154   Available: 9504962
Inodes: Total: 4685824    Free: 4401092

Clearly stat delivers the goods when it comes to file information, but what about that output format?  I can't think of anything worse to deal with from a script writer's point-of-view (actually I can, but let's not go there!).

Here's where the beauty of stat starts to shine through.  The output is completely customizable.  stat supports printf-like format specifiers.  Here is an example extracting just the name, size, and octal file permissions:

bshotts@twin7:~$ stat -c "%n  %s  %a" .bashrc
.bashrc  3800  644

The -c option provides basic formatting capabilities, while the --printf option can do even more by interpreting backslash escape sequences:

bshotts@twin7:~$ stat --printf="%n\t%s\t%a\n" .bashrc
.bashrc    3800    644

Using this format, we can produce tab-delimited output, perfect for processing by the cut command.  Each of the fields in the stat output is available for formatting.  See the stat man page for the complete list.

Further Reading

• The stat man page

The Linux Command Line:

• Chapter 10 (file attributes and permissions)
• Chapter 21 (cut command)
• Chapter 22 (printf command)

A Wikipedia article on the stat() Unix system call from which the stat command is derived:

• http://en.wikipedia.org/wiki/Stat_(Unix)

One of the features I have discovered during my testing of the Ubuntu10.04 beta is a nifty program called byobu, a slick wrapper for the screen terminal multiplexing program.  If you are unfamiliar with screen, please see the final installment of my Building An All-Text Linux Workstation series.

byobu was introduced in Ubuntu 09.10 and continues in 10.04.  The name "byobu" comes from a Japanese word meaning "folding screen."  Essentially, byobu adds an improved user interface to screen and takes advantage some advanced features in recent screen versions.  On a system so equipped, you launch byobu like this:

bshotts@twin7:~$ byobu

and you get a display like this:



We get a screen terminal session with system status notifications at the bottom of the terminal display.

byobu allows the use of function keys in addition to the usual Ctrl-a sequences supported by screen.  As indicated at the bottom right corner of the terminal, the F9 key invokes a menu:



Function Key Assignments

byobu assigns function keys to perform many of the common operations performed by screen, such as opening new terminal sessions, moving from session to session, and entering scroll back mode.  Selecting the "Help" item on the menu displays a map of the program's function key assignments:



Status Notifications

Also of interest is the list of available system indicators which is visible when the "Toggle status notifications" item is selected on the menu:


As you can see this is a pretty big list.

Other Features

It's also possible to create default windows, that is, terminal sessions that appear automatically when you start byobu.  This is handy if you have several applications that you want to start when you log in at the terminal.

Further Reading

• The screen and byobu man pages

byobu's home at Launchpad

• https://launchpad.net/byobu

A Wikipedia article on Japanese folding screens

• http://en.wikipedia.org/wiki/By%C5%8Dbu

Now that we have covered launching a script from the GUI, it's time to look at how to make our scripts a little more "graphical."  Interestingly, there are a number of tools that we can use to produce user interfaces for shell scripts.  In this installment, we will concentrate on the most mature of these tools, dialog.

The dialog program lets scripts display dialog boxes for user interaction.  It does this in a text-based manner, using the ncurses library.  It can produce a variety of user interface elements and can pass user input back to a script for processing.  The dialog package is not installed by default on most Linux distributions, but is readily available in most distribution repositories.

dialog has been around for a long time and has gone through a number of incarnations.  The current version (which is maintained by the Debian project) contains a number of features not found in earlier versions.

To demonstrate a few of the powers of dialog, we will write a script that displays a series of dialog boxes and provide some ideas of how to work with the output.  We'll start the script with the following code:


#!/bin/bash

# dialog-demo: script to demonstrate the dialog utility

BACKTITLE="Dialog Demonstration"

# message box

dialog  --title "Message Box" \
        --backtitle "$BACKTITLE" \
        --msgbox "This is a message box which simply\
        displays some text and an OK button." \
        9 50

The script invokes the dialog program with a series of options to set the title for the dialog box, the title for the background screen, the dialog box type (in this example a message box), the text to appear in the box and the size of the box (9 lines high by 50 characters wide).  When we run the script we get the following results:



Next we'll add the following code to the end of the script to generate another kind of dialog called a yes/no box.  In addition to the dialog box itself, we will also include some code to act on the results of the user's selection:

# yes/no box

dialog  --title "Yes/No Box" \
        --backtitle "$BACKTITLE" \
        --yesno "This is a yes/no box.  It has two buttons." \
        9 50

# examine the exit status and act on it
case $? in
        0)      dialog --title "Yes" \
                --backtitle "$BACKTITLE" \
                --msgbox "You answered \"Yes\"" \
                9 50
                ;;
        1)      dialog --title "No" \
                --backtitle "$BACKTITLE" \
                --msgbox "You answered \"No\"" \
                9 50
                ;;
        255)    dialog --title "Esc" \
                --backtitle "$BACKTITLE" \
                --msgbox "You pressed Esc" \
                9 50
                ;;
esac

The yes/no box offers the user three choices: yes, no, and a cancel which occurs if the Esc key is pressed.  To select a button, the user my use the Tab key to switch from button to button or, if dialog is being run inside a terminal emulator on a graphical desktop, the mouse may be used to select a button.

dialog communicates the user's choice back to the script via its exit status.  In the code that follows the yes/no box, we evaluate the exit status.  If the yes button is pressed, the exit status is 0, if the no button is pressed, the exit status is 1, and if Esc is pressed, the exit status is 255.




dialog can provide more than just simple button presses.  It can support edit boxes. forms, menus, file selectors. etc.  When returning more complex data, dialog outputs its results on standard error.  To demonstrate how this works, we will insert the following code near the beginning of the script (just after the BACKTITLE=... line) to define a function (read_tempfile) that will display the dialog output:


TEMPFILE=/tmp/dialog-demo.$$

read_tempfile() {

# display what is returned in the temp file

        local tempfile

        # read file contents then delete
        tempfile=$(cat $TEMPFILE)
        rm $TEMPFILE

        # message box to display contents
        dialog --title "Tempfile Contents" \
                --backtitle "$BACKTITLE" \
                --msgbox "The temp file contains: $tempfile" \
                0 0
}

Next, we will add the following code to the end of our script to display a menu box:

# menu

dialog --title "Menu" \
        --backtitle "$BACKTITLE" \
        --menu "This is a menu.  Please select one of the following:" \
        15 50 10 \
        1 "Item number 1" \
        2 "Item number 2" \
        3 "Item number 3" 2> $TEMPFILE

read_tempfile

This code displays a menu containing three items.  Each item consists of two elements; a "tag" which is returned when a choice is made and an "item string" which describes the menu selection.  In the example code above, we see that the first menu item has the tag "1" and the item string "Item number 1."  When we execute the code, dialog displays the following menu:



To capture and display the output of the menu, we redirect the standard error of dialog to the file named in the constant TEMPFILE.  The read_tempfile function assigns the contents of the temporary file to a variable (tempfile) and passes it to dialog to display in a message box.  After the menu is displayed and a selection is made by the user, the message box appears and displays the data returned to the script.

Next, we will add a checklist dialog by adding the following code to the end of the script.

# checklist

dialog --title "Checklist" \
        --backtitle "$BACKTITLE" \
        --checklist "This is a checklist.  Please from the following:" \
        15 50 10 \
        1 "Item number 1" off \
        2 "Item number 2" off \
        3 "Item number 3" off 2> $TEMPFILE

read_tempfile

As we can see, this code is almost the same as the code used to create the menu dialog.  The main difference is that the items in the checklist add a third field called "status."  The value of status may be either "on" or "off" and will determine if a selection is already selected or deselected when the program is run:




The user may select one or more checkbox items by pressing the space bar.

Next, we will add the following code to the end of the script to display a file selector:

# file selector

dialog  --title "File Selector" \
        --backtitle "$BACKTITLE" \
        --fselect ~/ 10 30 2> $TEMPFILE

read_tempfile

For the file selector, dialog is passed the base directory for the selection (in this example the directory ~/) and the size of selector.  For this example we specify 10 files in the list and a 30 character wide dialog.




I think the file selector is easy to code, but awkward to use.  You move around the dialog with the Tab key and select directories and files with the space bar.

One of the more novel types of boxes provided by dialog is the "gauge" which displays a progress bar.  To try out this feature, we will add this code to the end of the script:

# gauge

# generate a stream of integers (percent) and pipe into dialog

percent=0
while (( percent < 101 )); do
        echo $percent
        sleep 1
        percent=$((percent + 10))
done | dialog --title "Gauge" \
        --backtitle "$BACKTITLE" \
        --gauge "This is a gauge.  It shows a progress bar." \
        0 0

The gauge dialog does not output anything, rather it accepts a stream of integers (representing the percent of completion) via standard input.  To provide this, we create a while loop to generate the stream of integers and pipe the results of the loop into dialog.  When the script runs, the stream of numbers causes the progress bar to advance:



Note too, that we specified the size of the dialog as 0 0.  When this is done, dialog attempts to auto-size the box to fit in the minimum necessary space.

Finally, we'll add another of the novel dialogs, a calendar:

# calendar

dialog  --title "Calendar" \
        --backtitle "$BACKTITLE" \
        --calendar "This is a calendar." \
        0 0 \
        0 0 0 2> $TEMPFILE

read_tempfile
The calendar dialog box is given a date in day month year format.  If the year is specified as zero, as it is in this example, the current date is used.  The user can then select a date and the date is returned to the script.




This concludes our brief look at dialog.  As you can imagine, dialog can be very handy for adding  improved user interfaces for those scripts that require it.  While we have covered its general themes, dialog has many more options and features.  Check out the documentation link below for the complete story.

Further Reading

Documentation for dialog:

• http://invisible-island.net/dialog/manpage/dialog.html

Other dialog-like programs:

• http://library.gnome.org/users/zenity/stable/
• http://xdialog.free.fr/doc/index.html

In a recent post, we covered a technique that can produce colored text on the command line.  Today, we will look at a more general approach to producing not only text effects, but also gaining more visual control of our terminal.

A Little History

Back in the old days, when computers were connected to remote terminals, many brands of terminals existed and they were all a little different in terms of their feature sets and capabilities.  As a result, different terminals used different sets of commands to control them.

Terminals responds to codes (called control codes) embedded in the stream of text sent to them.  Some of these codes are standard and familiar like carriage return and line feed.  Others, like those to turn on bold text or underlining are not.  Terminals can, in fact, perform many kinds of functions.  As microprocessors became available and the advent of the personal computer loomed, terminals became increasingly "smart" and feature laden.

However the proliferation of terminal brands and feature sets posed a problem for software developers.  Software had to be painstakingly customized to support a particular terminal.  What was needed was a software system that supported hardware independence so that applications could use a standard set of commands to deal with any terminal.  This problem was addressed in two ways.  First a standard set of control sequences were developed by ANSI (American National Standards Institute) and adopted (in varying degrees) by terminal manufactures to give all terminals a common set of commands.  We looked at the ANSI commands in a an earlier post.  The second approach was development of an intermediary layer (much like today's notion of a device driver) that translates a standardized command into the specific control codes used by a particular terminal.

In the Unix world, there are two such systems, the original, termcap and the more recent terminfo. Both contain a database of control code sequences used by different kinds of terminals.

Enter tput

tput is a command that can query the terminfo database to see if a particular terminal can support a particular feature.  It can also accept terminal commands and output (via standard output) the control code sequences for that terminal.  tput is generally used like this:

tput capname [parameters...]

where capname is the name of a terminal capability and parameters are any option parameters associated with the specified capability.  For example, to output the sequence of instructions needed to move the cursor to the upper left corner of the screen (the "home" position):

tput cup 0 0

which means cursor position row 0, column 0.

Since tput actually outputs the sequence to standard output (you won't see the sequence since it is interpreted by your terminal emulator as an instruction), you can store the sequences in variables.  Here we will store the sequences to control bold text:

bold_on=$(tput bold)
bold_off=$(tput sgr0)

Now, to highlight some text, you could:

echo "This is ${bold_on}important${bold_off}."

and you get this:

This is important.

There are a huge number of terminal capabilities, though most terminals only support a small subset.  Besides changing text colors and positioning the cursor, it is possible to erase text, insert text, and control text attributes.  The terminfo man page lists all the terminal capabilities and the Bash Prompt HOWTO section 6.5 (see "Further Reading" below) describes the ones most useful for ordinary screen control.

Before we leave, here is a version of the prompt_colors script that uses tput to set foreground and background text colors:

#!/bin/bash

# prompt_colors -- demonstrate prompt color combinations.

for fore in {0..7}; do
    set_foreground=$(tput setf $fore)
    for back in {0..7}; do
        set_background=$(tput setb $back)
        echo -n $set_background$set_foreground
        printf ' F:%s B:%s ' $fore $back
    done
    echo $(tput sgr0)
done

Further Reading

• tput man page
  
• terminfo man page
  
• http://en.wikipedia.org/wiki/Computer_terminals
  
• http://en.wikipedia.org/wiki/Terminfo
  
• http://en.wikipedia.org/wiki/Tput
  
• http://tldp.org/HOWTO/Bash-Prompt-HOWTO/x405.html
  
• http://tldp.org/HOWTO/Text-Terminal-HOWTO.html