Archive for the ‘cli’ Category

I was curious to know how many books, on average, I read a month. I don’t expose this information directly on bookpiles either. You could extract it from the RSS feed. It’s one the features I would like to add when I understand what information I want to present and how it is best presented.

In the meantime, I ran a query in the database and came up with this:

2009-01 3
2009-02 2
2009-03 4
2009-04 3
2009-05 3
2009-06 3
2009-07 2
2009-08 3
2009-09 3
2009-10 3
2009-11 4
2009-12 1
2010-01 0
2010-02 1
2010-03 1
2010-04 6
2010-05 2
2010-06 5
2010-07 7
2010-08 2
2010-09 7
2010-10 4

I felt 80% done. Then, I realized I didn’t quite know how I would extract, from the command-line, the sum, mean, standard deviation, minimum and maximum value. Of course, I could run it through R. Or Excel… The question wasn’t how to do statistics in general — it was how to do it as a filter … easily … right now.

A little research didn’t turn out any obvious answer. (please, correct me if I missed an obvious solution)

I wrote my own in awk. (awk is present on ALL the machines I use)

min == "" {min=max=$1}
$1 < min  {min = $1}
$1 > max  {max = $1}
          {sum+=$1; sumsq+=$1*$1}
  print "lines: ", NR;
  print "min:   ", min;
  print "max:   ", max;
  print "sum:   ", sum;
  print "mean:  ", sum/NR;
  print "stddev:", sqrt(sumsq/NR – (sum/NR)**2)

Here’s what the output looks like:

I included it in my dotfiles: the awk code and a bootstrap shell script (used above).

Read Full Post »

Managing PATH and MANPATH


My PATH variable used to be a mess. I have used UNIX-like systems for 10 years and have carried around my configuration files in one form or another since then.

Think about Solaris, think about /opt (or /sw), and change the order based on different requirements.

I have seen a lot of people devise clever if-then-else logic with OS-detection. I have seen yet other, myself included, who tried to aim for the most comprehensive and all-inclusive PATH.

In the end, all that matters is that when you type a command, it IS in your path


As for MANPATH, the situation was even worse. I used to depend (and hope) that the OS-inherited MANPATH contained everything I needed. For a long time, I didn’t bother to set it right and just googled for the man pages if/when I was in need.

Invoking man for something I just installed often meant getting no help at all.

Where to look?

When it comes to bin and share/man directories, there are a handful of predictable places to look for. For PATH:

  • /usr/X11/bin
  • /bin
  • /sbin
  • /usr/bin
  • /usr/sbin
  • /usr/local/bin
  • /opt/local/bin
  • /opt/local/sbin

Notice the bin and sbin combinations. And for MANPATH:

  • /usr/X11/share/man
  • /usr/share/man
  • /usr/local/share/man
  • /opt/local/share/man

It should be clear that there is a lot of duplication there. Also, if you change the order of your PATH, you should probably change the order of your MANPATH so that the command you get the man page for is the command invoked by your shell. The GNU man pages are not very useful when you are using the BSD commands, on Darwin, for example.

A solution

Here’s the plan:

  1. Clear both PATH and MANPATH.
  2. Given a path, detect the presence of a bin, sbin and share/man subdirectories.
  3. Prepend the existing directories from step 2 to both PATH and MANPATH (as appropriate).

What you get:

  • Only existing paths go in PATH and MANPATH. No more just-in-case™ and for-some-other-OS™ paths polluting your variables.
  • Order of the paths is the same for both PATH and MANPATH. If you change the order in one, the order is changed for the other.
  • Easier to read configuration files. Colon-separated lists are no fun to parse visually.

Here’s something you can put in your .bashrc

# prepend_colon(val, var)
prepend_colon() {
  if [ -z "$2" ]; then
    echo $1
    echo $1:$2

# unshift_path(path)
unshift_path() {
  if [ -d $1/sbin ]; then
    export PATH=$(prepend_colon "$1/sbin" $PATH)
  if [ -d $1/bin ]; then
    export PATH=$(prepend_colon "$1/bin" $PATH)

  if [ -d $1/share/man ]; then
    export MANPATH=$(prepend_colon "$1/share/man" $MANPATH)

export PATH=""
export MANPATH=""

unshift_path "/usr/X11"
unshift_path ""
unshift_path "/usr"
unshift_path "/usr/local"
unshift_path "/opt/local"
unshift_path "$HOME/local"
unshift_path "$HOME/etc"

export PATH=$(prepend_colon ".local" $PATH)


I use $HOME/local to store machine-specific binaries/scripts. For example, that’s where I install homebrew on Mac OS X. That’s also where I would put cron scripts or other “I just use this script on this machine” type of things.

I use $HOME/etc to store binaries I carry around with my configuration files. That’s where I clone my dotfiles project.

Finally, the relative path .local is an interesting hack. It allows for directory-specific binaries. This solves the “I just use this script when I’m in that directory” problem. This trick is discussed in this blog post.

Read Full Post »

I have a bunch of shell/ruby scripts in a directory that I include into my PATH variable. The scripts live there because I don’t have a better location to put them. That’s fine when scripts are general enough to be used anywhere.

Some scripts are not so general and are meant to interact only with a few specific files and directories.

Putting things in perspective: this is a discussion about global variables and local variables when it comes to Bash. You want your variables to be scoped only that was is needed and not more.

Here’s an insight:

Include a .local directory in your PATH.

For example:

forgetful, a project I maintain that implements the Supermemo algorithm, takes CSV files as input. I could use a spreadsheet manipulate these files, but I prefer to use Vim and a bunch of unixy scripts to do what I want. In the directory where I keep my CSV files, I created a subdirectory called .local (could be called anything). When I’m in the directory, Bash will include the .local subdirectory when looking for executables … in essence, I get my “local” executables.

Notice how there’s a Rakefile in that directory? I think that’s a workaround that a lot of people end up using. I’ll probably strip out most of what the Rakefile is doing and shove it to the .local directory.

Read Full Post »

Using vim as a pager

I’ve talked casually about using Vim as a pager before. However, I’m still surprised to see how many people use Vim regularly and don’t know about this feature.

Here’s a quote straight from vim --help

vim [arguments] -               read text from stdin

Admittedly, it’s easy to overlook the hyphen in the explanation.

vim hyphen

Why Vim as a Pager?

If you’re using Vim already, there’s nothing else to install.

If you’re using Vim already, it’s already configured the way you like it.

More importanly, Vim detects the kind of file it is being piped and turns on the appropriate syntax highlighting. Why page in black and white? In this case, “less” is definitely less!

Improving the experience

As a pager, you want to use Vim in read-only mode.

some command | vim -R -

What the difference? Vim doesn’t ask you to save the file if you try to quit. Of course, you can still modify and write the file … the -R flag is just a more reasonable pager default.

PAGER variable and ANSI Escape Sequences

You probably don’t want to set the PAGER variable. Vim doesn’t understand ANSI escape sequences. As such, a command like “man vim | vim -R -” won’t show colors; it will show escape sequences.

vim and ansi

I haven’t found any quick and simple solution to make Vim show ANSI escape sequences, but it’s pretty easy to strip them out before passing the file to Vim:

man vim | col -b | vim -R -

I use less as PAGER. I use vim in explicit cases.


The view command gets installed at the same time as vim. It’s just a symlink to vim. Using view is exactly like typing vim -R.

There’s a certain aesthetic in:

some command | view -

But I find that typing vim -R - is easier on my finger’s muscle memory.

Read Full Post »

My very customized bash prompt

Bash’s $PS1 variable is what you see every time you get a prompt. It’s there, waiting for you to type something.

Some people go minimal … maybe just “$” like the Bourne shell. Others go crazy and cram as much information as possible … on multiple lines … in colors.

My $PS1 falls somewhere in the middle, but I realized today it was time for a change. One of the things I wanted was the current path. A lot of people like to put that in the “xterm title bar”, but I wanted it closer to the action.

After a bit of experimentation, I found that I could play with the PROMPT_COMMAND to faux-position the path in pale gray ON the same line as the prompt.

Here’s my variables.sh file which I put under ~/etc/bash/local/ — it works with the rest of my config files:

function prompt_command() {   

  printf \e[30;40;1m%*s\n\e[0m\e[1A $COLUMNS $PWD




export LANG=”en_US.UTF-8

export LC_ALL=”en_US.UTF-8

export TERM=”xterm-256color

export FIND_OPTIONS=”-name .git -prune -o -name .hg -prune -o

The prompt_command function uses “printf” to print ($COLUMNS wide) the $PWD. There are escapes sequences to color it gray. Finally, the “\e[1A” sequence moves the cursor 1 line up. Consequently, the prompt itself prints on the same line as the PROMPT_COMMAND.

It looks something like:


What do you put in your $PS1?

Read Full Post »

Rotating GNU Screens

I often have many open screen windows to monitor a machine: “top”, “tail -f …”, “thin …” and so on.

Wouldn’t it be great if you could open a terminal and have these screens rotated automatically?

I wrote this simple shell function: (to include in your .bashrc)

function screen_rotate() {
  local session_name=${1:?"missing session name"}
  local sleep_duration=${2:-5}

  while true; do
    screen -S $session_name -X next
    sleep $sleep_duration

The first argument is the screen session name. The second, optional, argument is the time spent on each screen window.

Read Full Post »

A year ago, I posted Like Slime, for Vim. There was a lot of interest in sticking with Vim but having a way to get something similar to Slime.

I tried to explain what the plugin was all about … but I always felt it would be better served by a screencast. Here’s what I came up with:

(Watch it bigger)

In short:

  • you can control GNU screen from the command-line
  • vim can, therefore, control GNU screen
  • you type in vim, type C-c C-c, and it appears in your screen session
  • just run clojure (bash, ruby, scala, …) in screen

Read the original post for more details. To save you time, here’s the Vim plugin: slime.vim.

Read Full Post »

I’ve been playing with autotest for my latest Rails project, and it’s great.

However, if you’re doing a “Ruby without Rails” project and your directory structure is different from the established conventions, you’re out to reverse engineer autotest itself and find outdated recommendations on Google.

It doesn’t have to be this way, here’s my .autotest:

require ‘autotest/redgreen’
require ‘autotest/timestamp’

# quit on first CTRL-c
Autotest.add_hook(:interrupt) do |at|
  at.wants_to_quit = true

# run all tests for ANY file
Autotest.add_hook(:initialize) do |at|
  at.add_mapping(/.*/) do |filename, _|


Read Full Post »

Like Slime, for Vim

UPDATE:The new version of slime.vim. The previous version is still available.

I started reading Practical Common Lisp yesterday. No discussion about Lisp can be complete without talking about Slime. Slime basically turns Emacs into an IDE for Lisp development. Peter Seibel thinks this is important enough to dedicate the second chapter to it. He even repackaged Lisp + Emacs + Slime as Lispbox to help people get started faster. For an excellent book (so far), available for free, and Lispbox, thank you Mr.Seibel.

After actually downloading, installing, and running Lispbox, I can see the advantages. Slime solves the problems you would have with any interactive REPL-type environment.

Let’s take Ruby’s irb as an example more people are going to relate to:

  1. you start irb
  2. you start a text editor (vim, textmate, emacs, …)
  3. you do a few tests in irb
  4. you copy and paste to a text editor
  5. you clean things up in the text editor
  6. you copy and paste back to irb
  7. you make a mistake
  8. you fix things up in the text editor
  9. you hesitate copying and pasting, because it’s painful now
  10. you write some tests
  11. you exit irb and run the tests to do your experiments

Let me present the dilemma another way: irb is great to get answers quick but it is also temporary because you know nothing you do in irb will be saved. However, the moment you start living in a text editor, you give up a lot of the power of REPL. Or, at least, your REPL becomes 10 seconds instead of 1 second. That changes the way you work. And that explains why Slime exists.

Slime creates a new interactive session with Lisp and you are able to copy and paste text from a text buffer to the session with one keyboard shortcut. That’s great! Now you can type, organize, pretty-print your code in a text editor, type C-c C-c to “refresh” the interactive session. This does not close and create a new session, the function definitions are reloaded into the current session. So, all your testing objects, those carefully crafted lists and hashes of objects (or whatnots) still exist—the world just changed around them.

Yeah, Slime is great. I’m just not an Emacs fan.

I did some research and Vim has no mode to support asynchronous sessions like Emacs. In essence, all that’s needed is a software that will spawn what you really want to run, say irb, control stdin/stdout/stderr, capture its own stdin/stdout/stderr and tunnel those to the child process. Also, it would be nice if it could open a port to receive external commands to be able to script stdin…

That’s when I remembered an article I read about scripting gnu-screen. To make a long story short, screen does everything we want, and more.

Here’s what we want to accomplish:

  1. start a named screen
  2. name the screen window
  3. start irb
  4. start another terminal
  5. start vim
  6. define a function/class/object
  7. have it “transported” to irb

Here are the instructions:

  1. screen -S session01
  2. C-a A—window01
  3. irb
  1. vim
  2. (type code)
  3. vip (select paragraph)
  4. “ry (copy to register r)
  5. :call system(“screen -S session01 -p window01 -X stuff ’” . @r . ”’”)

And BLAM, you just did some magic!

At this point, you are coming to 2 realizations:

  • this is WAY cool
  • you want this automated

Thankfully, I can help with the automation. Get slime.vim and put it in ~/.vim/plugin/ .

A few notes:

  • the magic key is C-c C-c (like Slime, surprise!)
  • the first time, you’ll be prompted for the “session name” and the “window name”
  • subsequent times will be automated
  • you can reprompt for “session name” and “window name” with C-c v
  • by default, C-c C-c will select the current paragraph and copy-paste it
  • but you can make your own selection first, and send it over with C-c C-c

As a final note, I’d like to drive the point that this can automate ANYTHING running in a screen:

  • bash
  • top
  • irb
  • python
  • any lisp/scheme REPL
  • mysql

One thing is for sure, this will definitely change the way I work.

(For extra points, write your own Textmate plugin … this hack is not limited to Vim!!)

Read Full Post »

Renaming Thousands of Files

If I told you I had to rename 1,000 files, change the extensions or
change hyphens to underscores:

  • how long would it take you?
  • what tools would you use?
  • what would you do?
  • how much would that answer change for 10,000/100,000/1,000,000 files?

Take a moment to think, please, before you keep reading.

This was a situation I was faced with this week. And it reminded me of Steve Yegge’s phone interview blog post You should read it for yourself, but here’s the problem statement:

Last year my team had to remove all the phone numbers from 50,000 Amazon web page templates, since many of the numbers were no longer in service, and we also wanted to route all customer contacts through a single page.

Let’s say you’re on my team, and we have to identify the pages having probable U.S. phone numbers in them. To simplify the problem slightly, assume we have 50,000 HTML files in a Unix directory tree, under a directory called ”/website”. We have 2 days to get a list of file paths to the editorial staff. You need to give me a list of the .html files in this directory tree that appear to contain phone numbers in the following two formats: (xxx) xxx-xxxx and xxx-xxx-xxxx.

How would you solve this problem? Keep in mind our team is on a short (2-day) timeline.

These are not “never-gonna-happen” situations. Your set of skills should include “entreprise” problem solving and “low-level” scripting.

For the curious, here’s how I solved the renaming problem:

find . -name '*.TXT' > src
cp src dest
vim dest
paste src dest > todo
vim todo
source todo

A good old “find”, some vim regular expression magic, “paste”, and more vim magic (to add “mv” to every line). Another advantage to this technique is that you’ll be able to “preview” the changes before you source the file.

Read Full Post »

Older Posts »