My config files have been on github for quite a while now. However, I named the project etc_config because I store those files in $HOME/etc. I renamed the project to dotfiles. This is both closer to my intent and more in line with what other people are doing.

I changed all the URLs I owned to point to the new project … but I don’t know what else on the github side is broken (watchers, feeds?)

Rake task completion is awesome. A quick google search will hook you up … (here’s one)

Rake completion must invoke rake and that can be SLOW. I use completion to save time … What’s the point on having bash freeze on rake -T for longer than it would take me to type the whole thing?

I took it upon myself to cache the output of rake -T and to use that if it’s available.

The magic is in the .rake_t_cache file. If it’s there, use it, if not … well … we haven’t gained/lost anything, the regular completion behavior is still in place.

Where does .rake_t_cache come from?

I have been thinking about a way to detect when the cache file needs to be regenerated. However, I leave it in the capable hands of the user. All the time saved completing tasks with the cache will compensate for the inevitable time when the cache will be stale and swear words will be required.

I used to work on a Java application that ran 24/7 and logged to a file on the system. The log file was rotated every week and it usually stood around 4GB.

When the shit hit the fan, I checked the log and tried to reverse-engineer how things got so bad. This is similar to what investigators do with a black box after a plane crash. How do you inspect a 4GB file with a text editor? You might be surprised to know how far Vim can take you in that direction. I have opened gigabyte-sized files before, and it worked … for some value of “worked”.

Luckily, the log4j format the application used contained the timestamp in ISO 8601 format. It looked something like YYYY-MM-DD hh:mm:ss. Thankfully, this is trivial to parse and guarantees that alphanumeric sorting (read: plain old sorting) will keep the dates in chronological order.

grep and sed

I’ll cover a simpler example and come back to dates later.

(seq might be called gseq on your system)

This created a 10000-line file with one number, from 1 to 10000, per line. I used this contrived example instead of ISO 8601 formatted dates because it was simple to generate and the relationship between the line number and the line content is obvious.

The next piece of the puzzle is grep. Grep has the -n/--line-number flag to “prefix each line of output with the line number”.

We’re going to extract from the line containing 444 to the line containing 2000. Of course, we know what those line numbers are because of how we generated this file. This is usually not the case.

Right, we want the first match… Part of the solution is to use a tighter regular expression. Also, and the reason I did this, is to realize that the file will keep being parsed after the first match is found. On huge files, waiting for grep to finish is both time-consuming and unnecessary.

The -m NUM/--max-count=NUM flag will “stop reading a file after NUM matching lines.”

Combining the line numbers, we can slice the log with sed:

Discussion

Why not skip grep and just RTF sed manual?

My reason: I want to visually confirm that my regular expressions matched the right lines. The time I would have saved bypassing grep would be wasted the first time I would open a file which didn’t contain what I really wanted.

Why not just grep for timestamp and use that?

That’s a subtle point. The log files contained YYYY-MM-DD hh:mm:ss at the beginning of almost every line.

Initially, I tried:

To get the log lines between 4am and 5am on a specific date.

This was simple to understand and explain, and it worked beautifully until we realized that it was almost every line … it was missing the stack traces. It was also missing, although rare in that application, other multi-line log messages.

So, I used:

and used sed to extract the lines in-between.

I’ve talked about SpiderMonkey before. Being able to instantly evaluate JavaScript code is great but you can use FireBug for that. I argued that the main reason to use SpiderMonkey is to script the command-line. Integrating with JSLint is an example of using your tools intelligently.

JSLint

JSLint is a tool that “looks for problems in JavaScript programs”. Here’s a list of things JSLint looks for (full list):

• missing semicolons
• missing curly braces ({})
• the use of with
• the unfiltered use of for in
• the use of eval
• the implicit use of global variables
• missing break statements
• double var definitions
• the appropriate use of = and == and ===
• unreachable code

JSLint is just a way to check your code. However, JSLint, as it stands, is a textarea on the web:

Every time you edit your code, do you want to go jslint.com, paste only the relevant portion of your code, fix the errors locally, rinse and repeat? Anything that causes friction won’t get done. Let’s minimize that.

With SpiderMonkey

As it turns out, JSLint is a JavaScript program that parses JavaScript (!). Knowing that you can evaluate JavaScript on the command-line with SpiderMonkey, you have all the ingredients you need to automate the process.

Most of the solution is here. Look for this button: (direct link)

I renamed the file to jslint.js and put it in ~/etc/bin. I then created a shell script to script the process:

I named it ~/etc/bin/jslint (chmod +x).

Stupid Example

Let’s take a stupid example: (stupid.js)

What’s wrong with this? Plenty:

After the fixes:

JSLint is just one tool. Let’s run this code through SpiderMonkey’s strict mode:

And this is for a trivial 7-line function. Running JSLint on hundreds of lines of code can be sobering experience. There’s even a warning on the site:

Seriously. As you become better with JavaScript and regularly check your code against JSLint (as a formality, of course), there seems to be no end to how nitpicky JSLint can be.

Finally, you can configure what JSLint will complain about by putting a specially-formatted comment in your file:

The undef option will make JSLint complain about using the global variable y which definitely looks like a mistake.

The full list of options is available here.

Web development means working at a very high level of abstraction. For the magic to work, a multitude of technologies must also work: networks, sockets, HTTP. Like all leaky abstractions, however, we can sidestep a lot of the complexity until things stop working.

Netcat does exactly what its name says: it cats stuff over a network. It can send or receive bytes over the network.

File transfer

client:

server:

This is probably one of the fastest and most casual way to transfer a file between two systems. Nothing (besides nc) needs to be installed, no authentication or encryption is performed.

I would reserve this use of netcat for post-apocalyptic server crashes where you need to transfer files but nothing is installed and zombies are about to come crashing in.

More realistically …

HTTP tricks

Let’s spy on safari:

This will happen after you try to open localhost:9999 in a browser. You can see all the headers, in the raw form. This is one level above using a packet sniffer like wireshark

Let’s save the request: (you’ll have to open localhost:9999 again)

Feed the request to google.com (or you own web server)

Have a look at the raw request: (vim response)

Yeah, it’s gzip compressed. More interestingly, we can use this to mock google:

Open localhost:9999 in a browser, get served, byte-for-byte, what google would have served you.

You can use the -k flag to keep the connection open after the file is served:

This will show you the subsequent requests (images, javascript, css)

Finally, you might want to serve the file more than once:

Discussion

I’ve used netcat in the past to debug gzip compression on nginx and lighttpd. With browsers and curl/wget all doing-the-right-thing with/without gzip compression, how can you really tell if it’s enabled or not?

You can also use it to spoof requests. Both a request or a response file can be trivially changed whereas the same effect could sometimes be achieved by making significant and/or time-consuming configuration changes to your setup.

Of course, this is not limited to HTTP. Extend the ideas here to fit your life.

Netcat is not the ultimate-solution™. It is called the hacker’s swiss army knife. It’s the kind of program you don’t need until you really do.

Lately, I’ve been working on JavaScript stuff at work. When it comes to JavaScript debugging, Firebug can take you a long way. Firebug is not, however, a very scriptable environment.

SpideMonkey is “Gecko’s JavaScript engine written in C”. For our purposes, SpiderMonkey is a command-line tool to execute JavaScript.

You can build it from source or look for it in your system’s package manager.

Before I continue, SpiderMonkey defines a few functions which are useful for non-browser environments. In the following examples, I’ll be using the print function which outputs a string to STDOUT.

Interactive mode:

Execute (-e) mode:

File (-f) mode:

File (-f) mode (multiple files):

Combined mode:

Apart from “print”, there are other useful functions:

Command        Usage                  Description
=======        =====                  ===========
load           load(['foo.js' ...])   Load files named by string arguments
readline       readline()             Read a single line from stdin
print          print([exp ...])       Evaluate and print expressions
help           help([name ...])       Display usage and help messages
quit           quit()                 Quit the shell
clear          clear([obj])           Clear properties of object

This is an abridged version of the available functions, here’s the full one.

The ability to import other files (load), read from STDIN (readline), output messages (print), and quit are exactly what’s needed for scripting. In fact, while I was googling for SpiderMonkey, I found this post which used SpiderMonkey as a primitive CGI script.

More realistically, because SpiderMonkey is MUCH faster to startup than Rhino, you can include it in a command-line workflow to unit test or lint your JavaScript.

I think I already established that Vim makes an excellent pager. Let me take it one step further: Vim is a customizable, programmable pager. (!)

There are plenty of cases where you want to pick one (1) thing out of a list. Vim can easily be made into a list picker.

A few examples

• pick a deep directory under the current directory (pick from find . -type d)
• pick a GNU screen session out of many (pick from screen -ls)
• pick a process to kill (pick from ps)

There are basically 2 components to these examples:

• what command will generate the list
• what command to run on the selection

The Code

Here’s the PickerMode plugin (put in ~/.vim/plugin/picker.vim)

Comments:

• cursorline highlight the line the cursor is on
• return saves the current line to ~/.picked

Here’s the bash code to invoke vim and execute a command on the selection:

Comments:

• the selection is written to a file called ~/.picked
• the existence of the file ~/.picked proves that you selected something
• functional programming in bash (!)

Using pick_with_vim

pick a deep directory under the current directory:

how to pick from screen:

Here’s a much simpler rewrite of “go” (my directory bookmark miniapp)

What now?

This code is available as part of my dotfiles on github. (though it is mixed with the rest)

I’ve talked casually about using Vim as a pager before. However, I’m still surprised to see how many people use Vim regularly and don’t know about this feature.

Here’s a quote straight from vim --help

vim [arguments] -               read text from stdin

Admittedly, it’s easy to overlook the hyphen in the explanation.

Why Vim as a Pager?

If you’re using Vim already, there’s nothing else to install.

If you’re using Vim already, it’s already configured the way you like it.

More importanly, Vim detects the kind of file it is being piped and turns on the appropriate syntax highlighting. Why page in black and white? In this case, “less” is definitely less!

Improving the experience

As a pager, you want to use Vim in read-only mode.

some command | vim -R -

What the difference? Vim doesn’t ask you to save the file if you try to quit. Of course, you can still modify and write the file … the -R flag is just a more reasonable pager default.

PAGER variable and ANSI Escape Sequences

You probably don’t want to set the PAGER variable. Vim doesn’t understand ANSI escape sequences. As such, a command like “man vim | vim -R -” won’t show colors; it will show escape sequences.

I haven’t found any quick and simple solution to make Vim show ANSI escape sequences, but it’s pretty easy to strip them out before passing the file to Vim:

man vim | col -b | vim -R -

I use less as PAGER. I use vim in explicit cases.

View

The view command gets installed at the same time as vim. It’s just a symlink to vim. Using view is exactly like typing vim -R.

There’s a certain aesthetic in:

some command | view -

But I find that typing vim -R - is easier on my finger’s muscle memory.

What happens when you type git diff? As with all interesting questions, the answer is “it depends…”

Here’s one thing you want git to do:

Vimdiff!

Step 1: add this to your .gitconfig

Step 2: create a file named git_diff_wrapper, put it somewhere in your $PATH

I still have access to the default git diff behavior with the --no-ext-diff flag. Here’s a function I put in my bash configuration files:

• --no-ext-diff : to prevent using vimdiff
• -w : to ignore whitespace
• -R : to start vim in read-only mode
• - : to make vim act as a pager

When it comes to vimdiff, you can get started with this tutorial.

“How did we get into this mess?” you exclaim looking at the code equivalent of:

There are a lot of reasons why projects end up looking like that. Beyond incompetence, there is the realization that you know more about a problem after you solve it.

Here’s a familiar scenario:

You extract an ugly part of a function. Out of the remaining code, you notice that it’s just a special case of a problem you have solved in a more general way elsewhere. A few tweaks later, you remove the special case, a whole bunch of code is gone.

The “Tetris” effect!

A good refactor feels the same way it does when you clear a couple of rows in Tetris. There are insights which allow you to provide the same (if not more) functionality in less code.

I remember reading Refactoring and being annoyed at the series of seemingly trivial changes that were being done to the code. However, one change at a time, the code was getting more and more manageable.

The same is true with Tetris: clearing the first row of block, you can start attacking the second and so forth. Hopefully you reach to bottom.

Which brings another interesting parallel with Tetris: if you don’t start simplifying things and cleaning up your mess, you end up with a big pile of unmanageable code. One block goes on top of another until you touch the top. Eventually, you have to start another game.

Compare that to incoming requirements which result in more features piling on top of each other. At some point, somebody starts talking about doing a rewrite. In effect, that would be like restarting the game. It’s always easier to throw blocks down an empty screen rather than fit them on top of an existing mess.

I’m not blaming anyone … there is a time and place for rewrites. Use judgment.

Can you also think of a time where a new requirement messed up the beauty of the model you had built. Something like this?

The more I think about it, the more I like it.