stow: Not just for package management any more

GNU stow is another one of those projects that took me a while to wrap my head around, but once I saw it in action, it made perfect sense. Of course, given that stow doesn’t really show any output, that statement is ironic on another level.

As I understand it, stow is intended as a kind of package manager for software you might build locally — which in my case, is quite a lot. stow works by creating symlinks from a program’s original location to a central depot of your creation.

The idea is that this reduces the chances of unforeseen conflicts, and makes managing random, scattered files easier, since most everything is in the same place. One program, one folder, and appropriate symlinks elsewhere in the system.

If that doesn’t make a lot of sense, don’t worry, because two hours before I wrote this, I was trying to wrap my head around it too. Now though, I think I see the value in it.

What clued me in was this post by Brandon Invergo, where he talks about creating a folder specifically for dotfiles with stow. Brandon is an easy read and he gives you a good visual illustration, so don’t worry if you’re not a fan of reading from blogs. Neither am I (ironic, isn’t it?).

If you step through his post, you’ll end up with a small tree of folders with the dotfiles of each program nested individually. Your home folder will still hold links to those files, and everything will still work as it should. So don’t panic. 😉

I tested it with two programs that I thought more or less bulletproof, even if my configurations were utterly vaporized: htop and snownews. And after Brandon’s instructions, wouldn’t you know it, everything worked fine.

Why would I want to do this? Well, like Brandon explains, this makes it much cleaner to synchronize your dotfiles against an online repository — for example, you can more conveniently dump your dotfiles on github. No more cherrypicking files and sending them singly.

But personally, I usually have a lump of folders and settings that I transfer between machines, to expedite setup or testing. Even just in the past two or three days, I’ve ended up manually copying files from one machine to another, and from that machine to a third. In the future, I expect I can install stow, rsync the dotfiles folder to the new machine and jump right in.

I suppose you could do this manually, file by file and link by link, and not need stow. But just thinking about that should make it obvious why stow is a good tool: Manually setting up all those links would be tedious to say the least.

I have the reassurance of Brandon and some other sites that even if you uninstall stow, your link systems will continue to work. That makes sense to me, even if I haven’t taken that step yet.

stow has the potential to be a game changer for you, if you need that kind of added flexibility with packages or with your personal configuration files. It won’t clean up your home directory — you’ll end up with just as many symlinks as you had configuration files, and in the same place — but it adds a layer of convenience that you might find immediately attractive.

paste: What I thought join would be

I just showed paste in the last post but I haven’t mentioned it on this site. I probably should have reversed the order here, since paste is one of the last coreutils toys I was holding back from the leftover slurry.

paste does what I thought join should do — concatenate two separate data files, line by line. Again, this is something easier done than said:

That looks almost identical to what join does; here’s where they differ:

paste at least hints that there were omissions in one column or the other. join, on the other hand, skipped over those items, and demanded they be sorted.

Of course, seeing paste and join side-by-side makes a lot more sense in why they’re named as they are. join links together corresponding entries according to a sorted order. paste just forces them together, even when something is missing.

I’d still like to see paste insert a tab where the first list is missing a line, but at least now I get the picture.

I handed datamash a small gold star for transposing its output, and paste has a similar function in its -s flag.

So you can run out vertical lists horizontally, if you are so inclined.

I’m quickly running out of coreutils titles, and I do so enjoy learning about them. Perhaps one day I shall start a blog that only steps through that and the util-linux package, and looks at each tool one at a time. … Nah, who’s got time for that? 😕

rename: The built-in filename sifting tool

Sorry for the one-day break; Thursdays are always a little bit hectic for me, and this being the last in the month was especially busy.

The bulk of util-linux is splashed across the previous post, but I have a few left over that I want to point out. rename is one of those, and at its best, rename is a quick and speedy tool for bulk renaming files. Here’s what it does, on a good day:

That’s a classy solution for bulk renaming where the same string needs to be substituted out in each file. rename makes it (more or less) a cinch to swap date strings, replace extensions or even make mass insertions and deletions to file names … with a little added command-line kung fu.

rename‘s shortcomings — and you knew I was going to point some out — occasionally crop up, though:

If you look closely, you’ll see that the last file name had its prefix changed, but not the extension. rename caught the first instance of “text,” but quit before it found the other.

rename also has very little in the way of error-checking. Once you send the command, the deed is done … and short of reversing your previous command, there is no preview-and-commit. And you must be cautious that your substitution doesn’t allow for files to be moved onto one another.

And it should probably go without saying that, unless you are a regex grand master, some of the more complex or subtle renaming that is possible with something like renameutils is lost on rename. Which isn’t to say it can’t be done, only to say that your success will depend on your proficiency. 😕

rename works though, and in minor substitutions it’s a breeze. And given its simplicity and straightforward arrangement, I can’t say too many bad things about it. Keep it in mind the next time you dump a couple hundred pictures off your digital camera, and need a system to order them. … 😯

echo: And an inescapable fragment of history

After much internal debate, I decided to include echo in this august parade. echo hardly does anything, but I use it so much that it would be a glaring omission if I didn’t include it.

You probably already know what echo is for: It outputs a string of text. That’s the first line of the man page, after all.

kmandla@6m47421: ~$ echo Hello world.
Hello world.

And now you’ve seen all that echo does … mostly. 😉

Like any good tool, it has a few things you can customize. For starters there is an -n flag, which prevents echo from breaking to a new line when it finishes. In other words:

kmandla@6m47421: ~$ echo -n Hello world.
Hello world.kmandla@6m47421: ~$ 

echo can also do some funny things, if you allow it with the -e flag. The man page has a full list, but for example, line feeds are possible. A line feed is not a newline, as this should show:

kmandla@6m47421: ~$ echo -e "Hello \fworld."
Hello 
      world.

And line feeds are not carriage returns, as this should show:

kmandla@6m47421: ~$ echo -e "Hello there, \rworld."
world.there,

If you remember computers from way, waaay back, you probably already knew the difference. If you don’t it might help to imagine a printer head moving across a page. Line feeds advance the paper without moving the head, and carriage returns bring the printer head back to the starting edge without advancing the paper. Put both of them together and you have a newline. And of course, all of this evolved from the days of manual typewriters, when your printer was 168cm tall and took coffee breaks twice a day.

I’ll leave it to you to explore all the subtleties of echo; there aren’t many, but it is a utility that you can use on a daily basis and still not see every option. And of course, echo is part of the world-famous coreutils suite. 🙂

mkfifo: Pipe panjandrum

I’m going to guess that you probably know what a pipe symbol does on most Linux systems — passes the output of one program to a second. It’s what allows you to do things like this

dmesg | grep ATA

and find local hard drives. Or gives me my topics for the day, with

ls vimwiki/ | shuf -n1

mkfifo is part of coreutils, and allows you to name a pipe, and reuse it over and over. It might sound odd, but it works in much the same way as the symbol.

mkfifo pipe

Now I have a file entry called “pipe” in my directory, marked with a “p” in the leftmost column.

kmandla@6m47421: ~$ ls -lha
prw-r--r--  1 kmandla users    0 Sep 21 07:42 pipe

Now I can jam something in that pipe, if I like.

kmandla@6m47421: ~$ ls vimwiki/ > pipe

And … apparently, nothing happens. My terminal is paused, somehow waiting for an action. The suspense is killing me. 🙄

Actually, until something collects the material in that pipe, it will pause there more or less indefinitely. So if I …

kmandla@6m47421: ~$ grep ch < pipe

The whole flood comes out, and we can all relax again. Crisis averted. 😐

It might seem rather pointless or irrelevant to name a pipe or even to make note of mkfifo for it. But you’ve already seen the subtlety in action, and maybe just didn’t think about it.

That first terminal emulator was paused, waiting for someone to unblock the pipe. A second terminal received the data and did something with it.

So what? So … not only can you jam a program’s output into a holding area, waiting for a recipient, but that also means you can pass data between terminals with a named pipe. So if you’re waiting for one program to end, you can send the output of another into stasis, until it’s ready.

And you can pass information between different programs running in different terminals, at virtual consoles, in and out of a multiplexer … possibly even between users or across distant machines if you’re clever. It’s not the best way to do those things, but it might work in a pinch. Experiment and see.

I know named pipes are not anything new, but little things like this are the reason I wanted to sift through coreutils again. If you need a better explanation of named pipes, and you don’t mind reaching back almost 20 years, Andy Vaught’s explanation from issue #44 of Linux Journal is a great starting point. Nice to see that things haven’t changed that much since ’97. … 😉

basename: What I had in mind

A while back, when I said I dumped coreutils back into The List because I thought there were still a few useful programs in there, I was thinking mostly of basename. So I’m kind of glad that shuf spat it out today.

I use basename a lot, although I can’t pin down any regular case that you might identify with. Here’s one that comes to mind though: Making sure the index page in my local wiki has all the programs included in the directory.

Occasionally I get sloppy and lose a title, or worse, have a file in the directory that doesn’t have a corresponding link in the index. That’s a problem because I can go months without knowing there’s an additional program in there.

I solve that by periodically dumping all the names of the files into an empty wiki page, and check links manually. vimwiki makes that fairly easy, since one press of the enter key will create a link, and the next will follow it. If the page is blank, it was a missing file, but if the file is there I can check that against my old list.

All this is terribly uninteresting. Let’s get to the good part. I use find to pluck out all the files from within /home/kmandla/vimwiki/ :

find . -type f

find naturally shows a preceding ./ for each file. I could fix that by adding -printf "%f\n" at the end, but vimwiki tacks .wiki on the end of each file, so I’d end up using basename anyway to get rid of that.

Ergo,

find . -type f -exec basename "{}" .wiki \;

In that case, basename trims off the leading path, and then the suffix .wiki from whatever it is given. The next step, which I won’t bother showing, is to dump all that into a separate file.

basename can cut off whatever you like from the end of a file, provided you predict it correctly. It’s a good way to trim file extensions, if you know what they are (if you don’t, you should be thinking about cut).

I have other uses for basename, and I can’t think of them right now. Perhaps in the future I’ll come back and add them on here, just for future reference. 🙂

join: Not everything is perfect

Some of the more elegant tools in coreutils — like comm, sort or uniq — are fairly straightforward and easy to understand.

join, on the other hand, seems to require some inspection. In its simplest form, it makes sense:

Each line from each file is paired and output together. But here’s where things get complicated … or at least my brain wants to follow a different track:

One file is a line longer, so join refuses to join them, or will only join the ones that match. My logic tells me it should join them anyway, but use some sort of placeholder to show information is missing.

If I ask for an additional line with the -a flag, I get the last line but it’s pressed to the left — and you can see how that makes it look like that line belongs to the first list. And the -a flag seems cumbersome, since it requires me to know beforehand that one list or the other is missing data.

And finally … if the lines aren’t prefixed, join shows nothing for its efforts. If they’re not sorted, the unsorted lines are spat out with a warning, in the center of everything else that worked. It’s a bit of a mess.

I understand how join works and I can abide by its rules and formatting, but it seems counterintuitive to me. Perhaps if I just want corresponding lines of two different files output on the same line, there’s another tool I should consider.

For what I’ve seen, join can do that, but it will take a little preparation from me. 😕

co: The command organizer

A couple of weeks ago, bm sounded like a good idea. This week, co sounds like a good one too. Where bm allows you to stash bookmarks and cue them quickly, co helps you build a database of particularly obtuse commands, tag them, annotate them, search them, filter them, and then recall them with only a few keypresses.

If co looks like a re-imagining of bash’s history tool, that’s understandable. A few important distinctions though:

• co doesn’t stash any commands until you tell it to, and then it only grabs the last thing you entered.
• When you tell it to save a command, you can add as many tags as you like, and a short “message” to remind you of the keyboard sorcery you performed.
• Once saved, you can list out the commands you saved, or filter that list by tag.
• You can perform a command again through co, by using its index number.

So instead of simply keeping a history of commands, co allows you to organize and manage the most useful ones.

As you can see above, it’s particularly helpful for long, complex commands that might otherwise require a lot of re-research to use again. co also has a rudimentary “interactive” mode, which allows you to step through your collection one line at a time, and even edit it on the fly.

One small note: There may come an odd situation where you ask co to save a command while you’re using more than one terminal. It’s possible (and I know it is, because it happened to me) that co will grab a command that was executed in another terminal, and save it instead of the one you just entered locally. Just so you know. 😐

co uses sqlite and ruby, and is not in Arch or Debian that I could find. I used gem install co in Arch to add it to my ~/.gems path; I’m sure one of you clever Internet heroes out there can build it properly for either of those distros, if you like it enough.

I really like co because it solves a dilemma for me: I hate programs that keep histories, but I do invent a lot of convoluted commands for esoteric situations.

co allows me to faithfully eradicate bash’s history at regular intervals, but still keep the command I use to generate sample text files filled with random words. 😕 Hey, that’s important stuff. 🙂

cheat, cheat, cheats and cheat: So many cheaters

I got a frenzied e-mail the other day insisting that I add “cheat” to the remaining titles in the C section. I agreed, but was a little surprised to see that it was a different “cheat” than what I had added months earlier.

So there are a lot of cheats out there. Here is the one I was sent a few days ago. … I think. … 😳

It’s a simple enough principle: Chris Lane’s cheat keeps a directory of text files, and calling “cheat” dumps them to the screen. In essence you have a healthy collection of cheatsheets with quick-fire display options.

What you put in a sheet is up to you; as Chris mentions, you could do just as well to keep recipes, inspirational quotes or bank account numbers in there. cheat itself doesn’t really know the difference.

Chris Lane’s cheat can accept a few external variables, such as a specific path to cheat files, or syntax coloring. Those can be helpful.

Here’s the cheat that I had found a few months ago, that is supposedly built to mimic Chris’s version.

It’s very similar, as you can see, and the real difference — as I understand it — is that jahendrie’s version is a bash rewrite, whereas Chris Lane’s original was made for python.

There are some special differences here and there; jahendrie’s version has a search option that seems to work differently from Chris’s. jahendrie also allows an in-file grep option, which might be preferable under some circumstances.

It’s very difficult for me to tell between Chris’s and jahendrie’s cheat. They work much the same and carry some similar options. I did notice that Chris’s version came bundled with cheatsheets for many terminal commands, but jahendrie’s apparently doesn’t. Please correct me if I’m wrong.

Here’s one more, this time called “cheats,” just to round out the trifecta.

There are some subtle differences here, which might not be obvious just from a screenshot. First, cheats allows for several cheatsheets on the same topic, with slightly different names. What you see as “dd 1” and “dd 2” came out of different files.

Second, and more importantly, cheats is a bash-only version with a very clever addition: the ability to actually prompt through a cheat, and build the command step-by-step. Finish the prompts, and cheats will run the command as you built it.

So in that sense, cheats takes the ideas shown in cheat and cheat one step further, and saves you retyping the reminder off the screen, or plunking around with testing options. It’s a nice touch, and would be useful for beginners in particular.

By default, cheats comes with a few examples for dd, du, git, sort, tar and a few others. You’ll have to add and build more, if you really want to flesh out your collection.

As a bonus, here’s cheat for ruby, which I found while trying to untangle the last three in my mind. 😐

 

I’m not a ruby programmer, so I have a feeling that a lot of what ruby-cheat offers is outside my grasp. On the other hand, ruby-cheat seems to know enough to dump its output straight into $PAGER, and let you navigate from there.

I won’t judge the ruby edition of cheat too harshly (in fact, not at all), since again, it’s not something I have a direct application for. Then again, when has that ever stopped me … ?

For what it’s worth, I have my own system for command-line cheats, and I would guess it’s similar to something you might have invented on your own. I keep a single text file with a list of sometimes-used-but-not-intuitive commands, and grep through them when I can’t remember the exact syntax.

It’s primitive, but it also allows me the freedom of pumping the command directly into the console, with help of eval. In short, eval $(grep exif hold/cmd.txt ) usually triggers a command, if I want it.

As a side note, I am admittedly a person who learns by experimenting with other people’s examples. It’s just my nature. To that end, you would think that something like cheat or cheat or cheats or cheat would be a quick adoption for me.

But I don’t know that any of these — cheats included, and that would probably be the one I would keep — is much more help than a decent man page. True, in a perfect world, every man page would include a few examples, and this is not a perfect world.

But I have a feeling these are only as useful as you make them. Pick any one and add a little of your own expertise to its database, and it will be a valuable addition. Of course, I could say that about everything. … 🙄

vimwiki: The reason, in due season

I promised I’d let on to why I’ve relied on vim all these years, and the time has finally come. As luck would have it, the only reason I put up with that cumbersome, unfriendly, cryptic and sadistic text editor is because of vimwiki.

I know, I’m a bit of a hypocrite for clinging to a particular text editor for years, all because of one silly plugin that it supports.

I can’t rationalize that, except to say that vimwiki has vastly simplified the task of managing The List — with a thousand program titles, each (supposedly ) with a one-line synopsis and a link to a home page, plus some notes. I couldn’t imagine trying to handle that in a flat text file, or something like hnb. It would not function nearly as cleanly.

vimwiki as evolved over the years I have used it, and I’m comfortable with it in its current rendition. Press enter on a word to convert it to a link, press enter on a link to jump to its page. Press backspace to work your way back through the breadcrumb trail.

Master that — within the convolutions of vim, of course — and you’ve gotten everything that you need to keep hyperlinked text files organized.

vimwiki also builds calendars and tables, exports to different formats and handles some markdown-ish syntax, although it’s incredibly rare that I need those features.

vimwiki will require some settings in your .vimrc that might prove confusing; the conceallevel in particular might make URLs contract and that was irritating for the first few days. Over the years I’ve learned to live with that.

vimwiki is smart enough to carry a few housekeeping features too, though. It’s a simple three-key command to delete or rename a page, which are both crucial functions in my project. And it’s smart enough to riffle through every other page, and correct links therein.

I know it’s not much of an endorsement, but vimwiki is probably one of the few note-taking tools that I immediately embraced, as soon as I saw how easy and clean it worked. The fact that I was willing to overlook all the fatal eccentricities of vim should be an indicator of how good I think it is.

After years of dedication and service, at last, a well-deserved K.Mandla gold star for vimwiki: ⭐ 😉

One last note: This is the only vim plugin I’ll discuss, mostly because there are literally hundreds out there. Some are good and some are bad, but mostly you know what you need and like. There’s no need for me to traipse through each one. Go on your own little adventure. 😉