Tag Archives: manage

stow: Not just for package management any more

GNU stow is another one of those projects that took me a while to wrap my head around, but once I saw it in action, it made perfect sense. Of course, given that stow doesn’t really show any output, that statement is ironic on another level. :\

As I understand it, stow is intended as a kind of package manager for software you might build locally — which in my case, is quite a lot. stow works by creating symlinks from a program’s original location to a central depot of your creation.

The idea is that this reduces the chances of unforeseen conflicts, and makes managing random, scattered files easier, since most everything is in the same place. One program, one folder, and appropriate symlinks elsewhere in the system.

If that doesn’t make a lot of sense, don’t worry, because two hours before I wrote this, I was trying to wrap my head around it too. Now though, I think I see the value in it.

What clued me in was this post by Brandon Invergo, where he talks about creating a folder specifically for dotfiles with stow. Brandon is an easy read and he gives you a good visual illustration, so don’t worry if you’re not a fan of reading from blogs. Neither am I (ironic, isn’t it?).

If you step through his post, you’ll end up with a small tree of folders with the dotfiles of each program nested individually. Your home folder will still hold links to those files, and everything will still work as it should. So don’t panic. 😉

I tested it with two programs that I thought more or less bulletproof, even if my configurations were utterly vaporized: htop and snownews. And after Brandon’s instructions, wouldn’t you know it, everything worked fine.

Why would I want to do this? Well, like Brandon explains, this makes it much cleaner to synchronize your dotfiles against an online repository — for example, you can more conveniently dump your dotfiles on github. No more cherrypicking files and sending them singly.

But personally, I usually have a lump of folders and settings that I transfer between machines, to expedite setup or testing. Even just in the past two or three days, I’ve ended up manually copying files from one machine to another, and from that machine to a third. In the future, I expect I can install stow, rsync the dotfiles folder to the new machine and jump right in.

I suppose you could do this manually, file by file and link by link, and not need stow. But just thinking about that should make it obvious why stow is a good tool: Manually setting up all those links would be tedious to say the least.

I have the reassurance of Brandon and some other sites that even if you uninstall stow, your link systems will continue to work. That makes sense to me, even if I haven’t taken that step yet.

stow has the potential to be a game changer for you, if you need that kind of added flexibility with packages or with your personal configuration files. It won’t clean up your home directory — you’ll end up with just as many symlinks as you had configuration files, and in the same place — but it adds a layer of convenience that you might find immediately attractive.

paste: What I thought join would be

I just showed paste in the last post but I haven’t mentioned it on this site. I probably should have reversed the order here, since paste is one of the last coreutils toys I was holding back from the leftover slurry.

paste does what I thought join should do — concatenate two separate data files, line by line. Again, this is something easier done than said:

2014-10-02-6m47421-paste-01

That looks almost identical to what join does; here’s where they differ:

2014-10-02-6m47421-paste-02

paste at least hints that there were omissions in one column or the other. join, on the other hand, skipped over those items, and demanded they be sorted. :\

Of course, seeing paste and join side-by-side makes a lot more sense in why they’re named as they are. join links together corresponding entries according to a sorted order. paste just forces them together, even when something is missing.

I’d still like to see paste insert a tab where the first list is missing a line, but at least now I get the picture.

I handed datamash a small gold star for transposing its output, and paste has a similar function in its -s flag.

2014-10-02-6m47421-paste-03

So you can run out vertical lists horizontally, if you are so inclined.

I’m quickly running out of coreutils titles, and I do so enjoy learning about them. Perhaps one day I shall start a blog that only steps through that and the util-linux package, and looks at each tool one at a time. … Nah, who’s got time for that? 😕

rename: The built-in filename sifting tool

Sorry for the one-day break; Thursdays are always a little bit hectic for me, and this being the last in the month was especially busy.

The bulk of util-linux is splashed across the previous post, but I have a few left over that I want to point out. rename is one of those, and at its best, rename is a quick and speedy tool for bulk renaming files. Here’s what it does, on a good day:

2014-09-26-6m47421-rename-01

That’s a classy solution for bulk renaming where the same string needs to be substituted out in each file. rename makes it (more or less) a cinch to swap date strings, replace extensions or even make mass insertions and deletions to file names … with a little added command-line kung fu.

rename‘s shortcomings — and you knew I was going to point some out — occasionally crop up, though:

2014-09-26-6m47421-rename-02

If you look closely, you’ll see that the last file name had its prefix changed, but not the extension. rename caught the first instance of “text,” but quit before it found the other.

rename also has very little in the way of error-checking. Once you send the command, the deed is done … and short of reversing your previous command, there is no preview-and-commit. And you must be cautious that your substitution doesn’t allow for files to be moved onto one another.

And it should probably go without saying that, unless you are a regex grand master, some of the more complex or subtle renaming that is possible with something like renameutils is lost on rename. Which isn’t to say it can’t be done, only to say that your success will depend on your proficiency. 😕

rename works though, and in minor substitutions it’s a breeze. And given its simplicity and straightforward arrangement, I can’t say too many bad things about it. Keep it in mind the next time you dump a couple hundred pictures off your digital camera, and need a system to order them. … 😯

echo: And an inescapable fragment of history

After much internal debate, I decided to include echo in this august parade. echo hardly does anything, but I use it so much that it would be a glaring omission if I didn’t include it.

You probably already know what echo is for: It outputs a string of text. That’s the first line of the man page, after all.

kmandla@6m47421: ~$ echo Hello world.
Hello world.

And now you’ve seen all that echo does … mostly. 😉

Like any good tool, it has a few things you can customize. For starters there is an -n flag, which prevents echo from breaking to a new line when it finishes. In other words:

kmandla@6m47421: ~$ echo -n Hello world.
Hello world.kmandla@6m47421: ~$ 

echo can also do some funny things, if you allow it with the -e flag. The man page has a full list, but for example, line feeds are possible. A line feed is not a newline, as this should show:

kmandla@6m47421: ~$ echo -e "Hello \fworld."
Hello 
      world.

And line feeds are not carriage returns, as this should show:

kmandla@6m47421: ~$ echo -e "Hello there, \rworld."
world.there,

If you remember computers from way, waaay back, you probably already knew the difference. If you don’t it might help to imagine a printer head moving across a page. Line feeds advance the paper without moving the head, and carriage returns bring the printer head back to the starting edge without advancing the paper. Put both of them together and you have a newline. And of course, all of this evolved from the days of manual typewriters, when your printer was 168cm tall and took coffee breaks twice a day.

I’ll leave it to you to explore all the subtleties of echo; there aren’t many, but it is a utility that you can use on a daily basis and still not see every option. And of course, echo is part of the world-famous coreutils suite. 🙂

mkfifo: Pipe panjandrum

I’m going to guess that you probably know what a pipe symbol does on most Linux systems — passes the output of one program to a second. It’s what allows you to do things like this

dmesg | grep ATA

and find local hard drives. Or gives me my topics for the day, with

ls vimwiki/ | shuf -n1

mkfifo is part of coreutils, and allows you to name a pipe, and reuse it over and over. It might sound odd, but it works in much the same way as the symbol.

mkfifo pipe

Now I have a file entry called “pipe” in my directory, marked with a “p” in the leftmost column.

kmandla@6m47421: ~$ ls -lha
prw-r--r--  1 kmandla users    0 Sep 21 07:42 pipe

Now I can jam something in that pipe, if I like.

kmandla@6m47421: ~$ ls vimwiki/ > pipe

And … apparently, nothing happens. My terminal is paused, somehow waiting for an action. The suspense is killing me. 🙄

Actually, until something collects the material in that pipe, it will pause there more or less indefinitely. So if I …

kmandla@6m47421: ~$ grep ch < pipe

The whole flood comes out, and we can all relax again. Crisis averted. 😐

2014-09-21-6m47421-mkfifo

It might seem rather pointless or irrelevant to name a pipe or even to make note of mkfifo for it. But you’ve already seen the subtlety in action, and maybe just didn’t think about it.

That first terminal emulator was paused, waiting for someone to unblock the pipe. A second terminal received the data and did something with it.

So what? So … not only can you jam a program’s output into a holding area, waiting for a recipient, but that also means you can pass data between terminals with a named pipe. So if you’re waiting for one program to end, you can send the output of another into stasis, until it’s ready.

And you can pass information between different programs running in different terminals, at virtual consoles, in and out of a multiplexer … possibly even between users or across distant machines if you’re clever. It’s not the best way to do those things, but it might work in a pinch. Experiment and see.

I know named pipes are not anything new, but little things like this are the reason I wanted to sift through coreutils again. If you need a better explanation of named pipes, and you don’t mind reaching back almost 20 years, Andy Vaught’s explanation from issue #44 of Linux Journal is a great starting point. Nice to see that things haven’t changed that much since ’97. … 😉

basename: What I had in mind

A while back, when I said I dumped coreutils back into The List because I thought there were still a few useful programs in there, I was thinking mostly of basename. So I’m kind of glad that shuf spat it out today.

I use basename a lot, although I can’t pin down any regular case that you might identify with. Here’s one that comes to mind though: Making sure the index page in my local wiki has all the programs included in the directory.

Occasionally I get sloppy and lose a title, or worse, have a file in the directory that doesn’t have a corresponding link in the index. That’s a problem because I can go months without knowing there’s an additional program in there.

I solve that by periodically dumping all the names of the files into an empty wiki page, and check links manually. vimwiki makes that fairly easy, since one press of the enter key will create a link, and the next will follow it. If the page is blank, it was a missing file, but if the file is there I can check that against my old list.

All this is terribly uninteresting. Let’s get to the good part. I use find to pluck out all the files from within /home/kmandla/vimwiki/ :

find . -type f

find naturally shows a preceding ./ for each file. I could fix that by adding -printf "%f\n" at the end, but vimwiki tacks .wiki on the end of each file, so I’d end up using basename anyway to get rid of that.

Ergo,

find . -type f -exec basename "{}" .wiki \;

In that case, basename trims off the leading path, and then the suffix .wiki from whatever it is given. The next step, which I won’t bother showing, is to dump all that into a separate file.

basename can cut off whatever you like from the end of a file, provided you predict it correctly. It’s a good way to trim file extensions, if you know what they are (if you don’t, you should be thinking about cut).

I have other uses for basename, and I can’t think of them right now. Perhaps in the future I’ll come back and add them on here, just for future reference. 🙂

join: Not everything is perfect

Some of the more elegant tools in coreutils — like comm, sort or uniq — are fairly straightforward and easy to understand.

join, on the other hand, seems to require some inspection. In its simplest form, it makes sense:

2014-08-22-6m47421-join-01

Each line from each file is paired and output together. But here’s where things get complicated … or at least my brain wants to follow a different track:

2014-08-22-6m47421-join-02

One file is a line longer, so join refuses to join them, or will only join the ones that match. My logic tells me it should join them anyway, but use some sort of placeholder to show information is missing.

If I ask for an additional line with the -a flag, I get the last line but it’s pressed to the left — and you can see how that makes it look like that line belongs to the first list. And the -a flag seems cumbersome, since it requires me to know beforehand that one list or the other is missing data.

And finally … if the lines aren’t prefixed, join shows nothing for its efforts. If they’re not sorted, the unsorted lines are spat out with a warning, in the center of everything else that worked. It’s a bit of a mess.

2014-08-22-6m47421-join-03

I understand how join works and I can abide by its rules and formatting, but it seems counterintuitive to me. Perhaps if I just want corresponding lines of two different files output on the same line, there’s another tool I should consider.

For what I’ve seen, join can do that, but it will take a little preparation from me. 😕