alot: With notmuch help involved, and little more

Up front, let’s say that alot is a “graphical” interface to notmuch, which you might know or remember as a mail indexer and search tool.

  

And as you can see, alot has a lot 😛 of the right stuff — good color, very flexible terminal space, on-screen help and an intuitive interface. Navigation is with enter keys and a few other mnemonics, and some keys have global behaviors, regardless of what you’re doing at any moment in time.

I would bet that, once your mail setup is complete, you’ll only need about three or four minutes to learn how to use alot, and after that, you’ll cruise through your mail folder at ludicrous speed.

A few things to remember though. … alot needs notmuch on board, which means you’re dealing mostly with the local mail concept, as opposed to relaying directly to an online mail source. If you’re looking for something that will pull down messages from GMail and let you pick through them, I don’t think this is it.

On the other hand, it does mean that you’re free to latch alot (and notmuch) on to your home-grown mail system — or as I showed above, latch notmuch onto something like offlineimap, which can yank messages off GMail and store them locally.

So technically speaking, there are options. The question becomes, how much do you want to step backwards through programs, just to take advantage of alot, when things like alpine can more or less handle the entire process in one fell swoop? Again, if you use a local mail system, alot might be preferable. For me … well. … 😐

All in all, I have to give alot a gold star for catching every point in my checklist for a good console application. I don’t think this is a game-changer for me though, since I’d have to rely on two or three other programs before alot became my mail reader, even if setup is fairly straightforward for all of them.

In AUR as alot and alot-git, but the git version wasn’t working at this exact moment in time. In Debian. And of course, as promised: ⭐ Enjoy! 🙂

andatool: Searching forever and ever

I had one more logging tool related to genstats and logintop10 that I wanted to point out, and unfortunately I can’t do much more than this when I show it.

That’s a disappointment on two counts, first being my relative dunderheaded attempts at feeding andatool a proper regex string. I’m dunderheaded about regex on the whole, so I suppose it’s not to be expected.

The other disappointment is in the fact that andatool is continually skimming through my /var/log/pacman.log file, forever and ever, looking for matching strings. As the log gets updated, the counts update … or they would, if I could get it working and show an animated image.

So you’ll have to use your imagination. Or you could build it yourself, give it a professional-grade regex string and a proper log to look at, and see how it works for you. Never trust me on these things. 😕

I can give you a few pointers though. You have to designate a file for andatool to skim through, with the -i flag, and you have to give it the full path. That might sound obvious, but I wrestled with that for a little bit.

Also, it seems more intuitive to use the -s flag each time, since that tells andatool to start at the beginning of the file and work the whole way through. I can imagine where it might be useful to omit it, but all of my best attempts were more interesting when I included that flag.

And I should mention that you’re not limited to one expression, so you could search through a log for several different instances, and watch each one update.

I don’t recall too many live log filter tools, so andatool seems worthy of mention. Not in AUR or Debian that I could find; don’t let that keep you from trying it though.

mps: Not unlike its brother

I was sorely tempted to gloss over mps, because I mentioned mps-youtube way back in September. But I’ve spent a short time with it and I think it’s worthy of mention in its own right.

mps sticks very close to mps-youtube in terms of operation and playback; enter a search term at the startup and mps will show a series of results. Cue the number of the track and mps feeds it into mplayer (or mpv), and the standard keys and controls are available to you.

The home page for mps suggests it can also create playlists, search for single tracks or through album lists, download tracks as well as stream, and a few other nifty tricks.

The home page also says the program works with python 2.7 and 3.3, but does not require any python dependencies. I’m a little fuzzy on that, but as a general rule of life, I subscribe to the principle that less dependencies is better.

I’ll keep this short since much of what mps does is similar to mps-youtube, and rehashing the features of one isn’t necessarily an endorsement of the other. If you liked the way mps-youtube worked — and I did, quite a lot as a matter of fact — mps is going to be familiar and enjoyable. Try one, try the other. 😉

cscope: The code navigator

Short post this time, since I have almost no frame of reference for cscope. I have all the coding ability of a day-old banana peel, so a tool that searches, arranges and navigates source code files is far and beyond me.

That’s cscope picking through the source code for curl, which was just an arbitrary choice. Nothing to be inferred in that.

It’s a very smooth tool though, and if you spend any time at all navigating large files or skimming through trees of code, I can see where cscope would be a huge asset.

From my very cursory inspection, you can set search strings or patterns through the prompts at the bottom of the screen. Matching lines will appear in the top half, and you can use the arrow keys to pick through them. Press enter, and cscope opens your $EDITOR for you to make changes.

Help is available at the ? key, and you can exit with CTRL+D. Just so you know. …

I know very little about coding, even less about cscope, and have only a brushing knowledge of some of the search tools aimed at developers — programs like ag or ack. I can see where cscope might be preferable though, since it offers an interface to your activity.

But of course, you are the best judge of that. 😉

tagfs, xtagfs, dhtfs and more: Tag, you’re it

For some reason, the last file tagging utility I mentioned, tagsistant, touched off a flood of suggestions on tagging titles. Four or five came from just one contributor alone, and I got e-mails about two or three more from other counties.

I didn’t know file tagging tools were so prolific. I’d always just relied on directory trees as the simplest way to arrange things, but now it seems I am living in 1988. 😐

Anachronistic me aside, I’m terribly grateful for all the suggestions. But the sheer volume — and the time it would take to build, set up, learn, playtest and evaluate all of them — means I must go the short route, and list them here as potentially interesting to you, the reader.

I don’t like doing this because there are undoubtedly some very useful tools in here, and it levels the field between the truly genius and the truly jejune. It’s hard to spot a real winner in a crop this dense though, so if you can attest to any one of these, please give us a steer.

• Dantalian: By the home page’s admission, a “multi-dimensionally hierarchical tag-based transparent lightweight file organization system.” This struck me as closest to tmsu, which I liked best of what I’ve used.
• debtags: If I understand correctly, debtags is the Debian solution for tagging the tens of thousands of titles in their collection, and that’s quite a testament. tagcoll, if I read the wiki right, is the go-to tool for manipulating the tags. I don’t know (but perhaps you do) if debtags or its underlying structure are applicable beyond that project itself.
• dhtfs: A tagging system that sports “dynamic directory hierarchies based on tags associated with files.” That suggests to me that the directory structure will evolve as the tags are applied, which is either attractive or horrifying, depending on you. Personally I’m curious in a morbid kind of way, because I like to manage the way things are arranged on my system. Perhaps I shall set up a dummy system and try it out, just for kicks.
• django-tagging: Kevin sent this one by e-mail, but I don’t have any experience with django, and I think this might go way beyond what I could investigate.
• flickerfs: This one might be oddball of the group: I believe this latches on to your flicker account and allows you to work the tags in use there. But I don’t use flicker, and so I might be way off base with this one. Thanks to Lars for suggesting it though.
• pytagsfs: I see this in Debian with the description, “arranges media files in a virtual directory structure based on the file tags,” which might mean it works something like dhtfs for audio files. I might give this one a try later. The home page listed on the package page doesn’t seem to be related to the project, though.
• stagfs: A “proof-of-concept non-hierarchical FUSE file system deriving structure from independent tag files.” The term “proof-of-concept” says to me that it wasn’t intended for actual use, but it might be a viable product, so don’t take my word for it. I am quite frequently wrong. 😦
• tag-fs: I have my doubts about this one, only because the downloadable source file is all of 36.7Kb, and the trunk is suspiciously concise. Also, it doesn’t appear to have seen updates since 2008, which is not necessarily a bad sign, only a little worrisome.
• tagfs: Not to be confused with the previous project, this has seen more recent updates and appears to be (have been?) a more complete effort. It also seems to have an approach like tmsu or tagsistant, and I can see where it might have inspired those two titles. Another of Eric Davis‘s suggestions from a month ago.
• tracker: tracker confuses me a little bit: Apparently this is a plugin for Nautilus, but I also see that there are things like tracker-utils in Debian, so maybe there are text-based tools that can run it too. If you use GNOME you may want to investigate further. If they can’t be split off from the graphical component, then. … 😦 Thanks to Jonas for the note.
• xtagfs: I have a strong suspicion this is intended as a tool for Macs, but I won’t discount it out of my relative ignorance about Mac-related software. And sometimes I see where tools written for Macs are implantable in Linux systems. You probably know better than me. … 😐

Of course, like I implied earlier, I simply don’t have the time right now to work with every title here, and so there may be one or two in there that are not only incapable or inappropriate, but impossible. My apologies if you stumble across something so useless as to be laughable. I feel obligated to share the information I receive, since people are kind enough to pass it my way. Just remember, as always, your mileage may vary. Happy tagging. 🙂

tagsistant: Tagging with a different approach

After my great delight at trying tmsu, I was willing to try out tagsistant on Eric Davis‘s suggestion. And between the two of them, I think Eric is right: tmsu might be a better fit for me too.

I can see similarities between tmsu and tagsistant; both use directory trees to arrange tags, and to show combinations between them. tagsistant seems very intuitive when it comes to finding mixed tags, with a simple plus sign showing combinations, and so forth.

tagistant “tags” files by “copying” them into a tag folder, which means adding an entire subfolder is a lot easier than tmsu was. And you can tag a directory itself, without necessarily applying the tag to files inside it. I can see where that might be preferable, and the home page suggests that will keep the database slim.

tagsistant is also file-manager friendly, and I suppose the same thing could be said about tmsu. Once I had a few files tagged and cross-tagged, I could work my way through the directory tree with Midnight Commander, and see how files were arranged between tags.

Probably the best part of tagsistant was the setup. The home page shows you in simple steps how to create a tag system, tag individual files and work the query process. It’s a very comfortable introduction.

My complaints against tagsistant are also simple and fairly straightforward: Perhaps biggest, there’s no way I could see to query the tag system without working through the folder tree.

With tmsu I could just ask it outright what tags were applied to a file, but I don’t see an analogue for that in tagsistant. Please point it out to me, if I’m just being dense.

Second, I see no expedient way to apply multiple tags at a time. The copy-file-to-tag motif is an interesting approach, but it doesn’t lend itself to adding three or four tags at once. I prefer tmsu’s approach, of just listing tags in quick succession.

That might go back to my first, and this last point, just that the tree structure for tagsistant is a useful format, but can be very cumbersome without a file manager. Tab completion helps, but each query up or down the tree is going to require some backspacing, correction and possibly even retyping. My advice? Pick a quick file manager.

(Of course, maybe if you could combine this with something like commacd, you might have a very powerful combination. … 😐 )

tagsistant is by no means an unusable tool, and depending on your files and tags, you may prefer this over other options. Definitely look through the advanced documentation, because it will help you with a lot of tagsistant’s finer points.

As for myself, I’ll stick with tmsu, unless one of Eric’s other recommendations wins me over. … 😉

cfind: A search tool built on a search tool

I had to read the home page for cfind more than once to figure out what it does, but I don’t think reading it any more will help me understand why it would be useful.

cfind “provides functionality similar to that of Google Desktop from the command line” — and that much I understand. But the little intricacies are escaping me.

cfind needs locate, which is part of mlocate in Arch and not installed by default. Before you can use locate, you have to run updatedb. After that, locate can scramble through the database of your files and pull out just about anything you want, in fractions of a second. That we saw long ago.

Before you can use cfind, you have to run cindex, it’s partner application. Once all those steps are complete, you can query cfind for search strings. So once more: updatedb, then locate if you want, then cindex, and finally cfind.

And here’s the kicker: cfind and locate will generate almost identical output, with two exceptions that I can see. cfind apparently only searches through text and TeX files, and cfind orders its results by “relevance.”

Here’s what it looks like when it’s finished, sort of.

There I sent the results of locate, on the left, and cfind, on the right, through paste with column -t | most for readability’s sake. 😉

I guess my confusion lies in the need for a tool to skim a database made from another database, when there’s already a tool that skims the original database. How is cfind better than using straight locate, and filtering the results for TeX or text files?

These are the questions that keep me awake at night. 😐

I appreciate that cfind might fill a niche, but I only rarely work with TeX files and don’t have enough text files to justify a search tool built on a search tool. And without any guidance on its algorithm for “relevance,” I don’t think I’ll be keeping this around.

For the record, this is the second tool I’ve come across in two years that lays claim to the “Google Desktop for the CLI” title. The other, you might remember, was doodle. Neither one seems to have usurped the standby, Unix-esque tools — find or locate — that are already available. 😦

the_silver_searcher: Intergalactic searcher smackdown

Uh-oh. I can already sense some friction between the_silver_searcher and ack, which you might remember as a sort of replacement for grep.

  

Left-to-right, ack, the_silver_searcher and grep. Please pay attention to the results of time.

Because it seems to me that both the_silver_searcher (which I will henceforth abbreviate by its executable ag, because that name is a little cumbersome to type) and ack pound their chests on their home pages, and suggest they are faster than grep. And ag definitely holds itself out as faster than ack. And I could swear ack claimed a speed difference over grep.

And yet in subsequent tests, I get results a lot like this …

	grep	ack	ag
time … “Time” list.txt	real 0m0.025s user 0m0.010s sys 0m0.003s	real 0m0.387s user 0m0.280s sys 0m0.010s	real 0m0.051s user 0m0.017s sys 0m0.007s

And grep is regularly faster than ack or ag. Q.E.D.

For the record, list.txt is a 75K list of television and movie titles that I scraped off the Internet just for this searching smackdown. No funky characters or encoding issues that I could see, and nothing but simple ASCII characters.

Maybe it’s my imagination and none of these programs attempted to be faster than another. And it’s important to remember that my tests are particularly gummy. I have an unnatural knack for screwing up some of the simplest setups, and an unscientific, unprofessional, unstatistical freestyle search-to-the-death would be no exception.

All that aside, ag strikes me as every bit as useful or friendly as ack, and seems to follow a lot of the same flag and output conventions. In fact, some of them appear identical. 😕

I suppose that means your choice of search tools will depend mostly on … your choice. In the mean time I’m going to turn up the heat on all three of these and check their results when I turn them loose on time {grep,ack,ag} main /lib/ … 😯 That will separate the men from the boys. 👿

P.S.: Thanks to riptor for the tip. 😉

tmsu: Guten Tag!

I got an e-mail from Paul a month or so ago, asking politely if I’d look over his tmsu program. I’m glad I did; it’s one of the niftier things I’ve come across for file management in the past year or so.

tmsu is a tagging tool, and a lot more. That might not sound too impressive at first (it didn’t to me), but I’ve had it installed for the past week or so, and I’m quickly becoming addicted to it.

What’s that mean — a tagging tool? Well, no doubt you’ve seen posts on this site that have tags at the bottom; I try to add tags that are vaguely related to the content, so you (or I) can search on topic very easily. You can even enter it from the address bar, and get a list of posts on that tag.

tmsu does much the same for files, plus more. Here’s an example. I have a folder of files that are all jazz music files, in ogg format. I can tag a file with “jazz” and “music” like this.

kmandla@6m47421: ~/downloads/Revolution Void/Increase the Dosage$ ls
01 Invisible Walls.ogg       05 The Modern Divide.ogg      09 Accelerated Lifestyle.ogg
02 Factum par Fictio.ogg     06 Double the Daily Dose.ogg  10 Headphonetic.ogg
03 Habitual Ritual.ogg       07 Weekend Amnesia.ogg        11 Nebulous Notions.ogg
04 Effects of Elevation.ogg  08 Obscure Terrain.ogg

kmandla@6m47421: ~/downloads/Revolution Void/Increase the Dosage$ tmsu tag "01 Invisible Walls.ogg" music jazz

No reply from tmsu, but that just means all went well. Now I can find exact files that have that tag, just by asking tmsu.

kmandla@6m47421: ~/downloads/Revolution Void/Increase the Dosage$ tmsu files jazz
./01 Invisible Walls.ogg

tmsu knows where I am in the folder tree, and shows the location relative to that. If I move elsewhere, tmsu’s reply changes to reflect it.

tmsu can also tell me what tags are applied to a file.

kmandla@6m47421: ~$ tmsu tags downloads/Revolution\ Void/Increase\ the\ Dosage/01\ Invisible\ Walls.ogg 
excellent  jazz  music

kmandla@6m47421: ~$ 

So, what’s so great about that? Well, it means I can tag a file with anything — dates, genres, contributing artists, release date, operating system, file format, program version, codebase, first initial, creator, download site, language, subtitles, video quality, personal ratings, prices, origin — you name it. And not just music files — anything. Plus, you’re not beholden to a rating system, application or specific mark as a tag. Tag it with whatever you like, not just what id3v9.1a says is a legitimate category for an mp3 file. 🙄

Here’s a rough example, using personal ratings of excellent, very-good, good, and poor, which is just arbitrary. I’ve abbreviated the lists considerably, of course.

kmandla@6m47421: ~$ tmsu files excellent
./downloads/Revolution Void/Increase the Dosage/01 Invisible Walls.ogg
...
./downloads/Revolution Void/Increase the Dosage/11 Nebulous Notions.ogg

kmandla@6m47421: ~$ tmsu files poor
./downloads/Revolution Void/Let 1,000 Flowers Bloom/01 Sympathy for Mr Vertigo.ogg
...
./downloads/Revolution Void/Let 1,000 Flowers Bloom/28 Ye Ye Ye.ogg

Those would all show up if I asked for “jazz” or “music” too, because I added those tags as well.

Now, if that didn’t grab you, here’s where tmsu suddenly changes from mild-mannered scientist into a giant green unstoppable mountain of muscle.

tmsu can mount a folder as a virtual filesystem, and create symbolic links that point back to the original file. Take a look.

kmandla@6m47421: ~$ tmsu mount temp/

kmandla@6m47421: ~$ ls temp/
queries  tags

The tags folder is probably the most obvious one: Inside that folder are all the tags you’ve created, and links back to their target files.

kmandla@6m47421: ~$ ls temp/tags/
alternative  excellent  good  hip-hop  jazz  music  poor  rap  rock  soul  very-good

kmandla@6m47421: ~$ ls temp/tags/excellent/
01 Invisible Walls.410.ogg       05 The Modern Divide.413.ogg      09 Accelerated Lifestyle.405.ogg  music
02 Factum par Fictio.404.ogg     06 Double the Daily Dose.412.ogg  10 Headphonetic.414.ogg
03 Habitual Ritual.406.ogg       07 Weekend Amnesia.408.ogg        11 Nebulous Notions.407.ogg
04 Effects of Elevation.409.ogg  08 Obscure Terrain.411.ogg        jazz

The numbers are file IDs, and the additional tag folders — “jazz” and “music” — point back to still more links. So I can sift even further into specific sequences of tags by following the folders, and for example, find “excellent/jazz/music”. 😉

That’s pretty cool, and if you have a massive collection of music or photos to manage, it will save you gobs of time and effort. But wait, there’s more. …

That other folder, queries … what about that? Let me show you first, and then we’ll try to puzzle it through.

kmandla@6m47421: ~$ ls temp/queries

kmandla@6m47421: ~$ ls temp/queries/excellent
01 Invisible Walls.410.ogg       05 The Modern Divide.413.ogg      09 Accelerated Lifestyle.405.ogg
...
04 Effects of Elevation.409.ogg  08 Obscure Terrain.411.ogg

kmandla@6m47421: ~$ ls temp/queries/
excellent

I abbreviated again, as you can see. My ls command created a folder inside queries, and inside there are links to everything that matches my request. Magical, on-the-fly searching and linking.

I haven’t tried using tmsu in conjunction with many graphical programs, but I believe you could create a folder inside queries from within a file selection dialog, and get the same results. I’d be interested to see how that works.

I haven’t found many shortcomings with tmsu; it appears to be very well thought out, and behaves in a very Unixy way. My biggest stumbling block was a way to tag an entire folder of files with spaces in their names; the spaces were confusing tmsu and my use of quotes wasn’t helping matters.

In the end I solved it in a similarly Unixy way, by drafting find into service:

find downloads/Revolution\ Void/ -type f -exec tmsu tag "{}" music jazz \;

And that was more than enough to get the job done.

I only have about 8Gb of music that I keep, and I don’t have much in the way of photos except for family snapshots. tmsu has made me think about how I store those though, and made me wonder if the endless branches of folders are really helping.

I would strongly recommend experimenting with tmsu if you have large masses of files that need quicker access than through a file tree, or even just as a way of managing a flat folder of files without resorting things into subfolders. Check it out for a few minutes at least, and see if it can make things easier for you. 🙂

notmuch: “Not much” is relative

I do my best to tinker with all the local mail tools that I collect, even when some of them are well beyond my reach. notmuch is a mail indexer, that supposedly is extensible to a practical mail reader, with the help of some outside applications.

Here’s what it can do on its own:

If you keep a local mail folder but need something to search and sort, notmuch might work well for you. I did that little bit with the help of offlineimap and one Google Mail account, and it didn’t take much more effort to get notmuch working. On its first run, notmuch gave me an interactive configuration wizard, and in the space of a minute, I was picking through my .mail folder.

notmuch has only a few commands — like search and show above — that return mail results, and from there you can manage them as you need to. I see that there are also apparently provisions for tagging messages, and dumping and restoring, a la an e-mail backup.

notmuch is another tool that, by itself, is not much. 😀 It will require something to fetch the mail, and probably you’ll want to either incorporate it into another program as a mail reader — I see that both emacs and vim are possibilities, as well as the classic mutt — and there’s still the task of replying to mail before notmuch becomes a full e-mail suite.

That’s a little more than I have time for, and to be honest, if you’re the kind of person who manages their e-mail at the local level, it’s probably something you’re better at setting up than I am. No offense taken.

But keep notmuch in mind when the time comes to sort through 3000 e-mails looking for that one person who sent you a tip about an esoteric console program. … 😉