Tag Archives: information

tmsu: Guten Tag!

I got an e-mail from Paul a month or so ago, asking politely if I’d look over his tmsu program. I’m glad I did; it’s one of the niftier things I’ve come across for file management in the past year or so.

tmsu is a tagging tool, and a lot more. That might not sound too impressive at first (it didn’t to me), but I’ve had it installed for the past week or so, and I’m quickly becoming addicted to it.

What’s that mean — a tagging tool? Well, no doubt you’ve seen posts on this site that have tags at the bottom; I try to add tags that are vaguely related to the content, so you (or I) can search on topic very easily. You can even enter it from the address bar, and get a list of posts on that tag.

tmsu does much the same for files, plus more. Here’s an example. I have a folder of files that are all jazz music files, in ogg format. I can tag a file with “jazz” and “music” like this.

kmandla@6m47421: ~/downloads/Revolution Void/Increase the Dosage$ ls
01 Invisible Walls.ogg       05 The Modern Divide.ogg      09 Accelerated Lifestyle.ogg
02 Factum par Fictio.ogg     06 Double the Daily Dose.ogg  10 Headphonetic.ogg
03 Habitual Ritual.ogg       07 Weekend Amnesia.ogg        11 Nebulous Notions.ogg
04 Effects of Elevation.ogg  08 Obscure Terrain.ogg

kmandla@6m47421: ~/downloads/Revolution Void/Increase the Dosage$ tmsu tag "01 Invisible Walls.ogg" music jazz

No reply from tmsu, but that just means all went well. Now I can find exact files that have that tag, just by asking tmsu.

kmandla@6m47421: ~/downloads/Revolution Void/Increase the Dosage$ tmsu files jazz
./01 Invisible Walls.ogg

tmsu knows where I am in the folder tree, and shows the location relative to that. If I move elsewhere, tmsu’s reply changes to reflect it.

tmsu can also tell me what tags are applied to a file.

kmandla@6m47421: ~$ tmsu tags downloads/Revolution\ Void/Increase\ the\ Dosage/01\ Invisible\ Walls.ogg 
excellent  jazz  music

kmandla@6m47421: ~$ 

So, what’s so great about that? Well, it means I can tag a file with anything — dates, genres, contributing artists, release date, operating system, file format, program version, codebase, first initial, creator, download site, language, subtitles, video quality, personal ratings, prices, origin — you name it. And not just music files — anything. Plus, you’re not beholden to a rating system, application or specific mark as a tag. Tag it with whatever you like, not just what id3v9.1a says is a legitimate category for an mp3 file. 🙄

Here’s a rough example, using personal ratings of excellent, very-good, good, and poor, which is just arbitrary. I’ve abbreviated the lists considerably, of course.

kmandla@6m47421: ~$ tmsu files excellent
./downloads/Revolution Void/Increase the Dosage/01 Invisible Walls.ogg
...
./downloads/Revolution Void/Increase the Dosage/11 Nebulous Notions.ogg

kmandla@6m47421: ~$ tmsu files poor
./downloads/Revolution Void/Let 1,000 Flowers Bloom/01 Sympathy for Mr Vertigo.ogg
...
./downloads/Revolution Void/Let 1,000 Flowers Bloom/28 Ye Ye Ye.ogg

Those would all show up if I asked for “jazz” or “music” too, because I added those tags as well.

Now, if that didn’t grab you, here’s where tmsu suddenly changes from mild-mannered scientist into a giant green unstoppable mountain of muscle.

tmsu can mount a folder as a virtual filesystem, and create symbolic links that point back to the original file. Take a look.

kmandla@6m47421: ~$ tmsu mount temp/

kmandla@6m47421: ~$ ls temp/
queries  tags

The tags folder is probably the most obvious one: Inside that folder are all the tags you’ve created, and links back to their target files.

kmandla@6m47421: ~$ ls temp/tags/
alternative  excellent  good  hip-hop  jazz  music  poor  rap  rock  soul  very-good

kmandla@6m47421: ~$ ls temp/tags/excellent/
01 Invisible Walls.410.ogg       05 The Modern Divide.413.ogg      09 Accelerated Lifestyle.405.ogg  music
02 Factum par Fictio.404.ogg     06 Double the Daily Dose.412.ogg  10 Headphonetic.414.ogg
03 Habitual Ritual.406.ogg       07 Weekend Amnesia.408.ogg        11 Nebulous Notions.407.ogg
04 Effects of Elevation.409.ogg  08 Obscure Terrain.411.ogg        jazz

The numbers are file IDs, and the additional tag folders — “jazz” and “music” — point back to still more links. So I can sift even further into specific sequences of tags by following the folders, and for example, find “excellent/jazz/music”. 😉

That’s pretty cool, and if you have a massive collection of music or photos to manage, it will save you gobs of time and effort. But wait, there’s more. …

That other folder, queries … what about that? Let me show you first, and then we’ll try to puzzle it through.

kmandla@6m47421: ~$ ls temp/queries

kmandla@6m47421: ~$ ls temp/queries/excellent
01 Invisible Walls.410.ogg       05 The Modern Divide.413.ogg      09 Accelerated Lifestyle.405.ogg
...
04 Effects of Elevation.409.ogg  08 Obscure Terrain.411.ogg

kmandla@6m47421: ~$ ls temp/queries/
excellent

I abbreviated again, as you can see. My ls command created a folder inside queries, and inside there are links to everything that matches my request. Magical, on-the-fly searching and linking.

I haven’t tried using tmsu in conjunction with many graphical programs, but I believe you could create a folder inside queries from within a file selection dialog, and get the same results. I’d be interested to see how that works.

I haven’t found many shortcomings with tmsu; it appears to be very well thought out, and behaves in a very Unixy way. My biggest stumbling block was a way to tag an entire folder of files with spaces in their names; the spaces were confusing tmsu and my use of quotes wasn’t helping matters.

In the end I solved it in a similarly Unixy way, by drafting find into service:

find downloads/Revolution\ Void/ -type f -exec tmsu tag "{}" music jazz \;

And that was more than enough to get the job done.

I only have about 8Gb of music that I keep, and I don’t have much in the way of photos except for family snapshots. tmsu has made me think about how I store those though, and made me wonder if the endless branches of folders are really helping.

I would strongly recommend experimenting with tmsu if you have large masses of files that need quicker access than through a file tree, or even just as a way of managing a flat folder of files without resorting things into subfolders. Check it out for a few minutes at least, and see if it can make things easier for you. 🙂

stat: Simple file information, and more

I have stat on my list as a remnant of the indomitable coreutils package, and looking back I’m not sure why I held it out of the massive missive I posted a month ago. 😕

stat has a few gimmicks worth showing. Here’s its basic effort.

2014-10-27-6m47421-stat

Just stat by itself reveals a lot of information about a physical file. Its size, the number of blocks, the device ID, inode and access and modification dates are all listed there, along with ownership and access privileges. Not bad.

The -f flag tells stat to supply information about the filesystem, and not just the file. So in the second readout, you’ll see the type, the free and total blocks, free and total inodes, etc.

Of course stat has a few other flags that will allow you to fine-tune its display or even customize the arrangement, sort of like the date command. It’s worth checking out what stat can show, mostly because it’s flexible in its output.

Which, now that I think about it, is probably why I didn’t just dump it into that big list. … :\ 🙂

pscpug: Nothing to do with pugs

The world needs good, accessible system monitors. It’s just a generalized rule. I can complain about an overabundance of music players or Tetris clones, but I don’t think anyone ever gets weary of seeing a new way of viewing their system information.

pscpug is a simple vertical scrolling process monitor that displays its results as a sparse bar graph.

2014-10-23-6m47421-pscpug

It took me a while to get a decent screenshot, mostly because the applications I use are usually text-based, and it seems their process usage over time seems fairly flat. Pale Moon didn’t let me down though.

pscpug is terrifically simple, and terrifically useful. Feed it the pid of an application and you get a bar graph that refreshes at intervals, showing CPU drag. That’s all. There are only three flags — one for a different refresh rate, one to suppress its closing display of statistics, and one to switch to a generic data collection mode.

No color, but I’m willing to overlook that. No line-drawing characters, which may or may not be a good thing, depending on your system.

It’s very simple to control, very simple to read, and very simple to run. And it would look good on a shelf with top, tload or maybe nload.

And that’s all I can say about it. It does what it promises and doesn’t make a mess of things. Nice work. 😉

sysdig: Information overload

I have seen several sites and software lists that include sysdig, usually with high praise for providing an unmatched level of insight into the inner workings of a system.

And that, I cannot dispute. I can’t think of a tool that spools quite the volume of raw data — and I do mean volume and I do mean raw — as sysdig can.

2014-10-08-6m47421-sysdig

That was just a smidgin — a smidgin, I tell you — of what sysdig started piping to my terminal. Vast volumes of internal clock checks, software requests, hardware reports … you name it. Everything stamped and logged, and open for scrutiny.

In that sense, sysdig does a terrific job of giving you the aforementioned unmatched level of insight into the inner workings of your system.

My problem is, there is so much raw data and so much detail in the information, I honestly don’t know what to do with it.

Seriously: Barely 30 seconds or so of sysdig’s output, piped into a plain text file, resulted in 20 megabytes — and I spelled that out as megabytes so there wouldn’t be any confusion — of raw text data. And that’s on a 12-year-old desktop system running a smattering of desktop applications. I can’t imagine what kind of volume would appear in a proper, high-end system doing real work, like a web server.

On top of that, sysdig itself is a rather taxing application … or at least it is on the hardware I run. While sysdig is doing its thing, I get lags while typing, skipping music playback, etc., etc. It’s obvious that tracking detail at that level is imposing a serious drag on the system. Observer effect, anyone?

sysdig in Arch will build a special module that must be inserted to get sysdig to work. I didn’t try Debian yet; my Linux Mint machine is offline for a day or so, for misbehaving. (I also occasionally punish my TV for saying rude things, so this is not unexpected.)

I won’t duplicate the glowing praise that sysdig gets in other circles, just because there seems to be a heavy tradeoff in using it, from my standpoint. Yes, it will let you see every clock sync between your hardware and software, and you’ll see exactly when the downbeat of “Bring da Ruckus” leaves your music player and departs your speakers.

But you’ll need a degree of power to keep both the system and sysdig afloat at the same time. I’m guessing if your machine predates the Pentium 4 generation, you might have trouble with that. Just so you know. 😐

lsp: Think of this as “ls, plus”

One of the under-the-radar winners of the past year was sl. No, not that sl. The other sl. The one nobody seems to know about because of the ASCII train. 🙄

sl attempted to convey a little more information than the ordinary ls command, by adding color and groups and a few more sparkly bits.

lsp is another attempt, and as you can see, it has a firm grasp on the idea.

2014-10-03-6m47421-lsp-01 2014-10-03-6m47421-lsp-02 2014-10-03-6m47421-lsp-03

By default, lsp aligns file names and file types down the center of the screen. If you add the -l flag, you’ll get the same arrangement but with separation between groups. Add -s and you get human-readable file sizes. -t shows timestamps. And so on.

I do admit I like lsp for smart use of color, and for a more readable arrangement on the screen. No more using my finger to trace across the display, looking for a file size in a long list. And if you don’t like that centered style, there is a flag to left-align the whole business.

lsp also does you the favor of telling you how many files are inside directories, totals for files and directories, resizing itself to smaller or oddball terminal dimensions, and a few other nice touches. It is obviously the product of some forethought.

For negative points, I feel a little bit guilty that a large amount of screen space seems left open by default. ls -lha --color seems to pack a lot more information into a smaller space, whereas lsp will need a vertical dimension, primarily.

Along the same lines, combining flags — particularly lsp -lt imbalanced the centering across the screen, ruining the effect. lsp should probably take into account short file names combined with long, extended timestamps, and center a little more to the left, when it’s possible.

I should mention that building lsp required installing go, but doesn’t require it to run. So if you’re not comfortable with the 238Mb (?!) that is taken up by go in the Arch version, you can rip out go and still use lsp.

lsp is not in Arch/AUR or Debian that I could find, and time stamps in the git repo suggest updates within the past few weeks. So it’s another new program for you, wandering in the wild. … 🙂

warcarrier: Looks good, almost works

I thought the craze for randomly driving around town and poking into networks was over, but maybe it’s not. This is warcarrier, an ncurses application dedicated to just that.

2014-09-20-6m47421-warcarrier

I guess I shouldn’t label wardriving as a lost art just yet, and I’m willing to give warcarrier the benefit of the doubt, given my rotten track record for network tools — particularly ones dedicated to the subtler art of network security.

So my abortive attempt above, and my general ineptitude at getting the requisite gpsd running, are no sign that warcarrier is deficient. At least a I hope not. The screenshots on the home page are quite promising.

I should note that what you see up there is, of course, in Arch Linux, built off the warcarrier-svn package from AUR. The home page has instructions based on a Debian version apparently, but I also see that the last update to the svn trunk was about a year and a half ago.

I don’t know if any of those things contributed to my botched attempts, but I suppose they’re worth mentioning if you hit the same problems.

Aside from all that, warcarrier looks good, and seems to have a command of the task at hand. Even if I don’t.

As a side note, I see references to a warcarrierOS, but I can’t seem to find any download links, either on the home page or elsewhere. Perhaps it was an idea that didn’t come to fruition. If you see it somewhere, please send me the address.

Not that I’m interested in driving around town and poking at networks. Just that it seems well done, and I can appreciate that. 🙂

httpry: Pry into your traffic

Network traffic analysis is a bit over my head, and since I generally only have single-user machines in the house, there’s no mystery about who accessed what site. For a better look at the innards though, httpry is a useful logging tool.

2014-09-12-6m47421-httpry-01 2014-09-12-6m47421-httpry-02

On the left, the tool at work; on the right, the fruits of its labor. As you can see, the log is mostly a plain-text dump of transactions, with relevant addresses and commands. In that sense, httpry is really just making a note of all the background noise that makes up your network traffic.

The home page for httpry says it’s not intended as an analytic tool, but it would be possible to perform some rudimentary filtering and screening with httpry, as you might guess from its options. There are also flags for specialized network settings, and for the daemon mode that httpry supports.

If you’re better attuned to network analysis than I, you’ll probably see some value in httpry, if only as a lightweight traffic logging tool. It can serve as the foundation for a more careful inspection, or just as a casual reminder that every interaction leaves a footprint or two. 😉

ipcalc and pipcalc: More than a one-letter difference

I’m going to lump ipcalc and pipcalc together today, for reasons that are probably immensely obvious just from looking at their names. And since neither one takes up much screen space in its default form, I can squish them both into one screenshot.

2014-09-11-6m47421-ipcalc-sipcalc

And there you have it. Both programs calculate IP information given a particular address, and return extended results. As far as I can tell neither one needed actual network access to accomplish that feat, so I believe they could be of use on an offline machine too.

And it’s likewise obvious from looking at that screenshot, that there are some intricacies in the way both programs display their results. ipcalc is considerably more detailed (and colorful 😉 ), and pipcalc is considerably more succinct.

Networking always has been, and probably always will be, a weak point for me, so how either of these tools — or sipcalc or prips, for that matter — are of practical use is a bit vague.

I will leave it to those of you with more experience and more need to tell me which of these four is the most useful. Me being me, I’m most likely to just pick the one with the most color. 🙄

bmon: Hey Ridley. …

After looking at nbwmon a couple of days ago, I skimmed through my list of remaining titles to see how many network monitors were left. I found one — bmon.

2014-09-10-6m47421-bmon

I remember bmon as one of the last posts I made before the old blog went into cryogenic hibernation. That alone surprises me since it has taken a couple of years to bring it back around again.

bmon is a great monitor to just let run. If you’re troubleshooting a line, some other monitors with flashy blinky lights might give you the information you need, but bmon will break things down in much greater detail.

It can also watch multiple interfaces and maintain statistics for each, doesn’t need superuser privileges to run, and has the good sense to warn you if it needs more screen space, rather than just crashing outright. 🙄

You can’t see it in the screenshot, but there are parts of bmon that have color. And pressing “?” gives a brief help window.

I started the version you see above with bmon -o curses, which starts an interactive session. bmon has a non-interactive (?) mode as well, but I won’t even touch on that. After flipping buttons and switches in the full interface, the other doesn’t really interest me. I am so predictable.

I’m on the fence with this one, because I feel like I’ve seen it before. But bmon has all the right moves — a full-screen interface, the fine-point details, onboard help, a smattering of color and brains enough to prevent its own implosion.

Ah, heck. I’m feeling generous. Have a world-wide-famous K.Mandla gold star: ⭐ 😉

archey3 and alsi: What all the cool kids are doing

When I kept appending screenfetch with the phrase, “or something like it” earlier today, I was doing that because screenfetch has kindred in archey3 and alsi.

2014-09-09-6m47421-archey3 2014-09-09-6m47421-alsi

archey3 on the left, and alsi on the right. And as you can see, they differ only in presentation, and in the ASCII art.

Well, that’s not entirely correct, since there are some underpinnings that are different. screenfetch used scrot by default for a screenshot. archey3 uses imagemagick‘s import tool, but alsi seems content with scrot as well. All three have options that allow you to change that.

From what I can see, screenfetch might be the only one that supports distros other than Arch. I might be wrong on that point though, so don’t hold me to it.

screenfetch was a bash script, archey3 is a python program, and alsi is written in perl. And as you can see, each one pulls out slightly different information on your system — some even polling hard drives or graphics processors.

One or two take png screenshots by default, and jpg in the others (as you might be able to tell from the poor readability around the blues and grays).

And that’s about where the comparison ends. Of the three, alsi appears to be the most flexible, as far as which outside programs you can use, and what ASCII art will be displayed. Don’t necessarily take that as an endorsement though.

At this point, the question becomes one of preference: Whose art do you like best, what colors do you prefer, and what information do you want to share? A question of aesthetics, really. 🙂

P.S., yes, there is an archey and an archey2. For purposes of convenience, can we agree that this covers both of those too?