lancat: Zero-configuration network transfer option

Quick on the heels of pyncp, I got a short note about Graham Edgecombe’s lancat.

 

As you might imagine, lancat works a lot like pyncp, with a couple of exceptions. Yes, it’s intended as another fire-and-forget, zero-configuration network transfer option. And as you can see in the pictures (which I snapped a little too early), one machine “sends,” and another “receives,” and they listen for one another in the mean time.

A couple of small differences though. And not just that lancat is written in ruby, and pyncp was in python (and ncp was in C).

First, lancat is apparently desingned to work like a pipe, with your source file redirected into it on the sending end, and out of it on the receiving end. This might lend itself to use in scripts or with other tools.

Second, lancat’s function is determined by a flag, not by interpreting a command in sequence. That might prove more convenient, or at least quicker than typing out a full command for pyncp/ncp to interpret.

Aside from that, lancat works much as might be expected, but comes with the same caveats as pyncp did: There is no provision for security (or compression) unless to force it before and after.

But there are tools for that, and given that lancat handles pipes and redirects well, it shouldn’t be a big issue. 😉

pyncp: Quick and easy network transfer

This will be quick and easy today, since pyncp is a clean and straightforward tool.

If you remember waaay back, almost two years ago, we looked at ncp as a quick alternative for network file transfer. pyncp accomplishes much the same thing, but apparently sticks to python for its magic.

There are no exquisite flags for pyncp, but the program has to be installed on both the source machine and the destination. One machine “pushes” a file with pyncp push file.txt, and the other “polls” for it with pyncp poll. You don’t need superuser privileges or passwords — you don’t even need to know how to manage the network. One pushes, one pulls. Or polls, I guess I should say.

That’s it. If you want an encrypted transmission, I believe you might have to do that manually, and decrypt after arrival. Same goes for compression, but if you’re managing a series of file transfers over an open network, you should be doing those things manually anyway.

I only tested pyncp over my home network, so I have no guarantees how it will behave out in the wide open world. pyncp is in AUR as pyncp-git; my Debian search turned up nothing.

rsync: Needs no introduction

I don’t think there’s much I can say about rsync that isn’t already common knowledge or preaching to the choir.

kmandla@6m47421: ~/downloads$ rsync -ah --progress source/ destination/
sending incremental file list
./
sample-01.txt
            925 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=9/11)
sample-02.txt
            835 100%  815.43kB/s    0:00:00 (xfr#2, to-chk=8/11)
sample-03.txt
            892 100%  871.09kB/s    0:00:00 (xfr#3, to-chk=7/11)
sample-04.txt
            901 100%  879.88kB/s    0:00:00 (xfr#4, to-chk=6/11)
sample-05.txt
            893 100%  872.07kB/s    0:00:00 (xfr#5, to-chk=5/11)
sample-06.txt
            900 100%  878.91kB/s    0:00:00 (xfr#6, to-chk=4/11)
sample-07.txt
            886 100%  865.23kB/s    0:00:00 (xfr#7, to-chk=3/11)
sample-08.txt
            832 100%  812.50kB/s    0:00:00 (xfr#8, to-chk=2/11)
sample-09.txt
            883 100%  862.30kB/s    0:00:00 (xfr#9, to-chk=1/11)
sample-10.txt
            888 100%  433.59kB/s    0:00:00 (xfr#10, to-chk=0/11)

kmandla@6m47421: ~/downloads$ 

rsync is, was, and has been one of my favorite tools for a very long time, and short of single-file, one target copies, it’s the one thing I use to copy, backup, synchronize or just plain double-check.

rsync works across networks, across directories and within file trees. It gives clean progress indicators, can run completely silent, can delete files that aren’t in the source folder, and will avoid updating files that don’t exist in the destination. Just tell it what you want.

I think that will do for now. Like I said at the start, if you know it, there’s no point in me gloating over it. And if you don’t … waste no time in trying it out. 😉

pssh: Still more parallelized tools

I think so far, every parallelized tool I’ve discovered in the P section has been new to me.

pssh is new to me too, even if it dates back to at least 2009, if not further.

pssh is a collection of ssh-oriented tools written in Python and mimicking a lot of the standard openssh-style fare. There is a strict pssh application, a psshscp tool for scp-ish adventures, a prsync utility and some others, along with a library to assist with creating new tools.

My escapades with the pssh tools was a little less than successful, something I am always willing to blame on myself first.

Part of my difficulty may lie in that the flag options for prsync (and I use that only as an example; some of the other tools also gave difficulty) are very different from vanilla rsync. prsync, as best I can tell, also demands that you declare a host and a user in the command or face error messages.

The odd thing being, if I tried to just sync two folders in my home directory, a la

prsync -r -h kmandla@127.0.0.1 -l kmandla source/ /home/kmandla/dest/

I met with an error exit code of 255 — which I can’t seem to track down in the man pages or on the web site.

Some other issues too; there were slight inconsistencies in the documentation. The man page for psshscp is titled “pscp.” The man page for pslurp says it’s an application to kill parallelized processes, but the extended description talks about copying and source and destinations and so forth. I admit I was confused. (All this in the Arch version, by the way.)

And beyond that — and I’m not afraid to display my ignorance here — I’m not sure if “parallelized” means “optimized for multiprocessor machines,” as was the case with pbzip2 and pigz, or “optimized for high bandwidth connections,” since most of these are aimed at networking tasks. For what I can tell.

It seems it should be the latter … or at least that’s what I’d be looking for. I’m probably splitting hairs here, but I can say that most of my bottlenecks when I use things like rsync or ssh are not at the processor. And that’s all the more I’ll say, at risk of embarrassing myself.

I’ll let you give them a try, and see if they behave any better for you, or if their focus is a little more clear. It’s good to know they’re available, and maybe they’ll brighten someone’s day. 😉

mmv: Nifty, but takes some getting used to

Mass-rename and mass-move tools are a bit tricky sometimes. I seem to recall having problems even with Windows tools years ago, where Explorer tried to rename all the files I had selected to the same name. That didn’t go over well.

Point being, aside from qmv and the rest of rename-utils, I don’t see many multiple file move tools. mmv is one of the rare ones though.

mmv gives its name to a quadfecta of tools — mmv, plus mcp, mad and mln. You can probably figure out what each one corresponds to, among Unix-ish tools.

mmv is a little cryptic at first, but once you get the hang of it, it works pretty well. The man pages are useful, but more useful for me was this blog post that showed examples. (I’m an assimilator. I follow examples and then break away to do follow my own style.)

To be honest, mmv won’t replace rename-utils for me. I can see where it might be preferable, for simpler, faster moves with quicker patterns and substitutions.

For my money, the real charm of qmv is being able to pipe everything through a text editor, and really get my hands dirty.

Looks like we’re about half way through the M section. There’s hope for me yet. 😐

gcp: Seemingly worthy, possibly flawed

I love being able to see what my computer is doing. What most often frustrates me is when a computer is apparently busy, but gives me no feedback to be sure.

This was the issue a few months ago, when I talked about Advanced Copy. gcp scratches that itch for me too.

And I would speak of gcp in the same glowing terms I used with Advanced Copy, except for one strange issue: I get error messages when I try to use this in an X-less environment. 😯

For posterity’s sake, or for debugging purposes, this is the error message that appears in a virtual console.

Traceback (most recent call last):
  File "/usr/bin/gcp", line 678, in 
    gcp = GCP()
  File "/usr/bin/gcp", line 205, in __init__
    raise e
dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NotSupported: Unable to autolaunch a dbus-daemon without a $DISPLAY for X11

It works fine in a terminal emulator in a graphical environment. But shifting to a console gives that error. I’m not good with debugging, but that’s a little sad.

Perhaps the issue is in the dependencies. Still, I’d sure hate to think that something as simple as a copy tool with a progress bar would rely on all of X to get the job done.

More than likely I have done something wrong. That’s usually the case. Yes, let’s blame it on K.Mandla. 😳

And I see a package called gcp in Debian, and that might be the same. Perhaps it works. Let me know if it does.

I looked for a bug tracker or some sort of feedback mechanism, but I don’t know if it’s available. In the mean time I’ll stick with Advanced Copy, even though gcp was quite promising.

dd, df and du: Pop quiz

Here are three tools you probably already know, if you’ve spent any time at the terminal since picking up Linux: dd, du and df.

All three come from coreutils, and I’m betting they’re installed on your system.

And in the interest of … being interesting, let’s turn this around and you give the answers.

1. Show the amount of disk space that’s free, in units that are easily readable by (normal) human beings. Bonus: Show only one type of filesystem.
2. Create an image of a floppy disc. 🙂 Bonus: Show the results on-screen … in hex. 😈
3. Show the size of folders in the /etc directory, in numbers that are easy to read, and include a total for the entire directory. Bonus: Show only the top ten, sorted highest to lowest, but no total.

You’re allowed to use external programs to finish the bonuses. 🙂

Like I said, you probably already know these tools and even if you can’t come up with answers on your own, you have the entire Intarnets at your disposal. Consider it an open-book test. 😉

Advanced Copy: Sorely needed, for a long time

I’ve had a silent, long-standing complaint about some of the core utilities in Linux. Where some are occasionally unexpressive, others are downright antisocial.

cp and mv are the main culprits. Even with their “verbose” flags, what you get is little more than one file name pointing at another. That’s no improvement.

Advanced Copy is very much an improvement though.

That’s the way it should be! That’s what I’m after: Transfer rates. File sizes. A proper progress bar and percent complete, with a second readout for overall progress, as needed. 😀

Now before all the Unix purists come along and bite my head off, let me just say there are many tools in the box, and this one I prefer. I know about pv and other piping toys, but I’d rather just work with this.

Fair enough? 😉