make argv $0 be 'tahoe', not 'twistd'

[warner]

To make it easier to identify a running tahoe node with standard tools like 'ps' and 'top', it would be awfully nice if argv[0] (aka $0) were 'tahoe'. At the moment, the way we spawn twistd means that a 'allmydata-tahoe start' command actually results in a process with an argv of /usr/bin/python /usr/bin/twistd -y tahoe-client.tac --logfile logs/twistd.log which is kind of gross.

To accomplish this, we'd need to invoke twistd directly (as a regular python function call), rather than using os.system() to spawn off a new process. This is tricky. I've done it in buildbot, so we can probably steal code from there, but it has ramifications for both win32 (#27) and for tailing-the-logfile (#71).

We've changed the name of the .tac file to contain the word "tahoe" (#156), so at least 'ps ax |grep tahoe' results in something useful now, but it would be nice to complete this effort.

[warner]

So, I stumbled across a feature in Application that might work well enough for our purposes here. By adding service.IProcess(application).processName = "tahoe"

to our .tac file, we go from an argv that looks like:

/usr/bin/python /usr/bin/twistd -y tahoe-client.tac --logfile logs/twistd.log

to one that looks like:

tahoe /usr/bin/twistd --originalname -y tahoe-client.tac --logfile logs/twistd.log

It looks kind of weird now, but 'tahoe' shows up as argv0, which is probably a good thing.

I'll do some more experimentation.

[warner]

Unfortunately, 'top' still shows the process as 'python', instead of 'tahoe'.

[warner]

Also, we'd like the node's basedir to show up in the argv list, so that 'ps ax |grep tahoe' would tell us which node is which.

[keturn]

we had to write a C extension to modify argv[0], so I'm curious if you find a cleaner way.

[zooko]

Hm, I was just playing with ​dupfilefind and I noticed that its entry in top is named "dupfilefind" instead of "python", even though its entry in "ps eax" is:

 8571 pts/5    D+     9:43 /usr/bin/python /usr/bin/dupfilefind --profiles .

[zandr]

Upon reflection, I think I want the nickname in argv for nodes that have them, and the base class (cpu-watcher, stats-gatherer, et al) for those that don't.

This was inspired by the behavior of Munin, which shows the hostname of the target of each munin-update process.

[zooko]

Since the Milestone is "undecided" and nobody is actively working on this ticket, it doesn't seem right to have its priority set to "critical".

[zooko]

If it *is* still of critical importance to a user of Tahoe-LAFS, such as Zandr, then please say so and we'll see if someone actually wants to spend time fixing it.

[zooko]

<exarkun> zooko: well, uh, how about setting processName to tahoe?

[zooko]

<hagna> zooko, exarkun: see http://jcalderone.livejournal.com/30438.html

[warner]

This might have been mostly fixed with the recent import+call+twistd.run() change. At least argv[0] should now be 'tahoe' instead of twistd. If you passed an explicit basedir (i.e. tahoe start FOO), then I think FOO will appear in your ps args.

Anyone want to check?

[francois]

Replying to warner:

Anyone want to check?

The cmdline as shown by ps ax looks fine but the process name displayed by top remains python.

$ ps ax | egrep [t]ahoe
21233 ?        S      0:03 /usr/bin/python /home/francois/dev/tahoe-upstream/support/bin/tahoe start /home/francois/.tahoe
$ top -b -n 1 -p 21233
top - 16:33:40 up 1 day, 23:01,  7 users,  load average: 0.56, 0.46, 0.39
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  5.3%us,  1.6%sy,  0.0%ni, 92.4%id,  0.7%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2056044k total,  1985072k used,    70972k free,    54296k buffers
Swap:  6024336k total,    45356k used,  5978980k free,   506808k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                        
21233 francois  20   0  210m  83m 2620 S    0  4.1   0:03.00 python                                                                         

$ cat /proc/21233/status | grep Name
Name:	python
$

[warner]

Hm, I guess most interpreted-language programs (those with #! lines) will show up that way. Should we change it? That is, do folks think it's useful to know which version/instance of python is being used to run your node? I suppose if it helps you reconstruct the running program, it might be, although then there may be other ambiguities involved (i.e. if you just run "tahoe start", such that it searches your $PATH for "tahoe", will "ps" show you which one it found?).

I imagine we might be able to have the process rearrange its sys.argv array to try and change what ps sees, but I'm not 100% sure that it'd be a good idea..

[davidsarah]

Here's what twistd does to change argv[0]: ​http://twistedmatrix.com/trac/browser/trunk/twisted/scripts/_twistd_unix.py?rev=27324#L173

(name is given by twisted.application.service.IProcess(application).processName.)

This requires execv and so is only done on Unix. (I don't think there's any way to change how the Processes tab of Task Manager lists processes on Windows.)

[davidsarah]

Reducing priority to minor, since the commandline now does give all significant information.

[warner]

Replying to davidsarah:

Here's what twistd does to change argv[0]: ​http://twistedmatrix.com/trac/browser/trunk/twisted/scripts/_twistd_unix.py?rev=27324#L173

Huh. That suggests that there isn't any way to affect argv from python (i.e. sys.argv is a one-time copy of the C data structure, and changes are not propagated back). Or at least that the Twisted devs decided it wasn't worth trying to use such a feature.

I'm not sure if that extra exec() is going to cause us problems. We're planning to set up a fairly specific environment before calling twistd.run(), and I don't think that re-execing the current contents of argv are likely to keep that same state around. It'll need some experimentation.

[davidsarah]

bin/tahoe-script.template: On non-Windows, invoke support/bin/tahoe directly as a script (rather than via python), so that 'top' for example will show it as 'tahoe'. On Windows, simplify some code that set argv[0], which is never used. fixes #174

[davidsarah]

I agree that an extra exec isn't worth it. However I think that on Unix we can invoke the support/bin/tahoe script directly, rather than via python, which should achieve the same effect. (That script's shebang line should point to the same Python interpreter as the bin/tahoe script, since both should be the full path to the Python that ran setup.py build.) Please repeat the checks in comment:15 to confirm.

I don't think we need a test for this, since the existing tests should catch any regressions. (It would be difficult to test, since sys.argv[0] is the same for a script invoked directly or via python.)

[warner]

davidsarah: applying your patch on my OS-X box and then using 'tahoe start' to launch a couple of nodes gives me 'ps aux' output that looks the same as it did with current trunk:

185:warner@host-4-50% ps aux |grep tahoe
warner   50886   0.0  0.4  2480848  17332   ??  S    11:22PM   0:00.15 /usr/bin/python /Users/warner2/stuff/tahoe/t2-174/support/bin/tahoe start ../MY-TESTNET/node-3

"top" on OS-X just shows "Python" in the COMMAND column. (OS-X's top is a bit weird. Actually, all OS-X unix tools are a bit weird. sysv vs bsd, I guess.)

Is that what was supposed to happen?

(if it did the right thing, I suppose I'd r+ the patch, OTOH I get a deep sense of dread when seeing that weird .template file get changed, and it raises my "this file shouldn't even exist" hackles. I'd be much more keen to r+ a patch that deleted it altogether :-)

For the record, I'll be happy if the 'ps aux' output includes "tahoe", since "ps aux |grep tahoe" is what I always do to find out if a tahoe process is currently running. And I'm also happy if the argv array includes the target directory. Both of these conditions appear to be true right now.

[davidsarah]

Replying to warner:

davidsarah: applying your patch on my OS-X box and then using 'tahoe start' to launch a couple of nodes gives me 'ps aux' output that looks the same as it did with current trunk:

185:warner@host-4-50% ps aux |grep tahoe
warner   50886   0.0  0.4  2480848  17332   ??  S    11:22PM   0:00.15 /usr/bin/python /Users/warner2/stuff/tahoe/t2-174/support/bin/tahoe start ../MY-TESTNET/node-3

"top" on OS-X just shows "Python" in the COMMAND column. (OS-X's top is a bit weird. Actually, all OS-X unix tools are a bit weird. sysv vs bsd, I guess.)

Is that what was supposed to happen?

No, it was supposed to change the command line to just "/Users/warner2/stuff/tahoe/t2-174/support/bin/tahoe start ../MY-TESTNET/node-3". Note that python setup.py build is needed, I should have mentioned that.

[davidsarah]

The patch doesn't help on OS X. Can someone try it on Linux?

[francois]

Replying to davidsarah:

The patch doesn't help on OS X. Can someone try it on Linux?

This patch was tried on my Ubuntu 10.10 box. The output of the ps command did not changed but the output of the top command did changed.

In both cases.

francois@korn:~$ ps aux |grep tahoe
francois  6778  1.3  4.0 213176 82868 ?        S    22:36   0:01 /usr/bin/python /home/francois/dev/tahoe-upstream/support/bin/tahoe start
francois  6873  0.0  0.0   9212   844 pts/4    S+   22:37   0:00 grep tahoe

Without the patch.

francois@korn:~/dev/tahoe$ top -b -n 1 | egrep "python|tahoe"
 7208 francois  20   0  208m  81m 2620 S    0  4.0   0:00.97 python             

With it.

francois@korn:~/dev/tahoe$ top -b -n 1 | egrep "python|tahoe"
 7354 francois  20   0  208m  80m 2580 S    2  4.0   0:00.92 tahoe              

[davidsarah]

francois said on irc:

yes I read the code and haven't found any obvious flaw in it

[david-sarah@…]

In 9815852a09582776:

bin/tahoe-script.template: On non-Windows, invoke support/bin/tahoe directly as a script (rather than via python), so that 'top' for example will show it as 'tahoe'. On Windows, simplify some code that set argv[0], which is never used. fixes #174

[david-sarah@…]

In 1190ce614303b6fb:

NEWS: 'top' for node processes, WUI formatting, removal of GUI apps, documentation updates, foolscap dependency. refs #174, #1219, #1225