Twisted Documentation: Flappserver: The Foolscap Application Server

Foolscap provides an "app server", to conveniently deploy small applications that were written by others. It fulfills the same role as "twistd" does for Twisted code: it allows sysadmins to configure and launch services without obligating them to write Python code for each one.

Example

## run this on the server machine
S% flappserver create ~/fs
Listening on 127.0.0.1:12345
Foolscap Application Server created in ~/fs
S% mkdir ~/incoming
S% flappserver add ~/fs file-uploader ~/incoming
Service created, FURL is pb://kykr3p2hsippfgxqq2icrbrncee2f6ef@127.0.0.1:12345/47nvyzu6dj6apyrdl7alpe2xasmi52jt
S% flappserver start ~/fs
Server Started
S%

## run this on the client machine
C% echo "pb://kykr3p2hsippfgxqq2icrbrncee2f6ef@127.0.0.1:12345/47nvyzu6dj6apyrdl7alpe2xasmi52jt" >~/.upload.furl
C% flapp-upload-client --furlfile ~/.upload.furl foo.jpg
## that uploads the local file "foo.jpg" to the server
C%

## run this on the server machine
S% ls ~/incoming
foo.jpg
S%

Concepts

"flappserver" is both the name of the Foolscap Application Server and the name of the command used to create, configure, and launch it. Each server creates a single foolscap Tub, which listens on one or more TCP ports, and is configured with a "location hint string" that explains (to eventual clients) how to contact the server. The server is given a working directory to store persistent data, including the Tub's private key.

Each flappserver hosts an arbitrary number of "services". Each service gets a distinct FURL (all sharing the same TubID and location). Each service gets a private subdirectory which contains its configuration arguments and any persistent state it wants to maintain.

When adding a service to a flappserver, you must specify the "service type". This indicates which kind of service you want to create. Any remaining arguments on the "flappserver add" command line will be passed to the service and can be used to configure its behavior. For each service that is added, a new FURL is generated and returned to the user (so they can copy it to the client system that wants to contact this service). The FURL can also be retrieved later through the "flappserver list" command.

Nothing happens until the flappserver is started, with "flappserver start". This is simply a front-end for twistd, so it takes twistd arguments like --nodaemon, --syslog, etc (use "twistd --help" for a complete list). The server will run in the background as a standard unix daemon. "flappserver stop" will shut down the daemon.

Services

The app server has a list of known service types. You can add multiple services of the same type to a single app server. This is analogous to object-oriented programming: the service types are classes, and the app server holds zero or more instances of each type (each of which is probably configured slightly differently).

Service types are defined by plugins, each of which provides the code to implement a named service.

The basic services that ship with Foolscap are:

file-uploader: allow files to be written into a single directory by the corresponding "flapp-upload-client". Files are streamed to a neighboring temporary file before being atomically moved into place. The client gets to choose the target filename. Optionally allow the creation and use of subdirectories.
exec: allow a preconfigured shell command to be executed by the corresponding "flapp-exec-client". Client receives stdout/stderr/rc. Command runs in a preconfigured working directory. Optionally allow the client to provide arguments to the command. Optionally allow the client to provide stdin to the command. Optionally provide locking around the command (allow only one instance to run at a time). Optionally merge multiple pending invocations.

Commands

`flappserver BASEDIR create [options]`

Create a new server, using BASEDIR as a working directory. BASEDIR should not already exist, and nothing else should touch its contents.

"create" options:

--port: strports description of the TCP port to listen on
--location: location hints to use in generated FURLs. If not provided, the server will attempt to enumerate all network interfaces and create a location hint string using each viable IP address it finds. If you have configured an external NAT or port forwarding for this server, you will need to set --location with the externally-visible listening port.

`flappserver BASEDIR add [options] SERVICE-TYPE SERVICE-ARGS`

Add a new service to the existing server that lives in BASEDIR. The new service will be of type SERVICE-TYPE (such as "file-uploader" or "exec"), and will be configured with SERVICE-ARGS.

A new unguessable "swissnum" will be generated for the service, from which a FURL will be computed. Clients must use this FURL to contact the service. The FURL will be printed to stdout, where it can be copied and transferred to client machines. It can also be viewed later using the "list" command.

The service instance will be created lazily, when a client actually connects to the FURL. There will be only one instance per service, which will last until the flappserver is terminated. (services are of course free to create new per-request objects, which can last as long as necessary)

The "add" command takes certain options. Separately, each SERVICE-TYPE will accept one or more SERVICE-ARGS, whose format depends upon the specific type of service being created. The "add" command options must appear before the SERVICE-TYPE parameter, while the SERVICE-ARGS always appear after the SERVICE-TYPE parameter.

"add" options:

--comment: short string explaining what this service is used for, appears in the output of flappserver list

`flappserver BASEDIR list`

List information about each service that has been configured in the given flappserver. Each service is listed with the unguessable "swissnum", followed by the service-type and service-args, then any --comment that was given to the add command, finishing with the access FURL:

% flappserver list ~/fs

47nvyzu6dj6apyrdl7alpe2xasmi52jt:
 file-uploader ~/incoming --allow-subdirectories
 # --comment text appears here
 pb://kykr3p2hsippfgxqq2icrbrncee2f6ef@127.0.0.1:12345/47nvyzu6dj6apyrdl7alpe2xasmi52jt

jgdqovf3tfd5xog34bxmkqwd3dxgycak:
 file-uploader ~/repo/packages
 pb://kykr3p2hsippfgxqq2icrbrncee2f6ef@127.0.0.1:12345/jgdqovf3tfd5xog34bxmkqwd3dxgycak

22ngipsyp2smmgguemf5hu45prz4jeui:
 exec --directory ~/repo make update-repository
 pb://kykr3p2hsippfgxqq2icrbrncee2f6ef@127.0.0.1:12345/22ngipsyp2smmgguemf5hu45prz4jeui

%

Create a new server, using BASEDIR as a working directory. BASEDIR should not already exist, and nothing else should touch its contents.

"list" options: none

`flappserver start BASEDIR [twistd options]`

Launch (and usually daemonize) the server that lives in BASEDIR. This command will return quickly, leaving the server running in the background. Logs will be written to BASEDIR/twistd.log unless overridden.

The "start" command accepts the same options as twistd, so use twistd --help to see the options that will be recognized. "flappserver start BASEDIR" is equivalent to "cd BASEDIR && twistd -y *.tac [options]".

`flappserver stop BASEDIR`

Terminate the server that is running in BASEDIR. This is equivalent to "cd BASEDIR && kill `cat twistd.pid`".

"stop" options: none

Services

`file-uploader [options] TARGETDIR`

This service accepts files from flapp-upload-client, placing them in TARGETDIR (which must already exist and be writable by the flappserver). The filenames are chosen by the client. Existing files will be overwritten. This service will never write client files above TARGETDIR. It will only write to subdirectories of TARGETDIR if configured with --allow-subdirectories, in which case the client controls which subdirectory is used (and created if necessary).

`exec [options] TARGETDIR COMMAND..`

This service invokes a preconfigured command in response to requests from flapp-exec-client. The command is always run from TARGETDIR.

"exec" options:

--accept-stdin: if set, any data written to the client's stdin will be streamed to the stdin of COMMAND. When the client's stdin is closed, the COMMAND's stdin will also be closed. If omitted, the client will be instructed to not read from its stdin, and COMMAND will not receive any stdin (the pipe will be left open, however).
--no-stdin: opposite of --accept-stdin. This is the default.
--send-stdout: if set, any data written by COMMAND to its stdout will be streamed to the client, which will deliver the data to its own stdout pipe. This is the default.
--no-stdout: if set, any data written by COMMAND to its stdout will be discarded, and not sent to the client.
--send-stderr: if set, any data written by COMMAND to its stderr will be streamed to the client, which will deliver the data to its own stderr pipe. This is the default.
--no-stderr: if set, any data written by COMMAND to its stderr will be discarded, and not sent to the client.

The numeric exit status of COMMAND will be delivered to the client, which will exit with the same status. If COMMAND terminates with a signal, a suitable non-zero exit status will be delivered (maybe 127?).

Future options will allow the client to modify COMMAND (in tightly controlled ways), and to wrap a semaphore around the invocation of COMMAND so that overlapping requests do not cause overlapping invocations. Another likely option is to coalesce multiple pending requests into a single invocation.

Clients

To talk to the services described above, Foolscap comes with two simple clients.

`flapp-upload-client [options] SOURCEFILES..`

This contacts a file-uploader service as created with flappserver add BASEDIR file-uploader TARGETDIR and sends it one or more local files. The FURL used can be passed directly to a --furl= argument, or indirectly by writing it into a file and then passing a --furlfile= argument. (the --furlfile form is useful to keep the secret FURL out of a transcript of the command being run, such as in a buildbot logfile).

The basename of each SOURCEFILE will be used to provide the remote filename. If there is only one SOURCEFILE argument, then the --target-filename= option can be used to override the remote filename. If the server side has enabled subdirectories, then --target-subdirectory= can be used to place the file in a subdirectory of the server's targetdir.

`flapp-exec-client [options]`

This contacts a command-executing service as created with flappserver add BASEDIR exec COMMAND and asks it to invoke the preconfigured command. It accepts the same --furl= and --furlfile= arguments as flapp-upload-client.

If the server was configured with --accept-stdin, the client will read from stdin until it is closed, continuously sending data to the server. If not, the client will ignore its stdin.

The client will write to its stdout and stderr as data arrives from the server. Once the server's process exits, the client will exit with the same exit code.