[tahoe-lafs-trac-stream] [Tahoe-LAFS] #1338: reduce CLI startup overhead by importing less

[Tahoe-LAFS]

#1338: reduce CLI startup overhead by importing less
-------------------------+-------------------------------------------------
     Reporter:           |      Owner:
  davidsarah             |     Status:  new
         Type:  defect   |  Milestone:  undecided
     Priority:  major    |    Version:  1.8.1
    Component:  code-    |   Keywords:  performance extensibility packaging
  frontend-cli           |
   Resolution:           |
Launchpad Bug:           |
-------------------------+-------------------------------------------------
Changes (by lpirl):

 * cc: tahoe-lafs.org@… (added)


Old description:

> {{{bin/tahoe}}} takes about 3.5 seconds to perform a trivial command
> (e.g. {{{bin/tahoe --help}}}) on my machine. Most of this overhead is due
> to importing dependencies: [source:src/allmydata/__init__.py] imports
> ''all'' of Tahoe's dependencies, in order to check their versions.
>
> Just removing the calls to {{{get_package_versions_and_locations()}}} and
> {{{check_all_requirements()}}} does not work; we end up importing most of
> these libraries anyway. However, most of the CLI commands that do not
> start a node, ''should not'' need to import anything except
> {{{twisted.python.usage}}}.
>
> By copying just the files that are needed for the CLI to work at all, and
> removing imports that are not really necessary, I was able to get the
> time to run {{{bin/tahoe --help}}} on my machine to 1.1 seconds, i.e. a
> factor-of-3 improvement. This is just a proof of concept at the moment,
> but I think this improvement could also be obtained for the real CLI.
> Note that just running a script that prints out the help text takes 0.8
> seconds, due to the startup overhead of the Python interpreter.
>
> Another major advantage of doing this is that the resulting CLI could be
> embedded as a library, without pulling in any of Tahoe's dependencies and
> their associated packaging woes.

New description:

 {{{bin/tahoe}}} takes about 3.5 seconds to perform a trivial command (e.g.
 {{{bin/tahoe --help}}}) on my machine. Most of this overhead is due to
 importing dependencies: [source:src/allmydata/__init__.py] imports ''all''
 of Tahoe's dependencies, in order to check their versions.

 Just removing the calls to {{{get_package_versions_and_locations()}}} and
 {{{check_all_requirements()}}} does not work; we end up importing most of
 these libraries anyway. However, most of the CLI commands that do not
 start a node, ''should not'' need to import anything except
 {{{twisted.python.usage}}}.

 By copying just the files that are needed for the CLI to work at all, and
 removing imports that are not really necessary, I was able to get the time
 to run {{{bin/tahoe --help}}} on my machine to 1.1 seconds, i.e. a factor-
 of-3 improvement. This is just a proof of concept at the moment, but I
 think this improvement could also be obtained for the real CLI. Note that
 just running a script that prints out the help text takes 0.8 seconds, due
 to the startup overhead of the Python interpreter.

 Another major advantage of doing this is that the resulting CLI could be
 embedded as a library, without pulling in any of Tahoe's dependencies and
 their associated packaging woes.

--

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1338#comment:2>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage