Ticket #4104: TahoeMeetings20240924.md

File TahoeMeetings20240924.md, 19.9 KB (added by blaisep, at 2024-09-25T15:15:36Z)

MeetingNotes?-2024-09-24

Line 
1---
2title: Tahoe-LAFS - Nuts & Bolts Meetings
3
4---
5
6# Tahoe-LAFS - Nuts & Bolts Meetings
7[Location](https://meet.jit.si/AccurateFiltersManipulateBefore)
8[Legacy meeting notes](https://tahoe-lafs.org/trac/tahoe-lafs/wiki/WeeklyMeeting)
9[Consolidation of notes](https://tahoe-lafs.org/trac/tahoe-lafs/ticket/4104)
10
11## Sep 24, 2024
12### Attendees
13
14- Blaise
15- B3n
16- Chris
17- hacklschorsch
18- Jeffhimself
19- Meejah
20- Pete
21
22### New Items:
23
24- The new tutorials and docs will default to systemd in examples for running as a service.
25- We'll start migrating issues. @b3n will provide a summarized plan next week. If necessary, we'll migrate to a self-hosted forgejo instanc.
26
27#### Migration: User Management topics
28- Account creation flow
29    - Auto-registration OFF,
30    - User Creation
31        - Auto create if Github auth'd
32        - Manual
33- Forgotten password flow (eg. Github IdP is down)
34
35#### Tahoe-S3
36- A client which maps the finger tree into S3 numbers (and deal with S3 size limits)
37- Finger tree used to handle failure scenarios
38- Immutables only
39- Doesn't create the shares
40- Doesn't replace the native Tahoe storage server
41- A challenge to implement in Haskell because it requires intimate knowledge of Tahoe internals, REST and Foolscap APIs
42
43#### ADR: Tech stack decisions
44- Whither Haskell?
45    - How much functionality is already implemented?
46    - Haskell is a good high-level language.
47- Whither Rust?
48    - If we build *low-level* components in Rust, there's opportunity for re-use.
49    - Rust has lots of tooling
50
51#### Improving Immutables
52- Single block writers are the problem (@ccx)
53- blockless multi-writer implementation is the answer
54
55Speaking of block storage, @blaisep wonders if [k8s longhorn](https://longhorn.io/docs/1.7.1/references/longhorn-client-python/) integration may be a crowd pleaser.
56
57
58## TILs
59
60- Duplicity has a backend option [`--tahoe`](https://fossies.org/linux/duplicity/duplicity/backends/tahoebackend.py)
61- Python pre-compiled for [iOS](https://github.com/PyO3/maturin/issues/1742#issuecomment-1686219705), courtesy of [maturin](https://www.maturin.rs/tutorial)
62- Room for improvement in describing our goals, roadmap milestones, features, etc. ("implement Tahoe-S3" vs "have a phone app")
63
64
65---
66
67## Sep 17, 2024
68### Attendees
69
70- Blaise
71- B3n
72- Pete
73
74### New Items:
75
76#### Trac migration
77The time has come. @b3n requests that next week we set in motion the steps to leave trac and use .... something else. (Eventually, we may use GitHub, but before we can migrate to GitHub we need to clean up the issue data anyway).
78
79#### Provisioning and configuration management
80On a ddifferent note, most of the meeting was a discussion comparing provisioning and config management tools. @pete has a project where he needs to produce packaged images for various distributions. Ultimately, the images will run containerized applications.
81This is an area with too many choices.
82
83Some of the resources that came up in the discussion may be interesting to others:
84- [DeBock's Ansible Roles](https://robertdebock.nl/ansible.html#roles). These are tested, [using molecule](https://ansible.readthedocs.io/projects/molecule/), against several Linux distos and versions.
85- [Packer](https://www.packer.io/) from HashiCorp, takes the "images as code" approach. Concerns about a terraform-style licensing issue came up.
86- [Distrobox](https://distrobox.it/) to create containers with custom configurations.
87- [Toolbx](https://containertoolbx.org/) is a wrapper for [podman](https://docs.podman.io/en/latest/) and can be considered as a simpler [alternative to distrobox](https://discussion.fedoraproject.org/t/thread-about-switching-to-distrobox/89151/18?page=2)
88- Finally, the consensus is to create a `justfile` using [Just](https://just.systems/man/en/) to control the tasks and invoke tools like packer and ansible when necessary. Also, consider the trick that the [Fedora Atomic Desktop](https://getaurora.dev/) use to alias `ujust` to point to [admin-provided tools](https://universal-blue.discourse.group/t/just-reorganization-part-1-released/417#automation-and-scripting-3).
89- We agreed that [dagger engine](https://docs.dagger.io/quickstart/#topics) would be useful for interacting with the running containers, but this was out of scope.
90
91---
92
93## Sep 10, 2024
94### Attendees
95
96- Blaise
97- hacklschorsch
98- Chris
99- Ccx
100
101### Step-by-step approach to getting started
102Instead of [showing everything at once](https://imgflip.com/i/930yve), build up the service one feature at a time. @blaisep  has Incoming PR for https://tahoe-lafs.org/trac/tahoe-lafs/ticket/4103
103
104
105### Express modularity using C4 notation for the diagrams
106- layers of abstraction Context, Container, Component, Code
107- [PlantUML includes](https://crashedmind.github.io/PlantUMLHitchhikersGuide/) C4 in the std lib
108- https://crashedmind.github.io/PlantUMLHitchhikersGuide/C4/C4Stdlib.html
109
110### Describing Tahoe to local user groups
111As prep for presenting to larger conferences
112@blaisep offers to do user group talks (later: conference presentations)
113- Historical context (eg. why did they create SDK first? Surveillance capitalism, enshitification, etc.)
114- Use case examples (creating examples relevant to end users)
115    - The "dumb pipe scenario" (provider-independent security), use cloud storage for personal data.
116    - Storage only is low value
117
118#### Challenges
119Either gaps in functionality or gaps in understanding
120- Describing the security boundaries (eg.vs corp walled gardens)
121- Key management, rotattion, revocation
122- The "dumb pipe scenario" (provider-independent security), use cloud storage for personal data.
123- No generic writer( @ccx ) can't write everything to/from everywhere.
124
125#### Comparisons
126Describing Tahoe in terms of something they already know.
127- Nostr:  @chris has written some [Nostr](https://nostr.org/) bots. It's not a dumb pipe in a security sense. It's meant to be censorship-resistance, not confidentiality. More of a brute force approach vs lots of cryptography. @hacklschorsch liked the way the Nostr delivery model (push data to your favorite relay...) is presented.
128Flo said NIP-104 will be a big improvement for private messaging
129https://github.com/nostr-protocol/nips/pull/1427
130 
131- IPFS: similar layer of abstraction. More client implementations. Scales differently? https://news.ycombinator.com/item?id=37750529
132- Signal: but for storage ( _can Signal self host?_ ) The reference implementation is a useful app, SDKs, Open protocol, ubiquitous.
133- Syncthing: (peer-to-peer) but with decentralization and capabilities (@ccx)
134
135---
136
137## Sep 3, 2024
138### Attendees
139
140- b3n
141- Blaise
142- hacklschorsch
143- Meejah
144- Ccx
145
146
147## Prior Items
148
149- Glossary items
150
151- Debian packaging maybe delayed because `pycddl` has never been packaged for Debian. We might make it in by the end of the year. @hacklschorsch
152    - (we should check if any of our other dependencies have never been packaged for Debian @meejah)
153
154- Trac migration (see: https://tahoe-lafs.org/trac/tahoe-lafs/wiki/MoveOffTrac#ProsandConsofself-hostedvsSaaS) looking at the results of migration, @b3n noticed a number of issues with author attribution and broken reference links.
155  - @b3n suggests that we consider a transition period where we run selfhosted and collaborate on the issues we care to migrate. (eg. the user creation needs to be automatic, yet if we set it to auto, then we also get junk accounts created) We can choose a threshold of noise and migrate or wait until it is perfect.
156  - "Our trac instance has defferred maintenance and accummulated cruft over the years. The community wants to retain the option to selfhost, even if we choose not to. So if it makes sense to self host, then we can exercise that option." @meejah
157
158- @hacklschorsch: Can we lower the Tahoe-LAFS merge process turnaround time?
159  - Optimistic Merging / other ideas for the Tahoe-LAFS GitHub repo? to make contributing leaner/more fun
160  - meejah: A PR that is green and has approvals is already merged quickly now.  If not, write on IRC
161  - Currently, meejah is the only active committer in Tahoe-LAFS.  The committers agree it would be good to have more committers, but it's not an easy task. Blaise nominates ccx, but ccx does not have time right now.
162
163
164## New items
165
166- Veilid: is it real, where are the intersections?
167
168@ccx : it's a library which bundles a number of p2p protocols. No existing app yet.
169
170@meejah : played with ethersync and he made it do a thing. It's early and doesn't do anything yet. For instance, no one would install MagicWormhole if they couldn't do file sharing.
171
172### Making it easier to develop on Tahoe
173Separation of concerns, breaking tahoe into modules would be a good next move. @ccx (@blaise start a trac issue?)
174
175---
176
177## August 27, 2024
178### Attendees
179
180- b3n
181- Chris
182- hacklschorsch
183- Meejah
184- Pete
185
186### Updates
187
188- Flo
189  - LA wants to hire a senior Haskeller to work on LAFS
190  - iOS: Obsidian is still investigating the "illegal instruction" compiler/linker issue
191  - Wrote some eMails back and forth with the Debian Tahoe-LAFS maintainer: Would be great if 1.20 would be clean of Python 2 code so it can be included in Debian again.  Also need to look at Debian Trixie code freeze date
192  - Works with Blaise on Tahoe-LAFS docs
193  - ... and would like to pair more in general
194 
195- Chris
196  - published a Rust crate for LAFS: https://crates.io/crates/lafs 
197 source repo: https://github.com/crwood/lafs
198  -  Local and much faster than the Python equivalent (.25s instead of 24s on Chris' machine)
199  -  It comes with Python bindings
200
201- b3n
202  - Codeberg importing issue acknowledged by them
203  - Would like to discuss self-host vs. SaaS for the upcoming GitTea with Meejah
204
205- Meejah
206  - Magic Wormhole protocol work
207  - FOWL Policy & Permissions API mostly done, soon to be merged & released.
208  - [Pear-On](https://sr.ht/~meejah/pear-on/) should work ATM - testers welcome
209
210
211## Previous items
212
213@blaisep Now owns [17 trac tickets from the backlog](https://tahoe-lafs.org/trac/tahoe-lafs/query?owner=blaisep&status=assigned&status=new&status=reopened&id=&keywords=~&reporter=~&col=id&col=summary&col=keywords&col=reporter&col=status&col=owner&col=type&col=priority&col=milestone&col=changetime&desc=1&order=id) which are addressed in this month's effort to produce the getting started kit (docs and Artifacts)
214
215
216## New Items
217
218- Trac [#4103](https://tahoe-lafs.org/trac/tahoe-lafs/ticket/4103#ticket) is a running summary of the items related to the Getting started guide.
219- Trac [4102](https://tahoe-lafs.org/trac/tahoe-lafs/ticket/4102#ticket) describes a workaround for a problem with one of the transitive dependencies (`eliotutils.py`).
220- @blaisep has limited availability until Sep 30 2024
221
222
223---
224
225## August 20, 2024
226### Attendees
227
228Blaise
229b3n
230ccx
231Chris
232hacklschorsch
233
234## Previous items
235
236### WTF Circle CI?
237* Is it broken? Maybe drop it?
238* Do we have much logic in the Circle CI?
239* It might be tests failing, rather than a systemic fail (credentials, dependencies, etc)
240* "AMP server"? (twisted RPC framework) @ccx
241* * https://app.circleci.com/pipelines/github/tahoe-lafs/tahoe-lafs/4958/workflows/d16324ae-293d-45c8-ba97-6ac39ef4cfc4
242* https://app.circleci.com/pipelines/github/tahoe-lafs/tahoe-lafs/4958/workflows/d16324ae-293d-45c8-ba97-6ac39ef4cfc4/jobs/86344
243
244
245### Towards a portable CI
246* keep logic in the repo
247* avoid huge Nix dependency (or maybe build a weekly Nix image)
248* https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/github_integration.html
249
250
251
252### Describe the build process
253* add prose in the dev guide to the files in the repo describing the build (@blaise)
254* update the release checklist: https://tahoe-lafs.readthedocs.io/en/latest/release-checklist.html
255
256### Mutables, Service of Happiness
257* @blaise asked where we were at with mutables...
258* using mutables as single writers (eg. Tahoe backup)
259* It's a big thing, eventually replace with a proper design for a multi-writer primitive
260* For now, stick with readonly or Magic Folder for examples and getting started guide.
261* > Mutables are not just even single-writer safe because partitions can cause rollbacks. Shouldn't be super common but it's still real world error case. @ccx
262
263### [MoveOffTrac](https://tahoe-lafs.org/trac/tahoe-lafs/wiki/MoveOffTrac) - [#4095](https://tahoe-lafs.org/trac/tahoe-lafs/ticket/4095)
264* Trac dump from last month has been migrated on the self-hosted Forgejo [instance](https://forge.lafs.eval.latfa.net/tahoe-lafs/trac-2024-07-25).
265* Working on user mapping: trying to figure how to link trac users to Gitea/Forgejo users, with a mapping to the Github user if different!
266* New Forgejo instance (copy/paste from Gitea) to be closer to Codeberg: https://forge.lafs.eval.latfa.net/tahoe-lafs/
267* Testing migration to Codeberg with pagination bug (workaround), performance issue (deadlock error) and user mapping (unlikely possible)
268* Keyoxide.org may help us to link all those account!?
269
270## New Items
271
272* Blaisep has trac !!!
273* TIL: Trac use "htdigest" (@meejah "i'll put a note in ~trac")
274
275---
276
277## August 13, 2024
278### Attendees
279
280Blaise
281b3n
282ccx
283Chris
284hacklschorsch
285Pete
286
287## Previous items
288
289- Blaise needs a trac account to CRUD issues, for docs, etc.
290- GitTea self register results in many spam accounts @b3n
291- We have users on Trac and users on GitHub, sometimes the names are different. @b3n
292- Trac Migration Plan [WIP](https://tahoe-lafs.org/trac/tahoe-lafs/wiki/MoveOffTrac)
293- migrated Trac eval instance (https://code.lafs.eval.latfa.net/tahoe-lafs/trac-2024-05-23/issues)
294
295## New
296- Meejah might not have admin to [github.com/tahoe-lafs](https://github.com/tahoe-lafs/tahoe-lafs/pull/1370) but so far that isn't yet a problem (eg. if Circle CI creds need updating.) @blaisep
297- Consider using Google, GitHub auth for auto register CodeBerg accounts
298- Consider upgrading/self hosting Trac instance (maybe it's not so bad after all)
299- Could we make trac readonly and do new work on the ["demo" instance](https://www.lafs.eval.latfa.net/) instance?
300
301
302### Migrating Issues: Goals and Constraints
303- Anything off of track is better
304- Github Issues are accessible, but there is risk of lock-in, enshitification
305- Gitea/Forgejo/CodeBerg is future-proof (no lock-in)
306- Inconsistent tribal knowledge about decisions made last year @blaisep et al.
307
308---
309
310
311## August 6, 2024
312### Attendees
313
314Blaise
315b3n
316Chris
317ccx
318hacklschorsch
319meejah
320Pete
321
322
323Thought leaders:
324Amber O'Hearn (not so much)
325Larry Lessig
326Brewster Kahle
327Christine_Lemmer-Webber (spritely)
328
329
330New Items:
331
332Pete's foundation doc: https://docs.google.com/document/d/1MpFUUB9sX-SA6rsHpq9ZkIhrnZaG3zwzI6KJ9I0VAgE/edit
333
334## Comparables
335
336* https://spritely.institute/about/
337* https://ocapn.org/
338* https://darkcrystal.pw/ (meejah)
339* ISRG https://www.abetterinternet.org/
340* https://divviup.org/
341* https://sans-io.readthedocs.io/
342
343
344### Mission statements
345
346"Infrastructure for Libre, gratis, private, intentional composability and sharing " @Blaise , @meejah
347
348Device introduction @meejah
349
350"social infrastructure tools (sharing and composability to support differrent social structures and relationships)"
351Bootstrap social relations, dependency maps.
352@Chris
353
354secure sharing with least privilege (zero trust?) @ccx
355(Higher resolution, more ephemeral than app-level privs)
356https://roy.marples.name/blog/posts/capsicum_vs_pledge_final_thoughts
357
358Discoverable, "~~decentralized~~" (Autonomous?) / disconnected in nature (eg. TLS without the PKI) @Pete
359Provider independent
360
361
362### Community Survey
363
364@Chris: Maybe do a poll "Should we build a Library or an application?" Right now it's the worst of both worlds. Maybe we could explore where we could get more bang; decompose into smaller building blocks may be better.
365eg, decouple wire protocol from the U I.
366https://sans-io.readthedocs.io/ @Pete
367
368Who is the audience? grandmas with photosharing? Political activists? crypto bros?
369
370---
371
372## July 30, 2024
373### Attendees
374
375Blaise
376hacklschorsch
377b3n
378Chris
379Pete
380ccx
381
382## Highlights:
383* Project governance structure
384
385## Agenda
386
387### Previous topics:
388
389* Docs PR in progress
390
391## New Topics:
392* A proposal for a governance structure (Pete)
393
394### An endowment and governance structure
395
396* Align the Tahoe mission with the right institutional structure.
397* Similar cases:
398    * The [Veillid Foundation](https://veilid.org/about-us/) is a very close analog.
399    * https://www.djangoproject.com/foundation/
400    * https://foundation.mozilla.org/en/who-we-are/
401    * https://www.abetterinternet.org/documents/2023-ISRG-Annual-Report.pdf
402    * https://www.abetterinternet.org/about/
403    * https://matrix.org/foundation/about/
404* Fund some dedicated project support
405
406### Who is the community?
407* Board members
408* What are the activities?
409* Intellectual property portfolio (trademarks, copyrights, patents)?
410
411### Who is NOT the community?
412Where has the project stalled and why?
413Why do people drop in, get frustrated, and leave?
414What are the blockers for adoption and how can we overcome them.
415
416### Tahoe public resources; what could we build next?
417* Public test grid ?
418* ZCap servers ?
419
420#### Other options for funding services
421
422* https://docs.opencollective.foundation/how-it-works/fees
423
424#### Dependent projects
425GridSync (golang) ?
426Magic Wormhole ?
427
428#### Dependencies
429* Twisted
430* Mutables
431* Python https://meta.pycqa.org/introduction.html
432
433---
434
435## July 23, 2024
436
437### Attendees
438
439Blaise
440hacklschorsch
441Meejah
442b3n
443Chris
444ccx
445
446
447## Highlights:
448* Mutables would be a good thing for the Foundation to fund.
449* The goal of [GBS](https://gitlab.com/tahoe-lafs/gbs-downloader) is to replace foolscap. Not there yet. (sucks to require a node online)
450* Create project for client filesystem
451
452## Agenda
453
454### Previous topics:
455
456Meejah:
457* Trac disk space restored
458* Granted admin privs
459
460hacklschorsch:
461Mobile LAFS [Wiki page](https://tahoe-lafs.org/trac/tahoe-lafs/wiki/TahoeLAFSMobile) with draft description of Features and Scope.
462
463B3n:
464Next, Trac migration path (DNS, )
465Codeberg bug
466
467
468### New topics:
469* B3n: Prepare Trac migration operating instructions (DNS settings, etc)
470* B3n: Help Meejah create Trac account (for Blaise).
471* Decision: Let's try to make the single mobile code base work (hacklschorsch)
472* Deep integration with the OS is not MVP  (hacklschorsch)
473* [Gitlab milestones](https://gitlab.com/tahoe-lafs/tahoe-lafs-mobile/-/milestones/7#tab-issues) don't have the OS integration yet.(ccx)
474* Create FUSE module for the dev environment (ccx)
475
476## Notes:
477
478Chris:
479* [Porting Tahoe hashlib](https://github.com/crwood/deterministic-keygen/issues/5) to Rust and found an issue:
480* To do a restore with user supplied keys, if a directory is already created we get an uncoordinated error. Possible solution is to derive the cap locally.
481* Curating a common set of test vectors
482
483#### Restore scenarios
484Todo: write up a description, describing permuted scenarios (Meejah, Chris)
485(We need better words to describe the operations and events)
486
487Four scenarios:
488* Client failed, grid is fine
489* Grid loses nodes
490* Grid loses partial data
491* Grid loses all data
492
493
494#### Mutables
495* Don't do service happiness (ccx's complaint)
496
497Can't detect all the restore scenarios. Sometimes all the shares end up on one server :frowning:
498What is the meaning of Happiness? Happiness is just a number (Meejah)
499
500#### Mobile
501* Need to write up the description in  [Wiki page](https://tahoe-lafs.org/trac/tahoe-lafs/wiki/TahoeLAFSMobile)
502* We need to describe how much of the functionality
503* Foolscap requires a node online, [GBS](https://gitlab.com/tahoe-lafs/gbs-downloader) does not
504* "User defined grid" ? (ccx)
505* What if we had a gridsync client (b3n)
506* Deep integration with the OS is not MVP  (hacklschorsch)
507* Obsidian has something called `backend` which is different.
508* [Obelisk overrides](https://gitlab.com/tahoe-lafs/tahoe-lafs-mobile/-/blob/main/obelisk/haskell-overrides.nix?ref_type=heads)
509* Let's have good interface definitions (ccx)
510* [Gitlab repo](https://gitlab.com/tahoe-lafs/tahoe-great-black-swamp) Code
511
512
513#### File system browser? (ccx)
514* Can we use the native [mobile libraries](https://emanual.github.io/Android-docs/guide/topics/providers/document-provider.html)?
515* How much do we need in the GUI?
516* (https://gitlab.com/tahoe-lafs/tahoe-lafs-mobile/-/blob/main/obelisk/haskell-overrides.nix?ref_type=heads)
517
518#### Create project: Investigate FUSE File System Support (ccx)
519* Read/Only native client
520* Read/Only magic folder
521* Read/Write Magic Folder
522* Built on top of GBS downloader
523
524---
525
526
527