Week in w3up: 2022-11-25

Week in w3up is a newsletter on the blog that shares updates and progress on improving w3up and integrating it into the core web3.storage platform.

This week in w3up! A short week with fewer developer resources than usual. We didn’t manage to meet our stretch goal of shipping to production, but we did manage to still get a whole lot done. A feast of improvements! 🍗

🎨 Database schema design solidifying

This week, a lot of hard work went into database design, which is incorporating anticipated future needs as well as our more immediate business goals, such as usage-based pricing. Naturally our designing has taken into account the current schema design, a prior proposal of a possible future design and we have hopefully come up with something that is better than both! The thoughtful discussions are taking place here, for the interested reader 😃.

We’ve also been musing over CAP theorem and have been wondering, “what if our database is just a bunch of materialized views over an append only log of UCAN invocations? 🤯”. The idea takes some influence from Scuttlebutt and if that sparked your interest then this article is a good read on how we might do something like that in AWS.

🏆 In general, having a log of events (UCAN invocations) allows us to:

Start accounting from Day 0 of implementation and figure out how to bill later.
Take snapshots at any desired frequency (daily, weekly, monthly) and reproduce the state of the world by playing forward invocations since then.

👀 Check out the latest proposed schema: https://github.com/web3-storage/upload-api/pull/48

Next week we are going to start looking into UCAN logging and handlers. Stay tuned!

💔 Side index / Freeway replication decoupling

This week, we decoupled the process of creating our side indexes and replicating uploaded CAR files to R2 and now have a generic lambda for copying from one bucket to another and a lambda for creating a side index and storing it in a bucket.

It means we get side indexes written to a bucket in AWS and get to reuse the generic copy from bucket to bucket lambda to copy the index over to R2!

Why do we want side indexes in AWS? Well, it means we could run a Freeway in AWS, race it with Cloudflare Freeway and mitigate against the time it takes to copy a CAR over to R2, catering for the folks who upload a file and then immediately attempt to fetch it from the gateway.

CID of the image

👀 Read more about this change: https://github.com/web3-storage/upload-api/pull/36#issuecomment-1326363388

🌌 Changing spaces

Last week, we made the transition from Account ➡️ Space. This week, a new proposal for accounts arrived as a tool for managing delegations and key pairs across multiple spaces, agents and devices.

👀 Read the latest ideas on the relationship between users, agents, and spaces: https://hackmd.io/@gozala/BkB4NDRLi

🔐 Access client and CLI update

w3access has it’s own CLI tool! I know right, who knew? It contains just the commands for creating and managing accounts (i.e. no uploading). It got an update this week, to orient around “spaces” but also to enable recovery in the case when you lose your keys! This includes work to the access client to allow this and closes the loop on the work done last week on the backend to enable this.

📃 Pagination now available in `upload-client`

Before the pagination update was 🚢’d, list calls only showed one page of uploads! Last week we made the backend changes to support this feature, now, users have the ability to call and view more than one page, allowing them to access lists of all their content 🙌

👀 Check it out: https://github.com/web3-storage/w3protocol/pull/204

⭕️ Big w3ui update incoming

There’s a huge update coming to w3ui that integrates the new access-client (with ucanto 0.9) and the new upload-client. It introduces the concept of spaces, and makes allowances for users to switch between them (thanks to the w3access agent). It also pulls in the latest upload-client changes allowing users to take advantage of pagination as well.

It works against our staging deployments of the upload-api and access-api so the code changes will remain in the PR until we deploy to production…there is also some documentation work to complete to finish the PR anyways!

However, what’s there is usable, and to prove it, we quickly knocked up a “space manager” app which allows you to create and switch between spaces. This may eventually become the basis of the w3access keyring app: