Happy (belated) New Year everyone! We have restarted our weekly w3UPDATES. Looking forward to having you join us for the journey 🚀 New Year, New w3up!
🧟♂️ Real Headless UI
We’ve merged a big update to the w3ui react components that makes building apps even easier. whispers sshhhh w3ui isn’t really headless yet (🙈OMG!), it’s “NoUI” really since the only things it exports are “providers”. This new update adds real headless UI** components like headlessui.com provides, so you can build an app that uploads stuff to web3.storage as simply as:
We’ve got some finishing touches to do before we can cut a release but it’ll be published in the next few days. The plan is to prove it out by using it to build w3console and then ship it for the other UI frameworks we support.
👀 https://github.com/web3-storage/w3ui/pull/136
🚗 🆙 w3cli gets Sugary CAR Upload
The new w3cli has been updated with support for CAR uploads. LOL wat!? we have that already!?! Ok, ok, you’re right, 😛 we have CAR upload via w3 store add (https://github.com/web3-storage/w3cli#w3-can-store-add-car-path) but it’s subject to the max single upload size that our bucket provider imposes - around 4GB. Bummer. Do not fear! w3 up --car is here! 🦸♂️🌈🌈
w3 up --car allows you to pass a CAR file of arbitrary size and it’ll automatically be sharded, it’ll call w3 can store add for each shard and finally it’ll automatically register an upload, tying together the shards, by calling w3 can upload add. YES BOSS. 👀 https://github.com/web3-storage/w3cli/pull/36
2️⃣→1️⃣ One UCAN Endpoint to Rule them ALL!
Through a quirk of history we have 2 UCAN services. They’re split logically as our “Upload API” and our “Access API” but really they’re just 2 parts of 1 service. In the big release we deployed before the holidays we did the work to switch to did:web keys (allowing key rotations amongst other things) and both services began using the same key. We’re now close to completing the work that allows invocations to either service via a single URL. 👀 https://github.com/web3-storage/w3protocol/pull/334
🧪 Testing Testing Testing
We’ve always had a big focus on testing and we often go so far as using hundreds to ensure we actually have 100% test coverage of our code. It’s not enough. One of the things we punted on at the end of the previous phase was End to End (E2E) testing…and damn, did it bite us more than once for doing so! This week we’ve been adding E2E tests to w3ui and the upload-api. Here’s a screenshot of the new w3ui E2E test results:
👀 https://github.com/web3-storage/w3ui/pull/242
♠️ Introducing w3filecoin
In web3.storage we rely on a tool called dagcargo to aggregate our user’s data together and make deals with Filecoin storage providers. The system was built 2 years ago as an MVP in just a few weeks and is tied heavily to the current web3.storage and nft.storage services.
The dagcargo service is being rebuilt as a new service with a clearer interface and tighter scope. The new service will be called Spade ♠️ and web3.storage will be one of the first users.
This week we’ve been thrashing out a design for how we integrate with Spade. Here’s how the infra is shaping up:
🧾 UCAN Execution Spec Progressing
Ucanto is a UCAN RPC system that we’re using in production now. The UCAN execution spec aims to formalise this RPC system but most importantly for us, specifies the idea of receipts - which is something we don’t have yet but know we need.
This week (and last, and also before the holidays) we’ve been investing in conversations pushing the draft spec forwards, including our own fork to propose better our requirements.
👀 https://github.com/web3-storage/specs/blob/feat/invocation/invocation.md
🛻 Pickup Rollout Begins
Pickup is the replacement for our IPFS Clusters. Our IPFS Clusters are now responsible only for pins that happen via our Pinning Service API i.e. not regular CAR file upload. This is currently a small portion of the data that is being added to web3.storage by users.
Pickup is a thin layer on top of our main service - it spins up ephemeral Kubo nodes for the purpose of fetching data using all the awesome tools IPFS and libp2p have for discovery and networking (e.g. hole punching). Once the data has been retrieved, it’s written as CAR files to a bucket and then enters our system via Elastic IPFS as any other upload does.
This week, we began the rollout of Pickup and it will slowly start “picking up” some of the traffic we see to the Pinning Service API. We’ll report back when Pickup is fully deployed!
👀 https://github.com/web3-storage/pickup
🚀 Great start to 2023 folks, love to you all ❤️
