This post focuses on outlines the vision of web3.storage. The state of the post reflects where the product will be in Q4 2022. For instance, we are currently in Beta of our new upload API that natively uses UCAN for auth, and will have this incorporated in the web3.storage client and website the coming weeks. In the meantime, some of what is outlined in this blog post might not yet be incorporated into the core web3.storage and NFT.Storage products.
Click here to read Part 1 of our introduction to the web3.storage platform!
The DAG House team is building the web3.storage platform to let developers take full advantage of the compelling advantages of the Data Layer and build the next generation of experiences for them and their users. In this section, we introduce the web3.storage platform “stack,” which involves the protocols of the Data Layer, and the specific products of the platform built to make them easy to use for developers.
📣 Today, NFT.Storage uses the web3.storage platform on its path to provide off-chain NFT storage as a public good, so if you use NFT.Storage you can take advantage of the data layer for free!
Protocols we use
The cornerstone of the stack is the protocols of the Data Layer. We gave an overview of these earlier, but getting more in depth gives a picture of all the advantages that these protocols provide. This section gets a bit into the weeds around a lot of acronyms, but the concepts themselves are familiar ones, so please do bear with us!
- IPFS: Peer-to-peer protocols that references data by its unique content identifier (CID)
IPFS itself enables networks where users can host or retrieve data using a CID, which is unique to the data, and access it as long as there is at least one peer on the network hosting a copy of the data
web3.storage participates in the public IPFS network, which is what most people reference when talking about IPFS - anyone can get data off the network with its CID
IPFS utilizes IPLD under the hood - it allows IPFS to easily link blocks of data together (to represent things like file systems, databases, and JSON, as well as how a single large file gets split up into smaller blocks)
- As a result, everything in IPFS can be represented as a graph of blocks
- This can create efficiencies around deduplication, efficient loading and diff’ing of data, and more
We generally exchange data in IPFS in the form of CAR files, which is a serialized set of CIDs and block data into a file
- Since it’s a file, this puts everything one needs to interact with IPLD-structured data into a convenient format that can be sent over any transport (HTTP, etc.)
- Since client-side CAR generation results in the CID being generated locally, the user can have full confidence no one tampered with it, even when they access it later
If you’d like to dive a bit deeper, check this out, which dives deeper into the technical details of the underlying data protocols
DID: A type of identifier that enables a verifiable, decentralized digital identity (decentralized identity document)
- Any private-public keypair can generate a unique DID
- Since DIDs are globally unique, they can be used to represent any actor in a system, opening things up even further
- For instance, within the web3.storage platform, users, storage accounts, virtual machines, and more can all be viewed as separate actors
UCAN: An auth token that self-contains all information for a service to cryptographically verify that a given actor has authorization to perform a certain action
UCAN tokens are issued using a DID and a signature from the DID’s private key; this signature can be used to verify the validity of the token
This means no external sources of truth are required to authorize services, and the same UCAN token can be used by any service that recognizes that DID for the services it requests
Further, UCANs allow for delegations of permissions across DIDs, meaning a user with a certain set of permissions can sign and delegate a subset of its permissions to other DIDs
And since anything with a private-public keypair can generate a DID, UCANs can be an auth layer that naturally plugs into existing identity primitives and services (e.g., crypto wallets)
With IPFS’s ability to immutably address any data and link it together, and UCAN’s self-containing authorization, these two protocols can also be used in tandem to unlock even more
- An IPFS graph can contain all data needed for a service to provide an operation, including the UCAN token (e.g., store this data! run this workload on top of this data!)
- With a standardized structure to make calls across interfaces and using CAR files as the transport to ensure all data needed is there, this opens up RPC layers that are interoperable across services
- After a workload is run, a proof containing the input, authorization, and output can be generated using IPFS
- So there’s no centralization of the workload, and anyone can provide the service as long as they’re compatible
Products in the web3.storage platform
So how does the web3.storage platform use these protocols today? Let’s tick through the products!
- w3up: Data storage service
Uploaded data is stored on an instance of Elastic IPFS, a cloud-native, scalable, open source implementation of IPFS that we wrote
- Everything you read on this page is enabled by the reliability and performance that Elastic IPFS provides - this is one of the web3.storage platform’s major innovations!
Data stored on w3up is also stored in multiple Filecoin storage deals, with the physical storage itself verifiable using cryptographic proofs and cryptoeconomics
w3up implements UCAN based authorization which decentralizes authentication and moves us away from web2 style API keys issued by the provider; any DID that registers can store data and delegate that ability to others
- ⚠️ The upload service on the current web3.storage website does not yet use UCAN. As of September, w3up is in beta, and will be integrated with the core web3.storage experience in Q4
w3up comes with some awesome clients
CLI and SDK: Makes it super easy for you to upload to web3.storage using UCANs and delegate others to upload on your behalf
- Like the old client library, a CAR file of the data is generated client-side, meaning you can verify that the CID of your data is the correct one
- Unlike the old client library, CAR generation is done in a streaming manner (speeding things up and alleviating memory constraints for large files)
- When the new API is implemented in the web3.storage and NFT.Storage websites, the login and upload consoles will be based on w3ui!
- w3link: Public IPFS HTTP Gateway that is optimized for performance
- Gateways act as bridges between the HTTP protocol that all web browsers understand and the peer-to-peer Bitswap protocol to make IPFS data accessible outside the IPFS network
- There are two URLs to access w3link:
[w3s.link]([http://w3s.link](http://w3s.link))(for general usage) and nftstorage.link (for NFT.Storage users)
- Data is referenced via CID in the HTTP URL (e.g.,
- w3link allows you to ensure extremely fast reads for data you care about, especially larger files, by allowing you to permanently cache your data on the edge network (a premium feature called SuperHot)
- It also will soon be checking whether content is malware (e.g., phishing websites) before serving it, limiting its ability to be abused. This detection service can be generalized for other gateways to use
- w3name: Cryptographically secure mutable references
Since IPFS generates a unique content identifier for a given piece of data using a hash function, just changing one bit of that data generates a completely new identifier
However, in modern web use cases, you often want the ability to use the same identifier to reference changing data (something we take for granted with HTTP, where we reference files by their filename)
You can do this using w3name, which implements the IPNS protocol that allows users to use a public / private keypair to generate and update a static address
There are other ways we’re thinking about doing mutability (e.g., mutable buckets with their own DID) that will come in the future!
Not yet public: There are other products that we currently consume internally, and will release public versions in the future
w3access + w3wallet
- These libraries are used to generate and manage a keypair associated with different devices and identities (w3wallet) and register and validate identities for web3.storage services (w3access)
- They are currently embedded in w3up, but will be modularized so that anyone can easily generate / manage keypairs in their preferred way (e.g., with a crypto wallet) and register corresponding DIDs for use with the web3.storage platform
w3query: Engine to run workloads on top of content addressed data + on top of other data that has globally unique identifiers, like blockchain data
- We use w3query internally to process uploads via web3.storage
- However, it’s an engine that can run on top of CIDs, contract IDs, NFTs, etc. - eventually, we envision a marketplace where folks can query in 3rd party services and use them to run compute workloads on their data from a single place
- In addition, the output of the workload can embedded in a verifiable proof that the workload was run, using CIDs to index the output value, the input data, and the query run (the interface for running a unit workload will be called bucket-vm)
And to reiterate the benefit of UCANs:
- Anyone can delegate other users to use our services
- Any other service can use the same open protocols to provide users value (including their own storage, compute service, etc.), making using multiple service providers or switching among them frictionless
- In addition, if you just want to interact with our service directly but benefit from using others’ services, you can delegate permission for us to interact with them on your behalf
Next up: The platform in practice
In this post, we introduced the products and products within the web3.storage platform. In the next post, we will discuss the how the platform is used in practice - how you might build the next killer app on top of it, and what more to expect in the future!