There are so many definitions of Web 3.0, and the current universally acknowledged definition is the next generation of the Internet that addresses the problems currently faced by Web 2.0, in which data is owned by individuals. Nowadays, governments around the world are passing legislation to achieve the personal ownership of data, but as long as the data is in the servers of other enterprises, it is difficult to achieve this. Only when the data storage problem is technically solved can the data be fully owned by individuals.
This is a different problem because it is impossible for an individual to set up a separate server for his or her data, and the best approach is to implement a data storage system that is not controlled by any single organization, in which personal data is stored and can be authorized to users, all of whom can read data from this system. The storer pays for storage charges based on the size and duration of the data stored, while the user pays for traffic charges based on the amount of traffic used. How to build this storage system that is not controlled by a single organization and can implement payments without a third party is perfectly answered by the development of blockchain technology.
Technical Conditions for Web 3.0 Implementation
Currently, Web 3,0 is still in its early stages and all kinds of technical solutions emerge endlessly, but with the exception of Ethereum which enables decentralized finance through smart contracts, all other applications are at the conceptual stage. The general public and even practitioners in the industry are not certain how far the current technology is from actual commercial applications and what technical conditions are still missing. The existence of this problem also makes Web 3.0 seem strange and distant to most people, who in extreme cases suspect that it is just a hoax.
There are already many projects that attempt to implement full Web 3.0 projects in the industry, such as Web3.Storage launched by FileCoin, which attempts to use Filecoin as a storage backend. There are also a large number of Web 3.0 application projects on Solana, such as Satellite.im, a decentralized instant messaging system, and Wum.bo, a decentralized social and streaming platform. However, it could be argued these projects are experimental in nature and have had limited success in the commercial application environment. Are they really fully decentralized? Are they able to satisfy the user experience of consumers? Whether these projects can stand up to the tests, whether they can meet the requirements of decentralization, how are they implemented so far, and are they still just an idea?
Web3Tube Design Specification
Aurora FS will provide a Web 3.0 application that can enable a complete commercial application to be implemented under a completely decentralized condition as much as possible, so that the general public can intuitively evaluate the effectiveness of Web 3.0 applications. Considering the current dominance of Web 2.0 in Internet systems, the decision was made to transform a Web 2.0 application with Web 3.0 technology.
This decentralized content platform allows the general public to experience Web 3.0, its application effect and commercial value, and also test the maturity of each decentralized technical module. The concept has been aptly named; Web3Tube.
Web3Tube’s vision is to design a streaming application based on Web 3.0, in which:
- Users log in directly using their blockchain wallet addresses.
- Users can create channels for videos.
- Users can post their own videos on Web3Tube and price them.
- Users can subscribe to other people’s channels for real-time notifications.
- Users can directly buy videos priced by others and pay for them directly through the blockchain.
- Users can comment on videos, like it or dislike it, and recommend it to others.
In this process, the uploaded videos are stored in the decentralized storage, and the content distribution (i.e. when consumers watch videos) is carried out through the decentralized content distribution network to ensure the performance and user experience of content distribution. At present, since the decentralized database technology is not yet mature, compromises need to be made.
This starts with using a centralized database to provide services externally through the P2P network proxy. When the client accesses this database, it connects to access the data and obtains the results using P2P technology. In this way, the database is centralized but anti-censorship, i.e., no third-party organization can prevent the client from accessing the database, except for the provider of this database (application). In the future, this centralized database will be gradually decentralized.
Web3Tube is built using multiple decentralized technologies. For example, AuroraFS is used for decentralized storage and content distribution, SOL is used for decentralized transactions, etc. The following diagram illustrates the interrelationship between different technology stacks:
Image Source: https://miro.medium.com/max/1400/1*pJabt7RiH8aQvCL3u1gfmw.jpeg
Decentralized Databases
At present, there are already some projects in the industry with the concept of decentralized databases. OrbitDB is another decentralized database, and its design concept is to synchronize database files between different IPFS nodes through the Pub/Sub mechanism of IPFS. OrbitDB is a peer-to-peer database, and each node needs to synchronize all the database files, which does not meet the server-side/light-node mode we need. Ties.DB is the first public database for decentralized structured data storage and allows advanced search and document modification. Tie.DB defines a set of protocols for implementing basic database operations such as adding, deleting, modifying, and querying, as well as the basic permission management on a decentralized network. While each BigChainDB project uses MongoDB as the backend, multiple nodes are synchronized using the Tendermint consensus mechanism. When a client requests data, it sends a request to the nodes of BigChainDB over HTTP and gets a response. BigChainDB is actually an attempt to combine a database with the blockchain to implement a decentralized database.
Decentralized Identification
Decentralized identification refers to a set of identities that are completely decentralized and allow individuals or organizations to have the full ownership, right of management and control of their digital identities and their data. In short, a user can prove his or her identity through a verifiable digital identity and access various blockchain projects with this identity. Besides, the control rights of the account are in the hands of the user, and consequently security and privacy are guaranteed.
Image Source: https://www.w3.org/TR/did-core/diagrams/figure-a.1-did-and-did-document-graph.svg
In Web3Tube, decentralized personal authentication is divided into two parts.
- The first part is the login process of Web3Tube. The wallet address is used as the login credential, which can be done by starting WalletConnect via Dapp and making WalletConnect connect to the wallets in the system (WalletConnect is currently supported by the vast majority of wallets)
- The second part is how to ensure that the data is owned by users when they generate it. For large unstructured data, such as the generated video files, they will be stored on a decentralized storage network; while small structured data, such as followers, like information, etc., will be stored in a decentralized database. In this part, the decentralized storage network and the personal storage of structured data are implemented by the corresponding technical module.
During the implementation of Web3Tube, these two parts are also implemented separately. The first part is implemented in the first stage, while the second part is implemented based on the implementation plan and roadmap of the corresponding technical module.
Decentralization of Communication
In current P2P systems, there are only two communication schemes: broadcasting and DHT-based data transmission. The former is to send data to all nodes, while the latter is to send data to the nodes closer to the destination address based on the DHT table. The current public chain projects in blockchain uses the P2P broadcasting technology, While the decentralized storage or content distribution projects such as IPFS/Swarm/BT use the DHT-based data transmission technology. The DHT-based data transmission technology is a data transmission scheme without security guarantees, which cannot achieve a balance between efficiency and reliability.
Decentralization of communication needs to implement unicast (data is transmitted from one node to another specified node) and multicast (data is transmitted from one node to a specified group of nodes, or a certain data response is got from a specified group of nodes) over a P2P network. To implement multicast over a P2P network, it is necessary to implement the routing system over the P2P network. Different from the implementation of routing systems in traditional networks, the implementation of routing systems over P2P networks needs to take into account the unreliability of nodes, which means that if there is a reward, any node may cheat to get the reward. If there is no reward, no node has an incentive to create routing table entries. If a small cost can be used to create a huge waste in the network, malicious nodes will constantly attack the network even if they don’t generate any income.
When there is a decentralized communication system, different systems such as decentralized storage, decentralized distribution system, decentralized database and blockchain can intercommunicate with each other. As shown below:
Through a decentralized communication system, the intercommunication of different decentralized subsystems is implemented. Note that in the diagram Solana and decentralized data are accessed through a P2P proxy, where the multicast technology is used to intercommunicate with the proxy, i.e., the clients (content producers and consumers) communicate by sending multicast messages to the database and the group in which SOLANA is located. By means of a P2P proxy, the problem that the SOLANA network and database cannot be accessed when RPC is blocked can be avoided, ensuring that the content producers and consumers have uncensored access to both subsystems.
In the diagram above, AuroraFS are not accessed through a proxy, as these two subsystems themselves are built on this decentralized communication system. Obviously, the above-mentioned decentralized communication can access more subsystems that differ from each other when needed. Conversely, any subsystem can use this decentralized communication system as the basic communication, which can seamlessly connect with other subsystems. In practical implementations, multiple subchains intercommunicate with each other through the multicast function in this decentralized network.
In Web3Tube, each client is directly connected to the P2P decentralized communication network, and communicates with each subsystem through the unicast and multicast functions of this network, thereby ensuring the anti-censorship data transmission.
Decentralization of Storage
Decentralization of storage is implemented by storing data on a P2P network. The core problem with decentralized storage is how to determine if a node actually stores data. ARWeave adopts a method of storing all data on the chain, i.e., each node stores all data, which leads to a large amount of duplicate storage and greatly increases the cost of storage. To solve this problem, ARWeave adopts a random synchronization method. Each node does not need to synchronize all block information, but only synchronizes the information of the previous block and some block from the past in the network. As this scheme has not been proven safe, ARWeave is just an attempt at decentralized storage. Another project of decentralized storage is FileCoin, which core idea is that through zero-knowledge proof, the node that stores data regularly provides proofs to the chain to prove that it has actually saved the data, and this timing proof is called PoST (Proof-Of-Space-Time).
To prevent external attacks, the node itself does not store data. When data is requested on the chain, the node makes requests to other nodes that store the data, and then returns it to the chain for verification. In order to solve these two problems, Filecoin adopts the form of large sector and multilayer encryption encapsulation, in which the sector size is 32GB or 64GB and the final sealed sector is formed through an 8-layer encryption. The purpose of this is to regenerate the evidence long enough to exceed the evidence generation time allowed by the system in the absence of data, thus avoiding both of the above attacks.
In FileCoin, one of the main reasons for these two attacks is to allow users to encapsulate data by themselves to form computing power. For example, users can encapsulate all zeros or data generated according to a certain algorithm, so that nodes do not store the original data, but regenerate data through the algorithm to prove when verification is required. In addition, there is no traffic payment mechanism. When verification is required, a node can read data from other nodes for free, and then generate proofs to deceive the verifier.
In AuroraFS, self-encapsulation of data is not allowed. This means that the data obtained by the storage nodes can obtain computer power on the chain only after the storage request is initiated by the storer and the chain, and matched to different storage nodes by the system. This solution solves the problem of user-generated data attacks, while the traffic payment solution solves the problem of external attack. In fact, with the AuroraFS p2p cloud storage system, this problem is further solved by the provision of authorized access.
In Web3Tube, AuroraFS is also used to store large files (videos, pictures, audios). When users store large file data, they first initiate a storage request on the chain. After the matching is automatically made on the chain, the matched storage nodes pull data from the user nodes, and then it is recorded on the chain after the local storage is completed.
Any file data is stored in a limited number of nodes in decentralized storage. When a great deal of data is read, these storage nodes may not be able to meet the bandwidth requirements for reading. Therefore, Web3Tube chooses to use AuroraFS to implement content distribution.
Decentralization of Content Distribution
In the traditional Internet, content distribution is implemented through CDN, i.e., the content is pre-distributed to different servers across the country (around the world). When a client requests access, it can directly request data from the corresponding server only after the scheduling is done based on the IP address of the client and the load of each server.
CDN is the current standard scheme for streaming media to implement high-performance reading, which is also a very centralized scheme. In Web 3, a system that is not only decentralized but also can implement distribution acceleration is required.
BT/IPFS is essentially a decentralized content distribution system that accelerates content distribution through P2P networks. The reason why they are not storage networks is that they have no file loss protection mechanism, i.e., the ability to detect insufficient backups of certain files and automatically add the backups, nor can they keep the data source nodes online through an incentive mechanism. In contrast to CDN, BT/IPFS is divided into two parts: the back-to-source nodes that ensure the data validity and the distribution nodes that improve the distribution ability. As the distribution nodes have a certain data caching ability, when the data request on a client has been deleted from the cache, the distribution nodes will read data from the back-to-source nodes and forward it to the client. BT and IPFS are essentially equivalent to the role of distribution nodes.
Due to the design scheme of BT/IPFS, data is stored on the storage device of the original nodes before being read by other nodes, while the data information (hash, the node that stores this hash) is pushed to the DHT network. When another node wants to read a certain piece of data according to the hash, it first finds the node according to the data information in the system (hash and the node that stores this hash), and then connects to this node to read the data. When the node reads the data, it also pushes the new data information (hash and this node information) to the network.
In this way, BT/IPFS cannot implement instant transmission (i.e. the data source node can be disconnected immediately after uploading data), and it cannot adapt to the C/S reading model of streaming media. The client node won’t provide reading services for other nodes after reading data).
Swarm (Bzz) proposed another model. BT/IPFS pushes file information (hash and the node that stores this hash) to the DHT network, while the Swarm system is a pre-distribution system, which directly slices data and pushes it to the DHT network, i.e., once data is uploaded, it will be distributed to different nodes in the network, and each node will cache the data once in the distribution process. Compared with IPFS, Swarm can implement instant transmission and fully adapt to the C/S reading model of streaming media (as it has been pre-distributed). Swarm’s disadvantage is that no matter how many times the data is read, it needs to be pre-distributed. This pre-distribution process will consume additional storage and bandwidth, and if this pre-distribution is free, it will not be able to prevent junk data attacks – malicious nodes block the network by uploading a large amount of junk data.
In addition, in P2P networks, the motivation for nodes to provide bandwidth and implement traffic distribution is important. Blockchain is the natural economic incentive system for P2P networks. Blockchain is the natural economic incentive system of the P2P network. As the decentralized rewards are given to nodes that provide bandwidth through the blockchain technology, nodes are willing to provide content distribution services for applications at low or even free prices in order to obtain bandwidth rewards at an early stage. The decentralized content distribution system uses the financial feature of blockchain to pay for data traffic, and quickly creates and forms a complete ecosystem. There are two challenges to the incentive system:
One is the problem of processing a large number of receipts on the blockchain. Since the nodes do not trust each other, the solution to achieve trust is that for two nodes that transmit data, each time a small piece of data (for example, 1MB) is transmitted, the recipient of the data gives the service party a receipt, so that if the recipient does not give a receipt after receiving the data, the service party can disconnect from the recipient, with the income of only a small piece of data lost. This solution solves the problem of mutual distrust between two nodes, but will generate a large number of receipts. Assuming that a node with 100Mb (about 10MB) bandwidth works at full speed, it will generate 10 receipts per second. If there are 1M service nodes working at full speed in the system, they will generate 10M receipts per second. These receipts are used as transaction records on the blockchain, as currently no blockchain can support them, so a lot of optimizations need to be done to meet the requirements of blockchain.
The second problem is about cheating. If this receipt is used as a service node to provide services and the corresponding rewards are given, then the nodes can join together to cheat. Some nodes among the joint cheaters give other nodes arbitrary receipts to make the system think that these service nodes generate very high data traffic and give corresponding rewards, so that these service nodes get traffic that should not belong to them.
Any decentralized content distribution system with rewards needs to solve these two issues. AuroraFS has solved not only these two problems, but also the performance problems of data reading and authorized access problems. Web3Tube uses AuroraFS as the solution of a decentralized content distribution system. For content consumers, they don’t care where the data is stored, but only need to read the expected data. Therefore, the SDK of AuroraFS is directly integrated in Dapp, it can directly request data from AuroraFS when needed.
Decentralization of Transaction
The concept of transaction is divided into narrow and broad sense. Blockchain can implement the decentralization of transactions, but due to the existence of impossible trinity for blockchain (i.e. decentralization, security, and scalability – TPS), the TPS performance of blockchain projects that can ensure decentralization is very low. For example, about 7-10 transactions per second for BTC, and about 15-20 transactions per second for ETH.
In Web3Tube, content creators set prices, consumers purchase, then download or play in real time, all of which belong to the above transactions. Web3Tube can use any of the above new public chains to implement decentralized transactions. For the first release version of Web3Tube, Solana was used as the public chain for decentralized transactions.
Sum Up
The above Web3Tube solution has analyzed the technical modules required for a fully decentralized application, analyzed and integrated the modules that rely most on decentralized technologies, and built a practically available Web 3.0 application that can be implemented in a commercial environment and has the value of applications. On the contrary, this application can be used to test the maturity of the current Web 3.0 technical modules.
As mentioned earlier, some of these technical modules are not so decentralized, or immature decentralized technical modules. In the future, Web3Tube can choose different technical modules again to build and form different forks, which also represent different degrees of decentralized maturity. Obviously, the fork with the highest maturity can represent the maturity of the whole Web 3.0 technology.
Web3Tube is an exciting and pioneering application of Web 3, which will soon be launched in its fullest form. The Phase III airdrop, encouraging new and existing node operators to fulfill tasks within the Web3Tube environment, is due to commence from March 25 with a warm-up phase. This will pave way into an activity phase, with activity, bandwidth, and engagement on Web3Tube all being contributing factors to AUFS token airdrops. The higher the contribution, the bigger the airdrop.
The latest announcements and information for setting up a node, and participating in the Web3Tube/Phase III airdrop event are found on the Aurora Gitbook, and across Telegram, Discord, and Twitter.
Docs & Node Specs: https://docs.aufs.io/
Join us on Telegram: https://t.me/AuroraFS
Follow us on Twitter: https://twitter.com/AuroraFS_Labs
Join our Discord: https://discord.gg/nDFnN6zScC
Media Contact:
Brendan Brown
Source: https://coinpedia.org/information/web3tube-the-content-streaming-dapp-based-on-web-3-0/