D.A.G.G.E.R. Testnet 1: Release Overview

After more than a year of research and development, GenesysGo is pleased to announce the launch of the D.A.G.G.E.R. Testnet Series, powering ShdwDrive v2 - a monumental achievement and major milestone on our journey to build the future of decentralized data storage. 

This article offers an overview of the technical objectives in focus as we continue to improve D.A.G.G.E.R. technology via a series of testnet roll-outs. We will cover what users can expect from these initial testnet releases as we march forward. The application being showcased throughout the testnets will be the limited-beta ShdwDrive v2 implementation of D.A.G.G.E.R..

Introducing D.A.G.G.E.R.

D.A.G.G.E.R. (Directed Acyclic Gossip Graph Enabling Replication) represents a groundbreaking innovation in distributed ledger technology (DLT) by focusing on scalable data-management (data storage) as a core principle when designing a high-speed consensus protocol. This is an often overlooked aspect of designing useful DLT infrastructure, and developers often have to choose between rapid transaction processing blockchains or scalable data-management blockchains. D.A.G.G.E.R. is a Byzantine Fault Tolerant consensus protocol optimized for high scalability, speed, bandwidth efficiency, and has a native storage framework that, when combined, delivers a cutting-edge DLT protocol that will usher in new and powerful applications. D.A.G.G.E.R.'s modularity and versatility as an underlying consensus engine and database network enable developers to create powerful new applications that are fast, secure, and data-scalable, such as our implementation of ShdwDrive v2.

At a high level, D.A.G.G.E.R. operates by having each node maintain its own copy of a shared directed acyclic graph (DAG) that cryptographically encodes the history of gossiped events in the network. Our Litepaper covers the technical details of D.A.G.G.E.R. extensively, but some key features include:

• Asynchronous, leaderless consensus
• Highly efficient bandwidth usage compared to traditional BFT designs
• Flexibility to support both permissioned and permissionless deployments
• Modularity enabling multiple use cases (e.g., ShdwDrive v2)

Litepaper: https://github.com/GenesysGo/dagger-litepaper

Experience D.A.G.G.E.R.: https://dagger-hammer.shadow.cloud/

Participate by running a D.A.G.G.E.R. Wield Node: https://docs.shdwdrive.com/wield

By combining DAGs, gossip, clever cryptographic sorting mechanisms, and native scalable data-management, D.A.G.G.E.R. can achieve high transaction speeds while delivering CDN-level storage throughput. The testnets will allow us to benchmark real-world performance and optimize various parameters of this core D.A.G.G.E.R. network underlay.

ShdwDrive v2 Built on Top of D.A.G.G.E.R.

ShdwDrive v2 is a data storage application built on top of D.A.G.G.E.R.. It aims to offer users distributed storage that is secure, affordable, and easy to use. By leveraging D.A.G.G.E.R. for consensus and data management, files are erasure-coded into shards and distributed across nodes for redundancy. Regular data audits and repairs ensure resilience, even if some nodes fail or go offline.

Since D.A.G.G.E.R. is a versatile consensus and data-management engine, a storage application such as ShdwDrive v2 not only showcases the power of D.A.G.G.E.R. technology but offers us a prime opportunity to improve D.A.G.G.E.R. by also improving the Shadow Drive v2 implementation. The initial ShdwDrive v2 features being implemented and made available to the public are intentionally limited, with a focus on experimenting, learning, and improvements. Additional ShdwDrive v2 features may be rolled out as we progress through the testnet series.

The Purpose of the Testnet

One of our key objectives is to observe how D.A.G.G.E.R. handles a mix of large and small transactions generated by thousands of simultaneous users interacting with an application. We also aim to evaluate its performance when file uploads and erasure coding processes are running concurrently. These tests will yield invaluable data on gossip communication channels, consensus mechanisms, scalability, peer-to-peer communications, and more. Additionally, identifying any gaps in our data-points on the network stack will guide us in enhancing our network telemetry.

The testnet serves as a proving ground for our technology, allowing us to demonstrate the capabilities of D.A.G.G.E.R. as a Byzantine Fault Tolerant (BFT) distributed storage protocol. As it powers chosen aspects of ShdwDrive v2, we will be testing the resilience of the protocol to make sure it will continue functioning correctly even if some nodes fail or behave maliciously.

The testnet also offers a controlled space to optimize performance, stress-test the system under high-volume “real” user activity, and gain valuable insights that will inform future developments. Each testnet targets multiple technical objectives, with some being present at the launch while others will roll out as either live updates or network restarts. The timing and order of these technical roadmap objectives may vary depending on our progress and learnings. It's a critical testing phase and an exciting step in our journey towards revolutionizing cloud storage.

Navigating the Uncharted

While our D.A.G.G.E.R. technology boasts a theoretical capability of up to one million transactions per second (TPS), we anticipate that real-world constraints will yield lower TPS during live testnets. Influencing factors such as internet bandwidth limitations, RPC traffic, browser constraints, and other known front-end inefficiencies could potentially reduce throughput to a range of 5,000 to 100,000 TPS for this initial testnet. 

A fascinating aspect of D.A.G.G.E.R. is that the wide variance in potential throughput is due to our control over the time-to-finality for blocks - a feature we aim to study and optimize during this test phase. This is yet another testament to the unique and adaptable design of D.A.G.G.E.R.!

As we embark on this exciting journey, it's crucial to set realistic expectations. We anticipate encountering challenges such as crashes, bugs, and unexpected occurrences that may necessitate restarting and troubleshooting. However, these hurdles are integral to the refinement of D.A.G.G.E.R. and are stepping stones towards revolutionizing decentralized data storage. We embrace these challenges as opportunities for growth and innovation, propelling us closer to our ultimate goal.

Communication and Feedback

We encourage testnet participants to report any issues or unexpected behaviors through our public GitHub repository. This allows us to quickly address problems and also handle any security matters following industry best practices.

Submit issues here: https://github.com/genesysgo/dagger-testnet-issues

Conclusion

The D.A.G.G.E.R. Testnet Series represents a major step toward our goal of creating versatile enterprise-quality distributed ledger technology. Furthermore, in showcasing ShdwDrive v2 as our application, we aim to demonstrate the scale and potential of D.A.G.G.E.R. as a core protocol upon which a new ecosystem of developers can flourish. We look forward to gathering invaluable insights from the community during this Testnet Series. Together, we can build an ecosystem that makes data open, accessible, and secure for all. Stay tuned for updates on our progress!

Technical Objectives for the Testnets

Note: The exact timing and availability of the items listed below are TBD. The objectives are subject to change based on development learnings and market priorities. We will be releasing a more detailed roadmap on October 20th.

High-Priority Objectives:

• Full-stack implementation of all five key D.A.G.G.E.R. modules
• Release of binaries for node operators, enabling ecosystem members to join the network and assist with testing
• Block explorer for tracking transactions and blocks
• Telemetry testing across the network for performance monitoring
• Deployment of snapshots for system upgrades

Additional Objectives:

• Improvements to snapshots and rejoining processes
• Lite client for graph retrieval and reconstruction
• Improve documentation on architecture, APIs, builds, configurations
• Create wallet integrations, blockchain bridges, demo dApps, and documentation for external developers
• Integration of SHDW/USDC utility
• ShdwDrive v2 Whitepaper
• Improve data indexing infrastructure
• Add features to reference explorer
• Implement defenses against common attacks like packet spamming, node impersonation, etc.

‍