DeepSeek Releases 3FS & Smallpond Framework

DeepSeek significantly boosted open-source capabilities on February 28, 2025, unveiling the Fire-Flyer File System (3FS) and the Smallpond data processing framework. These tools are designed to revolutionize data access and processing, particularly for AI training and inference.

? Day 5 of #OpenSourceWeek: 3FS, a powerful engine for all DeepSeek data access.

Fire-Flyer File System (3FS) – a parallel file system maximizing the bandwidth of modern SSDs and RDMA networks.

⚡ 6.6 TiB/s aggregate read throughput (180-node cluster) ⚡ 3.66 TiB/min…

— DeepSeek (@deepseek_ai) February 28, 2025

Table of Contents

• Fire-Flyer File System (3FS)
• Smallpond Framework
• Quick Start: 3FS and Smallpond
• Troubleshooting and Monitoring
• Summary

Fire-Flyer File System (3FS)

3FS is a high-performance, distributed file system built for modern SSDs and RDMA networks. It offers a robust shared storage solution, simplifying distributed application development.

Understanding RDMA

Remote Direct Memory Access (RDMA) bypasses operating system limitations, enabling direct data transfer between the memory of two computers. This results in faster, more efficient communication.

Key 3FS Features

• Unmatched Performance & Ease of Use:
  • 6.6 TiB/s aggregate read throughput (180-node cluster).
  • 3.66 TiB/min throughput on the GraySort benchmark (25-node cluster).
  • 40 GiB/s peak throughput per client node for KVCache lookups.
• Disaggregated Architecture:
  • Combines the throughput of thousands of SSDs with the network bandwidth of hundreds of storage nodes.
  • Offers locality-oblivious storage access for applications.
• Robust Consistency:
  • Employs Chain Replication with Apportioned Queries (CRAQ) for strong consistency, simplifying application coding.
• Standard File Interfaces:
  • Uses stateless metadata services based on a transactional key-value store (e.g., FoundationDB).
  • Maintains a familiar file interface, eliminating the need for new API learning.

Supported Workloads

• Data Preparation: Efficiently manages large volumes of intermediate outputs from data analytics pipelines.
• Dataloaders: Enables random access to training samples across compute nodes, eliminating prefetching or dataset shuffling.
• Checkpointing: Supports high-throughput parallel checkpointing for large-scale training.
• KVCache for Inference: Offers a cost-effective, high-throughput alternative to DRAM-based caching with significantly increased capacity.

Performance Benchmarks

Extensive testing validates 3FS performance. A read stress test on a large cluster achieved 6.6 TiB/s aggregate read throughput, even with concurrent training job traffic.

Smallpond Framework

Smallpond, designed to complement 3FS, is a lightweight, distributed data processing framework. It uses DuckDB as its compute engine and stores data in Parquet format on a distributed file system (like 3FS).

Key Smallpond Features

• High Performance: DuckDB provides native-level performance for efficient data processing.
• Scalability: Handles petabyte-scale data without memory bottlenecks thanks to high-performance distributed file systems.
• Simplicity: Easy deployment and maintenance due to the absence of long-running services or complex dependencies.
• Efficient Data Processing: A two-phase approach to sorting large datasets improves performance and efficiency (e.g., sorted 110.5 TiB across 8,192 partitions in under 30 minutes).
• Seamless 3FS Integration: Leverages 3FS's high throughput and strong consistency.

Quick Start: 3FS and Smallpond

3FS Installation

Clone the repository and install dependencies:

1. git clone https://github.com/deepseek-ai/3fs
2. cd 3fs
3. git submodule update --init --recursive
4. ./patches/apply.sh

Consult the 3FS documentation for further details.

Smallpond Quick Start

• Ensure Python 3.8 is installed.

• Install Smallpond: pip install smallpond

• Initialize a Smallpond session: import smallpond; sp = smallpond.init()

• Load Parquet data: df = sp.read_parquet("path/to/dataset/*.parquet")

• Repartition data (examples):

  • df = df.repartition(3)
  • df = df.repartition(3, by_row=True)
  • df = df.repartition(3, hash_by="host")

• Transform data (examples):

  • df = df.map('a b as c')
  • df = df.map(lambda row: {'c': row['a'] row['b']})

• Save data: df.write_parquet("path/to/output/dataset.parquet")

• Run a Smallpond job: sp.run(df)

Troubleshooting and Monitoring

Smallpond offers monitoring and debugging tools. Log analysis helps resolve execution issues. Comprehensive documentation, tutorials, and use cases are available through the official support channels.

Summary

The open-source release of 3FS and Smallpond represents a significant advancement in data processing. Their high performance, ease of use, and consistency empower developers and researchers. These tools provide a powerful infrastructure for modern, data-intensive applications.

The above is the detailed content of DeepSeek Releases 3FS & Smallpond Framework. For more information, please follow other related articles on the PHP Chinese website!