Understanding Deduplication in ENAS

UniFi Drive on ENAS supports block-level deduplication, reducing storage usage and improving efficiency. This guide explains how deduplication works and its recommended use scenarios. If you're looking to back up NAS data, click here. 

How Deduplication Works

Deduplication works by identifying identical data blocks and storing only one copy of each unique block. If the same data appears multiple times across your storage pool, ENAS keeps a single copy while maintaining full data integrity and accessibility. 

Deduplication does not:

• Compress data
• Encrypt data
• Reduce network transfer size
• Improve upload/download speed

It only reduces the amount of duplicate data physically stored on drives.

Recommended Use Cases

Deduplication works best in environments with large amounts of repeated or similar data, such as:

• Backup repositories (VM backups, system backups, snapshots)
• Virtual machine images based on similar OS templates
• Datasets with repetitive or duplicated data
• Environments where storage efficiency is more important than peak write performance

Performance Considerations

Deduplication can significantly reduce storage usage, but it comes with a performance trade-off. Because the system must scan and compare data during upload, you may notice:

• Slower upload and save times
• Increased system resource usage

If performance and speed are your top priorities, consider disabling this feature.

When Deduplication May Not Help

Deduplication provides limited benefit for:

• Large, unique media files (e.g., raw videos, high-resolution photos)
• Encrypted datasets
• Workloads with mostly unique data

In these cases, deduplication may add overhead without significant storage savings.

Requirements

• ENAS
• Metadata Group (Highly Recommended)
  • Deduplication relies heavily on system memory to maintain its deduplication table. To reduce memory usage and performance impact, it is strongly recommended to configure a Metadata Group.
    • A Metadata Group stores deduplication metadata on SSDs rather than in system RAM.
    • Minimum requirement: 2 × SATA SSDs (configured as a mirror)
    • To configure a Metadata Group: Navigate to Settings > Storage Pools and select the storage pool containing the Shared Drive with deduplication enabled. 

Enabling Deduplication

You can enable deduplication:

• When creating a new Shared Drive
• Or later in the settings of an existing Shared Drive

Estimate Storage Savings

Before enabling deduplication, you can run Estimate Deduplication Ratio to analyze the amount of duplicate data in the selected Shared Drive.

The deduplication ratio compares your total data size (logical) to the actual disk space used (physical).

• Ratio 2.0: Your data takes up half the storage space (≈50% is duplicate data)
• Ratio 3.0: Your data uses only one-third of the original space (≈67% is duplicate data)

Higher ratios indicate more duplicate data and greater potential storage savings.

As a general guideline, enable deduplication when the ratio is greater than 2.0. This means at least half of your data is redundant, and deduplication can significantly improve storage efficiency.

Important Behavior

• Deduplication applies only to data written while it is enabled.
• Files uploaded before enabling deduplication will not be processed retroactively.
• Files uploaded after disabling deduplication will not be deduplicated.

Important Notes & Considerations

Storage Usage Is Not Tracked Per Shared Drive

Deduplication can be enabled or disabled independently for each Shared Drive. However, deduplication operates at the storage pool level, identifying and eliminating duplicate data blocks across all Shared Drives in the same pool with deduplication enabled.

For example, if the same file exists in both Shared Drive A and Shared Drive B, only one physical copy of the duplicate blocks is stored in the pool. This reduces overall storage consumption across the pool.

Because deduplicated blocks are shared at the pool level, physical storage usage cannot be accurately attributed to individual Shared Drives. As a result, storage usage is reported at the pool level rather than per Shared Drive.

Memory Protection Mechanism

If no Metadata Group is configured, deduplication metadata is stored in system memory (RAM).

To protect system stability, ENAS automatically disables deduplication if the deduplication table exceeds 4 GB of RAM.

In this case:

• Existing data remains intact and fully accessible.
• No data is lost.
• Only new data will no longer be deduplicated.

It is strongly recommended to configure a Metadata Group to continue using deduplication safely.