Test Procedures

Our usual SSD test procedure was not designed to handle multi-device tiered storage, so some changes had to be made for this review and as a result much of the data presented here is not directly comparable to our previous reviews. The major changes are:

  • All test configurations were running the latest OS patches and CPU microcode updates for the Spectre and Meltdown vulnerabilities. Regular SSD reviews with post-patch test results will begin later this month.
  • Our synthetic benchmarks are usually run under Linux, but Intel's caching software is Windows-only so the usual fio scripts were adapted to run on Windows. The settings for data transfer sizes and test duration are unchanged, but the difference in storage APIs between operating systems means that the results shown here are lower across the board, especially for the low queue depth random I/O that is the greatest strength of Optane SSDs.
  • We only have equipment to measure the power consumption of one drive at a time. Rather than move that equipment out of the primary SSD testbed and use it to measure either the cache drive or the hard drive, we kept it busy testing drives for future reviews. The SYSmark 2014 SE test results include the usual whole-system energy usage measurements.
  • Optane SSDs and hard drives are not any slower when full than when empty, because they do not have the complicated wear leveling and block erase mechanisms that flash-based SSDs require, nor any equivalent to SLC write caches. The AnandTech Storage Bench (ATSB) trace-based tests in this review omit the usual full-drive test runs. Instead, caching configurations were tested by running each test three times in a row to check for effects of warming up the cache.
  • Our AnandTech Storage Bench "The Destroyer" test takes about 12 hours to run on a good SATA SSD and about 7 hours on the best PCIe SSDs. On a mechanical hard drive, it takes more like 24 hours. Results for The Destroyer will probably not be ready this week. In the meantime, the ATSB Heavy test is sufficiently large to illustrate how SSD caching performs for workloads that do not fit into the cache.

Benchmark Summary

This review analyzes the performance of Optane Memory caching both for boot drives and secondary drives. The Optane Memory modules are also tested as standalone SSDs. The benchmarks in this review fall into three categories:

Application benchmarks: SYSmark 2014 SE

SYSmark directly measures how long applications take to respond to simulated user input. The scores are normalized against a reference system, but otherwise are directly proportional to the accumulated time between user input and the result showing up on screen. SYSmark measures whole-system performance and energy usage with a broad variety of non-gaming applications. The tests are not particularly storage-intensive, and differences in CPU and RAM can have a much greater impact on scores than storage upgrades.

AnandTech Storage Bench: The Destroyer, Heavy, Light

These three tests are recorded traces of real-world I/O that are replayed onto the storage device under test. This allows for the same storage workload to be reproduced consistently and almost completely independent of changes in CPU, RAM or GPU, because none of the computational workload of the original applications is reproduced. The ATSB Light test is similar in scope to SYSmark while the ATSB Heavy and The Destroyer tests represent much more computer usage with a broader range of applications. As a concession to practicality, these traces are replayed with long disk idle times cut short, so that the Destroyer doesn't take a full week to run.

Synthetic Benchmarks: Flexible IO Tester (FIO)

FIO is used to produce and measure artificial storage workloads according to our custom scripts. Poor choice of data sizes, access patterns and test duration can produce results that are either unrealistically flattering to SSDs or are unfairly difficult. Our FIO-based tests are designed specifically for modern consumer SSDs, with an emphasis on queue depths and transfer sizes that are most relevant to client computing workloads. Test durations and preconditioning workloads have been chosen to avoid unrealistically triggering thermal throttling on M.2 SSDs or overflowing SLC write caches.

Introduction SYSmark 2014 SE
POST A COMMENT

97 Comments

View All Comments

  • Billy Tallis - Tuesday, May 15, 2018 - link

    FIO version 3.6, Windows binaries from https://bluestop.org/fio/ (and Linux binaries compiled locally, for the other reviews). The only fio settings that had to change when moving the scripts from Linux to Windows was the ioengine option for selecting which APIs to use for IO. On Linux, QD1 tests are done with synchronous IO and higher queue depths with libaio, and on Windows all the queue depths used asynchronous IO.

    In this review I also didn't bother secure erasing the drives between running the burst and sustained tests, but that shouldn't matter much for these drives.
    Reply
  • IntelUser2000 - Tuesday, May 15, 2018 - link

    So looking at the original Optane Memory review, the loss must be due to Meltdown as it also gets 400MB/s. Reply
  • Billy Tallis - Tuesday, May 15, 2018 - link

    The Meltdown+Spectre workarounds don't have anywhere near this kind of impact on Linux, so I don't think that's a sufficient explanation for what's going on with this review's Windows results.

    Last year's Optane Memory review only did synthetic benchmarks of the drive as a standalone device, not in a caching configuration because the drivers only supported boot drive acceleration at that time.
    Reply
  • IntelUser2000 - Tuesday, May 15, 2018 - link

    The strange performance may also explain why its sometimes faster in caching than when its standalone.

    Certainly the drive is capable of faster than that looking at raw media performance.

    My point with the last review was that, whether its standalone or not, the drive on the Optane Memory review is getting ~400MB/s, while in this review its getting 160MB/s.
    Reply
  • tuxRoller - Wednesday, May 16, 2018 - link

    As Billy said you're comparing the results from two different OSs' Reply
  • Intel999 - Tuesday, May 15, 2018 - link

    Will there be a comparison between the uber expensive Intel approach to sped up boot times with AMD's free approach using StorageMI? Reply
  • Billy Tallis - Tuesday, May 15, 2018 - link

    Yes, but since my AMD system is a Threadripper, it won't actually represent any cost savings compared to the systems tested in this review. Reply
  • evernessince - Wednesday, May 16, 2018 - link

    AdoredTV already did a video showing the performance improvements from StoreMi.

    https://www.youtube.com/watch?v=D3-SqJHYzC0

    AMD's solution works in the same way, in that as you run programs it stores data to the cache drive. The big difference is AMD's solution let's you use any SSD as a cache drive. This means it can be any size and it doesn't require an addition purpose. This is especially important, give the huge price tag of the larger optane drives.

    Speed wise though, assuming the Intel SSD is actually big enough to cache all your data, they are about equal. Of course, the AMD solution would be slower if you used a really low end SSD as your cache drive. It could also be much faster if you used a really good SSD though. The Intel optane drive has performance numbers similar to a 960 evo. The problem for Intel though are the small sizes and large prices. $200 for only 118GB of space is not a good solution. You could get double that space with a brand new 250GB 960 evo and it costs half as much. That's assuming you want to keep that drive for caching only, you could simply use your current SSD with the AMD solution and save $200+ altogether.

    I simply don't see a universe where Optane makes sense.
    Reply
  • CheapSushi - Wednesday, May 16, 2018 - link

    You realize you can use Optane like any other SSD right? You can even use it with StorageMI. Reply
  • MDD1963 - Tuesday, May 15, 2018 - link

    There will be no tiny Optane things inserted into/wasting an M.2 NVME slot making it SEEM like I have a 960/970; there will be a 960/970. :) Reply

Log in

Don't have an account? Sign up now