The DD Fallacy

Discussion in 'Server Stuff' started by Mun, Sep 3, 2014.

  1. Mun

    Mun Administrator

    In many VPS posts and reviews these days, extreme DD tests results are used to gimmick the public into thinking they are getting a better service then they actually are. Here is my attempt to show you exactly what I mean.

    Lets start off with the basic DD command of choice:
    Code:
    dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync; unlink test
    
    What this in theory does is write absolutely empty space to a file size of 1.1 GBs.

    This can be easily be proven by using tail:
    Code:
    mun@HP-Server:/media/1tb-raid$ ls 
    Games  oldcomp  test
    mun@HP-Server:/media/1tb-raid$ tail test
    
    ^Cmun@HP-Server:/media/1tb-raid$ 
    
    As you can see nothing is in that file.

    Moving on... lets do a quick test on really shitty hardware. Lets use my 4GB flash drive from San Disk.
    Code:
    Disk /dev/sdh: 4022 MB, 4022337024 bytes
    124 heads, 62 sectors/track, 1021 cylinders, total 7856127 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x00000000
    
    I first made the file system ext4, and then ran the usual DD command that is listed above. The performance was underwhelming.

    Code:
    root@HP-Server:/media/flash-drive# dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync; unlink test
    
    16384+0 records in
    16384+0 records out
    1073741824 bytes (1.1 GB) copied, 342.222 s, 3.1 MB/s
    
    Lets retry this now, with a little magic!

    Code:
    root@HP-Server:/media/flash-drive# dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync; unlink test
    16384+0 records in
    16384+0 records out
    1073741824 bytes (1.1 GB) copied, 19.6389 s, 54.7 MB/s
    root@HP-Server:/media/flash-drive# dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync; unlink test
    16384+0 records in
    16384+0 records out
    1073741824 bytes (1.1 GB) copied, 19.9902 s, 53.7 MB/s
    
    ... and that same flash drive can be as quick as a cheap hard drive.

    So what is happening? Simple, compression and devious commands. Since the file has nothing in it it is easily compressible to insane factors, to the point you are almost writing nothing physical data wise to the disk. Thus causing a magic trick of performance.

    Now you maybe thinking that is just a flash drive, that doesn't mean anything in terms of an actual hard drive. You, sadly would be mistaken. In actuality this is exactly what can be done with them, and to an even greater extent. Lets try this on my raid 1, WD Green 1TB raid X 2. (raid 1 being mirroring)

    Code:
    root@HP-Server:/media/1tb-raid# dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync; unlink test
    16384+0 records in
    16384+0 records out
    1073741824 bytes (1.1 GB) copied, 1.33738 s, 803 MB/s
    root@HP-Server:/media/1tb-raid# dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync; unlink test
    16384+0 records in
    16384+0 records out
    1073741824 bytes (1.1 GB) copied, 1.02586 s, 1.0 GB/s
    root@HP-Server:/media/1tb-raid# dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync; unlink test
    16384+0 records in
    16384+0 records out
    1073741824 bytes (1.1 GB) copied, 0.965919 s, 1.1 GB/s
    root@HP-Server:/media/1tb-raid# dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync; unlink test
    16384+0 records in
    16384+0 records out
    1073741824 bytes (1.1 GB) copied, 0.987057 s, 1.1 GB/s
    root@HP-Server:/media/1tb-raid# 
    
    As you can see, I am getting ridiculous speed on my 1tb hard drives as well, hard drive specs are below:
    Code:
    root@HP-Server:/media/1tb-raid# hdparm -I /dev/sdb
    
    /dev/sdb:
    
    ATA device, with non-removable media
        Model Number:       WDC WD10EARS-00Y5B1                    
        Serial Number:      WD-WCAV5F084053
        Firmware Revision:  80.00A80
        Transport:          Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6
    Standards:
        Supported: 8 7 6 5 
        Likely used: 8
    Configuration:
        Logical        max    current
        cylinders    16383    16383
        heads        16    16
        sectors/track    63    63
        --
        CHS current addressable sectors:   16514064
        LBA    user addressable sectors:  268435455
        LBA48  user addressable sectors: 1953525168
        Logical/Physical Sector size:           512 bytes
        device size with M = 1024*1024:      953869 MBytes
        device size with M = 1000*1000:     1000204 MBytes (1000 GB)
        cache/buffer size  = unknown
    Capabilities:
        LBA, IORDY(can be disabled)
        Queue depth: 32
        Standby timer values: spec'd by Standard, with device specific minimum
        R/W multiple sector transfer: Max = 16    Current = 16
        Recommended acoustic management value: 128, current value: 254
        DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6 
            Cycle time: min=120ns recommended=120ns
        PIO: pio0 pio1 pio2 pio3 pio4 
            Cycle time: no flow control=120ns  IORDY flow control=120ns
    Commands/features:
        Enabled    Supported:
          *    SMART feature set
              Security Mode feature set
          *    Power Management feature set
          *    Write cache
          *    Look-ahead
          *    Host Protected Area feature set
          *    WRITE_BUFFER command
          *    READ_BUFFER command
          *    NOP cmd
          *    DOWNLOAD_MICROCODE
              Power-Up In Standby feature set
          *    SET_FEATURES required to spinup after power up
              SET_MAX security extension
              Automatic Acoustic Management feature set
          *    48-bit Address feature set
          *    Device Configuration Overlay feature set
          *    Mandatory FLUSH_CACHE
          *    FLUSH_CACHE_EXT
          *    SMART error logging
          *    SMART self-test
          *    General Purpose Logging feature set
          *    64-bit World wide name
          *    {READ,WRITE}_DMA_EXT_GPL commands
          *    Segmented DOWNLOAD_MICROCODE
          *    Gen1 signaling speed (1.5Gb/s)
          *    Gen2 signaling speed (3.0Gb/s)
          *    Native Command Queueing (NCQ)
          *    Host-initiated interface power management
          *    Phy event counters
          *    NCQ priority information
          *    DMA Setup Auto-Activate optimization
          *    Software settings preservation
          *    SMART Command Transport (SCT) feature set
          *    SCT Features Control (AC4)
          *    SCT Data Tables (AC5)
              unknown 206[12] (vendor specific)
              unknown 206[13] (vendor specific)
    Security: 
        Master password revision code = 65534
            supported
        not    enabled
        not    locked
        not    frozen
        not    expired: security count
            supported: enhanced erase
        192min for SECURITY ERASE UNIT. 192min for ENHANCED SECURITY ERASE UNIT. 
    Logical Unit WWN Device Identifier: 50014ee204c93b0c
        NAA        : 5
        IEEE OUI    : 0014ee
        Unique ID    : 204c93b0c
    Checksum: correct
    root@HP-Server:/media/1tb-raid# 
    
    

    So, what does all this data mean? Simple, using a DD test to show performance is idiotic, and nothing more then a magic trick in most cases. And if you don't believe me, try it for yourself.

    How to try it for yourself: (Be very careful, I suggest only people with a good computer grasp try the below)
    First find a drive you have no data on, if you do have data on it then back it up please. This process will delete all of its contents.

    Run Ubuntu / Debian on a system of your choice (make sure not to run these commands on the main drive or it will go poof and kill your install)

    mkfs.ext4 as a comparison point:
    Code:
    mkfs.ext4 /dev/yourdevice
    
    Next run the dd test:
    Code:
    dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync; unlink test
    
    Run this a few time and keep track of your results.

    Next MAGIC!!!!!

    Install btrfs support:
    Code:
    apt-get install btrfs-tools
    
    mkfs.btrfs as second point:
    Code:
    umount /dev/yourdevice
    mkfs.btrfs /dev/yourdevice
    
    Edit fstab:
    Code:
    nano /etc/fstab
    #:: add
    /dev/yourdevice        /media/where_i_want_to_mount      btrfs   defaults,compress=lzo   0       1
    
    #:: save file and exit
    mount -a
    
    
    Now run the DD test again, and record the data:
    Code:
    dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync; unlink test
    
    As you can clearly see at this point, the DD test is completely off in comparison to the slow ext4.
     

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice