Data loss prevention

This article documents practices and tips for preventing data loss.

The goal of this article is defining a clear set of rules to follow for significantly reducing the probability of data loss. This includes unexpected data losses such as flash storage failure or unsolicited disposal.

Taking data loss prevention measures is like wearing a bycicle/motorcycle helmet and like a vehicle's seat belt: Even if a crash is rare and unlikely, taking precautions against it is worthed it merely for the sense of safety it provides.

Storing data at least twice

The greatest reduction of data loss risk comes from accessibly storing data twice or more (“S.E.T.” = “store everything twice”).

Even with reliable data storage, the human factor should be taken into consideration.

Another risk factor regardless of reliable hardware is malware intrusion.

An even safer strategy for storing data is the “3-2-1 backup strategy”, which implies storing data three times, on at least two different data storage types, and one off-site backup in case of an unlikely natural disaster. However, many people have no physical access to an off-site storage location.

Data storage types

Clouds are only redundant

Related section: § Online content

Cloud storage is for convenience, not archival.

One should never rely on an online service as one's primary place to store important data, because one does not have technical control over an online service.

Cloud storage is vulnerable to technical difficulties that may interfere with availability of data, and as many sites, including file hoster MegaUpload and video platforms VidMe and Vlare.TV have shown, legal and financial issues potentially endanger any data stored in an online service.

This is not limited to conventional cloud and file sharing services such as Google Drive, OneDrive (formerly SkyDrive), DropBox, MediaFire and Mega.Co.NZ, but also applies to email services (GMail, ProtonMail, Web.DE, GMX.net, EMail.T-Online.de, etc.) and social media platforms for hosting and/or sharing photo/video content (e.g. YouTube, Dailymotion, Instagram etc.).

When using such services for redundant backups, it is recommended to first review their default retention settings (especially on electronic mail services), retention policies and export options.

Flash storage is only for convenience

Flash storage has known benefits over other types of data storage, but data retention is not one of them.

Flash storage is also the least predictable data storage device type: Some flash storage devices might happen to work well beyond a decade, while others may start malfunctioning within two years, less than a year, or perhaps months of operation.

Flash storage can be affected by both incremental and abrupt failures, while, in case of failures, optical media is mainly affected by incremental failure only and hard disk drives mostly by abrupt mechanical failures.

Bit fading

This seal marks flash storage of low quality created by a wannabe flash storage vendor. Avoid flash storage highlighted with this seal like the plague (except for testing purposes).

Flash storage suffers from an effect known as bit fading, which causes transistors that store the data to lose their charge over time. This loss of data integrity is visible as glitch effects in photos and video footage, or hearable as noise in wavesound files, and gaps or hiccup sounds in compressed audio formats.

While only causing logical errors itself, the bit fading effect does deterioate with each rewrite cycle due to weardown of the transistors, leading to shorter information retention spans. Flash storage of higher density (multi-level cell (MLC) NAND storage such as MicroSD-XC cards) hold data in smaller transistors, which are prone to faster bit fading, and higher storage temperatures accellerate bit fading.

Although flash storage can not suffer from mechanical failure like hard drives can, and quality flash storage (e.g. those from the brands SanDisk, Transcend, Panasonic, Toshiba and Intenso) might be able to retain full data integrity for years under good conditions such as moderate (20°C) or cold (10°C or less) storage temperatures, the purpose of flash storage is being a physically robust, compact and fast, but short-term storage.

Bit fading in low quality flash storage such as that of the vendor Hama might lead to data loss within months and other random defects such as ceasing to function after being supplied with an unstable voltage, a so-called “brown-out”. Another possible cause of malfunction and instant data loss in low-quality flash storage is a voltage spike caused by an unstable power source.

Many models of flash storage, especially those of lower quality, do not report damaged data as such to the computer, and return the data as if it were undamaged. The only way for a computer to guess a damaged sector on such storage is by recognizing the flash storage's prolonged delay to return data from it, caused by a failed error correction attempt.

Retention

Data retention on flash storage is facilitated by refreshing the charge in the transistors that hold information, usually done by the flash storage's controller upon reading data. Flash storage of higher quality does automatically self-refresh data on idle power.

Its physical robustness makes flash storage a viable option as secondary backup. Preferably, data integrity should be verified at least yearly, more or less often depending on the importance of the data stored and how many copies exist on other media.

The precluded badblocks command on Linux, and the graphical third-party freeware HDD Scan for Windows can scan the readability of data.

Capacity fraud

Flash storage from questionable vendors may pretend to have a higher capacity than they actually do.

Not only is this an issue for cheap no-mame brand flash storage from online shops, but even flash storage from low quality vendors such as Hama that at least is expected to report its actual capacity to the computer may be compromised by a fraudulent supply chain, as has happened in 2008.

Software such as F3 (“fake flash fraud”) for Linux and h2TestW for Windows is able to verify the actual capacity of flash storage. Except for testing purposes, flash storage with fraudulent capacity reading should not be relied upon at all due to presumably poor manufacturing quality.

Optical media is your friend

Main article: Benefits of optical media

Optical media has a high fault tolerance, a supreme life expectancy and is the only type of data storage that is immune against sudden unpredictable failures such as mechanical failure (as in hard disk drives), electrical failure (as in flash storage) and magnetic vulnerability (hard disk drives and LTO tape storage).

Conventional optical media can be expected to retain full data integrity for at least ten years. Some variants such as Verbatim Gold (rated for 100 years) and M-Disc (rated for 1000 years) can be expected to reliably last significantly longer.

Optical media also has the cheap prices per piece, and CD/DVD writers are highly affordable, with prices commonly being below €/$/£ 20.

Although recoedable Blu-Ray discs themselves are very affordable as well, Blu-Ray hardware is somewhat more expensive, yet still a fraction of the prices of LTO tape drives, and with random access support.

Further more, the quality and data integrity of optical media is accurately measureable. It can be determined by measuring the rate of correctable data integrity errors, which is far more accurate than S.M.A.R.T., which relies on rough guesses.

With packet writing (also known as Live File System), incremental file actions known from hard disk drives and flash storage can be performed on optical media.

True write protection can be achieved by finalization of write-once media and reading optical discs through a ROM (read-only memory) drive whose laser beam is insufficiently strong to write any data.

Hard disk drives

Hard disk drives have a very high capacity per price.

While lesser used hard disk drives can be expected to function properly for at least a decade, mechanical failures are not reliabily predictable and sudden, as opposed to optical drives, which contains less delicate mechanical parts and a laser beam which can't physically touch the disc, rather than a magnetic head which floats imminently above the spinning disk, risking a so-called head crash in case of malfunction, which would physically scratch the disk.

In addition, hard drives have a relatively poor physical robustness due to delicate and relatively fragile internal parts.

Questionable vendors

See also: List of flash storage vendors

Storage by questionable vendors is at best suitable as temporary sandbox storage.

Storage by questionable vendors not only is more prone to failure due to lower manufacturing quality, but flash and hard drive storage by such vendors may fraudulently misreport their capacity, pretending to the computer to have a higher capacity than they actually have.

Common software that can verify the actual capacity of such storage is Fight Flash Fraud (or “F3”) for Linux, and h2testW for Windows.

One chance per vendor

Do not trust vendors again whose data storage has betrayed you.

A data storage device already starting to malfunction under stable operation conditions and at a young device age, such as bit fading data loss within months of purchase, suggests low quality manufacturing and possibly a questionable delivery chain of the vendor.

After the first incident with a specific vendor's data storage device that suggests low manufacturing quality, any data storage devices by that vendor, for anything beyond temporary, very short-term storage such as testing, storage of temporary data and file transfering, should not be trusted anymore.

Data on existing data storage from untrustworthy vendors such as Hama should be backed up immediately.

Handling of data

Prioritization

If one currently is restricted to a tight budget for purchasing data storage, it is beneficial to prioritize which data to back up.

Non-replaceable information and works created by oneself, including digital photos, video recordings, documents, XML-based project files (used by video editing software, for example) and possibly voice notes have the highest priority for being backed up. Smaller files such as plain text and rich text files can be backed up to more locations quickly and easily.

Information that can be replaced easily, such as software that can be re-downloaded from the Internet, has a lower priority for being backed up.

On damage, back up first

When noticing damaged data or any other strange behaviour of a data storage device, a common mistake is launching a file system repair software such as chkdsk or fsck, also known as file system surgery.

Instead, in order to migitate the damage, the best thing to do first is creating a full-disk image before conducting any write access to the device, after which file system actions and modifications can be conducted, if necessary.

Naming

It is recommended to use descriptive, dated and optionally numbered file and/or directory names to facilitate re-finding the files.

Revision history

Make sure to back up your work from time to time by saving it in a revision history by saving as or copying to a file name with time stamp and/or number.

For documents that are mainly plain text, the file size of each revision is marginal compared to the size of modern disk storage.

A revision history also allows resetting to an earlier version of one's work in case of an error, whether it be a human error or software bug (e.g. in video editing) or a writing error (e.g. caused by lack of free space or an unexpected system crash).

Data portability

This is more of a concern for online services such as mailing and messaging services.

Online services

Make sure that user data can be exported, which means downloaded to machine- or human-readable files, to allow creating backups and to prevent vendor lock-in.

Data portability in online services such as social media platforms has improved significantly since the passing of GDPR §20.

Software

Some software (e.g. messenger software Skype and WhatsApp) lacks native export functionality and stores messages in a closed-source proprietary database format, and user data stored by mobile phone software in a private app data folder, such as /data/ on Android mobile phones, which have a locked bootloader by default, is locked in, making backing up that user data near-impossible.

Devices

Another common hindrance to data portability in earlier mobile phones such as the 2005 Nokia 6111 were proprietary data connectors and the need for special proprietary software by the vendor to access user data such as photos stored on the device.

Even though most non-Apple mobiles phones since circa 2009 have a USB-B Micro connector, some earlier USB mobile phones such as the 2008 Nokia E71, which used the USB port only for data and not charging, depend on vendor's software to access some parts of the internal storage.

Another example device with complete lack of built-in data portability is the 2011 Telekom Entertain media receiver 303 television video recorder. Not only is it impossible to create backups of recording (despite of multiple existing USB ports) and move recordings elsewhere to free up disk space without having to delete existing recordings, but its operating system which refuses operation when no internet connection is available.

Service for said device has been deprecated in April 2019 due to having been superseded by the Media Receiver 401 device, meaning that all existing recordings are inaccessible since then. The recordings are stored on the device's hard disk in a proprietary format.

Only carry copies of data

On physical media, only copies of data should be carried aroud, because data storage carried around can get lost and is more exposed to the environment (e.g. rapid temperature changes inside a car's cabin).

Additionally, one could consider setting up a remote file server at home (using FTP or WebDAV), also known as a private cloud, if conventional cloud services can not meet one's demands.

Check backup before deletion

When backing up a big directory of many files, some files might not have been copied due to the length of file names, special characters in file names that are unsupported by the target file system and/or operating system, and misconfigured read access permissions on the origin file system.

Before deleting something from the origin storage (if necessary to free up space), check again if it is readable from the backup.

OS partitions ain't for backups.

The partition of the operation system's installation is not suitable for backups.

It may be used for redundant backups, but should not be relied upon as a long-term storage location, being subject to constant change and might even to a file system check and repair, which could be initiated by the operating system after an unexpected power-off. In addition, it has the highest likelihood of file corruption in case of malware intrusion.

This applies equally to the internal storage of mobile phones and other mobile devices including tablets, navigation devices, digital cameras, camcorders, audio recorders and pocket multimedia players, where a defect of the device could make any data stored on the internal storage inaccessible and only recoverable with heavy forensic work, if at all.

See also: Benefits of expandable storage

The first 100 megabytes

Most USB sticks and SD cards come pre-formatted with the widely supported but technically outdated FAT32 or exFAT file systems.

Before doing large file transfers from/to flash storage, if not already mirrored on the hard drive, it is recommended to create a disk dump of the data partition's first 100 MB, which includes the master file table (MFT), an index of file names and paths, logical block addresses (LBAs), fragmentation information and file attributes.

This is recommended because the master file table is prone to corruption from write access during unstable conditions such as a loose connection, unstable power supply (e.g. on a passive USB hub without stable voltage from an external power supply), unexpected removal and unexpected system power-off.

An example command with the disk dump utility which usually is precluded on all Linux systems (adapt if necessary):

sudo dd if=/dev/sdc of=/media/example/filename.img bs=2048 count=50000

Or using gddrescue (stop after a few seconds by pressing CTRL+C):

sudo ddrescue /dev/sdc /media/example/filename.img

One can find out the device paths using the lsblk (“list block devices”) command.

MTP is deprecated garbage

Main article: Criticism of Media Transfer Protocol

When transferring files from an MTP device such as a mobile phone through USB, direct moves should only be performed on files and folders without subfolders due to the poor stability of MTP (except for a countably small number of files at once).

Folders should be copied instead, the file count byte size compared on the device and the computer, and then deleted from the source MTP device (e.g. mobile phone) when matching.

If possible, high numbers of files should be split between multiple folders within the device itself using a file management software, prior to transfer using MTP.

One should consider transferring the files through File Transfer Protocol (FTP), which is more mature, has a better stability and supports parallelism.

Volatility

Although it sounds obvious to many, it is documented here as a reminder:

RAM drives and recycle bin directories are not for storing important data, a call history is no phonebook, the browsing history is not a bookmark storage and the keyboard software's clipboard buffer is no folder for text files.

For example, when having opened too many tabs on an internet browser, and one is not able to export the browsing session, usually on a mobile phone with locked bootloader (thus poor data portability), it may be tempting to start a clean browsing session in private (incognito) mode.

This however poses the risk of a total loss of the browsing session upon an unexpected power-off or browser crash, and makes some visited sites more difficult to find due to not being logged in the browsing history.

Volatile storage locations usually delete items after a specific time (e.g. 10 weeks of browsing history on Chromium-based browsers), number of new items (e.g. max. 20 clipboard items on the keyboard software precluded on Samsung Galaxy devices) or unstable conditions (e.g. Mozilla Firefox deletes browsing history automatically if it detects a shortage of remaining data storage space).

Anything one does not wish to lose should be copied or dumped into separate files.

Browsing history which may later facilitate re-discovering visited web sites can usually be found in the browser's profile folder under the names places.sqlite (Mozilla Firefox and its forks) and History (Google Chromium and its forks). These files can be backed up if necessary.

For browsing sessions, add-ons such as Tab List allow exporting a URL list of opened tabs into plain text. This could also be helpful to tab hoarders.

Websites that one wishes to look at later should not be stored in one tab's navigation history due to the risk of getting lost and the lack of most tab list exporting add-ons to export the tab history, on Firefox especially since version 57 (Quantum), which discarded the legacy XUL add-on format entirely.

To prevent the loss of online form submissions, ome should make use of a textarea cache add-on. In case the loss of a form has occured (e.g. from a failed form submission), the text contents might sill be recoverable from a RAM dump of the process.

When working on a project, it is recommend to set up a live synchronising directory which automatically backs up the work, to avoid losing progress in case of a technical failure.

Space storage starvation

Software is programmed to work well under stable conditions, which includes functional write access.

However, not all software handles low remaining space well, and even the operating system might act unpredictably and become unstable.

Text editing software that has not been optimized to handle space storage starvation might fail to save a file, possibly resulting in a zero-byte file.

Certain software might noticeably slow down due to malfunctioning caching during such condition.

Mozilla Firefox, a common web browser, evidently automatically erases the oldest entries in the in its browsing history (places.sqlite) during a full disk condition.

Some software might happen to reset its settings due to an unsuccessful write that blanked the configuration file.

Automatic saving in software such as word processors and graphics and video editing software does not work if the partition that holds their temporary directory can not be written to, risking a possible loss of work in case of an unexpected process ending, of which the risk is higher during such unstable conditions.

Low remaining space also leads to aggressive fragmentation of the file system, which negatively affects performance on a hard disk drive.

For these reasons, conditions of low remaining space storage should be avoided.

Archive files

Compressed archive

The main purpose of compressed file archives is compressing files for bundled file sharing.

Compressed archives can be useful for reducing the size of many low-entropy contents such as text documents or uncompressed bitmap image files, but make no big difference for already compressed data such as encoded video.

One should note however that the same number of damaged sectors on which the archive is stored can inflict significantly more damage to the file contents in an extracted state due to the same number of bits representing more information and shared information being stored only once in a solidly compressed archive to prevent wasting space storage.

In addition, retrieval from compressed archives, depending on used parameters, is usually significantly slower than retrieving uncompressed data.

Archive passwords

Although encryption for file archives (e.g. Zip, 7-Zip, RAR, etc.) can be useful for e.g. storing confidential information and allow safely storing malware samples for experimental purposes (for the latter purpose, a dummy password such a space bar character should suffice), they pose a risk of locking the user out due to forgetting the password.

If a unique password is used for a data archive that is not accessed for a long time (e.g. a few years), the risk of forgetting it before being retrieved in future increases.

Use archive passwords with care.

The same applies to other hardly recoverable passwords/PINs such as the Private folder feature of Samsung mobile phones.

Many small files

One should consider putting many small files (e.g. thousands of small files from saved HTML pages) into an archive to increase the stability of the file system and facilitate file transfering.

Date and time information

While some file formats (e.g. photos, videos and log files) store date and time information in the metadata (part of file contents) or the contents themselves, which is independent from the file system's header, other file types (e.g. voice notes) usually do not.

Some file actions might reset the date and time information in the file attributes, meaning that the existing date/time will be discarded and replaced with the momentary date and time.

The default cp Unix/Linux command with no parameters copies a file but resets its time attributes, unlike the default mv (move) command.

Many Android file management applications rely on this command, thus do not preserve the dats/time attributes upon copying files.

Graphical file explorer software, including Windows Explorer and common Linux file managers usually preserve file attributes with a few exceptions such as when transferring directories over MTP using Windows Explorer, which has an unstable implementation of such.

Methods to create file lists with date/time information include the commands dir [parameters] on Windows, ls -l [further parameters and stat.

Avoid clobbering

Clobbering means overwriting an existing file rather than appending to it or renaming the existing or new file.

This is the default behaviour of the cp and mv commands. In both cases however, the options -n (do not clobber), -b (rename target file first) and -i (interactive: ask before overwriting) can prevent an accidental overwrite.

The file saving dialogue of file selection managers (also known as “save as”) prompts the user whether to overwrite an existing file if the specified file name already exists. If yes is the pre-selected option, a moment of inattention might cause an unwanted overwrite of that existing file.

Another risk is export functionality in software that overwrites any existing target file, an example of which is the export functionality of the Battery Monitor Widget application for Android OS (developed by CCC71), which when used overwrites (clobbers) the existing bmw_history.txt file, rather than appending to it or dating and/or numbering the file names, like done by Android System Info by the developer Electric Sheep.

Preparation

In order to minimize data loss in case of damage, loss or theft of an apparatus and/or memory card, it is recommended to back up its memory each day before going on a trip.

At the same time, one should make sure to have sufficient remaining data storage space and electrical energy at all times to avoid missing out capturing moments.

File lists

File lists with additional information such as date/time and file size may facilitate forensic data recovery in case of file system header damage, and file lists from multiple drives at one place facilitate searching through files.

File system modifications

Before doing major file system modifications such as managing partitions (moving and resizing) and file system surgeries (using ChkDsk on Windows or fsck on Linux), a full disk image should be created temporarily to allow rolling back all changes in case of an error causing data loss, such as an unexpected power-off or a bug in the partition management software.

Additional risks

Unsolicited disposal

Living in a household shared with people who notoriously tend to dispose of obstacles, including other people's property without any warning or grace period (which technically is theft), have their data storage potentially put at risk.

This especially is a risk factor for very young videographers, data hoarders, archivists and digital preservationists in a household of minimalist parents.

Human factor

As stated above, even with reliable data storage, human factor should be taken into consideration, meaning that a person might accidentally overwrite or erase data in a moment of inattention and clumsiness.

Some argue that lost data be recoverable using software. However, not only is recovery from a backup much easier, certain and straightforward than (potentially expensive) forensic work, but data lost through accidental deletion usually is overwritten by new data from other software and/or the operating system imminently afterwards, making it irrecoverable.

On most solid state drives released since 2010, data even is immediately inaccessible after deletion due to being marked as non-existent (“trimmed”) by the storage controller, for a self-maintenance mechanism known as garbage collection, which aims to preserve writing performance.

Data recovery from an encrypted internal storage of a mobile device is close to impossible.

Poorly designed user interfaces

Some poorly designed user interfaces may cause the user to accidentally erase files.

Gallery software

As an example, some gallery software for mobile phones allow the deletion of items using a side swipe gesture, similarly to notifications. It is prone to accidentally being actuated.

After actuating, there is a five-second grace period: An on-screen toast notification with an undelete button appears on screen. If one misses that button with the finger, the toast notification disappears immediately and the side swiped item is erased.

Mozilla Firefox

Another example of unsafe user interface designs is the one-press deletion of selected items in the so-called library area of the Mozilla Firefox internet browser.

Browsing history, download history and bookmarks can be erased in a single step with a press on the del key, without confirmation. This is prone to happening accidentally.

The AUTOCHK time bomb

Related section: File system modifications

The file AUTOCHK.exe the Windows directories C:\Windows\System32\ and C:\Windows\SysWOW64\ is a ticking time bomb, and thus should be renamed in order to be disarmed.

If it deems the operating system's file system to have a problem, possibly caused by an unexpected power-off, it will automatically perform an unsolicited file system surgery, and possibly end up moving files it deems damaged or with invalid characters in the file name after writing to the partition from a Linux system to a directory named found.000 on the partition's root directory.

Several files and subfolders may be renamed to hexadecimally numbered names starting at file00000000.chk and dir_00000000.chk respectively, dismissing the original file names and directory structures.

File manager bugs

Some file managers may have a poor handling of data, such as erasing an entire source directory when a foler move is aborted unexpectedly such as exhausted space storage on the target path, without checking whether the files were moved successfully, causing unmoved files to be deleted.

Vendor lock-in

Storing data on proprietary data storage hardware (e.g. IoMega ZIP in the 1990s), proprietary formats (e.g. Acronis TIB Backup) poses a risk of vendor lock-in.

Online content

Related section: § Clouds are only redundant

Controversy

On social media and content sharing sites such as YouTube, content by users subject to a wave of criticism and/or constant controversy, as well as those with a history of randomly erasing their content, is endangered from both sides: Deletion by platform and deletion by uploader.

To prevent the former, controversial content creators should consider broadcasting their content to more than one platform.

Likewise, web resources one wishes to preserve for web citation should be archived using at least two website archival services, so future readers can access one site as a fallback resource in case the other one experiences technical issues.

Platform changes

Changes on platforms might lead to loss of information, such as features removed from YouTube and Quora's 2017 anonymity changes that detached anonymous content from accounts.

Paper books

Paper note books may contain important information and lots of work.

Although paper has excellent data retention, paper note books may get lost, for example during a moment of inattention caused by the relief at the end of a stressful school day.

Backing up paper books is usually done though a flat bed scanner, which has the highest resolution and accuracy. However, such a scanner requires scanning each page (or two pages at once on smaller note books) individually, which demands a lot of repetitive work and time.

A significantly faster method of backing up paper note books is to point a video camera at it, preferably one filming at least with 2160p (4K) resolution and mounted on a tripod pointing downwards, and then turning one page after another.

At 4K resolution (3840×2160 pixels), each still frame of the video acts as a solid 8.3 megapixel still photo with slight video compression artefacts, depending on bit rate.

To reduce the required ISO light sensitivity (which causes undesirable noise, thus loss of detail), this should be done in a bright environment, ideally illuminated by sunlight.

Make sure the video camera is fully focused on the paper's surface, and horizontally aligned with the paper.

Also see

• Benefits of optical media

Related navigation boxes: Electronics • Mobile phones • Data storage • User experience and user interfaces

📰 Article(s) of the same category(ies)

• Benefits of optical media

---