Compressing a big folder and wanting progress

This post is over a year old, some of this information may be out of date.

Everyone runs a bad migration at least once in their career and this happened to me a short while ago.

To get back deleted data, you can spin up a google cloud vm with mysql and extract the incremental snapshot to that test database. Run the queries to get the deleted data back and export those results. You can use these exports to cherry pick your rows to write them back to your production database.

A problem I encountered was massive data files was taking very long to compress and I didn't have a visual indication of how far along this process was.

`pv`

A small command line utility [https://linux.die.net/man/1/pv] to visualise the progress of a running command.

To install run sudo apt-get install pv

Compress

sudo su
tar cf - <folder> -P | pv -s $(du -sb <folder> | awk '{print $1}') | gzip > data.tar.gz

\# Outputs something like this
\# 3.06GiB 0:03:01 \[15.8MiB/s\] \[========> \] 5% ETA 0:57:05

Extract

pv /path/to/data.tar.gz | tar xzf - -C .

\# Outputs something like this
\# 3.06GiB 0:03:01 \[15.8MiB/s\] \[========> \] 5% ETA 0:57:05

This gave me a good indication of how long compression and extraction was running.

Cover photo by Christian Weiss.