1/17/2024 0 Comments Tar compress multithread![]() ![]() In my scenario it is about just as important to activate multithreading in xz than it is to parallelize the whole pipeline if you can combine this with pigz and/or multiple files, you can even be a bit faster than a quarter of the sum of the single steps. Just by setting up four simple quarter pipelines with &, you get 0.8s, same as for a 100M file with xz -T4. And even without multithreading, even for xz:įor f in f25-?.gz do time gzip -dc "$f" | xz -c >"$f".xz & done This seems even slightly faster with 0.7s. then tick 'yes' and mark as Solved Last edited by dru8274 09-07-2012 at 09:27 PM. It makes use of multiple processors to speed up its operation and in general works considerably faster than bzip2. So, typical tar options to compress and extract files with tar plus pigz might be. Here I use four 25M files instead:įor f in f25-?.gz do time pigz -p4 -dc "$f" | xz -c -T0 >"$f".xz & done Lbzip2 is a multithreaded utility for handling ‘bzip2’ compression, written by Laszlo Ersek. With multiple files, but not too multiple, you can get highest overall parallelism with 4 shell background processes. If you want to run multithreaded you should. 0.2 seconds when I put all together: $ time pigz -dc -p4 f100.gz | xz -c -T4 >out.xz gzip also has a parallel version (pigz) that will let the program use multiple cores, making it much faster. "sys" now shows some overhead - real is the sum of 1/4 user plus sys. The "user" value must be divided by 4, the way it is reported/defined. Overall result: 300% faster, almost linear per core. ![]() With all four cores activated with -T4 (or -T0) $ time xz -c -T4 f100 >/dev/null In this post, well look at how to compress files with the tar command in Linux, along with some examples of tar in action. Usage: lbzip2 -d <2> pbzip2 -d <2> -d option is used for decompression.$ time xz -c f100 >/dev/nullĩ9% of the time is spent compressing on one core. 4 Answers Sorted by: 24 lbzip2 and pbzip2 are the tools which you can use for parallel compression and decompression. To use MT also means: wait until all input (data) is read into memory, and then start to compress in "parallel".Īfter including pigz into my tests I analyze the perfomance step by step I have a 100M file f100. With the -T0 multithread option you tell xz two things at once. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |