LZ4 - Very Fast compressor => x2.07 @ 295 MB/s

View previous topic View next topic Go down

LZ4 - Very Fast compressor => x2.07 @ 295 MB/s

Post  Yann on Fri 16 Oct - 23:03

Note : LZ4 homepage has changed, please update your bookmark. The new homepage is here : http://fastcompression.blogspot.com/p/lz4.html

LZ4 is a particularly fast compressor using well-known LZ77 (Lempel-Ziv) algorithm.

Originally based on LZP2, it provides much better compression ratio on text files and reaches impressive decompression speed, beyond 500MB/s for text and 1GB/s (!) for binaries (test system : Core 2 Duo E8400). These speeds are scalable with multi-threading modes.
According to several compression benchmark websites, LZ4 is the fastest compressor and decoder.

LZ4 now features several compression modes, with different trade-off. The higher compression ratios use the MMC (Morphing Match Chain) search algorithm (explained here).

All compression modes generate *.lz4 compressed files.


Download :
v1.2b : LZ4 (win32)
- Corrected : MMC on specific patterns
- Corrected : Pipe mode (stdin)
- Corrected : zero-byte file
- benchmark mode can process multiple input files and provide overall summary


Benchmark evaluation :
File sample totalizing about 4GB.
In-memory benchmark results (command -b), excluding I/O considerations (such as hard drive).
Benchmark platform : Core 2 Duo E8400 (3GHz), Window Seven 32-bits
Compression level
Compression Ratio
Speed
Decoding
Single Thread



c0
2.075
295 MB/s
1.02 GB/s
c1
2.34335.4 MB/s
1.09 GB/s
c2
2.403
17.6 MB/s
1.15 GB/s
With 2 Threads



c0
2.075
575 MB/s
1.81 GB/s
c1
2.34369.0 MB/s
1.84 GB/s
c2
2.403
34.8 MB/s
1.91 GB/s

See benchmark results below


Last edited by Yann on Thu 29 Mar - 17:25; edited 84 times in total

Yann
Admin

Number of posts : 174
Registration date : 2008-05-01

http://phantasie.tonempire.net

Back to top Go down

Previous releases

Post  Yann on Fri 16 Oct - 23:03

"General" version :
v1.2b : LZ4
- Corrected : MMC on specific patterns
- Corrected : Pipe mode (stdin)
- Corrected : zero-byte file
- benchmark mode can process multiple input files and provide overall summary
v1.2 : LZ4
- c2 compression level speed increase
- decompression speed increase
- pipe mode (stdin & stdout) and "null" redirection
- benchmark can process multiple input files
v1.1 : LZ4
- c1 mode : multi threading available
- Stats animation while compressing
- Silent mode (-s)
v1.0 : LZ4
- Integrate all compression modes

"Ultra Fast" versions :
v0.9 : LZ4
- Multi-threading available for file compression
v0.8 : LZ4
- Significant speed boost on binary files
- Improved multi-threading code (benchmark mode only)
v0.7 : LZ4
- Multi-threading compression & decompression (benchmark mode only)
v0.6 : LZ4
- Improved decoding speed
v0.5 : LZ4
- small compression speed improvement
v0.4 : LZ4
- Improved decoding speed
v0.3 : LZ4
- small compression speed and ratio improvement
v0.2 : LZ4
- initial release

"Optimal Compression" versions :
v0.1 : LZ4OC "Optimal Compression"
- initial release; Simple Hash Chain with max search limit 256 (note : superseded by faster HC versions using MMC)

"High Compression" versions :
v0.9 : LZ4HC "High Compression"
- De-Overlap : Improved decoding speed
v0.8 : LZ4HC "High Compression"
- Multi-Level promotions : substantial compression speed improvement
v0.7 : LZ4HC "High Compression"
- Fusion (full segment & MMC integration) : small compression speed improvement
v0.6 : LZ4HC "High Compression"
- Fusion0 : Improved compression speed
- Improved decoding speed
- Corrected progress animation with large files (>4GB)
v0.5 : LZ4HC "High Compression"
- Segments generalisation : Improved compression speed; minor compression rate improvement
v0.4 : LZ4HC "High Compression"
- Segments 0 : Large compression speed improvement on binary files; minor compression rate improvement
v0.3 : LZ4HC "High Compression"
- Changed : improved compression speed, using Morphing Match Chain (MMC) search algorithm
v0.2 : LZ4HC "High Compression"
- Changed : improved compression ratio
v0.1 : LZ4HC "High Compression"
initial release, using simple hash-chain search methodology


Last edited by Yann on Fri 21 Oct - 20:55; edited 24 times in total

Yann
Admin

Number of posts : 174
Registration date : 2008-05-01

http://phantasie.tonempire.net

Back to top Go down

Benchmarks

Post  Yann on Fri 16 Oct - 23:08

1) LZ4 on Squeeze Chart, by Stephan Busch : fastest compressor

2) LZ4 on Compression Ratings, by Sami Runsas : fastest compressor

3) LZ4 on Large Text Compression Benchmark, by Matt Mahoney : fastest decoder, and pareto frontier for compression speed/ratio cheers

4) Fast compression algorithms comparison : by Richard Sim

5) Graphical comparison of fast compressors (outdated)


Last edited by Yann on Mon 25 Apr - 13:34; edited 8 times in total

Yann
Admin

Number of posts : 174
Registration date : 2008-05-01

http://phantasie.tonempire.net

Back to top Go down

Morphing Match Chain

Post  Yann on Sun 7 Nov - 21:33

Morphing Match Chain explanation :

Homepage = http://fastcompression.blogspot.com/p/mmc-morphing-match-chain.html

Discussion : encode.ru post


Last edited by Yann on Tue 8 Mar - 23:35; edited 1 time in total

Yann
Admin

Number of posts : 174
Registration date : 2008-05-01

http://phantasie.tonempire.net

Back to top Go down

2 threads en décompressions

Post  stfox on Tue 8 Mar - 10:28

J'ai pas accès à ma messagerie, alors je te laisse un petit commentaire :
j'ai remarqué qu'en décompression LZ4 utilise au minimum 2 threads, même si on lui spécifie -t1,
étonnant ? Tu peux le voir dans le gestionnaire des tâches en affichant la colonne thread, mais
aussi par le fait qu'un dual core tourne à 100% alors qu'avec un seul thread il ne devrait tourner
qu'à 50% Wink

A++
Bruno

stfox

Number of posts : 13
Registration date : 2009-01-07

Back to top Go down

Re: LZ4 - Very Fast compressor => x2.07 @ 295 MB/s

Post  Yann on Tue 8 Mar - 23:29

Alors, je suppose que tu parles du mode I/O (normal) ?

Dans ce cas, j'ai peut-être une explication :
En plus du thread de compression ou de decompression, il y a un thread I/O.

Il y a toujours un thread I/O. Il s'ajoute au nombre de threads spécifié par -t#.

C'est surtout par simplicité que j'ai choisi ce design.
La liste de tâches est alors "encadrée" par le thread principal qui s'occupe de l'I/O.

Comme dans la pub : on peut faire plus lent, mais c'est plus compliqué...


Yann
Admin

Number of posts : 174
Registration date : 2008-05-01

http://phantasie.tonempire.net

Back to top Go down

Re: LZ4 - Very Fast compressor => x2.07 @ 295 MB/s

Post  Sponsored content Today at 12:35


Sponsored content


Back to top Go down

View previous topic View next topic Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum