Lrzip version 0.641

Bugfix release for lrzip not fully compressing files on version 0.640 with no other changes.

Downloads, including access to standalone static binary:

http://lrzip.kolivas.org/

Git tree:

https://github.com/ckolivas/lrzip

 

Enjoy!

お楽しみ下さい

-ck

Lrzip version 0.640

I haven't had much time till recently to dedicate to my application lrzip, but I have a soft spot for it since I use it extensively so I couldn't let it go to bitrot, and luckily I'm between other projects so have spent a few days on it. Here is an updated version with numerous bugfixes, some for security issues, some for crashes on corrupt archives, some to avoid creating corrupt archives, some for build issues, and random other bugfixes too numerous to list in detail. So this is a stable bug release and I encourage everyone to update.

Downloads, including access to standalone static binary:

http://lrzip.kolivas.org/

Git tree:

https://github.com/ckolivas/lrzip

The only change to its behaviour is it now requires liblz4-dev to build because it uses the super fast lz4 compression algorithm for compressibility testing - it does NOT use it for any compressed archives as this would be incompatible with the existing format, and almost certainly will not offer any advantage in speed over lzo because the bulk of the time is the rzip pre-compression library stage. It also now can build the crc checking as an assembly component on x86_64, though this is largely not going to affect speed in any significant fashion. Long term I will probably remove this ancient and not-very reliable checking from future versions if I break the existing format. I will also probably drop MD5 sums.

Peter Hyman has also been working behind the scenes to add countless little features to lrzip for the duration that I haven't given it any attention and has been running his own tree with all his changes. 

 https://github.com/pete4abw/lrzip/

Alas most of those are new features and break the existing lrzip file format, and have come with new issues of their own that would require quite some effort to blend into the existing codebase and iron out bugs. I'd also want new archives to be created compatible with older versions as much as possible preferentially so compatibility code would need to be added.

Thus I concentrated entirely on just bringing out another stable release before looking at any of his code, including having to roll back some of his changes I'd already incorporated into my master tree. In time I hope to get to his changes to incorporate them.

Enjoy!

お楽しみ下さい

-ck

lrzip version 0.631

Announcing an updated version of lrzip.

Tarballs:
http://ck.kolivas.org/apps/lrzip/
Git tree:
https://github.com/ckolivas/lrzip

This is a minor bugfix release.

- Encryption complexity has been altered to match CPU speed rate rises that have NOT paralleled Moore's law.
- Some of the command line parameters did not work properly in compatibility mode.
- Compressed files did not retain the same date as the original file.
- The -p parameter did not accept arguments and would not work.

Enjoy!
お楽しみ下さい
-ck

lrzip 0.630

It's been a long time since I've updated lrzip as version 0.621 was very stable. Having had a long time for many people to test it in lots of environments has allowed a few rare bugs to be shaken out, and a few issues showed up on different hardware/OS, so an update was finally required. In addition it gave me an opportunity to implement a feature many had requested - gzip command line support. This is now done when lrzip is called via the command line 'lrz' instead of its full name lrzip.

Get it here:
lrzip.kolivas.org 
Git repository:
https://github.com/ckolivas/lrzip


The short short feature changelog:
- gzip command line support when lrzip is called with the name lrz instead of lrzip
- recursive directory compression support with -r (this was required to meet gzip compatibility)

The short short bug changelog:
- Reports of being unable to malloc ram and failing should all be fixed now.
- Inability to decompress very large (multiple chunk) encrypted archives is now fixed. Fortunately the issue was on decompression, not compression so if you have generated files that meet this criteria they are safe.

Full changelog:
* checksum.buf should only be changed after the semaphore wait
* Update README
* Add documentation for recursive mode
* Implement gzip compatible -r recursive option
* Add initial argument processing for recursive option
* Tidy
* Add one more verbose for compat mode
* Add support for various combinations in compat mode
* models is array of chars. char's signess is implementation specific. It's
unsigned on ARMv7. Unsigned char cannot represent negative values. GCC 6
complains about it:
* Fix decompression of multiple chunk encrypted archives
* Tidy gotos
* Show correct lengths during testing on big endian and compressed archives
* Update copyright dates
* Allow less than maxram to be malloced for checksum to fix Failed to malloc
ckbuf in hash_search2
* Base temporary output buffer on maximum mallocable, not maxram
* Enable subdir objects for future automake compatibility
* Add support for -m option in lrztar
* Big endian fix for Solaris Sparc courtesy of joelfredrikson.
* Fixed typographical error, changed accomodate to accommodate in README.
* A whitespace fix on lrztar.
* Add sanity check to prevent trying to malloc more ram than a
system/environment is capable of
* Cosmetic help change for compat
* Add rudimentary manpage for lrz
* Fix lrz symbolic linkage
* Do not fail if we are unable to write temporary files, giving a warning only
that it might fail if we don't have enough ram
* Try /tmp/ if none of the temporary environment directories or the current
directory are writeable
* Set STDOUT correctly in compat mode
* Style police
* Fix false warning on decompressing from stdin without keep files
* Fix false warning on compressing from stdin without keep files
* Don't show extra message in compat mode decompress
* Show correct appname when called in compat mode
* Add support for progress, fast and best flags in compat mode
* Add compatibility mode with gzip when called as lrz
* Correct adding slash to control->tmpdir. off-by-one error.
* Update manpage for long options

Enjoy!
お楽しみ下さい
-ck

lrzip-0.621

As if often the case, once I find time to work on a project, a few releases back to back usually follow. The same happened with lrzip, and here is version 0.621

http://lrzip.kolivas.org/

and the git source code page:

https://github.com/ckolivas/lrzip

For fun I compressed all stable linux kernels from 1.0 to 3.19 with lrzip as posted here:

Commemorative linux-1.0-3.19 compressed tarball
This was a 29GB tarball that compressed to 355MB and can be downloaded here:
linux-1.0-3.19.tar.lrz


What's new
- Substantial speed ups for the rzip stage in both regular and unlimited modes.
- Lrzip now supports long command line options.
- Proper support for the various forms of TMPDIR environment variables.
- More unix portability fixes.
- OSX fixes.
- Fixed order of lrzip.conf search.
- Addressed all warnings created with pedantic compiler settings and clang
- Fixes for some stderr messages being swallowed up.
- Fixed being unable to decompress to STDOUT when in a non-writable directory.
- Changed broken liblrzip callback function API to match lrzip proper.


Changelog
* Fix libzpaq.cpp warnings
* Fix warnings in LzmaLib.c
* Making the high buffer only one page size is faster for sliding mmap mode
* Fix incompatible log callback in liblrzip
* Use PRId64 instead of lld in fscanf
* Use int64_t for i64
* Fix ISO C warning in lrzip.c
* Fix ISO C warning in lrzip.c
* Fix ISO C warning in runzip.c
* Fix ISO C warnings in rzip.c
* Fix iso c warning in util.c
* Fix EOL to unix on libzpaq
* control->tmpdir fixes
* Null terminate long options to find unhandled options
* Trivial date
* Add long option support
* Brace failure lead to broken dump to stdout
* fflush messages on print_err
* Fix inverse logic
* Honour the proper unix environment variable for temporary directory and then
try variations
* Attempt to decompress file in ram only if we are unable to create temporary
files when decompressing to stdout
* Avoid checking return values repeatedly on compress that only lead to failures
anyway
* Microoptimisation
* Don't check for failure condition that can no longer occur
* Don't check twice for failure conditions that are fatal in hot paths
* Cache the chunk bytes value to avoid setting it on each read_header call
* fake_mremap is only used when defined to mremap
* Remove unused cksem functions
* Fix remaining use of mutexes lock/unlocking in different threads with cksems,
corecting cksem usage on osx
* Update copyright dates
* Make match_len a function completely removing all indirect calls to get_sb,
significantly speeding up the single_get_sb case
* Make full_tag a pointer allowing us to avoid a function call for get_sb
* Call sliding_get_sb directly in sliding_next_tag
* Make next_tag a pointer to allow ordinary mapping to avoid an extra function
call
* fix order of lrzip.conf search

Lrzip 0.620

I finally found some time to give lrzip some much needed love again. Fortunately the last release, 0.616, proved very stable for the vast majority of workloads so there was never any great need to give it attention, though little things slowly cropped up, so I've accumulated all the bugfixes till now to release a new stable version, 0.620.

Freecode has long since died so here's the link to the (ghetto) home and download page:


http://lrzip.kolivas.org/

and the git source code page:

https://github.com/ckolivas/lrzip


Summary of changes:
- It would previously crash if trying to decompress from STDIN with a file that was too large to fit in ram which has now been fixed.
- It would previously fail to decompress files that were too large to fit in ram decompressed, now fixed.
- There were some scenarios lrzip would run out of ram when there was plenty to allocate, now fixed.
- Some other unix platforms would consider locking a mutex in one thread and releasing it in another a bug, so I've converted the use of those mutexes to anonymous semaphores.
- In order to maintain compatibility with platforms that don't properly support anonymous semaphores (OSX I'm looking at you), I've added the use of my custom fake semaphores as discussed here: unnamed-semaphores-and-pososx
- Some files would have their size reported wrongly with -i, now fixed.
- Added the ability to limit the use of ram with -m since lrzip happily uses all of it normally.
- Other minor changes and fixes for rare corner cases.


The changelog:
* Increase maxram when we abandon use of temporary input/output buffers
* Don't delete the tmpinfile when decompressing from stdin before allowing seek
to end to succeed in checking md5
* Use temporary file from read_seekto when STDIN will not fit in the ram input
buffer
* Remove unused read_i64 function
* Add message about issue tracker in BUGS
* Use a common exit path iin lrzip_compress/decompress and fix lr leak on
successful return
* Fix parenthesis placement inside of unlikely().
* Clear sa_mask and sa_handler before calling sigaction().
* Fix for lrzip -i. Decompressed size wrong
* added '-m' command line option
* Fix wrong README file being included in Makefile
* Pass strict sizes to decompress length, rounding up only the amount we're
allocating to not confuse decompression libraries
* Convert the thread locking to use cksems
* Add cksems to util.h
* Fix 'Failed to malloc ckbuf in hash_search2' with very large files.
* Round up compression and decompression buffers to page size since malloc will
allocate them that large anyway.
* Increase the compressed buffer size given to libzpaq in case of incompressible
data since it does not check if it's trying to write beyond the end of the
buffer.
* Provide a helper function to round a value up to the nearest page size for
malloc optimisations.

Lrzip 0.616

Due to some issues with lrztar, I've posted an updated lrzip with a fixed lrztar.

Version 0.616 is now up, and should appear here soon:
https://freecode.com/projects/long-range-zip

This post will likely be overshadowed by the next one I'm about to post.

Enjoy!
お楽しみください

Unnamed semaphores and POSOSX

During the development of my bitcoin mining software, cgminer, I've used just about every synchronisation primitive due to it being heavily multithreaded. A few months back I used some semaphores and the first thing I reached for was the much more useful unnamed semaphores commonly in use today. Classic SYSV IPC semaphores are limited in number, require allocating of shared memory, stay in use till destroyed or the system rebooted etc. etc. that make them real awkward to use and far less flexible so I never even considered using them. For some reason, though, I had a vague memory of trying to use them on lrzip years ago and deciding not too. Well that memory came back to bite me.

Cgminer is cross platform, working reasonably well on various architectures with Linux, windows (via mingw32) and OSX mainly, though other brave souls have used it on all sorts of things. I've often heard OSX described as the "Fischer Price" unix, AKA "My first unix" because of its restricted subset of unix capabilities that it has, although I'm led to believe it claimed to have POSIX compliance at some stage - though I never really investigated it nor does it really matter since Linux is only POSIXy at best.

So the interesting thing was that I had written some code for cgminer which used unnamed semaphores and it compiled fine across the 3 main platforms, but it failed miserably when it came to working on OSX. Of note, the unnamed semaphore functions conform to POSIX.1-2001. All of the functions compiled perfectly fine, but the application refused to run properly, and finally when I got some of the OSX users to investigate further, every single unnamed semaphore function, such as sem_init, sem_post, sem_wait etc, would return a unique OSX error which when deciphered it was actually "Unimplemented feature". Quite amusing that to get POSIX compliance it only had to implement the functions, but not the actual features of those functions... You may go wild with speculation as to why this may be. This is why I coined the term POSOSX.

After toying with the idea of using SYSV semaphores and being disgusted at the thought, I finally decided that I should just implement really basic fake unnamed semaphores using pipes on OSX to imitate their behaviour.

Simplified code from cgminer for OSX follows (real code checks return values etc.):

struct cgsem {
    int pipefd[2];
};

typedef struct cgsem cgsem_t;
 
void cgsem_init(cgsem_t *cgsem)
{
    int flags, fd, i;

    pipe(cgsem->pipefd);

    /* Make the pipes FD_CLOEXEC to allow them to close should we call
     * execv on restart. */
    for (i = 0; i < 2; i++) {
        fd = cgsem->pipefd[i];
        flags = fcntl(fd, F_GETFD, 0);
        flags |= FD_CLOEXEC;
        fcntl(fd, F_SETFD, flags);
    }
}

void cgsem_post(cgsem_t *cgsem)
{
    const char buf = 1;

    write(cgsem->pipefd[1], &buf, 1);
}

void cgsem_wait(cgsem_t *cgsem)
{
    char buf;

    read(cgsem->pipefd[0], &buf, 1);
}

void cgsem_destroy(cgsem_t *cgsem)
{
    close(cgsem->pipefd[1]);
    close(cgsem->pipefd[0]);
}

Lrzip version 0.615

So finally I found some time to give my other pet project, lrzip, some love and attention that it sorely deserved. Over the past year since the last release, a number of bugs have shown up and some generous souls have contributed bugfixes where possible for them, or at least instrumented where they come from.

This latest release addresses all known bugs that were reported in that time, and includes a few performance micro-optimisations.


The project page is here (0.615 will appear there shortly):

https://freecode.com/projects/long-range-zip

The news update follows:

Fixed -O not working on lrztar.
Made it less likely to run out of ram when working with STDIN/OUT.
Fixed running out of ram when using -U on huge files.
Fixed corrupt archives being generated from incompressible data.
Fixed corrupt archives being generated from very small files.
Fixed endianness on various platforms for MD5 calculation to work.
Fixed rare corruption when compressing with lzma from STDIN.
Fixed all blank data being generated when compressing from STDIN on OSX.
Performance micro-optimisations.
Fixed corrupt archive being generated when all the same non-zero bytes exist on
large files.

Enjoy!
お楽しみください

lrzip-0.614

This release is a quick hotfix for broken lrztar in lrzip 0.613.

https://freecode.com/projects/long-range-zip

lrzip-0.613

lrzip 0.612 has been out in the wild for a while now and the good news is that there have been very few bug reports in that time. After allowing enough accumulated issues collect in my inbox, I've created a pure-bugfix maintenance release in version 0.613:

long-range-zip 0.613

One bug of note was that the md5 calculation on files that had compressed blocks greater than 4GB in size was wrong. This was very suspicious for a 32 bit overflow error. Indeed Serge Belyshev did some excellent detective work and found the culprit to be in the glibc implementation of md5, which is used by lrzip. This only affects using the md5 library components, not the md5sum command line utility which uses a different rolling algorithm so glibc userspace never hit it. The bug in question was amusing in the way it shows one of the many naive ways we dealt with 32 bit limitations in the past. It assumed anything larger than a 32bit chunk was just 2^31 + (chunk size modulo 2^31). That means it would never work with a chunk larger than 2^32. The fix has been pushed upstream and is now incorporated into lrzip.

Another bug, as reported on this blog by a commenter, was that of creating corrupt very small archives (less than 64 bytes). This has been fixed by disabling the back end compression when the chunk is less than 64 bytes and just using the rzip first stage.

A lot of the other work in this release was just getting it to compile  on osx. Numerous issues showed up as always, and I didn't have access to an osx machine on the previous release to fix it. This time I used my wife's laptop ;) . One of the issues, for example, was that osx didn't see itself as #ifdef unix, which I thought was a little amusing. Another unexpected surprise was that the default osx filesystem is not case sensitive which caused a conflict lrzip.h vs Lrzip.h. Alas I have no other BSDs to try compiling it on so I'm not sure if they're fixed with this.

Interestingly, I still have to disable md5 calculation on the osx build. The md5 is calculated the same on compression and decompression within lrzip, but it disagrees with the result returned from the ports version of md5! This defeats the whole purpose of including md5 in it since the point of it is to have a command line result to compare to. I'm guessing there's an endianness dispute there somewhere and haven't ever tracked it down, since osx has done an endian flip in the past. lrzip still uses crc32 checking of each block internally so it's not like there isn't any integrity checking.

Finally what would a release be without some new benchmarks? Nothing performance-wise has changed in lrzip since the last version, but I have access to a 12 thread CPU machine with 32GB of ram now, so I did some quick benchmarks with the classic 10GB virtual image I've been using till now.

Compression  Size           Percentage  Compress Time  Decompress Time
None         10737418240     100.0
gzip          2772899756      25.8       3m56s          2m15s
pbzip2        2705814394      25.2       1m41s          1m46s
lrzip         1095337763      10.2       2m54s          2m21s

Note that with enough ram and CPU, lrzip is actually faster than gzip (which does compression in place) and comparable on decompression, despite a huge increase in compression. pbzip2 is faster than both but its compression is almost no better than gzip.

lrzip-0.612

Just for good measure, here's yet another lrzip release.

lrzip 0.612 on freecode

This time the main update is a new zpaq library back end instead of using the ageing zpipe code. There are a number of advantages to using the libzpaq library over the old zpipe code.

First, the old code required a FILE type stream as it was written with STDIO in mind, so it was the only compression back end that required the use of some lesser known but handy, yet (virtually) linux only memory features like fmemopen, open_memstream and friends. These were not portable for osx and others so they were emulated on those platforms through the incredibly clunky use of  temporary files on disk. Using the new library has killed off the need for these features making the code more portable.

Second, the code is significantly faster since it is the latest full c++ version of the zpaq code. Unfortunately it also means it takes a LOT longer to compile this part of lrzip now, but that's not a big deal since you usually only compile it once ;)

Third, it supports 3 different compression levels, one of which is higher than the previously supported one in lrzip. As lrzip uses 9 levels of compression, I've mapped the 3 levels to -L 1-3, 4-7 and 8-9 since -L 7 is the default and that provides the "mid level" compression from zpaq.

Finally, the beauty of the zpaq compression algorithm is the reference decoder can decompress any zpaq compressed data of any profile. This means you are able to use the latest version of lrzip with compression -L 9 (max profile), yet it is still backwardly compatible with older 0.6x versions of lrzip, not requiring an updated minor version and file format. The release archive I provide of lrzip-0.612.tar.lrz is self compressed with the new max profile. Even though there is significantly more code than ever in the lrzip release tarball, it has actually shrunk for the first time in a while.

So all that talk is boring and all, so let's throw around some benchmark results which are much more fun.

From the original readme benchmarks, I had compressed the linux 2.6.37 tarball, so I used that again for comparison. Tests were performed on an Intel quad core 3Ghz core 2 duo.

Compression Size  Percentage Compress Decompress
None     430612480 100
7z        63636839 14.8  2m28s  0m6.6s
xz        63291156 14.7  4m02s  0m8.7
lrzip     64561485 14.9  1m12s  0m4.3s
lrzip -z  51588423 12.0  2m02s  2m08s
lrzip -l 137515997 31.9  0m14s  0m2.7s
lrzip -g  86142459 20.0  0m17s  0m3.0s
lrzip -b  72103197 16.7  0m21s  0m6.5s
bzip2     74060625 17.2  0m48s  0m12.8s
gzip      94512561 21.9  0m17s  0m4.0s

As you can see, the improvements in speed of the rzip stage have made all the compression back ends pretty snappy, and most fun of all is that lrzip -z on this workload is even faster on compression than the multithreaded 7z and is significantly smaller. Alas the major disadvantage of zpaq remains that it takes about as long to decompress as it takes to compress. However, with the trend towards more CPU cores as time goes on, one could argue that zpaq compression, as used within lrzip, is getting to a speed where it can be in regular use instead of just research/experimental use, especially when files are small like the lrzip distributed tarball I provide.

I also repeated my old classic 10GB virtual image benchmarks

Compression Size  Percentage Compress Time Decompress Time
None     10737418240 100.0
gzip      2772899756  25.8  05m47s  2m46s
bzip2     2704781700  25.2  16m15s  6m19s
xz        2272322208  21.2  50m58s  3m52s
7z        2242897134  20.9  26m36s  5m41s
lrzip     1372218189  12.8  10m23s  2m53s
lrzip -U  1095735108  10.2  08m44s  2m45s
lrzip -l  1831894161  17.1  04m53s  2m37s
lrzip -lU 1414959433  13.2  04m48s  2m38s
lrzip -zU 1067169419   9.9  39m32s  39m46s

Using "U"nlimited "z"paq options, it is actually faster than xz now. Note that about 30% of this image is blank space but that's a not-uncommon type of virtual image. If it were full of data, the difference would be less. Anyway I think it's fair to say that it's worth watching zpaq in the future. Edit: I've sent Matt Mahoney (zpaq author) the latest benchmarks for lrzip and how it performs on the large file benchmark and he's updated his site: http://mattmahoney.net/dc/text.html I think it's performing pretty well for a general compression utility.

lrzip-0.611

It usually doesn't take long to find bugs in a new significantly larger release and that was the case with lrzip 0.610. Since I'm loathe to leaving a buggy release lying around, I've released a new version hot on the heels of the old one.

lrzip 0.611 on freecode

http://lrzip.kolivas.org

lrzcat and lrzuntar were broken on the last release and they've now been fixed. lrzuntar also had the nasty habit of overwriting existing directories without warning so I've modified the code so it will not overwrite it without the -f option.

With the report of slowdowns on the last release, almost certainly due to incorporating the liblrzip library support, since nothing else changed, I figured it was time to do some hot spot profiling. So I pulled out oprofile and found where most of the time was spent in the code during the rzip stage. Then I went in and carefully rewrote small areas of the hottest functions as though they were critical code paths in the CPU scheduler and managed to find a few small speed improvements. Most of the improvements won't be noticeable unless you're running one of the faster compression modes like lzo, or none at all with -n, but it is faster. I also moved the checksum routines (crc32 and md5) into separate threads as they now use a significant amount of CPU time of their own during the rzip phase, and this should speed things up slightly too.

Then I went to the decompression side of things and did some profiling on the runzip stage and got a surprise. Most of the time during decompression is simply spent on the md5 update code. If I disabled the md5 update code, it was much faster during the runzip stage. After arguing with myself for a day, I figured it was still better to have integrity checking enabled and consider the addition of a fast mode for decompression since that will actually be almost lightning quick.

Interestingly, when profiling decompression, I was using the test option (-t) which does not write anything to disk, and things changed quite dramatically when I changed to actual decompression to disk. It took four times longer to decompress a 1GB archive to disk than it did to just test decompression in ram. Now this all seems obvious if you consider how long it takes to write something to disk, but that was not the case at all. In fact, virtually none of the data is written to disk by the time decompression is complete; it is just all in "dirty ram" as writeback. This did not change whether I used a -ck kernel with a low dirty ratio setting or the mainline default of 10. After some more prodding around I discovered that doing a simple write() of a 1GB buffer took over 7 seconds on modern hardware. This is only 140MB/s for a copy from memory to memory! It should be 20 times faster than that. I even tried taking the write function out of the equation and doing an mmap on the file and then memcpy from the buffer to the mmaped ram and it took the same amount of time, so the slowdown was not in the write() function call. After speaking to a few people and googling, it appears I'm not alone in this finding and it may be only happening in recent linux kernels. At this point I gave up trying to find what was causing this slow decompression on lrzip since it seemed unrelated to the lrzip code, and concentrated on getting this release out. I wonder if this is related to the writeback code that I was actually so looking forward to in 3.2ish. However others reported the problem continues as far back as 2.6.38 whereas it's not there in 2.6.35. I'll wait and see.

Anyway it may be possible to get lrzip integration into libarchive and therefore a number of package managers and decompression programs that use this library. The GPL license in lrzip may be a sticking point, though, and the authors of lzo and the original rzip have not responded about queries about the possibility of making the library LGPL which would make it easier to incorporate into the BSD licensed libarchive. So for now, it's stuck on GPL version 2.

Enjoy.

lrzip-0.610

I haven't been idle all this time. In all honesty I've spent a lot of time playing with my bitcoin mining code in cgminer and after a couple of hardware failures with my mining rig I was finally able to turn my attention back to lrzip which has been really fairly quiet in the bug reporting department since the last stable release.

Here is version 0.610 (when freecode updates):
http://freecode.com/projects/long-range-zip

Or directly:
http://lrzip.kolivas.org

The major feature addition in this latest version is largely thanks to the work of Michael Blumenkrantz in the form of a liblrzip library! This means you can now link in lrzip compression and decompression into any other application. There is a range of APIs available to use this capability from the simplest gzip equivalent lrzip_compress and lrzip_decompress functions to ones that expose all the low level features knobs and dials in lrzip. There are also a couple of demo examples of using these functions included in the source code, but in their simplest form they should be very easy to use. I look forward to seeing package managers, archivers and maybe even git and so on having lrzip support in the future ;) Being a large chunk of new code it is not entirely impossible that bugs or issues with using the library may show up.

WHATS-NEW summary:
The new liblrzip library allows you to add lrzip compression and decompression
to other applications with either simple lrzip_compress and lrzip_decompress
functions or fine control over all the options with low level functions.
Faster rzip stage when files are large enough to require the sliding mmap
feature (usually >1/3 of ram) and in unlimited mode.
A bug where multiple files being compressed or decompressed from the one
command line could have gotten corrupted was fixed.
Modification date of the decompressed file is now set to that of the lrzip
archive (support for storing the original file's date would require modifying
the archive format again).
Compilation warning fixes.
Make lrztar work with directories with spaces in their names.


The full changelog follows:
* Implement complete set of liblrzip libraries, documentation and example uses
with support for simple lrzip_compress() and lrzip_decompress() or complete
fine-grained control over all compression and decompression options.
* Use as much of the low buffer as possible with a single memcopy before going
fine grained byte by byte.
* Preserve the compressed time on decompression where suitable.
* Store a copy of the control struct to be reused on subsequent files to prevent
variables being modified in the control struct on the first file that corrupt
compression/decompression of the 2nd file.
* Explicitly select C99 to avoid certain warnings.
* Generic modifications to silence -Wextra warnings.
* Fix typos.
* Use an array of parameters in lrztar to allow working with directories with
spaces in their names.

Enjoy.

lrzip-0.608

Right on the heels of the lrzip 0.607 bugfix release is yet another release, this time with a performance boost!

http://lrzip.kolivas.org

TL;DR: Faster compression on lrzip

When I converted lrzip to work with stdin/stdout and with infinitely sized mmap windows, I needed to do an awful lot of layering of the access to the data internally to decide whether to access the data directly in the original mmapped region of ram, whether to mmap a new region of ram, or to use the moving smaller upper mmapped ram on just the portion being accessed. While this made it possible to implement features that were originally considered virtually impossible, it unfortunately also slowed it down substantially during the rzip dictionary search stage.

For example, a simple memcpy had to turn into this convoluted function:

        for (i = 0; i < n; i++) {
            memcpy(sinfo->s[stream].buf + sinfo->s[stream].buflen + i,
                   get_sb(control, p + i), 1);
        }

As you can see, it had to check it byte by byte in the get_sb function to decide where the memory was. I had tried my best to make the get_sb() function as small as possible since I knew this was a hot path, and it had ended up looking like this:

    i64 low_end = sb.offset_low + sb.size_low;

    if (unlikely(sb.offset_search > low_end))
        remap_low_sb(control);
    if (p >= sb.offset_low && p < low_end)
        return (sb.buf_low + p - sb.offset_low);
    if (p >= sb.offset_high && p < (sb.offset_high + sb.size_high))
        return (sb.buf_high + (p - sb.offset_high));
    /* p is not within the low or high buffer range */
    remap_high_sb(control, p);
    return (sb.buf_high + (p - sb.offset_high));
Not too bad, but clearly this is going to cost a heck of a lot more than just doing a simple memcpy!

Anyway it had concerned me that the more "features" I added to lrzip, the slower it got. Of course this allows one to have unlimited compression windows making for unparalleled compression ratios of massive files with long range redundancy, but most of the time people won't be using this feature and that code is just going to cost them adversely :\

A little bit of playing with profiling recently showed me that the situation was even worse than I had imagined. During the rzip phase, get_sb accounted for 37% of the consumed CPU cycles! Anyway, a simple set of proxy functions seemed the simplest answer, where it would use different versions of memcpy and get_sb depending on whether the file fits in ram or not.

static uchar *sliding_get_sb(rzip_control *control, i64 p)
{
    if (p >= sb.offset_low && p < sb.offset_low + sb.size_low)
        return (sb.buf_low + p - sb.offset_low);
    if (p >= sb.offset_high && p < (sb.offset_high + sb.size_high))
        return (sb.buf_high + (p - sb.offset_high));
    /* p is not within the low or high buffer range */
    remap_high_sb(control, p);
    return (sb.buf_high + (p - sb.offset_high));
}

static uchar *single_get_sb(rzip_control *control, i64 p)
{
    return (sb.buf_low + p);
}

/* We use a pointer to the function we actually want to use and only enable
 * the sliding mmap version if we need sliding mmap functionality as this is
 * a hot function during the rzip phase */
static uchar *(*get_sb)(rzip_control *control, i64 p);

static void sliding_mcpy(rzip_control *control, unsigned char *buf, i64 offset, i64 len)
{
    i64 i;

    for (i = 0; i < len; i++)
        memcpy(buf + i, sliding_get_sb(control, offset + i), 1);
}

static void single_mcpy(rzip_control *control, unsigned char *buf, i64 offset, i64 len)
{
    memcpy(buf, sb.buf_low + offset, len);
}

/* Since the sliding get_sb only allows us to access one byte at a time, we
 * do the same as we did with get_sb with the memcpy since one memcpy is much
 * faster than numerous memcpys 1 byte at a time */
static void (*do_mcpy)(rzip_control *control, unsigned char *buf, i64 offset, i64 len);

Voila, 30% speedup for the rzip stage on small files. This does not bring us back to the previous performance, but close to it. Unfortunately the extra level of indirection actually slowed it down even more on larger files. Then I discovered I could move one compare out of the get_sb function and recover that last bit of speed. So overall, large file rzip stage is actually slightly faster too now.

Finally I decided to not install the bash completion script by default since some distributions include a bash completion package and they conflict on install. So instead I've kept it in the tarball, but don't install it by default. A couple of other trivial changes, and I figured I should just release a stable faster release, so here's lrzip 0.608 (when freshmeat syncs up)

http://freshmeat.net/projects/long-range-zip

lrzip-0.607

It's been a while since lrzip received any attention while I got distracted by cgminer. However a bug report where there was an archive that was not decompressible drew my attention. After a run of debugging and testing (thanks Entil!) I tracked down the bug. The nice thing was the archives were actually all intact, it was purely on decompression that it could be a problem, and it was extremely rare for an archive to be made in this way. So I fixed the bug, updated the lzma library to the latest, added an lrzip bash completion script and a couple of other minor things and have released version 0.607.

lrzip on freshmeat
http://lrzip.kolivas.org
http://github.com/ckolivas/lrzip

Meanwhile Michael Blumenkrantz has been working on a liblrzip library for lrzip! That means once we finalise the format, you'll be able to compile in lrzip support into your applications, using functions like:

lrzip_compress(void *dest, unsigned long *dest_len, const void *source, unsigned long source_len)

This work is currently in the liblrzip branch on the github tree.

lrzip-0.606

I broke lrzuntar in version 0.605 when I disabled automatic stdout on lrzip so this is a tiny bugfix release to make lrzuntar work again.

lrzip on freshmeat

lrzip-0.605

A few minor bugs have shown up since version 0.604 that people have spotted, so I've attended to them and added a new lrzcat symbolic link and released version 0.605. Here's the changelog:

Addition of lrzcat - automatically decompresses .lrz files to STDOUT. lrzip and lrunzip will no longer automatically output to STDOUT when output is redirected. Progress output will no longer spam the output unless the percentage has changed. lrzip now has no lower limit on file sizes it will happily compress and is able to work with zero byte sized files. The percentage counter when getting file info on small files will not show %nan. The executable bit will not be enabled when compressing via a means that can't preserve the original permissions (e.g. from STDIN).

I'm now getting reports that lrzip is sneaking into many distribution repositories, that the 'file' command is finally getting support for .lrz files, and that rpm is even including support for lrzip compressed files. Great news, and thanks to the packagers and other developers.

Get version 0.605 from here: LRZIP 0.605 on freshmeat

Happy Mother's day if you're celebrating it in your part of the world.

lrzip-0.604

A one bug fix release.

Changelog:
Detach threads after creating them on the compression side. Not joining them meant that compressing massive files requiring hundreds of threads would eventually hit the resource limit of number of threads created even though the threads themselves would exit.

English:
lrzip will no longer fail with a "resource temporarily unavailable" error when compressing files over 100GB that require hundreds of threads to complete.

Get it here: LRZIP

lrzip-0.603

Trying to polish off version 0.6x of lrzip to be nice and stable and working as planned, I've made a few more updates addressing a few issues that have come up, along with some outside help. Here's the short changelog:

lrzip now detects when output is being redirected without a filename and will automatically output to stdout. Apple builds, which had errors on compressing files larger than 2GB in size, were fixed. lrztar now properly supports -o, -O, and -S. The lrzip configuration file now supports encryption. lrzip will now warn if it's inappropriately passed a directory as an argument directly.

Probably the most fun part of this is the first feature upgrade to do with stdout, which I use regularly now since I store all my kernels and patches as .lrz, I can now do:

lrunzip patch-2.6.38.4.lrz | patch -p1

Also, graysky made some nice graphs and I feel obliged to put them up here:

Of course, with much larger files and more CPUs and RAM the discrepancy becomes much greater with lrzip but that doesn't change the fact this is a real world test.

So grab it here:
LRZIP ON FRESHMEAT

As an aside, debian unstable now has 0.602+ in its repo, and the upcoming elite release of slackware also has 0.602.