Sunday, 2 August 2015

BFS 463, linux-4.1-ck1

Finally a resync to linux-4.1 . Sorry I was just too preoccupied to get around to doing this, and I haven't directly addressed a few known problems that have workarounds, and it comes with a warning.

BFS by itself:


-ck branded linux-4.0-ck1 patches:

4.1-ck1 patches

The usual collection of resyncs and minor updates including pending fixes post 462.

This includes a fix for some uniprocessor build problems courtesy of Serge Belyshev. If you still have boot problems with uniprocessor builds the workaround is to create an SMP kernel.

I've finally bit the bullet and removed the block flush code from within the main schedule() call, in keeping with how mainline does it. This is a problem that has recurred every time I've removed this change from previous kernels and had to re-add it every time. Complete hangs under particularly heavy IO used to be the problem, please report back if these come back with this kernel, hence the warning.

On the previous kernel, some had crashes unless they enabled NUMA. I have no idea what caused these and have done no specific changes to address it. I don't want people to enable NUMA unnecessarily but if you have crashes this is the first thing to try and please report back.


Thursday, 16 April 2015

BFS 462, linux-4.0-ck1

Announcing a resync and update of BFS for linux-4.0

BFS by itself:


-ck branded linux-4.0-ck1 patches:

4.0-ck1 patches

The usual collection of resyncs and minor updates only.

It includes the following changes:
- Minor tweaks to uniprocessor build (though enabling SMP will fix breakage if it still exists).
- Fix for tracing build failure
- SMT nice update to ignore kernel threads
- Decrease log level of locality information to debug

EDIT Fix for 4.0.2+: bfs462-rtmn-fix.patch


Tuesday, 10 March 2015


As if often the case, once I find time to work on a project, a few releases back to back usually follow. The same happened with lrzip, and here is version 0.621

and the git source code page:

For fun I compressed all stable linux kernels from 1.0 to 3.19 with lrzip as posted here:

Commemorative linux-1.0-3.19 compressed tarball
This was a 29GB tarball that compressed to 355MB and can be downloaded here:

What's new
- Substantial speed ups for the rzip stage in both regular and unlimited modes.
- Lrzip now supports long command line options.
- Proper support for the various forms of TMPDIR environment variables.
- More unix portability fixes.
- OSX fixes.
- Fixed order of lrzip.conf search.
- Addressed all warnings created with pedantic compiler settings and clang
- Fixes for some stderr messages being swallowed up.
- Fixed being unable to decompress to STDOUT when in a non-writable directory.
- Changed broken liblrzip callback function API to match lrzip proper.

* Fix libzpaq.cpp warnings
* Fix warnings in LzmaLib.c
* Making the high buffer only one page size is faster for sliding mmap mode
* Fix incompatible log callback in liblrzip
* Use PRId64 instead of lld in fscanf
* Use int64_t for i64
* Fix ISO C warning in lrzip.c
* Fix ISO C warning in lrzip.c
* Fix ISO C warning in runzip.c
* Fix ISO C warnings in rzip.c
* Fix iso c warning in util.c
* Fix EOL to unix on libzpaq
* control->tmpdir fixes
* Null terminate long options to find unhandled options
* Trivial date
* Add long option support
* Brace failure lead to broken dump to stdout
* fflush messages on print_err
* Fix inverse logic
* Honour the proper unix environment variable for temporary directory and then
try variations
* Attempt to decompress file in ram only if we are unable to create temporary
files when decompressing to stdout
* Avoid checking return values repeatedly on compress that only lead to failures
* Microoptimisation
* Don't check for failure condition that can no longer occur
* Don't check twice for failure conditions that are fatal in hot paths
* Cache the chunk bytes value to avoid setting it on each read_header call
* fake_mremap is only used when defined to mremap
* Remove unused cksem functions
* Fix remaining use of mutexes lock/unlocking in different threads with cksems,
corecting cksem usage on osx
* Update copyright dates
* Make match_len a function completely removing all indirect calls to get_sb,
significantly speeding up the single_get_sb case
* Make full_tag a pointer allowing us to avoid a function call for get_sb
* Call sliding_get_sb directly in sliding_next_tag
* Make next_tag a pointer to allow ordinary mapping to avoid an extra function
* fix order of lrzip.conf search

Tuesday, 3 March 2015

Lrzip 0.620

I finally found some time to give lrzip some much needed love again. Fortunately the last release, 0.616, proved very stable for the vast majority of workloads so there was never any great need to give it attention, though little things slowly cropped up, so I've accumulated all the bugfixes till now to release a new stable version, 0.620.

Freecode has long since died so here's the link to the (ghetto) home and download page:

and the git source code page:

Summary of changes:
- It would previously crash if trying to decompress from STDIN with a file that was too large to fit in ram which has now been fixed.
- It would previously fail to decompress files that were too large to fit in ram decompressed, now fixed.
- There were some scenarios lrzip would run out of ram when there was plenty to allocate, now fixed.
- Some other unix platforms would consider locking a mutex in one thread and releasing it in another a bug, so I've converted the use of those mutexes to anonymous semaphores.
- In order to maintain compatibility with platforms that don't properly support anonymous semaphores (OSX I'm looking at you), I've added the use of my custom fake semaphores as discussed here: unnamed-semaphores-and-pososx
- Some files would have their size reported wrongly with -i, now fixed.
- Added the ability to limit the use of ram with -m since lrzip happily uses all of it normally.
- Other minor changes and fixes for rare corner cases.

The changelog:
* Increase maxram when we abandon use of temporary input/output buffers
* Don't delete the tmpinfile when decompressing from stdin before allowing seek
to end to succeed in checking md5
* Use temporary file from read_seekto when STDIN will not fit in the ram input
* Remove unused read_i64 function
* Add message about issue tracker in BUGS
* Use a common exit path iin lrzip_compress/decompress and fix lr leak on
successful return
* Fix parenthesis placement inside of unlikely().
* Clear sa_mask and sa_handler before calling sigaction().
* Fix for lrzip -i. Decompressed size wrong
* added '-m' command line option
* Fix wrong README file being included in Makefile
* Pass strict sizes to decompress length, rounding up only the amount we're
allocating to not confuse decompression libraries
* Convert the thread locking to use cksems
* Add cksems to util.h
* Fix 'Failed to malloc ckbuf in hash_search2' with very large files.
* Round up compression and decompression buffers to page size since malloc will
allocate them that large anyway.
* Increase the compressed buffer size given to libzpaq in case of incompressible
data since it does not check if it's trying to write beyond the end of the
* Provide a helper function to round a value up to the nearest page size for
malloc optimisations.