Friday, 13 February 2026

Lrzip version 0.660

I finally found the time to give lrzip a long overdue update which I hadn't touched in 4 years. Apologies for my extended absence from this project.

 Get it from here:

 https://github.com/ckolivas/lrzip/tree/v0.660

 This is a maintenance bugfix release for the accumulated issues developed in that time. Many of the fixes are security fixes for potentially maliciously crafted archives, thus it is strongly recommended existing users of lrzip update.

 The brief WHATS-NEW notes summarise the notable changes:

  •  Address multiple potential security issues with crafted or corrupt archives.
  • Accept password usage as a parameter on decompressing encrypted files.
  • Fix CPU detection not respecting affinity.
  • Fix segfault with zpaq and incompressible blocks.
  • Fix inappropriate warnings being displayed.
  • Minor cleanups and extra code sanity checks.


Saturday, 4 November 2023

EEVDF & the mainline linux kernel scheduler

 A number of people have already asked me my opinion on the development of EEVDF on top of CFS for the mainline kernel. All of my previous schedulers - staircase first, followed by BFS, and finally MuQSS, were all EEVDF designs, so in principle at least you can imagine I'm mildly intrigued and pleased with this direction. I think it's the best known way to tackle interactivity and responsiveness in a CPU process scheduler.

Any qualms I may have about it would be the reluctance to move processes/threads from one CPU to another to achieve said goal of tackling the earliest eligible virtual deadline process first. As it is common for processes to be relatively "sticky" to per-CPU runqueues for cache warmth and throughput reasons, this ends up being orthogonal to the demands of scheduling for minimal latency first. 

In my original BFS design there was only one runqueue  for all CPUs which was the optimal design for a global EEVDF design but this would eventually not have scaled to many CPUs in throughput. This led to the development of MuQSS for which I moved to multiple runqueues, but soon found that sharing runqueues for latency reasons was more important than worrying about the last bit of throughput. This is why in configuration it was possible to choose the degree to which runqueues were shared to choose to optimise primarily around throughput or latency - the more sharing, the more latency focused the scheduler behaved. Sharing runqueues between shared cache CPUs provided the best compromise at the time, though modern CPUs have far more cores and threads which all share various levels of cache. 

Much like there are sorting algorithms which excel at different sizes (nothing beats insertion sort for up to ~16 variables), I expect runqueue sharing would exhibit a similar phenomenon and that it would actually be disadvantageous to have many runqueues for small numbers of CPUs. My random prediction based on older anecdotal observation is that number is also up to about 16 threads/cores per runqueue (provided they're all sharing at least some form of cache.)

As I've not looked at mainline kernel code at depth in years, and not at all at this new EEVDF development I cannot comment with any authority at all on the code nor implementation at this stage, but it's certainly an admirable goal and I'm cautiously optimistic about it.

Wednesday, 9 March 2022

lrzip version 0.651

 As often happens shortly after a substantial release, some minor issues are discovered and a small update is required so here is 0.651. The main issue was potentially confusing locale dependent output which has been reverted. Hopefully it's been short enough a period from 0.650 that distros will not have adopted that one yet.

Get it here:

http://ck.kolivas.org/apps/lrzip/

or via git here:

https://github.com/ckolivas/lrzip

 What's new:

  • Remove redundant files
  • Revert locale dependent output
  • Add warnings for low memory and threads

-ck


Sunday, 27 February 2022

lrzip version 0.650

A number of accumulated bug reports had collected since the last lrzip release and since I regularly use lrzip I want to make sure it stays bug free as far as I am aware, even if I'm not planning any new features for it. As some of the changes are potentially security fixes, I urge any user to update.

Get it here:

http://ck.kolivas.org/apps/lrzip/

or via git here:

https://github.com/ckolivas/lrzip 

Here is the what's new list:

  • Minor optimisations.
  • Exit status fixes.
  • Update and beautify information output.
  • Fix Android build.
  • Enable MD5 on Apple build.
  • Deprecate and remove liblrzip which was unused and at risk of bitrot.
  • Fix failures with compressing to STDOUT with inadequate memory.
  • Fix possible race conditions.
  • Fix memory leaks.
  • Fix -q to only hide progress.
  • Add -Q option for very quiet.
-ck