lrzip-0.552 Random fixes.

It had been a while since lrzip had received attention, and a number of things came up that made me give it some attention. First, it wouldn't compile on FreeBSD because it doesn't implement mremap. So I wrapped that in a fake mremap the way I managed the OSX one. Then OSX would compile the existing code fine but would fail to actually work because it would say "unimplemented" when you tried to run it due to OSX not having unnamed semaphores implemented. I've mentioned this before, but named semaphores are a real pain to work with and unnamed ones are very convenient. So I wrapped all the threading calls for OSX to not actually thread and ignore all the calls to semaphore functions. It means OSX doesn't benefit from the extra threading added after version 0.530 but at least it actually works now.

Finally, there was an alarming report I received about file corruption on decompression from a user who had a 20GB log that he was compressing. The generated archive would decompress fine with version 0.530 but it randomly would silently corrupt somewhere in the decompression with any version after that and it was only noticeable after comparing to the original file. Now, try as hard as I could at home, I couldn't reproduce this bug. So with the help of this user doing possibly hundreds of tests for me and with debugging code I found a trail of possible causes. Disabling the decompression threading code surprisingly didn't fix it, confirming the bug was elsewhere. After an extensive search I found there were some mmap calls in lrzip that weren't being careful about being aligned to page size, hence writing to the buffer generated would have random results. It's surprising that no more corruption was seen but presumably the particular buffer in question never had that much data so it would fit in whatever amount happened to be allocated. Presumably the larger file is what made it easier to trigger. That would certainly have explained random position failures, but it didn't really explain why it only started happening after 0.530.

Anyway after converting the mmap calls to ordinary malloc calls, decompression would now actually fail with a crc error. Lrzip does a crc check on the data generated, and compares it to the crc stored in the archive. However if a string of zeroes is generated, and then the crc is also read as zero, it can look ok. I'm assuming this is why it was silently corrupting it. The crc errors at least gave me a trail of possible sources for the error. After much scratching of heads, heavy drinking and sleepless nights, I found that in the original rzip code that lrzip came from was a function that was designed to read a certain amount of data, then return how much it had returned. The thing is, the function that called it (unzip_literal) didn't check the return value and just assumed it always read all the data asked of it. Hitting this codepath is obviously extremely rare but there it was. Fixing that resolved the instant crc error and made it mostly reliable.

Now I'd love to say it's 100% reliable, but after running the decompression hundreds of times just to be certain, it failed on one occasion. So there's still potential for failure, possibly somewhere else, in the code on decompressing extremely large archives of 20GB or more. Given the number of fixes I put into lrzip, and that there were obvious bugs in the previous version, I've released the cumulative fixes as version 0.552. I've also put a warning in there that all very large decompressed files should be checked with md5sum or equivalent to ensure they are not corrupted.

A handful of other minor fixes went in too, so this is basically a bugfix release only. So if you're using lrzip, I highly recommend upgrading to version 0.552.

Freshmeat link (sometimes takes a while for new release to appear):
lrzip

Now for more debugging.

Becoming a Ubu nut. -ck packages for Ubuntu

I've been on debian stable for many years as my primary desktop, being quite happy with relatively old software provided most things worked as expected. The only things I ever updated outside of the repositories were firefox, openoffice and only random other small things. However I finally have had enough. KDE 4.4 broke me. No, I'm not interested in using a different desktop. I tried and tried recently and they just never felt right, mainly because I've been on KDE for so long that it works the way I want it to (when it did work!). KDE 4.4 still had some showstopper bugs for my usage that affected me daily, and reporting them upstream has had no effect, because they're all fixed in 4.5 or later, and will not be backported to 4.4 on debian. Now it's not entirely debian's fault since they have to freeze at some point - I do understand that. It's just that kde4.4 was still... meh.

So I bit the bullet and installed (K)Ubuntu 10.10 on my primary desktop, which till now I'd only used on a laptop, but it had been quite respectable. After an uneventful install and reboot, my keyboard and mouse would not work. I tried different usb keyboards and ps/2 ones to no avail. Ironically the install DVD worked fine, and using the rescue mode I tried the first logical thing - I installed a -ck kernel that I had from my previous install. This fixed it. So anyway I'm not going to go on about distros because that's a long hard painful discussion that there is no answer to, and all I can say is there are different distros out there and just use what's comfortable for you. I'm also not going to try and review the distro. There are enough people out there that do this already. I just wish the KDE 4.x composite + nvidia driver combination causing high CPU usage and slowdown issues with kwin and xorg would be fixed :\ Search for kwin high CPU and nvidia and you'll see loads and loads of posts on heaps of forums with different distros and no solution. I'm betting it's nvidia's fault. (Disabling blur makes it use less CPU, but sluggishness kicks in after a while regardless).

So anyway the point of this is that I figured - what the heck, I may as well make some distro kernels for Ubuntu that people keep asking me about. So I've quickly packaged up an amd64 2.6.35.11-ck1 and 2.6.37-ck1 kernel. The older one I recommend for most people, and the newer one for relatively more adventurous souls.

For now they can be found here (no ppa or anything at the moment, just .deb packages):
Ubuntu Packages

Note that if you have a 32 bit Ubuntu install, but a 64 bit capable CPU (as all are in the last few years) you can still force install these kernels and they should work fine on 32 bit userspace. You might have trouble if you install other drivers though. There's a good chance these packages will work on debian as well, but I can't guarantee anything. Please report back if you try that combination out.

Enjoy!

EDIT: I added the backports ppa by kubuntu so that I can install kde 4.6 and all remnants of sluggishness are gone! Thank goodness. It looks like it was kde after all.

2.6.37-ck1, BFS 0.363 for 2.6.37, Grouping by UID

It looks like 2.6.37 made it out in time before I left for my trip, so here's some goodies to keep you all busy, from the emails I just sent to announce them on lkml:

---

These are patches designed to improve system responsiveness and interactivity with specific emphasis on the desktop, but suitable to any workload.


Apply to 2.6.37:
http://www.kernel.org/pub/linux/kernel/people/ck/patches/2.6/2.6.37/2.6.37-ck1/patch-2.6.37-ck1.bz2

Broken out tarball:
http://www.kernel.org/pub/linux/kernel/people/ck/patches/2.6/2.6.37/2.6.37-ck1/2.6.37-ck1-broken-out.tar.bz2

Discrete patches:
http://www.kernel.org/pub/linux/kernel/people/ck/patches/2.6/2.6.37/2.6.37-ck1/patches/

All -ck patches:
http://www.kernel.org/pub/linux/kernel/people/ck/patches/


Web:
http://kernel.kolivas.org

Code blog when I feel like it:
http://ck-hack.blogspot.com/

Each discrete patch contains a brief description of what it does at the top of
the patch itself.


The most significant change is an updated BFS CPU scheduler, up to version
0.363. See the announce of that patch for the changelog. The rest is a resync.


2.6.37-sched-bfs-363.patch
sched-add-above-background-load-function.patch
mm-make_swappiness_really_mean_it.patch
mm-zero_swappiness.patch
mm-enable_swaptoken_only_when_swap_full.patch
mm-drop_swap_cache_aggressively.patch
mm-kswapd_inherit_prio-1.patch
mm-background_scan.patch
mm-idleprio_prio-1.patch
mm-lru_cache_add_lru_tail.patch
mm-decrease_default_dirty_ratio.patch
kconfig-expose_vmsplit_option.patch
hz-default_1000.patch
hz-no_default_250.patch
hz-raise_max.patch
preempt-desktop-tune.patch
cpufreq-bfs_tweaks.patch
ck1-version.patch


---

The BFS (name shall not be said for PG requirements) CPU scheduler for 2.6.37
is now available.

Since the last release, a lot more work was put into maintaining fine grained
accounting at all times (which should help on 32 bit machines, uniprocessor
and 100Hz configs), minor changes have been made to make CPU offline code more
robust for the purposes of suspend to ram/disk, and some small scalability
improvements were added for SMT CPUs (eg i7). These changes are unlikely to
have dramatically noticeable effects unless you were already experiencing a
problem or poor performance.

A direct link to the patch for 2.6.37 is here:
http://ck.kolivas.org/patches/bfs/2.6.37/2.6.37-sched-bfs-363.patch

All BFS patches here:
http://ck.kolivas.org/patches/bfs

Version 363 has been ported to 2.6.35 and 2.6.32 and available from that
directory in lieu of the long term release nature of these kernels.


On a related note, a small multi-user server feature request was commissioned
for BFS that I was happy to work on, which I'd like to also make publicly
available.

Here is the changelog:

---

Make it possible to proportion CPU resource strictly according to user ID by
grouping all tasks from the one user as one task.

This is done through simply tracking how many tasks from the one UID are
running at any one time and using that data to determine what the virtual
deadline is, offset by proportionately more according to the number of running
tasks. Do this by creating an array of every UID for very quick lookup of the
running value and protect it by the grq lock. This should incur almost
immeasurably small overhead even when enabled. An upper limit of 65535 UIDs
is currently supported.

Make this feature configurable at build time and runtime via Kconfig, and
through the use of sysctls

/proc/sys/kernel/group_by_uid
to enable or disable the feature (default 1 == on), and

/proc/sys/kernel/group_uid_min
to decide the minimum uid to group tasks from (default 1000)

Nice values are still respected, making it possible to allocate different
amounts of CPU to each user.

This feature is most suited to a multi-user shell type server environment and
is NOT recommended for an ordinary desktop.

---
The patch is available for the moment here:
http://ck.kolivas.org/patches/bfs/test/bfs363-group_uids.patch


A reminder that this is NOT a desktop, laptop or embedded device type feature.
The purpose of this feature is to make it impossible for any one user to get
more CPU than any other user on a multi-user login. This is suitable for
multiuser shared GUI/X session or shell type machines, and incurs almost
negligible overhead.


---
I'll be offline shortly and in Japan for a few weeks so I'll be unlikely to respond to any emails in that time.

Enjoy!

BFS version 0.363

Welcome to 2011!

The testing on BFS ported to 2.6.37-rc8 has been reassuring, and no real show stopper bugs have shown up. The remaining changes required to make it release ready for 2.6.37 have now been committed, along with some other very minor changes, so I've bumped the version up to 0.363. The main change was implementing the fine grained interrupt accounting which will have very little, if any, impact on regular users. These changes are ONLY suitable for 2.6.37, so they have not been ported back to the BFS I'm maintaining for earlier kernels. The rest of the changes suitable for older kernels have gone into 363 for them.

Here is the changelog as it affects existing BFS 360 users:

Make CPU offlining more robust by simply removing all affinity for processes
that no longer have any CPUs they can run on. This allows the machine stop
thread to complete offlining CPUs and makes for a little less overhead in hot
paths.

Allow SCHED_IDLEPRIO to wake up idle CPUs in try_preempt. This would have
caused minor slowdowns for IDLEPRIO tasks only on relatively quiescent systems.

Remove inappropriate likely()s.

Update cpustat for irq - may have been under-reporting interrupt load.

Cosmetic changes.

Bump version to 0.363

Most of these changes should have no visible behavioural effect to the user, apart from the following:

For those on BFS 360, if you were having warnings or even OOPSes on suspend to ram/disk or wakeup from them, or if you were having trouble suspending or resuming, this change might help.

The other change guarantees that CPUs will be busier on SMP machines when tasks are being run IDLEPRIO, so it will increase throughput, but ONLY if you run tasks IDLEPRIO.


Incremental: 2.6.36-bfs-360-363.patch

For 2.6.36ish:
2.6.36-sched-bfs-363.patch

2.6.35.10:
2.6.35.10-sched-bfs-363.patch

2.6.32.27:
2.6.32.27-sched-bfs-363.patch

Shortly I'll be going to Japan for a few weeks as I do almost every year now, so I'll be offline for a while.