5.14 and the future of MuQSS and -ck once again

 Having missed the update for the 5.13 kernel entirely, I thought I'd just skip ahead to merge up with 5.14 and started looking at/working on it today. The size of the changes are depressingly large and whilst it's mostly trivial changes, and features I wouldn't implement in MuQSS, I'm once again left wondering if I should be bothering with maintaining this patch-set, as I've mentioned before on this blog.

 The size of my user-base seems to be diminishing with time, and I'm getting further and further out of touch with what's happening in the linux kernel space at all, with countless other things to preoccupy me in my spare time. 

 As much as I still prefer running my own kernel on my hardware, I'm having trouble motivating myself after the last 18 months of world madness due to Covid19 and feel that I should really sadly bring this patch-set to a graceful end. My first linux kernel patches stretch back 20 years and with almost no passion for working on it any more, I feel it may be long overdue.

 Unfortunately I also do not have faith that there is anyone I can reliably hand the code over to as a successor as well, as almost all forks I've seen on my work have been prone to problems I've tried hard to avoid myself.

 There is always the possibility that mainline linux kernel will be so bad that I'll be forced to create a new kernel of my own out of disgust, which is how I got here in the first place, but that looks very unlikely. Many of you would have anticipated this coming after my last motivation blog-post, but unless I can find the motivation to work on it again, or something comes up that gives me a meaningful reason to work on it, I will have to sadly declare 5.12-ck the last of the MuQSS and -ck patches.

Final word. If you want to get the most out of the mainline kernel without trying to port MuQSS, then using at least the hrtimer patches from -ck and 1000Hz should make a significant difference.

-ck

Moving

Busy just moving house so there will likely be an extended delay before I can sync up with the latest kernel. Sorry about any inconvenience.

-ck

linux-5.12-ck1, MuQSS version 0.210 for linux-5.12

 Announcing a new -ck release, 5.12-ck1  with the latest version of the Multiple Queue Skiplist Scheduler, version 0.210 These are patches designed to improve system responsiveness and interactivity with specific emphasis on the desktop, but configurable for any workload.

This was a resync and build bugfix from 5.11-ck1. The only new change to the -ck patch is the ability to reselect ondemand and conservative governors with Intel Pstate, and to deselect schedutil.

 

 linux-5.12-ck1:

patch-5.12-ck1.xz

Git tree:

5.12-ck

MuQSS only:

0001-MultiQueue-Skiplist-Scheduler-v0.210.patch

Git tree:

5.12-muqss

Web: kernel.kolivas.org

Enjoy!
お楽しみ下さい
-ck

Lrzip version 0.641

Bugfix release for lrzip not fully compressing files on version 0.640 with no other changes.

Downloads, including access to standalone static binary:

http://lrzip.kolivas.org/

Git tree:

https://github.com/ckolivas/lrzip

 

Enjoy!

お楽しみ下さい

-ck

linux-5.11-ck1, MuQSS version 0.208 for linux-5.11

Announcing a new -ck release, 5.11-ck1  with the latest version of the Multiple Queue Skiplist Scheduler, version 0.208 These are patches designed to improve system responsiveness and interactivity with specific emphasis on the desktop, but configurable for any workload.

This was a resync and bugfix from 5.10-ck1. 

 linux-5.11-ck1:

patch-5.11-ck1.xz

Git tree:

5.11-ck

MuQSS only:

0001-MultiQueue-Skiplist-Scheduler-v0.208.patch

Git tree:

5.11-muqss

Web: kernel.kolivas.org

Enjoy!
お楽しみ下さい
-ck

Lrzip version 0.640

I haven't had much time till recently to dedicate to my application lrzip, but I have a soft spot for it since I use it extensively so I couldn't let it go to bitrot, and luckily I'm between other projects so have spent a few days on it. Here is an updated version with numerous bugfixes, some for security issues, some for crashes on corrupt archives, some to avoid creating corrupt archives, some for build issues, and random other bugfixes too numerous to list in detail. So this is a stable bug release and I encourage everyone to update.

Downloads, including access to standalone static binary:

http://lrzip.kolivas.org/

Git tree:

https://github.com/ckolivas/lrzip

The only change to its behaviour is it now requires liblz4-dev to build because it uses the super fast lz4 compression algorithm for compressibility testing - it does NOT use it for any compressed archives as this would be incompatible with the existing format, and almost certainly will not offer any advantage in speed over lzo because the bulk of the time is the rzip pre-compression library stage. It also now can build the crc checking as an assembly component on x86_64, though this is largely not going to affect speed in any significant fashion. Long term I will probably remove this ancient and not-very reliable checking from future versions if I break the existing format. I will also probably drop MD5 sums.

Peter Hyman has also been working behind the scenes to add countless little features to lrzip for the duration that I haven't given it any attention and has been running his own tree with all his changes. 

 https://github.com/pete4abw/lrzip/

Alas most of those are new features and break the existing lrzip file format, and have come with new issues of their own that would require quite some effort to blend into the existing codebase and iron out bugs. I'd also want new archives to be created compatible with older versions as much as possible preferentially so compatibility code would need to be added.

Thus I concentrated entirely on just bringing out another stable release before looking at any of his code, including having to roll back some of his changes I'd already incorporated into my master tree. In time I hope to get to his changes to incorporate them.

Enjoy!

お楽しみ下さい

-ck

linux-5.10-ck1, MuQSS version 0.205 for linux-5.10

Announcing a new -ck release, 5.10-ck1  with the latest version of the Multiple Queue Skiplist Scheduler, version 0.205 These are patches designed to improve system responsiveness and interactivity with specific emphasis on the desktop, but configurable for any workload. 

 

Probably the most interesting thing to happen as pointed out to me by Damentz was that the Intel i915 scheduler is based on the scheduling algorithm from MuQSS:

 [Intel-gfx] [PATCH 36/56] drm/i915: Fair low-latency scheduling  

 

It seems they understand the incredible simplicity of the underlying scheduling algorithm that guarantees both latency and fairness intrinsically.

 

This was a very minor resync from 5.9-ck1.

linux-5.10-ck1:

-ck1 patch:

patch-5.10-ck1.xz

 

Git tree:

5.10-ck

 

MuQSS only:

Download:

5.10-muqss-205.patch

 

Git tree:

5.10-muqss

 

 

Web: http://kernel.kolivas.org
 

Enjoy!

お楽しみ下さい

-ck

linux-5.9-ck1, MuQSS version 0.204 for linux-5.9

Unfortunately these past few months have been marred by lockdown and family issues, culminating in the ultimate death of my father just over a month ago (unrelated to covid19 but made that much worse because of its effects on everything in our city) so linux kernel was the furthest thing from my mind and a 5.8 resync never happened. He'll be sorely missed, and if this were something more substantial I'd dedicate it towards him but it doesn't do him justice.

Announcing a new -ck release, 5.9-ck1  with the latest version of the Multiple Queue Skiplist Scheduler, version 0.204 These are patches designed to improve system responsiveness and interactivity with specific emphasis on the desktop, but configurable for any workload. 

 

This was a massive resync being two versions deep and is only lightly tested so far so take the usual precautions.

linux-5.9-ck1:

-ck1 patch:

patch-5.9-ck1.xz

Git tree:

5.9-ck

MuQSS only:

Download:

5.9-muqss-204.patch

Git tree:

5.9-muqss

Web: http://kernel.kolivas.org

This release is a minor bugfix and resync from 5.7-ck1.

Enjoy!

お楽しみ下さい

-ck

More delays and motivation issues

 Hi all. 

 

Sorry I haven't gotten out a MuQSS and -ck release for linux-5.8. Some of you have been emailing me to check on my well-being in this crazy world. I appreciate the concern, and apart from family issues, I've been fine. It's fair to say that my motivation for keeping up with linux kernel development has been flagging for some time now and the current world situation is not helping. Hearing the news extol the virtues of linux-5.8 being the "biggest release ever" does not particularly aid my situation. If it were just a massive drop of new drivers I could understand that, but usually it just means yet more rewrites of major infrastructure within the kernel in the quest to "make it better." Personally I don't think it's such a great thing, but that's a debate best left for elsewhere. I do plan to stay in sync with 5.8 and future kernel releases, but I'm not sure when I'll be motivated to resume that resyncing process. My biggest concern with the massive churn is me screwing something up in a way that leaves users of my code open to security issues or fatal data corruption at some stage because I haven't been careful enough to protect against this happening. For this reason I've often considered abandoning the code entirely but some supportive individuals have stated they find comfort in the relative stability and continued utility of MuQSS's code in the increasingly volatile kernel churn world which is reassuring and encouraging enough for me to at least plan to stay in sync. 

 

As time goes on and more and more features get added to the scheduler that have nothing to do with ordinary desktop and mobile platform usage, at some stage distributions will be tempted to become dependent on one or more of those features and if I don't develop MuQSS much further to incorporate my own version of those features, it will become redundant. Given the completely different scheduler architecture of MuQSS versus CFS means I can't simply just port over the code most of the time; I have to write my own complete feature equivalent version and these are far from trivial. The accounting code is completely different, most of the CGROUP features aren't even implemented, and deadline scheduling is not available at all for example. If more of these appear in the future and eventually become showstoppers, then unless some miracle happens to make me find the motivation to work on them, it will be the death of it.

Cheers all. Stay safe and well,

-ck

linux-5.7-ck1, MuQSS version 0.202 for linux-5.7

Announcing a new -ck release, 5.7-ck1  with the latest version of the Multiple Queue Skiplist Scheduler, version 0.202. These are patches designed to improve system responsiveness and interactivity with specific emphasis on the desktop, but configurable for any workload.

linux-5.7-ck1:

-ck1 patch:

patch-5.7-ck1.xz

Git tree:

5.7-ck

MuQSS only:

Download:

5.7-muqss-202.patch

Git tree:

5.7-muqss

Web: http://kernel.kolivas.org

This release is a minor bugfix and resync from 5.6-ck2.

Enjoy!

お楽しみ下さい

-ck

linux-5.6-ck2, MuQSS version 0.2 for linux-5.6

Announcing a new -ck release, 5.6-ck2  with the latest version of the Multiple Queue Skiplist Scheduler, version 0.2. These are patches designed to improve system responsiveness and interactivity with specific emphasis on the desktop, but configurable for any workload. This is a maintenance release to address a build failure on -ck1 when built with full dynticks, and fix some cosmetic CPU load accounting issues. Upgrading is not required unless you are affected by the aforementioned issues or are rebuilding for a new stable release. It's worth pointing out that changing the reported load might have repercussions with how CPU frequency scaling behaves.

linux-5.6-ck2:

-ck2 patch:

patch-5.6-ck2.xz

Git tree:

5.6-ck

MuQSS only:

Download:

5.6-muqss-200.patch

Git tree:

5.6-muqss

Web: http://kernel.kolivas.org


As an aside, it has been brought to my attention that the MESA code uses SCHED_IDLEPRIO for what it considers low priority threads. In the mainline kernel this only makes them lower priority than regular tasks, but on MuQSS which has true idle scheduling, it can potentially lead to stalls under conditions of load. Once a thread has stalled for an extended time, it is possible that it will not progress normally depending on how the code expects to run. This could lead to GUI stalls in applications that use MESA, of which there are quite a few now, such as firefox. I've been considering submitting a change to the MESA code in the hope they approach this differently, but I am a pragmatist and expect the turnaround time and acceptability of the changes may be drawn out and unsatisfactory. So I am considering alternatively softening the idle scheduling and making it configurable to behave more like mainline's by default and optionally be set to be true idle scheduling. In the meantime, I've prepared some hacked mesa packages for those on ubuntu 20.04 variants that disable this behaviour, but this is a cludge only for the time being:

mesa-ubuntu20.04 packages

Here is a much better patch for Mesa that converts threads to nice 19 SCHED_BATCH instead:

0001-Linux-Change-minimum-priority-threads-from-SCHED_IDL.patch


 Enjoy!

お楽しみ下さい

-ck

linux-5.6-ck1, MuQSS version 0.199 for linux-5.6

Announcing a new -ck release, 5.6-ck1  with the latest version of the Multiple Queue Skiplist Scheduler, version 0.199. These are patches designed to improve system responsiveness and interactivity with specific emphasis on the desktop, but configurable for any workload. Since MuQSS has reached version 0.199, I'm left wondering what I should make the next version. As it is likely to be just as trivial an update as this one was, version 0.200 will not actually be a significant update to some new fancy scheduler, it will just coincidentally bump the minor  version up. That said, MuQSS has been stable for a very long time so I should have bumped it to version 1.0 quite a while back.

linux-5.6-ck1:

-ck1 patch:

patch-5.6-ck1.xz

Git tree:

5.6-ck

MuQSS only:

Download:

5.6-muqss-199.patch

Git tree:

5.6-muqss

Web: http://kernel.kolivas.org

This is a resync from 5.5-ck1.

Enjoy!

お楽しみ下さい

-ck

Covid-19 Delays

As you've probably gathered, I've been preoccupied with Covid-19 issues delaying any attempt to stay in sync with the latest kernel. I didn't even know there was a new one till recently.

Here's what I've been busy with:

https://www.abc.net.au/news/2020-04-06/doctors-designing-medical-equipment-to-face-coronavirus-covid-19/12120588

I'll be back when things stop being so crazy.

お大事にしてください
Please take care,
-ck

linux-5.5-ck1, MuQSS version 0.198 for linux-5.5

Announcing a new -ck release, 5.5-ck1  with the latest version of the Multiple Queue Skiplist Scheduler, version 0.198. These are patches designed to improve system responsiveness and interactivity with specific emphasis on the desktop, but configurable for any workload.

linux-5.5-ck1:

-ck1 patch:

patch-5.5-ck1

Git tree:

5.5-ck

MuQSS only:

Download:

5.5-muqss-198.patch

Git tree:

5.5-muqss

Web: http://kernel.kolivas.org
This is a resync from 5.4-ck1, with some minor tidying and fixes.

Enjoy!

お楽しみ下さい

-ck

Happy new decade.

Happy new decade everyone.

Here's an interesting blogpost a few people on IRC have pointed out to me. Best save reading it for after you've sobered up in the new year ☺

https://probablydance.com/2019/12/30/measuring-mutexes-spinlocks-and-how-bad-the-linux-scheduler-really-is

The anecdotal video performance part of that whole post may well be the most interesting part of it, but there's definitely food for thought there regarding locking primitives and MuQSS's performance. I've pointed out in a comment (yet to be published) to the author that yield is configurable on MuQSS and use of it is likely responsible for the poor performance of the sub-benchmarks involving it. Either way it makes for interesting reading and I anticipate a mainline mutex and especially spinlock performance speed chase in the near future. This can only lead to good things for everyone.

linux-5.4-ck1, MuQSS version 0.196 for linux-5.4

Announcing a new -ck release, 5.4-ck1  with the latest version of the Multiple Queue Skiplist Scheduler, version 0.196. These are patches designed to improve system responsiveness and interactivity with specific emphasis on the desktop, but configurable for any workload.

linux-5.4-ck1:

-ck1 patches:

5.4-ck1

Git tree:

5.4-ck

MuQSS only:

Download:

5.4-muqss-196.patch

Git tree:

5.4-muqss

Web: http://kernel.kolivas.org


This is a resync from 5.3-ck1, done promptly to make up for the last release's delays.

Enjoy!

お楽しみ下さい

-ck

linux-5.3-ck1, MuQSS version 0.195 for linux-5.3

Announcing a new -ck release, 5.3-ck1  with the latest version of the Multiple Queue Skiplist Scheduler, version 0.195. These are patches designed to improve system responsiveness and interactivity with specific emphasis on the desktop, but configurable for any workload.

linux-5.3-ck1:

-ck1 patches:

5.3-ck1

Git tree:

5.3-ck

MuQSS only:

Download:

5.3-muqss-195.patch

Git tree:

5.3-muqss

Web: http://kernel.kolivas.org


This is a resync from 5.2-ck1 plus the Ryzen/LLC fixes courtesy of Eduards Bezverhijs (thanks very much!) virtually unchanged. A reminder if you're new to using my patches, MuQSS performs best when in combination with the full -ck patchset as they're all complementary changes.

Sorry about the delay, I was in the thick of a project I had to complete first.

You will find that it may not completely apply to later 5.3.x kernels because of a very small change to a Makefile. It's trivial to fix, but please note my patches are always designed around 2 point releases, in this case 5.3, and I never try to resync with the many 3 point stable releases that follow.

Enjoy!

お楽しみ下さい

-ck

5.3 Delays

Sorry about not releasing a new kernel. Motivation and time issues have prevented me from updating to the latest kernel. I will get around to it eventually.

linux-5.2-ck1, MuQSS version 0.193 for linux-5.2

Announcing a new -ck release, 5.2-ck1  with the latest version of the Multiple Queue Skiplist Scheduler, version 0.193. These are patches designed to improve system responsiveness and interactivity with specific emphasis on the desktop, but configurable for any workload.

linux-5.2-ck1:

-ck1 patches:

5.2-ck1

Git tree:

5.2-ck

MuQSS only:

Download:

5.2-muqss-193.patch

Git tree:

5.2-muqss

Web: http://kernel.kolivas.org


This is mostly a resync from 5.1-ck1. A reminder if you're new to using my patches, MuQSS performs best when in combination with the full -ck patchset as they're all complementary changes.

Enjoy!

お楽しみ下さい

-ck

linux-5.1-ck1, MuQSS version 0.192 for linux-5.1

Announcing a new -ck release, 5.1-ck1  with the latest version of the Multiple Queue Skiplist Scheduler, version 0.192. These are patches designed to improve system responsiveness and interactivity with specific emphasis on the desktop, but configurable for any workload.

linux-5.1-ck1:

-ck1 patches:

5.1-ck1

Git tree:

5.1-ck

MuQSS only:

Download:

5.1-muqss-192.patch

Git tree:

5.1-muqss

Web: http://kernel.kolivas.org


This is mostly a resync from 5.0-ck1 with some build and boot fixes courtesy of Serge Belyshev (thanks!)

Enjoy!

お楽しみ下さい

-ck