Linux-7.1-ck1, MuQSS version 0.3 for linux-7.1

It's been 10 long years since the last -ck patchset. I wasn't planning on ever updating it because of the amount of time involved to resync with each new kernel version. However, I'd missed its improvements, and Claude AI has made it a fraction of the workload it used to be. So, here is the first updated patchset which is mostly just a resync of the old patches for the latest linux kernel.

 https://github.com/ckolivas/linux/tree/v7.1-ck1 

The only new changes are MuQSS now uses P/E core aware idle balancing.

Only lightly tested and may not build on other configs, so use usual precautions. 

お楽しみください

-ck 

•  Add -ck1 version.
•  Make nohz_full not be picked up as a default config option and add recommendation to help.
•  Reinstate default Hz of 100 in combination with MuQSS and -ck patches.
•  Make hrtimer granularity and minimum hrtimeout configurable in sysctl. Set default granularity to 100us and min timeout to 500us.
•  Don't use hrtimer overlay when pm_freezing since some drivers still don't correctly use freezable timeouts.
•  Replace all calls to schedule_timeout_uninterruptible to use schedule_msec_hrtimeout_uninterruptible.
•  Replace all calls to schedule_timeout_interruptible to use schedule_msec_hrtimeout_interruptible.
•  Convert msleep to use hrtimers when active.
•  Convert all low value schedule_timeouts to their hrtimeout equivalents.
•  Create highres timeout variants of schedule_timeout functions.
•  Make preemptible kernel default.
•  Make threaded IRQs optionally the default which can be disabled.
•  MultiQueue Skiplist Scheduler v0.300

Lrzip version 0.7.0

https://github.com/ckolivas/lrzip/tree/v0.7.0

• New file format mostly backward compatible except for streamed large STDIO compressed files and encrypted files.
• Update embedded libzpaq from 5.00 to 7.15 - faster & better compression.
• Update embedded LZMA SDK from ~4.63/9.x to 26.02 - faster & better compression.
• Default -e encryption is AES-256-GCM + PBKDF2-HMAC-SHA512 (magic[22]=3).
• Add --legacy-encrypt for 0.6-compatible AES-128-CBC (magic[22]=1).
• More robust encryption protection against malicious files, but much slower.
• Compressing via STDIO no longer writes temporary files, using the new streaming file format instead.
• Updated lrztar to accept most lrzip options.
• Many optimisations and cleanups.
• 32 bit build completely deprecated and removed.
  

Lrzip version 0.660

I finally found the time to give lrzip a long overdue update which I hadn't touched in 4 years. Apologies for my extended absence from this project.

 Get it from here:

 https://github.com/ckolivas/lrzip/tree/v0.660

 This is a maintenance bugfix release for the accumulated issues developed in that time. Many of the fixes are security fixes for potentially maliciously crafted archives, thus it is strongly recommended existing users of lrzip update.

 The brief WHATS-NEW notes summarise the notable changes:

•  Address multiple potential security issues with crafted or corrupt archives.
• Accept password usage as a parameter on decompressing encrypted files.
• Fix CPU detection not respecting affinity.
• Fix segfault with zpaq and incompressible blocks.
• Fix inappropriate warnings being displayed.
• Minor cleanups and extra code sanity checks.

EEVDF & the mainline linux kernel scheduler

 A number of people have already asked me my opinion on the development of EEVDF on top of CFS for the mainline kernel. All of my previous schedulers - staircase first, followed by BFS, and finally MuQSS, were all EEVDF designs, so in principle at least you can imagine I'm mildly intrigued and pleased with this direction. I think it's the best known way to tackle interactivity and responsiveness in a CPU process scheduler.

Any qualms I may have about it would be the reluctance to move processes/threads from one CPU to another to achieve said goal of tackling the earliest eligible virtual deadline process first. As it is common for processes to be relatively "sticky" to per-CPU runqueues for cache warmth and throughput reasons, this ends up being orthogonal to the demands of scheduling for minimal latency first. 

In my original BFS design there was only one runqueue  for all CPUs which was the optimal design for a global EEVDF design but this would eventually not have scaled to many CPUs in throughput. This led to the development of MuQSS for which I moved to multiple runqueues, but soon found that sharing runqueues for latency reasons was more important than worrying about the last bit of throughput. This is why in configuration it was possible to choose the degree to which runqueues were shared to choose to optimise primarily around throughput or latency - the more sharing, the more latency focused the scheduler behaved. Sharing runqueues between shared cache CPUs provided the best compromise at the time, though modern CPUs have far more cores and threads which all share various levels of cache. 

Much like there are sorting algorithms which excel at different sizes (nothing beats insertion sort for up to ~16 variables), I expect runqueue sharing would exhibit a similar phenomenon and that it would actually be disadvantageous to have many runqueues for small numbers of CPUs. My random prediction based on older anecdotal observation is that number is also up to about 16 threads/cores per runqueue (provided they're all sharing at least some form of cache.)

As I've not looked at mainline kernel code at depth in years, and not at all at this new EEVDF development I cannot comment with any authority at all on the code nor implementation at this stage, but it's certainly an admirable goal and I'm cautiously optimistic about it.