Friday 25 October 2019

linux-5.3-ck1, MuQSS version 0.195 for linux-5.3

Announcing a new -ck release, 5.3-ck1  with the latest version of the Multiple Queue Skiplist Scheduler, version 0.195. These are patches designed to improve system responsiveness and interactivity with specific emphasis on the desktop, but configurable for any workload.

linux-5.3-ck1:
-ck1 patches:
Git tree:
MuQSS only:
Download:
Git tree:


Web: http://kernel.kolivas.org


This is a resync from 5.2-ck1 plus the Ryzen/LLC fixes courtesy of Eduards Bezverhijs (thanks very much!) virtually unchanged. A reminder if you're new to using my patches, MuQSS performs best when in combination with the full -ck patchset as they're all complementary changes.
Sorry about the delay, I was in the thick of a project I had to complete first.

You will find that it may not completely apply to later 5.3.x kernels because of a very small change to a Makefile. It's trivial to fix, but please note my patches are always designed around 2 point releases, in this case 5.3, and I never try to resync with the many 3 point stable releases that follow.
Enjoy!
お楽しみ下さい
-ck

34 comments:

  1. Thank you for the update.

    Note, that that the patch fails to apply against 5.3.7 at tools/objtool/Makefile, for the same reason 5.2-ck1 failed against 5.2.18.

    The patch can be again updated to apply against the latest kernel releases with the following:
    sed -i -e '/^-CFLAGS/ s,+=,:=,' -i -e '/^+CFLAGS/ s,+=,:=,' patch-5.3-ck1

    ReplyDelete
  2. This comment has been removed by a blog administrator.

    ReplyDelete
  3. Thank you for the update.

    I am trying to incorporate the -ck patchset into the disto I use. They have a kernel config which is generated through the use of their tooling. Do I need to manually select any kernel config options or will the patchset set what is required (for a laptop) automatically? For example something like, CONFIG_FORCE_IRQ_THREADING=y.

    A better question might be what are the necessary kernel config options for proper -ck patchset experience.

    dk

    ReplyDelete
    Replies
    1. The default config options are the most compatible and give you most of the experience. The IRQ threading for example very rarely causes issues which is why I don't default to having it on.

      Delete
  4. Thank you for the update.

    It does not seem as this includes the patch described here: https://bbs.archlinux.org/viewtopic.php?pid=1863063#p1863063 with patch posted here: https://github.com/ckolivas/linux/pull/17/commits/3d6e52f1415a5bcdc6b06f10f9a48388597fcd60

    Given i have not actually tested -ck on 5.3 yet, but since BMQ incorporates this patch with current release, i would think it is needed.

    ReplyDelete
  5. patch -p1 < ../5.3-ck1/0002-Fix-Werror-build-failure-in-tools.patch
    patching file tools/objtool/Makefile
    Hunk #1 FAILED at 35.
    1 out of 1 hunk FAILED -- saving rejects to file tools/objtool/Makefile.rej

    ReplyDelete
    Replies
    1. Look at the first post up top...

      Delete
    2. The patch is unfortunately always designed around the 2 point release, in this case 5.3. The sed script provided above by ooo (thanks!) can make it work with the latest 3 point release.

      Delete
    3. OK, got it.
      Thanks.

      Delete
  6. Thank you very much.

    ReplyDelete
  7. Hello there!

    First off, just want to say thank you for all the hard work you do, I use the linux-ck-skylake from the repo-ck repo on arch and it is amazing.

    One request though, and if this is the wrong place to make such a request then I apologise, but have you considered applying the fsync patchset? It's the only thing I miss from linux-zen, otherwise, yours trumps it in every way, haha

    Again, thank you so much. have a great day!

    ReplyDelete
    Replies
    1. You're welcome. My experience with maintaining other peoples' patches has not been that satisfactory so I prefer to only stick to patches that I've created for convenience, efficiency, familiarity, reliability, and workload.

      Delete
    2. I have to agree, as fsync imo does not really have that much to do with scheduler patches. Patchsets like the fsync patches is more distro/other specific, and maintaining upgrades from Valve to this before its mainline is not really efficient.

      Andrew: You are better off trying to get distro's like Arch/Ubuntu/"insert-name-here" to default the fsync patches. TKGlitch kernel have this as an PKGBUILD option and so on.

      Delete
    3. http://ck-hack.blogspot.com/2019/10/53-delays.html?showComment=1570567482876#c6240887462522979028

      Delete
    4. Hello all,

      Thank you so much for getting back to me so promptly, I will write this in the linux-ck forum support thread and see what can be done.

      All the best!

      Delete
  8. Has anyone else been having poor performance during heavy load? When I am compiling a large repo package, I eventually hit a point where any program refuses to open. I can move my mouse around normally but my whole system's I/O gets choked until I cancel the compilation. This behavior is present on all runqueue types, full patchset applied, and with/without compulsory IRQ threading.

    ReplyDelete
    Replies
    1. For the record, I have been noticing a downtrend in performance for the Linux kernel. I've felt it with BFS, BMQ, pf, MuQSS, and the vanilla scheduler and it's disheartening. MuQSS shined around the 4.16-4.19 versions extremely well. Would it be recommended to stick with a 4.19 LTS kernel or will I be missing important changes that were introduced to MuQSS? Thanks.

      Delete
    2. Mitigations for (mostly) Intel CPUs is the main culprit for slowdowns.
      Try disabling them, if Your environment allows that.
      Also, You can check Phoronix tests, they publish performance tests after most mitigations.

      Delete
    3. I confirm - at least, similar situations have occurred.

      Delete
    4. I do not believe the Spectre mitigations have this severe impact on I/O. If it did, it would be apparent on other schedulers.

      There is a binutils patch by HJ Lu that lessens the impact of mitigations on the kernel. You will need a patched version of binutils AND your kernel. I believe testing or tweaking should be done with the I/O schedulers available.

      Delete
    5. The most likely culprit is something that uses idle scheduling as a poorly chosen component of its design. Someone recently pointed out that mesa (for example) does it here:
      https://gitlab.freedesktop.org/mesa/mesa/blob/master/src/util/u_queue.c#L334

      Delete
    6. Yes, I noticed that too on my Ryzen 2500u

      Delete
    7. Con, should we be prefixing toolsched (http://ck.kolivas.org/apps/toolsched/toolsched-0.17/) to programs that use SCHED_IDLE?

      Delete
    8. I miss the speed and responsiveness of the BFS 3.12.57 kernel...
      5.3.13 is not too bad.
      I guess it depends on the config and boot parameters also.

      Delete
  9. Hi

    Looks like 5.3-ck1 has some race condition.
    I am getting random hard freezes under high multithreaded load (all ht-cores are at 100% for long time). Both default and rqshare=smt tested. CPU temperature is at ~70C, so no overheating.

    5.2-ck1 also has the same issue.

    Idle or not fully loaded system is ok.
    Any hints on how to debug?

    ReplyDelete
    Replies
    1. I've been running my tests under 5.1.18-ck1 for 3 hours already. And it works fine so far.

      Do you have any idea what changed in 5.2-ck1?

      Delete
    2. Not offhand since the core MuQSS code did not change. There are always mainline changes that have to be included with each major release. Presumably it is related to those either interacting somehow with MuQSS or I have not satisfactorily modified them to work properly with it.

      Delete
    3. I've noticed that my CPU cache doesn't get cleared at all with newer versions until I SIGINT whatever is using the most resources, then everything 'waiting in line' to be run pops up instantly. Is this something that might be related? How can I triage this? The issue does not replicate with a vanilla kernel or BMQ.

      Delete
    4. Sure.
      Can I help somehow to find the issue?

      Delete
    5. Try adding CONFIG_RCU_BOOST=y config option to your kernel.

      Delete
  10. Hello.
    people, please, remember, what recommended SLAB allocator, for usage with patches? SLAB,SLUB,SLOB?

    ReplyDelete