Monday 24 October 2016

linux-4.8-ck4, MuQSS CPU scheduler v0.116

Yet another bugfix release for MuQSS and the -ck patchset with one of the most substantial latency fixes yet. Everyone should upgrade if they're on a previous 4.8 patchset of mine. Sorry about the frequency of these releases but I just can't allow a known buggy release be the latest version.

4.8-ck4 patchset:
http://ck.kolivas.org/patches/4.0/4.8/4.8-ck4/

MuQSS by itself for 4.8:
4.8-sched-MuQSS_116.patch

MuQSS by itself for 4.7:
4.7-sched-MuQSS_116.patch

I'm hoping this is the release that allows me to not push any more -ck versions out till 4.9 is released since it addresses all remaining issues that I know about.

A lingering bug that has been troubling me for some time was leading to occasional massive latencies and thanks to some detective work by Serge Belyshev I was able to narrow it down to a single line fix which dramatically improves worst case latency when measured. Throughput is virtually unchanged. The flow-on effect to other areas was also apparent with sometimes unused CPU cycles and weird stalls on some workloads.

Sched_yield was reverted to the old BFS mechanism again which GPU drivers prefer but it wasn't working previously on MuQSS because of the first bug. The difference is substantial now and drivers (such as nvidia proprietary) and apps that use it a lot (such as the folding @ home client) behave much better now.

The late introduced bugs that got into ck3/muqss115 were reverted.

The results come up quite well now with interbench (my latency under load benchmark) which I have recently updated and should now give sensible values:

https://github.com/ckolivas/interbench

If you're baffled by interbench results, the most important number is %deadlines met which should be as close to 100% as possible followed by max latency which should be as low as possible for each section. In the near future I'll announce an official new release version.

Pedro in the comments section previously was using runqlat from bcc tools to test latencies as well, but after some investigation it became clear to me that the tool was buggy and did not work properly with bfs/muqss either so I've provided a slightly updated version here which should work properly:

runqlat.py

Enjoy!
お楽しみ下さい
-ck

52 comments:

  1. Hi Con,

    thanks for your fantastic job with BFS and now with MuQSS.

    I have been an anonymous user of BFS for years now and I have been doing my own testing on MuQSS.

    MuQSS have been working flawless until V0.111 in my laptop. When I tried to compile V0.115 I noticed a regression in the compile kernel. Basically the kernel with V0.115 when on full load starts to stall progressively and eventually freezes the laptop.

    My concept of full load is to compile a new kernel with 4 processes, Web browser running and running glxgears with optirun (nothing outstanding).

    The laptop is an HP with Processor: Intel(R) Core(TM) i7-4510U CPU @ 2.00GHz, 8GB ram and using p-state powersave. The laptop has a hybrid GPU (Intel+NVidia).

    I trace the problem back to a commit between V0.111 and V0.112, more exactly the commit fdd879d37e6ca088410511e9f1146c328700e92a.

    I'm writing this message from a kernel with V0.111 and all the patchs up to V0.112 with exception of the one in the commit fdd879d37e6ca088410511e9f1146c328700e92a.

    Also tested V0.116 and the problem still persists in this version.

    Any idea what could be happening?

    Cheers

    PB

    ReplyDelete
    Replies
    1. Well for what it's worth, that commit only does something when you suspend/resume your machine so the next question is do you only get it after one suspend/resume cycle?

      Delete
    2. No. The behavior happens after a fresh start of the machine. Normally I never suspend/resume the machine. Now I'm more puzzled about what could be happening. I even compiled a kernel from source without any type of patchs except the MuQSS using as base the Debian default kernel. The results are the same.

      Where do you think I should be looking at this stage? That commit, in my machine, is doing something else outside the suspend/resume functions.

      Cheers

      PB

      Delete
    3. It could be some issue with the hybrid gpu driver brought out through some bizarre coincidence on muqss. Additionally the only code in that patch that affects anything outside the suspend is reverted in this one liner I created for you:
      http://ck.kolivas.org/patches/muqss/Test/muqss116-test_task_not_wake_cpu.patch
      Anything's worth a shot.

      Delete
    4. I will give it a try and then post here the results. Unfortunately that will only happen by the end of this week.

      Cheers
      PB

      Delete
    5. I was able to do a quick test after all. Your patch did the trick. Kernel 4.8.4 compiled with MuQSS 0.116 + patch passed the first test. The others at this stage already had failed the test. :)

      Notice the following entries in the dmesg:

      [ 59.842291] BFQ WARNING:last 4611686022722404696 budget 13835058059577180154 jiffies 4295016646
      [ 59.842295] diff 4611686018427388108

      Does this entries have any particular meaning for you? Can it give a clue what was the issue?

      Now more testing only at end of the week.

      Cheers
      PB

      Delete
    6. That's great thanks for testing it! It means somehow your hardware/driver combination is setting affinity on something repeatedly and running into problems. You can run with that patch it should be safe. As for the bfq warning, sorry I know very little about bfq and you'd need to check with the author of the bfq patch here http://algo.ing.unimo.it/people/paolo/disk_sched/ . You can simply switch io schedulers on the fly if it's a problem too.

      Delete
    7. @PB,

      Please check this if it's of any help to You: https://groups.google.com/forum/m/#!topic/bfq-iosched/BEunjSrC4w8

      For me it solved the problem with warning.

      Br, Eduardo

      Delete
    8. @ck:
      Yep, BFQ is in next door. Paolo's door. I just looked to the warning and with the hurry didn't think properly... My fault.
      What is in your opinion what is the best approach to discover the possible combination of hardware/driver creating the problem?

      @Eduardo:
      Thanks for the tip. Is known if the warning has consequences? Or can be disregarded?

      Delete
    9. I would suggest applying the patch :)

      Br, Eduardo

      Delete
  2. You got it Con. No more regression with intel pstate.

    MuQSS112+intel-pstate powersave (linux 4.8.1)
    make j2 317.21
    bz2 62.11
    xz 104.78

    MuQSS116+intel-pstate powersave (linux 4.8.4)
    make j2 341.32
    bz2 60.23
    xz 105.60

    And thanks for the modified runqlat tool. I'll test it later.
    Can I use this one also on cfs for a fair comparison ?

    Pedro

    ReplyDelete
    Replies
    1. Thanks Pedro, and yes you can use it on cfs. It looks for the same function in both schedulers and does NOT try to report the latency of the idle task now (which was an amusing bug to find.)

      Delete
    2. I tried your modified version of runqlat. It runs fine on MuQSS, but I can't get it to work on CFS (the original runqlat run fine on CFS). I don't understand the error message:

      write(/sys/kernel/debug/tracing/kprobe_events, "p:kprobes/p_enqueue_task enqueue_task") failed: No such file or directory
      Traceback (most recent call last):
      File "/usr/local/bin/runqlat.py", line 164, in
      b.attach_kprobe(event="enqueue_task", fn_name="trace_enqueue")
      File "/usr/lib/python3.5/site-packages/bcc/__init__.py", line 418, in attach_kprobe
      raise Exception("Failed to attach BPF to kprobe")
      Exception: Failed to attach BPF to kprobe

      The kernel configurations are the same, with SMT_NICE disabled on MuQSS.

      Pedro

      Delete
    3. Oh right enqueue task is inlined and different on cfs.
      Just revert this line:
      b.attach_kprobe(event="enqueue_task", fn_name="trace_enqueue")

      to what it was in the original runqlat.py for it to work on cfs, and leave the rest unchanged from my edits.

      Delete
    4. Thanks. It works now.

      By the way I've put the results for MuQSS116 at the usual place:
      https://docs.google.com/spreadsheets/d/1ZfXUfcP2fBpQA6LLb-DP6xyDgPdFYZMwJdE0SQ6y3Xg/edit?usp=sharing

      I added a 'make j1' test. The difference on this one between intel pstate powersave and performance (+20%) even on CFS is quite surprising.

      Pedro

      Delete
    5. I've put new results of the runqlat tests with MuQSS 116 in both interactive and non-interactive mode.

      For the curious, I also tested throughput with different frequency governor on CFS. They make significant differences with make j1, but at high load they seem roughly equivalent. Unfortunately I have no means to see the differences in power usage.

      Pedro

      Delete
    6. Added a comparison with SMT_NICE enabled or disabled. The overhead of SMT_NICE enabled is about 1%.

      Pedro

      Delete
  3. With v116 I'm now (again) seeing "NOHZ: local_softirq_pending 10" messages under load. First saw those with early versions, but I'm sure they went away at some point (with or after v112 I think). Some kind of starvation?

    ReplyDelete
    Replies
    1. I thought that was a 4.8 "feature" and nothing to do with muqss...

      Delete
    2. Maybe so, though all I can say is that I've never seen it once with either BFS or (briefly) CFS on 4.8.x.

      Delete
    3. I was being slightly facetious on that one. It's a warning from a CPU that has been told it's okay to go idle where it discovers there are still softirqs to service. Since rescheduling is done cross-cpus lockless on muqss, it may just be a simple matter of disabling the warning but I'll investigate to make sure something isn't being missed.

      Delete
    4. Those are some of the warnings I get during CPU hangs and stutters opening wine osu!. This is my most recent one:

      10月 25 22:24:23 systemd[1]: Started CUPS Scheduler.
      10月 25 22:24:55 kernel: NOHZ: local_softirq_pending 40
      10月 25 22:24:55 kernel: NOHZ: local_softirq_pending 40
      10月 25 22:24:55 kernel: NOHZ: local_softirq_pending 40
      10月 25 22:24:55 kernel: NOHZ: local_softirq_pending 40

      Delete
    5. Thanks, I'm still investigating what causes these and if they're significant. The fact it correlates with stutters for you is a first.

      Delete
    6. Con,

      for me they tend to manifest under higher load for the CPU


      just had a kernel compilation and wanted to do some editing on PDF file (via WINE + PDF XChange Editor),

      out of the sudden the win program (PDF XChange Editor) crashed - which isn't something out of the ordinary,

      but then everything got VERY sluggish.

      The mouse pointer hardly moved for lots of seconds then moved a tiny bit (almost slurred), then nothing and several seconds later it moved again.

      Compiz was running with sched_yield NONE, aking a look at the "cursor" of gnome-terminal

      it blinked for 2 seconds (2 blinks), then several seconds nothing, then again 2 blinks and long time nothing ...


      Afaik the state of MuQSS was at a7c0de9472e18df1ab604e24b82b5fc6e6b97d43 (0.116)

      Kernel compilation load didn't seem to be accounted in this state (switching to vt2 after several minutes) and checking via htop [load was 4-4.5],

      there also was some delay but terminal input more or less worked


      after 12+ minutes of kernel compilation and still no finishing (usually takes around 8 minutes)

      The only thing I could get out of dmesg was the following (not really sure if it helps):

      [ 1100.731633] NOHZ: local_softirq_pending 02
      [ 1168.311173] NOHZ: local_softirq_pending 202
      [ 1168.311235] NOHZ: local_softirq_pending 202
      [ 1168.312015] NOHZ: local_softirq_pending 202
      [ 1168.312029] NOHZ: local_softirq_pending 202
      [ 1168.312208] NOHZ: local_softirq_pending 202
      [ 1168.312412] NOHZ: local_softirq_pending 202
      [ 1168.455853] NOHZ: local_softirq_pending 202
      [ 1168.456871] NOHZ: local_softirq_pending 202
      [ 1168.456957] NOHZ: local_softirq_pending 202
      [ 1908.524277] BUG: workqueue lockup - pool cpus=0-7 flags=0x4 nice=0 stuck for 36s!
      [ 1908.524282] Showing busy workqueues and worker pools:
      [ 1908.524286] workqueue events_freezable_power_: flags=0x84
      [ 1908.524287] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
      [ 1908.524290] pending: disk_events_workfn
      [ 1908.524297] workqueue writeback: flags=0x4e
      [ 1908.524298] pwq 16: cpus=0-7 flags=0x4 nice=0 active=1/256
      [ 1908.524299] pending: wb_workfn
      [ 1908.524322] workqueue kcryptd: flags=0x2a
      [ 1908.524322] pwq 16: cpus=0-7 flags=0x4 nice=0 active=4/8
      [ 1908.524324] pending: kcryptd_crypt, kcryptd_crypt, kcryptd_crypt, kcryptd_crypt
      [ 1908.524331] workqueue edac-poller: flags=0x6000a
      [ 1908.524332] pwq 16: cpus=0-7 flags=0x4 nice=0 active=1/1
      [ 1908.524333] pending: edac_mc_workq_function [edac_core]

      Delete
    7. As Manuel suggested in the comments below, this may be fixed in git with one of the extra patches post 116. There are some minor bugfixes in there; one or more of them may be related.

      Delete
    8. Okay, running it right now,

      thanks :)

      Delete
    9. I cannot comment on those dmesg lines, as I don't get them. (Possible reasons: Still stuck with 4.7.10, using HZ_1000 all the time and no shiny modern governors available for my old 2core cpu.)

      But using the mentioned 4 commits makes a positive difference on here anyways.

      BR, Manuel Krause

      Delete
  4. When opening osu! through wine, ck3 patch would freeze the whole system, ck4 would make the whole system stutter close to a hang.

    This is the journal from the moment of the crash. Not too sure if it logged it correctly:
    10月 25 13:30:27 pulseaudio[474]: E: [pulseaudio] bluez5-util.c: GetManagedObjects() failed: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: t
    10月 25 13:30:27 dbus[350]: [system] Failed to activate service 'org.bluez': timed out
    10月 25 13:30:32 dbus[350]: [system] Activating via systemd: service name='org.bluez' unit='dbus-org.bluez.service'
    10月 25 13:30:44 dbus-daemon[453]: Activating service name='org.gnome.GConf'
    10月 25 13:30:44 dbus-daemon[453]: Successfully activated service 'org.gnome.GConf'
    10月 25 13:30:57 dbus[350]: [system] Failed to activate service 'org.bluez': timed out
    10月 25 13:31:23 dbus[350]: [system] Activating via systemd: service name='org.freedesktop.UDisks2' unit='udisks2.service'
    10月 25 13:31:23 systemd[1]: Starting Disk Manager...
    10月 25 13:31:23 udisksd[2015]: udisks daemon version 2.1.7 starting
    10月 25 13:31:24 dbus[350]: [system] Successfully activated service 'org.freedesktop.UDisks2'
    10月 25 13:31:24 systemd[1]: Started Disk Manager.
    10月 25 13:31:24 udisksd[2015]: Acquired the name org.freedesktop.UDisks2 on the system message bus
    10月 25 13:31:41 wpa_supplicant[392]: wlp6s0: CTRL-EVENT-SCAN-FAILED ret=-22
    10月 25 13:32:13 systemd[1]: Starting CUPS Scheduler...
    10月 25 13:32:14 dbus[350]: [system] Activating via systemd: service name='org.freedesktop.ColorManager' unit='colord.service'
    10月 25 13:32:14 systemd[1]: Starting Manage, Install and Generate Color Profiles...
    10月 25 13:32:14 dbus[350]: [system] Successfully activated service 'org.freedesktop.ColorManager'
    10月 25 13:32:14 systemd[1]: Started Manage, Install and Generate Color Profiles.
    10月 25 13:32:14 colord[2239]: failed to get session [pid 2238]: No such device or address
    10月 25 13:32:15 systemd[1]: Started CUPS Scheduler.
    10月 25 13:35:44 wpa_supplicant[392]: wlp6s0: CTRL-EVENT-SCAN-FAILED ret=-22
    10月 25 13:37:40 systemd-logind[347]: Power key pressed.
    10月 25 13:37:40 systemd-logind[347]: Powering Off...
    1

    ReplyDelete
  5. I have noticed that the game ark survival evolved runs bad with ck kernel and amd Cool n Quiet enabled. Im getting very short fps dropdowns to 10fps or 15fps every 1 or 2 seconds. with stock kernel I get at worse 20fps but only when moving fast around otherwise it's around 30fps or more. disabling amd cool and quiet is also working. I have noticed that in ark the cpu usage is spiking a lot maybe sth isn't fast enough to clock the cpu higher? the cpu clocks ingame around 1400mhz and 4200mhz. I have a amd fx-8350 and a nvidia gtx1070 (just got it and was a bit sad about those stutters)

    ReplyDelete
    Replies
    1. Not sure how cool-n-quiet works and how it differs from regular cpufrequency governors but it probably needs different information to work properly. I will investigate. Thanks.

      Delete
    2. @Anonymous,

      I might have very similar problem with VRQ (Alfred's kernel https://cchalpha.blogspot.fi/), but in Diablo 3.
      When I start game (via wine, of course) using VRQ kernel, whole game stutters very bad, FPS is around 15 (normally about 80-90). When it stutters, everything freezes (even mouse) for a second or so, then resumes and so on.
      Maybe it's smth specific which mux started doing as well. At least this really interests me and I think may interest Alfred as well.

      What we have in common is HW manufacturer, I have Phenom II 975 BE.

      I have to note that none of previous kernels I used (ubuntu, mainline, BFS and mux < 115) except VRQ, had this problem, but now I'll go and test latest mux as well. Yesterday LOL started stuttering badly as well (after I installed mux 116), but I thought this may be some other issues, like mesa from git.
      I'll keep an eye on system when this happens again.

      Br, Eduardo

      Delete
    3. Are you running amd cool-n-quiet as well then? If so, try disabling it to see if the problem goes away and is there any way to use a regular cpu frequency governor, like ondemand, instead?

      Delete
    4. @Ck,

      Forgot to add that I play games using "performance" governor, eg. max freq all the time. I have enabled Cool'n Quiet since ages and no other kernels show the issue, except VRQ (will test latest mux soon).
      Latest tests show that "ondemand" behaviour, using VRQ, does not show the problem. But I'll test again w/o C&Q + VRQ/MUX and post results.

      Br, Eduardo

      Delete
    5. Please do not report issues with VRQ here. It has nothing to do with my development and only confuses the discussion.

      Delete
    6. I just tried the latest commits and no change.I tried ark again and using stock kernel gives me 5 - 10 more fps maybe thats why it doesn't stutter but why is it faster with stock?

      Delete
    7. @Ck,

      Sorry for noise about VRQ, it just seemed very similar.
      No issues with mux 116 + diablo 3 + cool n quiet.

      Br, Eduardo

      Delete
    8. @Anonymous, Ck,

      I think I have an answer on this.
      Cool n quiet for AMD means dynamic frequency/voltage/etc. scaling. I just tried disabling it and frequency stays at max speed at all times, which means it's about the same as running with "performance" governor.
      In addition, when I run my gaming tests, mainline kernel is always better with "ondemand" governor by a large margin, BFS/MUX using "ondemand" are comparably slow.
      Check https://docs.google.com/spreadsheets/d/1EayezAsGlJdXjZbS3b9m7YtvtRF-DJ3xrT3hYCvfymQ/edit?usp=sharing , page "Perf. (DRI2), OND GOV", You'll see BFS/MUX reaches about 30 FPS while standard Ubuntu kernel reaches 42 FPS.
      This explains gains and losses with Anonymous tests.

      I can't explain this (and won't speculate), but Con can for sure :)

      Br, Eduardo

      Delete
    9. @Ck,

      im not such a linux expert but as far as I understand Cool n Quiet is a bios function like intel eist. It underclocks the cpu when not in use. So im already using ondemand. I will try tommorow some other games because ark is a bad game to test I think I saw sth similar on Saints Row 4. and yes I tried to disable Cool n Quiet and it worked but maybe it's just the extra fps boost that did the trick.

      Delete
    10. @Ck,

      switched back to ck kernel without the 4 patches and again back with the patches. And even though I said they don't change sth they make a little difference especially when I have higher fps in ark it works good and the stutters don't appear. So the only problem I have now is the relativ huge performance loss with ondemand. I will also try some unigine benchmarks if disabling cool n quiet, switching kernel or using performance governour makes any difference.

      Delete
    11. @Ck,

      my results you have to download them to see:
      https://www.dropbox.com/sh/dybposdl9t52u7o/AACT64VoaLQUoV3Yh6iAsS9ea?dl=0

      you can see that ck kernel with ondemand gives me around 200 less points in unigine-heaven compared to archlinux stock kernel but using performance governour works the same as using no cool and quiet and gives both kernels equal performance again.

      Delete
  6. You might try applying all the four patches which have been commited after v0.116 was tagged (https://github.com/ckolivas/linux/commits/4.8-muqss).

    ReplyDelete
    Replies
    1. Yes, I'd like to second that proposal for those people experiencing lags with plain v0.116:
      On my system these atm. 4 added commits cure a lazy mouse pointer after periods of being unmoved and graphical window content update delays at switching to top within KDE (also in the presence of video playback, what formerly stuttered at that very moment).
      Using these addon commits is IMO worth a try.

      BR, Manuel Krause

      Delete
    2. @CK:
      Also after making heavy use of /dev/shm backed by swapping-to-second-disk, in the presence of flash-streamed TV in firefox, the recovery of video and other KDE windows is almost immediate.
      This is a really remarkable advantage, as we may all know about the swap bottleneck.

      BR, Manuel Krause

      Delete
  7. @CK:
    I commented success too early.
    Got the following after third resume attempt:
    http://pastebin.com/dReCcQGm

    Maybe you'd understand it.
    BR, Manuel Krause

    ReplyDelete
  8. Would just like to report that latest 4 patches (after 0.116 -ck4) fixed problems with games for me, mainly Dota 2.
    (Jitter, mini-freezes every 10 seconds and FPS fluctuations)
    I am using latest legacy Nvidia drivers 340.98.
    So thank you.

    ReplyDelete
  9. @ck:
    Out of curiosity I've also added the two newest commits upon v0.116 (all six) and enabled the new possible settings:
    CONFIG_TICK_CPU_ACCOUNTING=y
    CONFIG_RCU_NOCB_CPU=y
    CONFIG_RCU_NOCB_CPU_ALL=y
    (Being in doubt, whether the latter two appear useful.)

    At resume I get the following warning, you may want to have a look at: http://pastebin.com/FPAbxAv0

    BR, Manuel Krause

    ReplyDelete
    Replies
    1. Thanks. Those 2 extra features are of no use. They just weren't supported by muqss until recently. Fortunately that warning is harmless on resume but I should silence it.

      Delete
    2. This warning here remained harmless, means: no negative effect. You mean, the RCU_NOCB* ones are useless?
      BR, Manuel Krause

      Delete
    3. Yes. The main reason for those changes was to improve latency on low Hz builds. Latest git muqss has low latency on 100Hz which I'm going to start recommending for the next version.

      Delete
    4. https://www.kernel.org/pub/linux/kernel/people/andrea/misc/31-44-100-1000/31-44-100-1000.html

      Didn't realize there was so much of a performance hit,

      so you say with the new changes it should be fine even for e.g. FPS games that require low latency and low-latency audio ?

      Good news :)

      Delete
  10. New muqss and -ck version coming out soon so if you were about to build a new kernel for 4.8.5, hold out a few hours :)

    ReplyDelete