Tuesday, 13 September 2016

BFS 497, linux-4.7-ck4

For the first time in a very long time, I'm announcing yet another -ck release up to ck4 along with yet more substantial updates for BFS for linux-4.7 based kernels.

BFS by itself:
4.7-sched-bfs-497.patch

-ck branded linux-4.7-ck4 patches:
linux-4.7-ck4

Thanks(?) to the massive changes to the mainline kernel I'd been forced to rewrite significant components of BFS to work properly with them, specifically the cpu frequency governors. At the same time I've had quite a bit of energy and enthusiasm for working on BFS in a way I haven't had in a long time. As a result, this updated version not only addresses the remaining cgroup stub patch bug (mentioned on the previous announcement) but implements further improvements and clean ups to go with those improvements.

Alas I still have no explanation for the random lockups some people are seeing, but I have seen reports of it happening on mainline kernels as well now, so while I'm always suspicious of my own code, there is also the chance that BFS exacerbates an issue in mainline. Something that appears common is onboard Intel graphics with the Haswell chipset.

Additionally I had reports of people being unable to suspend with BFS from 4.7 but I haven't heard back from them on later versions.

The short summary of improvements in this version are less overhead, higher throughput and less latencies.

I've rewritten the skiplist implementation to not require a malloc/free on insertion/removal of a new node which seemed to noticeably improve throughput at high loads.
Now that CPU frequency governors know what the scheduler is doing, the approach of BFS of old of knowing what the governor was doing and working around it is no longer helpful and I've removed the whole sticky task and offset for throttled CPUs and throughput has actually improved instead.
I've also added some micro-optimisations and cleanups.
I've added a minor change for offlining CPUs to prevent tasks trying to schedule to them.

The set of patches in ck4 is the largest in the ck patchset since the early 2.6 patchset days. I've also included the patch from Alfred (thanks!) to fix the warning that happens with suspend which is mostly harmless.

Each patch included has a mini changelog at the top.

I'm also keen to get feedback from people on if they see any noticeable interactive/responsiveness regressions by disabling the interactive flag as follows:

echo 0 > /proc/sys/kernel/interactive

Enjoy!
お楽しみ下さい
-ck

38 comments:

  1. Thanks Con. 4 bfs releases in 2 weeks, well done !
    I feel the need to run throughpout tests again against bfs 497.
    The performance is the same as with bfs 490, and almost on par with bfs 467 (linux 4.4).
    https://docs.google.com/spreadsheets/d/1ZfXUfcP2fBpQA6LLb-DP6xyDgPdFYZMwJdE0SQ6y3Xg/edit?usp=sharing

    I'll run the rq latencies test latter.
    I think this will close the chapter of bfs on linux 4.7.

    Pedro

    ReplyDelete
    Replies
    1. I have tested gaming performance, sorta, using Unigine Valley, system config & results are there: https://docs.google.com/spreadsheets/d/1EayezAsGlJdXjZbS3b9m7YtvtRF-DJ3xrT3hYCvfymQ/edit?usp=sharing

      If that's of any use :)

      br, Eduardo

      Delete
    2. I've put the rq latencies results.
      cfs has been tested with SCHED_AUTOGROUP disabled. Both kernels were using acpi-cpufreq+performance governor.

      Pedro

      Delete
    3. Min FPS are notably higher with bfs490 and bfs497. So we can expect that games will be smoother with new bfs?

      Thank you all

      Delete
  2. Thanks for those. I don't anticipate another release now for 4.7 unless I find the reason for the hangs and can fix it.

    ReplyDelete
    Replies
    1. @ck
      It seems that the code changes are fininally settle down. I'm also working on an embedded version of skip list which similar to the skip list changes in 0497, and some other improvement about skip list. I'll release the code after cleaning up the commits in a day or two.

      Delete
    2. I have finished the code of my embedded version and new implementation of skip list for BFS. In short, there is improvement comparing to the baseline version. If you are instersting, please visit http://cchalpha.blogspot.com/2016/09/about-skip-list-in-bfs.html and http://cchalpha.blogspot.com/2016/09/new-implementaion-of-skip-list-for-bfs.html for detail.

      Delete
  3. @Con,

    I'm using bfs 497 for half a day and first lockup is here. It does not seem to be an issue directly with bfs, but most likely it triggers this error which is some sort of graphic memory management problem (that's as far I know).
    On previous BFS versions as well as Alfred's VRQ versions or Ubuntu standard kernels there are no issues like this. There were no system updates in between using 490 and 497 either.

    NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [compiz:1961]
    Call Trace:
    [] _raw_write_lock+0x2e/0x40
    [] drm_vma_node_revoke+0x19/0x80 [drm]
    [] drm_gem_object_release_handle+0x35/0xa0 [drm]
    [] drm_gem_handle_delete+0x62/0x90 [drm]
    [] drm_gem_close_ioctl+0x20/0x30 [drm]
    [] drm_ioctl+0x152/0x540 [drm]
    [] ? drm_gem_handle_create+0x40/0x40 [drm]
    [] ? dentry_free+0x4e/0x90
    [] ? __dentry_kill+0x181/0x220
    [] ? mntput_no_expire+0x2c/0x1b0
    [] do_vfs_ioctl+0xa1/0x5c0
    [] ? ____fput+0xe/0x10
    [] ? __fget+0x77/0xb0
    [] SyS_ioctl+0x79/0x90
    [] entry_SYSCALL_64_fastpath+0x1e/0xa8

    This is on Intel HD4000 (IvyBridge). I'll try experiment with i915 module options to mitigate the issue.
    I doubt intel guys will consider to check bug report as this is not on CFS :(

    br, Eduardo

    ReplyDelete
  4. I think that I heard problems with rsync and for other person with nvidia too and haswell, I only know exactly my problem, with stock kernel and bfs 472 nothing but the updates maky my pc unusable due to the freezes, gaming browsing( I use hardware accel in firefox nightly and chromium with vaapi support) it may exacerbate the hangs but the problem is here in some way..( I couldn't try yet the new version but some people in aur page has posted freezes problems and more in the arch forum, tell us for testing or something

    ReplyDelete
    Replies
    1. I am in the 4.7.3-5 ck kernel haswell without the patch since aproximately half and hour, any freeze until now, before the freeze with gaming occurs in minutes with browsing usually in half and hour but for now it appears stable.. for me. i'll post the results

      Delete
    2. the version without the patch is working perfectly in my machine, thanks for your work con, I couldn't test the patch because it' isn't in the repo, but for me it isn't any need of use because the bfs 497 is perfect stable and reliable included temps(but it is colder in the city that days too xD)

      Delete
    3. I am the one from Arch forum who had problems using rsync to backup ~22G while running linux-ck-piledriver 4.7.3. I am now running rsync tests with Graysky's latest release, 4.7.3-5, from linux-ck repo, and so far after hours I haven't encountered freezes.

      Good job!

      Delete
  5. With /proc/sys/kernel/interactive set to 0 the distribution of cpu time is broken here:

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    23510 xxxxxx 21 0 156 16 8 R 99.0 0.0 0:11.10 burnP5
    23513 xxxxxx 20 0 156 16 8 R 33.6 0.0 0:03.75 burnP5
    23511 xxxxxx 21 0 156 12 8 R 33.2 0.0 0:03.74 burnP5
    23512 xxxxxx 21 0 156 16 8 R 33.2 0.0 0:03.73 burnP5

    ReplyDelete
    Replies
    1. Yeah it looks it doesn't it. That's the cost of sacrificing perfect fairness and interactivity for a little bit more throughput.

      Delete
  6. While using this, I had an issue where Discord (a VOIP client) just started lagging out on my Intel Core m5 with it getting insufficient CPU, which didn't happen on the stock scheduler or on previous versions of BFS.

    ReplyDelete
    Replies
    1. Interesting and no doubt related to the cpufreq changes. Which governor are you using? pstates? I assume you have hyperthread enabled on your machine as well? If you know how to, try briefly setting the pstate governor to performance instead of powersave. Additionally try disabling hyperthreading in your bios temporarily. Neither of these is a solution, but to help me diagnose what the issue might be.

      Delete
    2. Using p-states. Unfortunately, this is the Core m5 suffering the issues so:
      A) Setting it to performance mostly just cripples the machine to the point Discord doesn't work anyway about as well.
      B) The BIOS is crap enough I can't disable hyperthreading. (It also crashes regularly on boot due to ACPI issues due to a rampantly out of spec ACPI table. I hate this BIOS so much.)

      Delete
    3. I can test it with performance sometime tomorrow - I'll give it a shot, this thing just relies heavily on powersaving to not thermal up to the point it has to lower clockspeed on a regular basis.

      Delete
    4. Gah. Can you try a patch then instead?

      Delete
    5. Sure. The other problem with testing the Discord thing upon reflection is that Discord worked great for about 40 minutes (sorry, I was in a hurry putting down the initial post so I didn't think to mention that) and then just completely broke due to the CPU starving it until I restarted the system into the regular setup.

      Delete
    6. Then are you sure you're actually blaming the right thing? Could be unrelated screwage with Discord. You could also try disabling interactive mode in the interim which won't require a new kernel.

      After that, add the two patches in the testing directory. They add a tunable that will allow the cpufreq load estimator to test a variety of different mechanisms.
      You can set values from 0-5 in:
      /proc/sys/kernel/smt_load
      3 is how ck4 is set. Try 0 and 5.

      Delete
    7. Yeah, fair enough. I'll try the interactive mode thing the next time I've got a good long shot at using Discord. Sorry about the replies here, I just kind of jumped to blaming this given how the earlier 4.7 revision had turned out for me.

      Delete
  7. Works great for me, feels faster. Gentoo-4.7.4-ck, Haswell, nvidia, intel graphics onboard. Thanks for the great work!

    ReplyDelete
  8. I hit that kind of bug:

    [ 202.694145] INFO: rcu_preempt detected stalls on CPUs/tasks:
    [ 202.694150] Tasks blocked on level-0 rcu_node (CPUs 0-3): P4030
    [ 202.694151] (detected by 3, t=60002 jiffies, g=25053, c=25052, q=1020526)
    [ 202.694154] baloo_file_extr R running task 0 4030 3950 0x00000000
    [ 202.694156] ffff880077433e00 ffff8800414e1e70 ffff880197baa640 0000000200016900
    [ 202.694158] ffff8800414e1980 ffff8800414e3fc0 ffff880077434000 ffff88019ed16940
    [ 202.694160] 0000000000000000 ffff8800414e1980 0000000000000000 ffff880077433e18
    [ 202.694161] Call Trace:
    [ 202.694168] [] preempt_schedule_common+0x1f/0x40
    [ 202.694170] [] preempt_schedule+0x26/0x30
    [ 202.694172] [] ___preempt_schedule+0x16/0x18
    [ 202.694174] [] _raw_spin_unlock_irqrestore+0x26/0x30
    [ 202.694176] [] __sched_setscheduler.constprop.43+0x2c2/0x520
    [ 202.694179] [] ? __do_page_fault+0x1f5/0x510
    [ 202.694180] [] do_sched_setscheduler+0x80/0xc0
    [ 202.694182] [] sys_sched_setscheduler+0x12/0x20
    [ 202.694183] [] entry_SYSCALL_64_fastpath+0x1a/0xa4
    [ 202.694184] baloo_file_extr R running task 0 4030 3950 0x00000000
    [ 202.694186] ffff880077433e00 ffff8800414e1e70 ffff880197baa640 0000000200016900
    [ 202.694187] ffff8800414e1980 ffff8800414e3fc0 ffff880077434000 ffff88019ed16940
    [ 202.694188] 0000000000000000 ffff8800414e1980 0000000000000000 ffff880077433e18
    [ 202.694190] Call Trace:
    [ 202.694192] [] preempt_schedule_common+0x1f/0x40
    [ 202.694193] [] preempt_schedule+0x26/0x30
    [ 202.694194] [] ___preempt_schedule+0x16/0x18
    [ 202.694196] [] _raw_spin_unlock_irqrestore+0x26/0x30
    [ 202.694197] [] __sched_setscheduler.constprop.43+0x2c2/0x520
    [ 202.694198] [] ? __do_page_fault+0x1f5/0x510
    [ 202.694200] [] do_sched_setscheduler+0x80/0xc0
    [ 202.694201] [] sys_sched_setscheduler+0x12/0x20
    [ 202.694203] [] entry_SYSCALL_64_fastpath+0x1a/0xa4

    ReplyDelete
    Replies
    1. That's probably related to the bug that this patch fixes: bfs497-revert-othercpufreq.patch

      Delete
    2. Ok, but

      $ dmesg | grep -i "BFS CPU"
      [ 0.736436] BFS CPU scheduler v0.497 by Con Kolivas.

      and

      $ uname -r
      4.7.4-ck4

      Which means I already had this path applied.

      Delete
    3. No you have NOT applied the patch I have directed you to.

      Delete
    4. This comment has been removed by the author.

      Delete
    5. I will, given it fixes the remaining freezes, but I was waiting to see if there were any other changes that needed to go in.

      Delete
    6. Con, kernel build fail with bfs497 and all the patches in pending applied when SMT_NICE=disabled (SMT_NICE=y builds fine).
      I managed to build successfully with the following modification. However I'm not sure it is correct.

      Pedro

      @@ -745,6 +745,8 @@
      /* Sorry, you lose */
      return false;
      }
      +#else
      +#define smt_schedule(p, this_rq) (true)
      #endif /* CONFIG_SMT_NICE */
      #ifdef CONFIG_SMP
      /*

      Delete
    7. Thanks Pedro. Added to the pending/ director and will go in the next release.

      Delete
  9. Hi, I have tried BFS with 4.7.4 kernel and it works good, but there is some "freezing" lag when cpu is on load and I try to change song in player. Can that be solved somehow ? Thanks.

    ReplyDelete
  10. On Haswell E here. I'm running with the performance governor turned on. Though not sure if it's getting initialized properly. In the boot logs I receive "ENERGY_PERF_BIAS: Set to 'normal', was 'performance'." Though, cpupower still shows the governor as performance.

    The only thing I noticed with the freeze is it happens if I have a long running process taking up all of the cores, and startup another process that also requires heavy cpu usage.

    This is how it always happens for me. CLion is loading a particularly large cmake file. (Unreal just uses one giant cmake file) Then in Virtualbox I start up a Windows 10 VM. After a few seconds I freeze. It also happens if Discord is running during heavy cpu use.

    ReplyDelete
    Replies
    1. There is an extra patch in the pending/ directory that should fix this.

      Delete
  11. Con, broken ARM build reported.

    https://kojipkgs.fedoraproject.org//work/tasks/6478/15706478/build.log

    Could you please take a look at it?

    ===
    kernel/sched/bfs.c: In function 'context_switch':
    kernel/sched/bfs.c:2079:3: error: implicit declaration of function 'switch_mm_irqs_off' [-Werror=implicit-function-declaration]
    switch_mm_irqs_off(oldmm, mm, next);
    ^~~~~~~~~~~~~~~~~~
    ===

    ReplyDelete
    Replies
    1. Did you check the pending/ directory? The fix in there should do the trick.

      Delete