Friday, 7 October 2016

MuQSS - The Multiple Queue Skiplist Scheduler v0.108

A new version of the MuQSS CPU scheduler

Incrementals and full patches available for 4.8 and 4.7 respectively here:
http://ck.kolivas.org/patches/muqss/4.0/4.8/


http://ck.kolivas.org/patches/muqss/4.0/4.7/

Yet more minor bugfixes and some important performance enhancements.

This version brings to the table the same locking scheme for trying to wake tasks up as mainline which is advantageous on process busy workloads and many CPUs. This is important because the main reason for moving to multiple runqueues was to minimise lock contention for the global runqueue lock that is in BFS (as mentioned here numerous times before) and this wake up scheme helps make the most of the multiple discrete runqueue locks.

Note this change is much more significant than the last releases so new instability is a possibility. Please report any problems or stacktraces!

There was a workload when I started out that I used lockstat to debug to get an idea of how much lock contention was going on and how long it lasted. Originally with the first incarnations of MuQSS on a 14 second benchmark with thousands of tasks on a 12x CPU it obtained 3 million locks and had almost 300k contentions with the longest contention lasting 80us. Now the same workload grabs the lock just 5k times with only 18 contentions in total and the longest lasted 1us.

This clearly demonstrates that the target endpoint for avoiding lock contention has been achieved. It does not translate into performance improvements on ordinary hardware today because you need ridiculous workloads on many CPUs to even begin deriving advantage from it. However as even our phones now have reached 8 logical CPUs, it will only be a matter of time before 16 threads appears on commodity hardware - a complaint that was directed at BFS when it came out 7 years ago but they still haven't appeared just yet. BFS was shown to be scalable for all workloads up to 16 CPUs, and beyond for certain workloads, but suffered dramatically for others. MuQSS now makes it possible for what was BFS to be useful much further into the future.

Again - MuQSS is aimed primarily at desktop/laptop/mobile device users for the best possible interactivity and responsiveness, and is still very simple in its approach to balancing workloads to CPUs so there are likely to be throughput workloads on mainline that outperform it, though there are almost certainly workloads where the opposite is true.

I've now addressed all planned changes to MuQSS and plan to hopefully only look at bug reports instead of further development from here on for a little while. In my eyes it is now stable enough to replace BFS in the next -ck release barring some unexpected showstopper bug appearing.

EDIT: If you blinked you missed the 107 announcement which was shortly superseded by 108.

EDIT2: Always watch the pending directory for updated pending patches to add.
http://ck.kolivas.org/patches/muqss/4.0/4.8/Pending/

Enjoy!
お楽しみ下さい
-ck

72 comments:

  1. Gah! I just finished building 107... :-P An update, after our previous discussions, I looked at some load testing, and noticed that during a kernel build, CPU1 was always at 0%. So I switched back to vanilla and built 107; for x64 to see if it fixed the CPU issue. It appears that it did, as its building now under MuQsS 107, with CPU{0,1} both at 100%, and Firefox couldn't care less. :) Will report back tomorrow, and hopefully get to my i686 cross-compile too.
    -jwh

    ReplyDelete
    Replies
    1. I meant, "...building now under MuQSS 108"
      -jwh

      Delete
  2. With v0.108 on kernel 4.7.6 I get the following warning during bootup:
    http://pastebin.com/h89PDcKv

    BR, Manuel Krause

    ReplyDelete
    Replies
    1. Darn that xfs problem keeps coming back. It's definitely a lingering bug and I need to figure out why, even if it runs fine afterwards as it's a sign that potentially something can go wrong. I know what the bug means (the worker is running on the wrong CPU temporarily) but I thought I'd fixed every way that might happen.

      Delete
    2. I've uploaded a patch which should hopefully help with the xfs warning. muqss108-001-check_affinity_switch.patch

      Delete
    3. @Con:
      I thought this above posted warning's trace has nothing to do with xfs. I just figured out, that it's in my .config as module. Immediately thrown CONFIG_XFS_RT to n.
      I don't make any use of xfs on my machine, btw.

      Let's see what your fixes change.
      Thanks and BR, Manuel Krause

      Delete
    4. It doesn't matter if you use it or not, it's the presence of the module which creates the warning. If you remove xfs from your config then you won't be able to test it again.

      Delete
    5. @Con:
      I'm lost in compilation.
      Newly re-compiled kernel (without any changes I intended above, and still without your patch) doesn't show the issue now at this bootup.

      Don't know... Applying your patch anyways?

      BR, Manuel Krause

      Delete
    6. It's likely to be intermittent only. Don't stress over it. Just apply the patch and see if you ever get the warnings again.

      Delete
    7. Thx for the recreational reply, project's task is in the cpu's working circuits now.
      BR, Manuel Krause

      Delete
  3. Yet another round of tests:
    https://docs.google.com/spreadsheets/d/1ZfXUfcP2fBpQA6LLb-DP6xyDgPdFYZMwJdE0SQ6y3Xg/edit?usp=sharing

    This time I used intel_pstate+performance instead of acpi-cpufreq+performance. It enables longer uptime since I don't need to reboot to switch frequency driver, and on cfs both drivers+performance gives approximately the same results (see '4.7 bfs502').

    Throughput is good on my 2c/4t cpu. However I'd like to see some results with many cores to see where muqss really shines against bfs.
    I don't report any errors after 6 hours of uptime.
    Well done Con ! Maybe now the flow of patches will dry up :)

    On a side note, I noticed you put a 'bfs512-fixes2_1.patch' for bfs512. Does this patch include all the fixes you gathered with muqss ? Because 'bfs512 + bfs512-fixes.patch' was giving me errors with xfs.

    Pedro

    ReplyDelete
    Replies
    1. Thanks for those results, as always, Pedro. Indeed it appears that the interactive=0 mode is quite redundant and I'm thinking of getting rid of it entirely unless I make some dramatic changes to it, which basically I'd prefer not to bother. The bfs fixes2 patch still won't fix the xfs problem (which affects both bfs and muqss it seems.)

      Delete
  4. Hey Manuel; fwiw, I do not get that workqueue.c warning. Also, I compared the v107 and v108 dmesg outputs (in case the warning showed up in the latter), and they are essentially identical. I'm on an old Athlon64 X2 system though; you're on Intel I recall? I also don't have TOI, and I didn't compile in BFQ; I'm using CFQ atm (rotational drive). All working well so far Con!
    -jwh

    ReplyDelete
    Replies
    1. @-jwh, @CK:
      Yes it's an intel dual core without HT capability. But it's still at kernel 4.7.6 and my .config is unchanged since MuQSS first tests beginning. In the last step I've gone from 106 to 108.
      But this is absolutely NO hot issue, I don't suffer from any disadvantage vs. 106 after getting this warning.
      BR, Manuel Krause

      Delete
    2. Ah ok, I'm on 4.8.0...fyi, kernel 4.8.1 came out today. "I can never win..." :-P
      -jwh

      Delete
    3. Somehow "they" have trashed the 4.7.7 release, today. Commits went off the tree, but no patch available.
      My TOI patch base is outdated so much... but the modded version kept working til 4.7.6, o.k. only without encryption. I adjust it from kernel to kernel version.
      Annoying work, for 4.8 I'm not gone through all obstacles so far.
      "We can win, if we want..." 8-D
      BR, Manuel Krause

      Delete
  5. Thank you Con :)

    will give that a try on 4.8 ...

    Using MuQSS 106 with patches 1-6 + 8

    and there seemed to be a regression of some sort there:

    usual kernel compilation with genkernel took 2.5-3x times as usual (22 minutes vs. 7.5 - 8 minutes)

    stage4 system backup (essentially tar -cp / with an exclusionary file list routed through lzma2 (7z)) took twice as long:

    40 minutes vs. around 20 minutes,

    there's one additional kernel folder which could have raised it to around 23 to 25 minutes but doubling of time doesn't explain it.

    Also no process was running.

    ZFS was set to use some more memory but it was hardly used so memory contention, fragmentation, etc. also can't explain it.


    Meanwhile compilation of the ZFS kernel modules and proprietary nvidia-driver took around the same time like with MuQSS 103 or BFS 497

    Will keep you posted if I encounter anything unusual after upgrading to MuQSS 108

    ReplyDelete
    Replies
    1. Actually there's something else that can explain that:

      I've modified ZFS' behavior heavily to cut down latency but at the same time throughput kind of suffers in part (for certain workloads)

      So it could also be that

      Delete
    2. No there was a regression in 106 that is now fixed that is also contributing, thanks KofT

      Delete
    3. Oh wait I didn't see that you applied the pending patches. In which case that would be identical to 108 so cancel that.

      Delete
    4. Latency so far appears quite superb (really, really low):

      running two audio players at the same time (VLC + audacious) against pulseaudio 9.0,

      while editing a PDF document in WINE via PDF-XChange Editor and occasional browsing in the web (looking up e.g. words).

      Desktop is run on nvidia proprietary drivers with
      'threadirqs' appended to kernel and priority raised of relevant IRQs, WINE, audacious, vlc and compiz and X to cut down latency

      and compiling firefox or webkit-gtk

      All that leads to NO to absolute minimal disturbances in playback (actually only twice so far and one equaled to a less than a second, 1/3 or 1/4 of a second "blip" during playback of audacious; the second now was a delayed sound segment [1-2 seconds] being interspersed while regular playback continued in audacious - these seemed to happen way more often in the past).

      Might need to raise audio quality in pulseaudio to see how MuQSS can keep up with it

      webkit-gtk and chromium are the worst offenders since with CFS and some BFS versions usually latency and smoothness especially of PDF Annotating in WINE, audio playback and mouse movement really suffers during those extreme cases

      Compilations are reniced but usually under CFS it doesn't really help or seem to have an effect.

      No regression in compilation time for firefox or sqlite, compilations are done in RAM (zram) on a newly created Btrfs after every boot - so that mostly excludes i/o limitations


      Note that several security enhancing features for 4.8 of RAM randomization got enabled which might also raise latency and increase load somewhat - but so far all looks well,

      Once enough experience was collected I'll do another stage4 build (which I usually don't do that often but let's see how it compares)

      Thanks !

      Delete
    5. Thanks KoT, sounds good! It appears you missed patch 009 which also helps performance, unless you've since updated further. Also it wasn't that much work to port it to 4.7 after all so I've done so.

      Delete
    6. @Con:
      Seems like that the pending muqss106-009-fix_rq_load.patch didn't find it's way into the 4.7-sched-MuQSS_108.patch, while it's properly included in the 4.8 version.

      BR, Manuel Krause

      Delete
    7. I'm using 4.8 kernel with 108,

      great to read that 4.7 also got the full treatment :)

      Had some weird program GUI freezes (could be compositing, compiz related or scheduler)

      xfce4-panel content froze and content of chromium tabs also froze - only killing and/or closing of them and launching again would fix it,

      chromium occurred a few more times


      Besides that the desktop seems to behave normally


      Will see if I have a chance to add those 2 pending patches and see how behavior changes

      Thanks

      Delete
    8. Did you get any warnings or anything unusual in dmesg when it happened?

      Delete
    9. No, unfortunately not

      usually there are stall or RCU messages when these kind of things appear

      but it was totally silent :/


      Meanwhile I upgraded to 4.8.1 and the 2 pending patches

      Kernel compilation even was a bit faster than ever (7 minutes 22 secs)

      Seems like on 4.8 kernel MuQSS is behaving as it should :)

      Thanks

      Delete
    10. Aww crap,

      the most important usage case is totally broken

      "smooth motion" video playback:

      playing back HD videos via mpv with raised FPS to 48 or 60 FPS

      (made possible with vapoursynth)

      or as other option

      SVP4 for Linux (which also uses vapoursynth)

      https://www.svp-team.com/wiki/SVP:Linux


      The video lags a lot, sound and video output do NOT match most of the time

      Delete
    11. Not entirely sure if it's the newly enabled & added memory randomization placement changes,

      proprietary nvidia drivers

      the scheduler

      or a combination of the above 3 and more factors :/

      Delete
    12. Interactive mode
      (echo 1 > /proc/sys/kernel/interactive)

      clearly seems to be the culprit somehow, btw


      If it's disabled:
      echo 0 > /proc/sys/kernel/interactive


      the performance is MUCH better that it almost works fully (smooth video playback, sound + video matching most of the time)

      Delete
    13. Try with exactly the same base version of mainline and base config and nvidia driver and compare?

      Delete
    14. They are ?

      I've no reference point of MuQSS since I unfortunately had missed out on testing smooth video playback with MuQSS before (4.7.6 + MuQSS 106) due to lack of time and occupation

      the last "official" BFS I tried out was 497 which worked fine with interactive mode (I tend to leave it at default, thus enabled)


      nvidia drivers are the same, base config, well - the security RAM enhancements I can disable which would make it close to the previous kernel config

      but that testing has to wait, the kernel works satisfactory right now for all the tasks I more or less on a daily basis (as time permits)


      The only change is:

      that interactive seems to give much worse performance, higher latency and less throughput (if that makes sense) - but only in certain usage cases

      and running compiz with
      export __GL_YIELD=USLEEP

      is totally unusable right now since there's too much lag, oftentimes the video totally stalls (including the whole content of the LCD screen) for several seconds - sometimes even half a minute


      I'm not sure how __GL_YIELD=USLEEP works but from the sound of it, it heavily involves the cpu scheduler, too (?)


      kwin_x11 with compositing also seems to have gotten worse than before,

      xfwm4 and kwin_x11 without compositing seems to be fine and even smoother than before


      So something in the communication between kernel and gpu might have changed in 4.8 kernels


      Would be great to hear/read from others how their experience with MuQSS 108 on Kernel 4.7 vs. 4.8 with nvidia drivers is

      Delete
    15. >is totally unusable right now since there's too much lag, oftentimes the video totally stalls (including the whole content of the LCD screen) for several seconds - sometimes even half a minute

      that only applies to MPV smooth video (high FPS) video playback btw,

      e.g. normal playback and playback 1080p 60fps in chromium, Youtube is totally fine with compiz using gl_yield


      Thanks

      Delete
    16. Sounds like a problem with a yield implementation rather than anything else. If your driver doesn't use sched_yield it wouldn't be a problem. There are literally thousands of discussions regarding the (in)correct use of sched yield and the battle has raged for decades. I'll take a look at yield and see what it's doing. Meanwhile you could try running your application SCHED_ISO to see what that does (just as a data point.)
      schedtool -I -e mpv

      Delete
    17. It is on by default,

      I'm dragging the video files onto the launcher in Xfce4:

      env PULSE_LATENCY_MSEC=180 ionice -c2 schedtool -I -e mpv --profile=pseudo-gui -- %U

      is the full launch command


      Thanks

      Delete
    18. Ah in that case try NOT using sched_iso. Since it IS a realtime policy, if the application is written loosely enough that threads can lead to priority inversion, it can worsen performance out of sight.

      Delete
    19. Hm, that's an interesting advice and kind of makes sense,

      will try it later today WITHOUT sched_iso then


      I'm usually always raising IRQs:
      https://gist.github.com/kernelOfTruth/856f41450bdd8cbfcd5c55760e4b750e


      and under certain weird cases (e.g. FPS gaming, GRID Autosport, PDF Annotation via WINE etc.)

      the following helps:

      schedtool -I -n -15 `pidof X`

      schedtool -I -n -15 `pidof audacious`
      schedtool -I -n -15 `pidof vlc`

      schedtool -I -n -14 `pidof wineserver`
      schedtool -I -n -14 `pidof compiz`
      schedtool -I -n -14 `pidof kwin`
      schedtool -I -n -14 `pidof xfwm4`




      This time I ran the following full stack - which is obviously overkill, shouldn't be necessarily (and I believe also partially might be wrong):

      schedtool -R -p 3 `pidof X`
      schedtool -R -p 4 `pidof wineserver`
      schedtool -R -p 4 `pidof compiz`


      schedtool -I -n -15 `pidof X`

      schedtool -I -n -15 `pidof audacious`
      schedtool -I -n -15 `pidof vlc`

      schedtool -I -n -14 `pidof wineserver`
      schedtool -I -n -14 `pidof compiz`
      schedtool -I -n -14 `pidof kwin`
      schedtool -I -n -14 `pidof xfwm4`


      and the video DID NOT stutter or stall ONE BIT

      sound, video output was fine and in sync all the time


      But until next boot priorities are kind of out of whack ;)

      (and in the past it lead to instabilities, stalls and hangups after longer runtime duration [several days])

      Thanks

      Delete
    20. Yes,

      MPV is heavily multi-threaded and puts lots of load on the CPU(s)

      Delete
    21. Yeah I use mpv a lot myself. Also worth disabling yield entirely as a trial.
      __GL_YIELD=NOTHING

      Delete
    22. Con, MuQSS is really acting up on this boot:

      there are random stalls while browsing bookmarks in chromium,

      switching between tasks (via alt+tab) using compiz as compositor it hangs there for several seconds (10-30 seconds when running sched_iso, WITHOUT sched_iso it's only 0.5-3 seconds) then continues,

      and this happens every few app switches


      tried running chromium and compiz under sched_iso which made it significantly worse

      Delete
    23. Thanks KoT, sched_iso is not really tested on muqss so I have to investigate. Additionally you might want to try the 003 patch I just uploaded.

      Delete
  6. Added a 002 pending patch for 108 as well which does something I forgot to do and improves performance a little more (I originally posted this in the wrong muqss version announce, even I'm losing track.)

    ReplyDelete
    Replies
    1. Just wondering wouldn't it be more comfortable to have a single public git repo (github?) where you can simply push your changes, plus you can simply tag a commit which for example shows what release it is? Also having branches is much easier to search for patches.

      Delete
  7. I ran simple Unigine tests on my machine and version 108 seems to be best, it's within the error margin, but still...
    https://docs.google.com/spreadsheets/d/1EayezAsGlJdXjZbS3b9m7YtvtRF-DJ3xrT3hYCvfymQ/edit?usp=sharing

    Would it make sense to run tests with smth in the background, like "stress" utility?

    Br, Eduardo

    ReplyDelete
  8. @Con,

    Can You please look at this crash: http://pastebin.com/5m22kAar .
    I was compiling another kernel in VirtualBox meanwhile watching a video in Youtube and it crashed, it slowly stopped to respond to anything I do and I had to poweroff the laptop.
    Kernel: 4.7.6 + 108 + wbt + bfq.

    If You need more info, please ask.

    regards,
    Eduardo

    ReplyDelete
    Replies
    1. Hey Eduardo. The backtrace looks all filesystem related so I can't rule out bfq being responsible?

      Delete
  9. @Con,

    Thanks for quick answer, I'll switch to CFQ and try stressing the system again, let's see what happens.
    Sorry for disturbance :)

    So far MuQSS is behaving good, performance wise it's very good as well. Thanks for your work.

    br, Eduardo

    ReplyDelete
  10. I gave another shot at 'muqss108 + 2 pending patches' with the runqlat utility.
    I had to revert 2 commit in the kernel to get runqlat working (from this bug report https://github.com/iovisor/bcc/issues/728).
    As a reminder, the test consist in building ffmpeg with increasing number of -j and running runqlat. This time, the results are in msecs and I ran runqlat for a much longer duration (120s instead of 10s) in the hope to catch high latencies if there are any.

    I put the results in the spreadsheet. I'd like to make better charts, but google sheets doesn't have a lot of options.
    The latencies are much higher with muqss. I don't know if these results are expected. They are reproducible.

    Pedro

    ReplyDelete
    Replies
    1. Yes it's possible for now, tough I doubt you mean milliseconds as according to that the latencies were over 32 SECONDS at one stage. I have an idea what might be causing them and will be posting a new pending patch soon.

      Delete
    2. Thanks for the answer.
      Yeah I know it's odd but it's really milliseconds, as I used the -m option for runqlat.
      Here is the script I used:
      http://pastebin.com/KTjaaRaU

      Pedro

      Delete
    3. I guess you can try the 003 pending patch I just uploaded and see if that does anything for it? Thanks!

      Delete
    4. With the 003 pending patch it's a bit better, especially at low loads. See updated results.

      Pedro

      Delete
    5. Thanks. It helps but there's still something fundamentally wrong. I'll keep looking.

      Delete
    6. Thanks Pedro I think I've found the problem thanks to your testing and have uploaded a 004 patch which (hopefully) should address it.

      Delete
    7. ...maybe not. I'll keep looking.

      Delete
  11. @ck:
    I've run into strange troubles on here with kernel 4.7.7 + MuQSS 108+ the 2 pending patches + BFQ + WBT + my TOI forward port.
    Symptoms are
    o empty firefox startup time extremely long, >30s vs. a few
    o TOI hibernation doesn't resume ("Compressed_read returns -22")
    I know, it's quite keen to expect attention with this unsupported setup.

    In a kind of bisecting approach, I've reverted last 2 pending 108 patches and the muqss106-009-fix_rq_load.patch -- what led to TOI hibernating well, while the firefox startup remains unchanged long.

    Does this make any sense for you? On my side, I simply currently don't know, whether to continue "bisecting" to chase the FF issue or, what makes TOI failing.

    Clueless, thanks for any hint/ advice,
    BR, Manuel Krause

    ReplyDelete
    Replies
    1. My results are quite impredictible/ unreproducible.
      Sometimes FF starts up fast, sometimes not. Sometimes TOI fails, sometimes not. 50/50 for both issues.

      Going back to 106, what was the last known safe haven after Alfred's last VRQ.

      BR Manuel Krause

      Delete
    2. Manuel could you please try without bfq first and then without wbt next. I know you all love bfq but there are just too many patches adding too many variables to this testing for me to know what's going on, especially with bfq.

      Delete
    3. @ck:
      Mmh, I don't want to leave 4.7 without BFQ or WBT, as these haven't led to any issues until & incl. MuQSS 106.

      I understand your reasonable suggestion, though.
      BR, Manuel Krause

      Delete
    4. For me it was a good decision to have stopped testing yesterday evening.
      Some hours ago, I've then added all the 7 piled up 108 pending patches (+muqss106-009) upon my otherwise unchanged (sorry ;-) ) 4.7.7 setup. Although uptime is quite short, I want to let you know early, that everything seems to be working well again. Going to test it longer.
      (And, when errors occur, begin to re-test with the other patches removed.)

      Thank you very much for your successful work,
      BR, Manuel Krause

      Delete
    5. @ck+:
      For normal usage everything went very well, only my TOI failed at the 3rd hibernation's resume.
      Currently I'm cross-testing this newer kernel with same setup+.config with Alfred's very last 4.7 patch, to see if 4.7.5 to 4.7.7 transition introduced malfunctions, before removing TOI/ BFQ/ WBT. Unfortunately, I'm quite sticking to this formally good working combination. :-/

      Does someone know, whether in-kernel suspend-to-disk got better/faster over the last year?

      BR, Manuel Krause

      Delete
  12. I'm not doing any formal testing or benchmarking, but I did build MuQSS 108 for the 4.7.7 kernel on PCLinuxOS. It is the most responsive desktop that I've seen so far. Great work!
    (I also am using bfq.)

    Galen Seaman

    ReplyDelete
  13. It would be nice if someone could make ubuntu/debian binaries for testing....

    ReplyDelete
    Replies
    1. https://wiki.ubuntu.com/KernelTeam/GitKernelBuild
      https://www.kernel.org/doc/Documentation/applying-patches.txt

      Delete
  14. Hey Con; on my i686 build, I tried to disable SMT/SMP/HT but the build failed. Enabling SMT fixed it; I'm wondering if the equivalent of bfs497-fix_smt_nonice.patch might be needed or something?

    Also, when I re-enabled SMT, the kernel config asked to choose the CPU governor; is there any difference to MuQSS on what is selected here (like ondemand vs schedutil)?

    Fyi, on the x64 dual-core (its thru patch 3 atm), when I cross-compiled i686, I forgot to change its make config (set to -j4 as it used to be a Phenom X4). Load average reached almost 6, and I didn't notice any sluggishness at all!

    ReplyDelete
    Replies
    1. Thanks jwh7. I'll look into the build issue but I'll worry more about fixing all builds when all the remaining bugs are shaken out - it won't cause any demonstrable detriment to enable SMT as a workaround for now. There's no difference in terms of what governor you should choose on muqss. Glad to hear you're benefiting from the interactivity too.

      Delete
  15. I'm getting huge load averages with your latest Pending patches applied. kernel version is 4.7

    ReplyDelete
    Replies
    1. Yes I acknowledge that bug of huge load averages, thanks.

      Delete
  16. Haha, those are AWESOME :D

    load average: 2349669.27, 7343369107.80, 2588893649.73

    Top50 fastest computers are NOTHING compared to MY mini quantum computer ;)


    stage4 backup completed in reasonable time, btw (roughly 21 minutes) with 108 + 9 pending patches applied

    playback with MPV and vapoursynth (48 fps) on compiz with opengl compositing

    99% also works fine

    As adviced no sched_iso was used


    Thank you VERY much :)

    ReplyDelete
  17. @Con, For future reference, here are the error (and warning) messages from my aforementioned failed MuQSS v108 build w/ no SMT/SMP/HT/MC...
    https://gist.github.com/jeremywh7/944c10e189300086f1de58b8fa7fc0b4

    ReplyDelete
    Replies
    1. Thanks for those, I'll fix them shortly and wrap up all the pending patches into a new release.

      Delete
  18. where is bbswitch-ck, i need it for my hybrid graphics

    ReplyDelete