Sick compile time
-
Due to policy in our company, I was forced to switch server OS to Gentoo.
Ok now Im with root shell trying to install KDE. As KDE Plasma alrewady contains x11, there is no need to install it separately.
emerge -avND kde-plasma
It successfully emerges 128 out of 268 pkgs. It reacheswebkit-fck
which puulsqt5-rickkk-q
as dependency and it takes forever to compile.I googled and it appearsa that it takes > 24 hrs to compile even on the fastest computers. And devs knows it from circa 2011 and had done nothing to shorten compile time.
-
@Eddy said in Sick compile time:
Hi,
And what is your question? Is this Qt related? Maybe the Gentoo forums are more appropriate to find help.
Eddy
My question is why
kde-*
(meaning every packet of KDE) takeswebkit-gtk
as dependency or is there a way to a) shorten buildtime or b) maskwebkit-gtk
in the way it does not pullqt*
as dependency?My machine is Xeon E3-1225 V2 @ 3.20 GHz so pretty powerfull
qt-webkit:
genlop -p qt-webkit
Estimated time of building qt-webkit is 2 hours
webkit-gtk:
genlop -p webkit-gtk
Estimated time of building webkit-gtk is 16 hours
Why is that?
-
@wlisik I've run gentoo for many many years. I never saw gtk's webkit being part of KDE. You must have some other thing pulling it in as a dependency.
KDE uses Qt and Qt's webkit.
As for build times, on my more recent systems building Qt takes about 15 minutes. So I don't know why your (or other's) system would take 24 hours. Are you making sure you use multiple jobs? If you do just a plain
make
then it will be really slow. If you domake -j12
replace 12 with # of cpu cores + 1 for your system then it should be exponentially faster.Just for reference I'm using an i7-5820k 3.3ghz overclocked to 3.7 or so.
Gentoo is not for the faint of heart though. It will indeed take a long time to build up a whole system. Being very experienced with gentoo and building in general it still took me at least a day to build a full system from the ground up. I can do my new distro (arch) in about an hour. That is why I switched. I get the same cutting edge linux out of arch but without having to build everything. ;)
Oh 1 more idea. Are you using kde plasma 5? Or is it kde 4 or something? Some of the older KDE's might have a dependency on gtk's webkit, but even then I don't remember that being an issue. It was a long build (maybe 2 hours) and there was always something that needed it so I always had to build it anyway.
I would pretty much get my system all set up with the basics then start my emerge on my apps and window manager and go to bed. This step would take a lot of hours. Libreoffice is a killer, so is thunderbird, chromium, and of course webkit.
-
I can confirm 24 hours build time of KDE on Gentoo. But that was on an Intel Celeron around the year 2000 :-)
-
@ambershark said in Sick compile time:
@wlisik I've run gentoo for many many years. I never saw gtk's webkit being part of KDE. You must have some other thing pulling it in as a dependency.
KDE uses Qt and Qt's webkit.
As for build times, on my more recent systems building Qt takes about 15 minutes. So I don't know why your (or other's) system would take 24 hours. Are you making sure you use multiple jobs? If you do just a plain
make
then it will be really slow. If you domake -j12
replace 12 with # of cpu cores + 1 for your system then it should be exponentially faster.Just for reference I'm using an i7-5820k 3.3ghz overclocked to 3.7 or so.
Gentoo is not for the faint of heart though. It will indeed take a long time to build up a whole system. Being very experienced with gentoo and building in general it still took me at least a day to build a full system from the ground up. I can do my new distro (arch) in about an hour. That is why I switched. I get the same cutting edge linux out of arch but without having to build everything. ;)
Oh 1 more idea. Are you using kde plasma 5? Or is it kde 4 or something? Some of the older KDE's might have a dependency on gtk's webkit, but even then I don't remember that being an issue. It was a long build (maybe 2 hours) and there was always something that needed it so I always had to build it anyway.
I would pretty much get my system all set up with the basics then start my emerge on my apps and window manager and go to bed. This step would take a lot of hours. Libreoffice is a killer, so is thunderbird, chromium, and of course webkit.
if you are experienced with Gentoo, you should be able to list all deps for
kde-plasma
. Bothwebkit-gtk
andqt-webkit
are listed as dependecies. -
@wlisik Well, it is your server. But I would not install a huge desktop environment on a server, especially if it is a production server. Do you use the server as a desktop machine as well, or what is the reason to install KDE there?
-
@jsulm said in Sick compile time:
@wlisik Well, it is your server. But I would not install a huge desktop environment on a server, especially if it is a production server. Do you use the server as a desktop machine as well, or what is the reason to install KDE there?
the reason for this is that server is not a production server (serves test/dev purposes) and is hosting multiple VMs (accessible via internal LAN). KDE 'won' our elections for best DE so I installed it.
-
@wlisik said in Sick compile time:
if you are experienced with Gentoo, you should be able to list all deps for kde-plasma . Both webkit-gtk and qt-webkit are listed as dependecies.
I'm very experienced with gentoo. It was my primary OS for both desktop and 2 of my servers for 8 years or so.
GTK's webkit was always pulled in during my builds by something so I never bothered to check if it was a pure dependency of KDE or not. I still find that odd as KDE should not use it.
Anyway, all that is not really the point. You were saying it was slow compilation. Which means you probably aren't utilizing your CPU to it's fullest. Is a large gentoo installation build slow? Yes. But that is because it builds so many packages. Like I said before I set mine before bed and it's done in the morning. Never have I had it take longer than 10 hours or so.
So make sure you have set your jobs properly by adding
MAKEOPTS="-j12"
to your make.conf. Don't do 12 unless you have a 6 core cpu like I do. Substitute whatever your server has cpu wise +1. I.e. the value ofnproc
.If you didn't use jobs that alone will speed you up tremendously. The only other thing you can do is buy a faster CPU/overclock your current one. Or you can do like I did and switch to Arch. It's all the awesome cutting edge coolness of Gentoo without the large CPU requirement during install/updates. ;)
One more thing, if you are running this as a VM and not on bare metal that will be super slow as well. From your original post I gathered it was on the box itself, but just thought I'd mention that. Virtualization is really slow even on good systems.
-
My
nproc
shows6
so I have in my/etc/portage/make.conf
:MAKEOPTS=-j7
. Its been there since begining.
As of Arch - I know it and its in our internal repo so customers are free to use it. But to be first and main OS - big no.
As of virtual-things::Its nice to be able to fit OS inside one single file. It doesnt have to be slow whatsoever. I know many containers which are rendered via TAN and arew even faster than host. It depends on how skilled the admin is.
Super skilled admin is able to make every VM running smooth. It takes +/- 1hr to achieve this.
Skilled admin needs 10hrs
So-called admin will say fuck you it cannot be done. -
@wlisik said in Sick compile time:
even faster than host
That would be impossible. The raw CPU would be the fastest. There's no way to get a virtual machine running a non-native OS to be even close to the bare metal. Even if you could pretend that the virtualization software had no code what-so-ever and therefore no overhead you'd still only be able to achieve as fast as the host. Even that is impossible though.
And don't confuse running smooth as running fast. All my VMs run smooth, none of them are even close to bare metal in speed. :) And I have some of them heavily optimized.
As for your build times, I'm not sure why they are so slow if you have a decent cpu and 6 cores. That's the reason I finally gave up on gentoo. Things just took too long to deal with in some cases. Sometimes you just need to have things working and can't spare 24 hours to let things compile. ;)
I still love gentoo, it's my second favorite linux. Was my first for 8 years but sometimes it's just not worth the hassle.
I've never even heard of a company using gentoo much less forcing it as their standard linux. That's super rare. Awesome, but super rare. :)
Linux in the corporate world seems to be ubuntu on a desktop and redhat or centos on servers, lol. I hate all those distros, but after many years doing cross platform development, that is what I've seen at every major corporation I've worked for.
-
@wlisik said in Sick compile time:
@jsulm said in Sick compile time:
@wlisik KDE on a server?
why not?Because putting an X-server and a desktop environment on a machine greatly enlarges its attack surface.
-
@Wieland said in Sick compile time:
@wlisik said in Sick compile time:
@jsulm said in Sick compile time:
@wlisik KDE on a server?
why not?Because putting an X-server and a desktop environment on a machine greatly enlarges its attack surface.
NOT if you know what you are you doing.
-
@wlisik said in Sick compile time:
@Wieland said in Sick compile time:
@wlisik said in Sick compile time:
@jsulm said in Sick compile time:
@wlisik KDE on a server?
why not?Because putting an X-server and a desktop environment on a machine greatly enlarges its attack surface.
NOT if you know what you are you doing.
Um lol... I've administered and run linux full time for 16 years now professionally, and 4 years beyond that in college casually. I'm pretty sure I know what I'm doing.
That being said, what @Wieland said is 100% true. Adding X windows is a HUGE security vulnerability. And unless you spend a bunch of time patching code to change how X works, then you have those vulnerabilities as well on your server. If you did that you wouldn't really even use X you'd have your own X11 compatible system your wrote with security in mind.
Even if X didn't have a fair amount of security flaws in it's design and implementation it is still a fact just based on more code/applications means more chance for security flaws. We're all coders here. We all know that every line of code you write has the potential to be exploited and attacked. Being a coder you know this as well. If you add almost a full GB of applications to your system then you are absolutely opening security holes. That's just a plain fact.
Don't forget companies like Apple, google, and Microsoft have the money to hire the best of the best to develop and handle security for them. They all get critical security flaws, sometimes daily. I know some google engineers personally and have worked with them. They are way smarter than 95% of all the other engineers I know.
I work in a security related field a lot too with credit card machines, POS systems, fuel controllers, etc. Most of these products need 3rd party PCI DSS compliance. I know about security and code security. And all our embedded devices run linux. So trust me when I say @Wieland is right. ;)