| 04:36.17 | *** join/#oe infobot (ibot@96-86-209-99-static.hfc.comcastbusiness.net) |
| 04:36.17 | *** topic/#oe is OpenEmbedded Developer Lounge | Web: http://openembedded.org | Repositories: http://git.openembedded.org/ | Primary Repo Mirrors: https://github.com/openembedded | This is not a distro or machine support channel |
| 05:35.22 | *** join/#oe AndersD (~AndersD@h-98-128-162-82.NA.cust.bahnhof.se) |
| 06:11.44 | *** join/#oe LocutusOfBorg (~locutusof@ubuntu/member/locutusofborg) |
| 06:13.56 | *** join/#oe rob_w (~bob@unaffiliated/rob-w/x-1112029) |
| 06:17.14 | *** join/#oe davidlt (~davidlt@78-63-27-146.static.zebra.lt) |
| 06:23.30 | *** join/#oe eFfeM (~fmeulenbr@f160137.upc-f.chello.nl) |
| 06:33.31 | *** join/#oe LetoThe2nd (uid453638@gateway/web/irccloud.com/x-vccfadksvuvkvdgc) |
| 07:19.06 | *** join/#oe cpriouzeau (~cpriouzea@165.225.94.93) |
| 07:29.04 | *** join/#oe ao2 (~ao2@host-79-46-138-206.retail.telecomitalia.it) |
| 07:31.06 | *** join/#oe leon-anavi (~Leon@78.130.197.211) |
| 07:46.17 | *** join/#oe florian_kc (~florian_k@Maemo/community/contributor/florian) |
| 08:17.14 | *** join/#oe RobertBerger (~rber@ppp-2-84-14-95.home.otenet.gr) |
| 08:19.46 | *** join/#oe eFfeM (~fmeulenbr@f160137.upc-f.chello.nl) |
| 08:21.57 | dv_ | hm would it make sense to have a PARALLEL_BUILD variable as an alternative to PARALLEL_MAKE? PARALLEL_BUILD would just be the number |
| 08:22.03 | dv_ | PARALLEL_MAKE is something like "-j 8" |
| 08:22.11 | dv_ | the thing is, not every tool that can parallelize uses the -j flag |
| 08:22.20 | dv_ | one example is xz |
| 08:22.34 | dv_ | or does such a variable exist already? |
| 08:35.26 | *** join/#oe tnovotny (~tnovotny@176-74-132-138.netdatacomm.cz) |
| 08:36.26 | hrw | dv_: https://marcin.juszkiewicz.com.pl/2008/04/07/speeding-up-bitbake-builds/ is 12.5 years old but has what you wanted |
| 08:37.00 | hrw | still has some hits ;d |
| 08:38.01 | XorA | can speed up massively with debootstrap :-D |
| 08:38.15 | dv_ | hrw: no, BB_NUMBER_THREADS is not what I mean |
| 08:38.37 | dv_ | BB_NUMBER_THREADS is specifically about the number of *bitbake* threads |
| 08:41.51 | hrw | dv_: PARALLEL_MAKE + BB_NUMBER_THREADS are easiest ways to speedup your build. |
| 08:42.01 | hrw | assuming that you already have cpu power, memory and storage |
| 08:42.24 | dv_ | hrw: you are missing the point |
| 08:42.32 | hrw | enlight me |
| 08:42.37 | dv_ | <dv_>the thing is, not every tool that can parallelize uses the -j flag |
| 08:42.43 | dv_ | <dv_>one example is xz |
| 08:42.55 | dv_ | <dv_>PARALLEL_MAKE is something like "-j 8" |
| 08:43.00 | dv_ | try passing "-j 8" to xz |
| 08:43.23 | dv_ | xz has a -T flag, not a -j |
| 08:43.33 | hrw | I leave xz to do it's job as bitbake runs 10-20 other threads at same time keeping all cpu cores busy |
| 08:43.42 | dv_ | so I am not asking for a solution here, I am proposing an improvement |
| 08:44.23 | dv_ | xz is run with -T 0 "makes xz use as many threads as there are CPU cores on the system" |
| 08:44.34 | dv_ | and maybe I don't *want* xz to use so many threads |
| 08:45.00 | dv_ | maybe I set PARALLEL_MAKE and BB_NUMBER_THREADS to values that combined leave some cores idle so I can run some other stuff in parallel |
| 08:45.22 | dv_ | and this falls apart once xz starts up |
| 08:45.37 | dv_ | easiest solution: pass the parallel build number to its -T flag |
| 08:46.19 | hrw | XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" |
| 08:46.32 | hrw | you can set XZ_THREADS=1 in local.conf |
| 08:47.19 | dv_ | hm, true. |
| 08:47.50 | hrw | always worth checking conf/bitbake.conf in OE repo |
| 08:48.20 | dv_ | still, PARALLEL_MAKE seems a bit like code smell to me. all I really want to convey with it is how many parallel jobs builds should use. the "-j" flag is superfluous of that and purely a make artifact. |
| 08:49.01 | hrw | dv_: feel free to create patch which adds it |
| 08:49.21 | dv_ | I will, but I asked here about opinions |
| 08:49.34 | dv_ | <dv_>hm would it make sense to have a PARALLEL_BUILD variable as an alternative to PARALLEL_MAKE? <- note the "would it make sense" part |
| 08:49.42 | hrw | sure, it may lack PARALLEL_NINJA or sth like that but several build systems go multithread by default now |
| 08:50.49 | hrw | dv_: would be nice to have a list of places where such new var would be useful. make, xz, what else? |
| 08:51.07 | hrw | ninja says: -j N run N jobs in parallel (0 means infinity) [default=14 on this system] |
| 08:51.15 | hrw | while system has 12 threads |
| 08:51.26 | dv_ | interestingly, gzip seems to also support multithreaded compression, but I see no flag to control this |
| 08:51.34 | hrw | because by default ninja goes $(nproc)+2 |
| 08:52.29 | hrw | gzip/bzip2 are single threaded in default implementations. there are multithreaded implementations of both |
| 08:52.52 | dv_ | yeah, like the pigz implementation. is this being used in oe-core? |
| 08:52.54 | dv_ | looks |
| 08:54.09 | dv_ | ah, pigz has the -p flag |
| 08:54.13 | hrw | and still - adding extra vars to get few tools go multithread while you can run multiple threads sounds like overkill to me |
| 08:55.06 | hrw | how it changes something is package.tar.gz will unpack on all 12 cores when I can use those 12 cores to run 12 bitbake threads at same time to unpack 12 different tarballs |
| 08:55.09 | hrw | for example |
| 08:56.06 | hrw | in times when 32GB ram is quite a standard in normal development machine and 64+ goes into build machines. and nvme storage is a standard too |
| 08:58.30 | hrw | looks at tensorflow build on 46 cpu cores eating 50GB of ram... |
| 09:05.44 | *** join/#oe davidlt_ (~davidlt@78-63-27-146.static.zebra.lt) |
| 09:05.57 | hrw | dv_: ? |
| 09:10.20 | dv_ | back |
| 09:10.36 | dv_ | <dv_>still, PARALLEL_MAKE seems a bit like code smell to me. all I really want to convey with it is how many parallel jobs builds should use. the "-j" flag is superfluous of that and purely a make artifact. <- this is the entire core of my argument |
| 09:10.46 | dv_ | it is not about huge number or core or anything like that |
| 09:10.51 | dv_ | the xz thing was just one exaple |
| 09:10.53 | dv_ | *example |
| 09:11.18 | dv_ | *huge number of cores |
| 09:11.26 | dv_ | needs some coffee :) |
| 09:12.23 | dv_ | admittedly, I deviated from my own point by going too much into xz ... |
| 09:12.28 | hrw | ;) |
| 09:12.58 | hrw | dv_: PARALLEL_MAKE is older than BB_NUMBER_THREADS iirc. |
| 09:13.18 | dv_ | uh ... so? |
| 09:13.27 | dv_ | ah wait. *older*. |
| 09:13.31 | dv_ | yes, i do need some coffee =) |
| 09:13.33 | dv_ | brb |
| 09:13.33 | hrw | dv_: in 2004-6 single/dual machines were most popular |
| 09:14.57 | hrw | and still I think that going into 'how to multithread single apps' (so PARALLEL_BUILD proposal) < 'run multiple apps at same time' (BB_NUMBER_THREADS) |
| 09:23.16 | hrw | bbl |
| 09:51.43 | *** join/#oe leon-anavi (~Leon@78.130.197.211) |
| 10:06.34 | JaMa | dv_: meta/lib/oe/utils.py also has parallel_make_argument which will re-parse PARALLEL_MAKE to whatever format you need |
| 10:07.09 | JaMa | e.g. like go is using in GO_PARALLEL_BUILD ?= "${@oe.utils.parallel_make_argument(d, '-p %d')}" |
| 10:08.10 | *** join/#oe eFfeM (~fmeulenbr@f160137.upc-f.chello.nl) |
| 10:09.38 | dv_ | ahh |
| 10:46.47 | hrw | JaMa: nice! |
| 10:48.57 | *** join/#oe mrnuke (~mrnuke@c-98-195-139-126.hsd1.tx.comcast.net) |
| 10:49.55 | JaMa | still has some hits ;d |
| 10:50.38 | JaMa | ah, that was for blog article, not you personally :) |
| 10:55.02 | hrw | d: |
| 10:58.03 | *** join/#oe eFfeM (~fmeulenbr@f160137.upc-f.chello.nl) |
| 11:43.15 | *** join/#oe leon-anavi (~Leon@78.130.197.211) |
| 11:44.41 | *** join/#oe berton (~berton@181.220.78.182) |
| 12:28.15 | *** join/#oe leon-anavi (~Leon@78.130.197.211) |
| 12:28.32 | *** join/#oe khem (~khem@unaffiliated/khem) |
| 12:53.50 | *** join/#oe enr0n (sid391711@gateway/web/irccloud.com/x-wlfypzgmawecogbi) |
| 13:45.26 | *** join/#oe eFfeM (~fmeulenbr@f160137.upc-f.chello.nl) |
| 13:51.42 | *** join/#oe sakoman (~steve@rrcs-66-91-142-162.west.biz.rr.com) |
| 13:52.20 | *** join/#oe ericch (~ericch@pool-108-34-251-214.prvdri.fios.verizon.net) |
| 15:05.18 | *** join/#oe creich (~creich@p200300f6af231710000000000000039b.dip0.t-ipconnect.de) |
| 15:57.40 | *** join/#oe agaikova (user21236@neotame.csclub.uwaterloo.ca) |
| 15:57.45 | agaikova | Hello friends :) |
| 16:00.58 | Jin^eLD | hi :> |
| 16:57.27 | *** join/#oe florian_kc (~florian_k@Maemo/community/contributor/florian) |
| 17:00.27 | *** join/#oe hrw (~hrw@redhat/hrw) |
| 19:20.44 | *** join/#oe florian_kc (~florian_k@Maemo/community/contributor/florian) |
| 19:33.49 | *** join/#oe ao2 (~ao2@host-79-46-138-206.retail.telecomitalia.it) |
| 21:26.16 | *** join/#oe ejoerns (4d153bed@ip4d153bed.dynamic.kabel-deutschland.de) |
| 21:29.19 | *** join/#oe florian_kc (~florian_k@Maemo/community/contributor/florian) |
| 21:31.59 | ejoerns | Hi all |
| 21:32.13 | ejoerns | Could one of you say if there is any existing way to re-use the images created by wic's rootfs split? E.g. bootloader image from /boot or home filesystem image created from /home. I can only see them living temporary in ${WORKDIR}/build-wic, but would like to use and pack them all as separate artifacts for image-based system updates. |
| 21:33.37 | marex | ejoerns: do you would get from system-state-A => system-state-B kind of update mode, to system-state-{A,1,a,...} => system-state-{B,1,a,...} => ... => kind of update horror ? :) |
| 21:33.48 | marex | ejoerns: atomic updates are good , really |
| 21:34.05 | marex | ejoerns: but the images are in tmp/deploy/images/*/ , so you can pick them from that deploy dir |
| 21:35.21 | ejoerns | @marex welll... trust me, I know that ;) I want to move from system state A {rootfs A, appfs A} to system state B {rootfs B, appfs B}. Boot I would say is orthogonal as it can be atomically updated but is not part of A/B play |
| 21:35.53 | ejoerns | in tmp/deploy/images/ i have the disk image created by wic, but not the fs images |
| 21:36.56 | marex | ejoerns: well then that might be a configuration setting ? |
| 21:40.48 | ejoerns | marex: so for pure wic it would be "-o" option. But for image_type_wic.bbclass it is hard-coded to "${WORKDIR}/build-wic" at the moment. And the artifact names are hard to parse as they are only named by their partition number. |
| 21:44.03 | ejoerns | in general it does not seem to be part of the concept of wic at the moment to re-use the fs images. They seem to be more like internally used temporary artifacts only. Please correct me if I am wrong. |