IRCloggy #moarvm 2018-11-27

Logs Search ←Prev date Next date→ Channels Documentation

Provider of IRC logs since 2005.
WARNING: As Freenode became unjoinable and lost all warnings in topics, we cannot log channels on Freenode anymore.

2018-11-27

← lizmat left02:08
← avar left05:34
→ avar joined05:45
← avar left05:45
→ avar joined05:45
p6bannerbot set mode: +v05:45
p6bannerbot set mode: +v05:46
→ ilogger2 joined06:34
p6bannerbot set mode: +v06:35
→ domidumont joined07:03
p6bannerbot set mode: +v07:04
→ robertle joined07:48
p6bannerbot set mode: +v07:48
jnthn timotimo: I forget exactly what it does; it may just let old ones be GC'd09:30
Would have to check the code09:30
timotimo that seems to be the case09:30
i put a breakpoint on its gc_mark and for every gc run there was only ever one that got marked09:31
so that shouldn't cause unlimited memory growth09:31
jnthn Probably I did that because it's "obviously safe", whereas re-use always raises questions :)09:31
Well, is its gc_free being called? :)09:31
nwc10 jnthn: ASAN considers e317922036598184745c9b2afc1eafb44597e75b to be boring09:36
(Sketch out creating deopt/materialization info)09:36
jnthn Yay :)09:44
nwc10 please try harder :-)09:45
← domidumont left09:51
→ domidumont joined09:51
p6bannerbot set mode: +v09:52
timotimo putting the breakpoint in the wrong spot will get you to a spot where you can "c" as often as you want and you won't make progress, because for some reason the other thread won't release the condition variable10:01
ok, i've got some more breakpoints and i'm seeing lots and lots of spesh logs being created and the distance between nursery_alloc_limit and nursery_alloc only shrinking by a very small amount10:14
so maybe we should deduct the size of the spesh log buffer from the nursery alloc limit10:15
but even then, this shouldn't be unbounded growth10:16
i'll wait for a gc run to happen10:16
→ lizmat joined10:20
p6bannerbot set mode: +v10:20
→ zakharyas joined10:40
p6bannerbot set mode: +v10:41
timotimo OK, it's not unbounded, the bound is just very, very, very high10:46
RSS is currently at 490, the nursery still has 94992 bytes of space10:47
the other thread also occasionally creates spesh logs, it's at 126472 bytes left before a GC would happen10:47
i'd almost consider freeing the buffer right when the spesh worker is done with it, tbh10:48
and letting the gc take care of the object itself, only freeing the buffer if it's not turned into a null pointer yet10:48
m: say 93152 / 8010:49
camelia rakudo-moar 7a82deb6e: OUTPUT: «1164.4␤»10:49
timotimo m: say (1164 * 16384 * 24) / (1024 * 1024)10:50
camelia rakudo-moar 7a82deb6e: OUTPUT: «436.5␤»10:50
timotimo ok, it'd still allocate this many megabytes of ram before the GC kicks in10:50
→ squashable6 joined11:12
p6bannerbot set mode: +v11:13
Geth ¦ MoarVM/decrease_spesh_log_memory_growth: c933b51669 | (Timo Paulssen)++ | 2 files11:43
¦ MoarVM/decrease_spesh_log_memory_growth: free spesh log entries after consuming them11:43
¦ MoarVM/decrease_spesh_log_memory_growth:11:43
¦ MoarVM/decrease_spesh_log_memory_growth: this prevents threads that do little to no11:43
¦ MoarVM/decrease_spesh_log_memory_growth: allocations but lots of spesh logging in an11:43
¦ MoarVM/decrease_spesh_log_memory_growth: otherwise idle program from growing to multiple11:43
¦ MoarVM/decrease_spesh_log_memory_growth: gigabytes of ram used solely for spesh log entries.11:43
¦ MoarVM/decrease_spesh_log_memory_growth:11:43
¦ MoarVM/decrease_spesh_log_memory_growth: This happened with rakudo's ThreadPoolScheduler11:44
¦ MoarVM/decrease_spesh_log_memory_growth: supervisor thread when a program with multithreading11:44
¦ MoarVM/decrease_spesh_log_memory_growth: or async workloads was left idle, like for example11:44
¦ MoarVM/decrease_spesh_log_memory_growth: a Cro app.11:44
¦ MoarVM/decrease_spesh_log_memory_growth: review: https://github.com/MoarVM/MoarVM/commit/c933b5166911:44
timotimo m: say (16384 * 24) / (1024 * 1024)11:44
camelia rakudo-moar 7a82deb6e: OUTPUT: «0.375␤»11:44
Geth ¦ MoarVM: timo++ created pull request #1002: free spesh log entries after consuming them11:47
¦ MoarVM: review: https://github.com/MoarVM/MoarVM/pull/100211:47
timotimo ^- pull request contains a bit more prose11:47
anyway, that makes my program's memory growth stop11:55
if it's still at 219 in an hour, i'd say it's done :) :)11:56
nwc10 and you plan to test this by going out for lunch?11:57
timotimo late breakfast, i believe12:02
→ evalable6 joined12:16
p6bannerbot set mode: +v12:16
timotimo AlexDaniel: wanna try out that branch to see if it makes things better?12:19
AlexDaniel timotimo: I don't know any quick and easy way to test it12:20
timotimo well, how slowly does it grow?12:21
AlexDaniel well uhh… after like 2 weeks I think the memory usage grows to like two gigs of ram… or something like that12:21
timotimo mhh12:23
that's about what i expect from spesh log growth12:23
timotimo AFK12:23
→ shareable6 joined12:31
→ reportable6 joined12:32
p6bannerbot set mode: +v12:32
p6bannerbot set mode: +v12:33
AlexDaniel also just got a machine with 64 GB of RAM12:45
maybe these memory issues don't matter as much after all :D12:45
timotimo it'd still be nice to run on raspberry pis (or beaglebones) without blowing all the memory on unused data13:00
oh no!13:02
it grew to 200 megs! :(13:02
→ avar joined13:19
← avar left13:19
→ avar joined13:19
p6bannerbot set mode: +v13:19
p6bannerbot set mode: +v13:19
nine timotimo: but does it still grow as much?14:02
timotimo it has now grown to 181 megs14:17
or rather it has nworged to 181 megs14:17
lizmat nworged >14:42
?14:42
[Coke] m: say "grown".flip ~ "ed"14:44
camelia rakudo-moar 7a82deb6e: OUTPUT: «nworged␤»14:44
lizmat aha!14:45
timotimo it's still at 181 megs right now14:46
though i don't know if it went up some time in between14:46
← domidumont left14:47
→ domidumont joined15:02
p6bannerbot set mode: +v15:03
← zakharyas left15:05
← domidumont left17:30
← robertle left17:36
timotimo it nworged to 180 in the mean time18:17
→ domidumont joined18:33
p6bannerbot set mode: +v18:34
samcv timotimo, minor nitpick: can you make sure you use titlecase in your commit messages from now on?19:10
timotimo oh19:13
i bet there's a git commit hook on the internet that i could steal19:13
samcv heh19:13
timotimo title case means every word is capitalized, right? except for maybe words like "the" or "a"?19:14
samcv luckily i have a (T)itlecase option in my update-changelog script19:14
oh no just the first one19:14
→ AlexDaniel` joined19:14
p6bannerbot set mode: +v19:14
samcv nine, can you give me an overview of the reason for the MAST changes?19:15
or someone else that knows19:15
https://github.com/MoarVM/MoarVM/commit/4fbaeee5f3f06425f715b8d9a913abe51c0c8cd719:15
timotimo so far we had a set of classes like MAST::Label, MAST::Operand, ... and instruction lists that hold them19:16
and that got fed as one big batch to mastcompiler.c19:16
now the QAST is turned into mbc as directly as possible19:17
this can greatly decrease the maximum memory used19:17
samcv: do you think that PR can go into master before this release?19:17
maybe i should at least stresstest it19:18
samcv link to PR19:18
since what i linked is already merged, so guessing you mean a differentn one19:19
→ zakharyas joined19:20
timotimo https://github.com/MoarVM/MoarVM/pull/100219:20
p6bannerbot set mode: +v19:21
samcv timotimo, i'm going to merge. let me know how the stresstest goes19:22
Geth ¦ MoarVM: c933b51669 | (Timo Paulssen)++ | 2 files19:23
¦ MoarVM: free spesh log entries after consuming them19:23
¦ MoarVM:19:23
¦ MoarVM: this prevents threads that do little to no19:23
¦ MoarVM: allocations but lots of spesh logging in an19:23
¦ MoarVM: otherwise idle program from growing to multiple19:23
¦ MoarVM: gigabytes of ram used solely for spesh log entries.19:23
¦ MoarVM:19:23
¦ MoarVM: This happened with rakudo's ThreadPoolScheduler19:23
¦ MoarVM: supervisor thread when a program with multithreading19:23
¦ MoarVM: or async workloads was left idle, like for example19:23
¦ MoarVM: a Cro app.19:23
¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/c933b5166919:23
¦ MoarVM: 88a776d1ab | (Samantha McVey)++ (committed using GitHub Web editor) | 2 files19:23
¦ MoarVM: Merge pull request #1002 from MoarVM/decrease_spesh_log_memory_growth19:23
¦ MoarVM:19:23
¦ MoarVM: free spesh log entries after consuming them19:23
¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/88a776d1ab19:23
timotimo i haven't updated rakudo recently, so a few individual tests fail, but the only symptom that could come from this is terrible memory corruption or very bad mishebaviour19:26
maybe nwc10 can run it with asan? :)19:26
jnthn We can perhaps think about removing src/mast/ stuff, given I guess we're not using it now we generate the bytecode directly19:28
(After the release, of course)19:28
timotimo i'm glad i got around to figuring out the memory growth thing19:29
i've gotta go grocery shopping, but maybe i can ssh in later and report on the stresstest19:30
jnthn timotimo++19:31
Geth ¦ MoarVM: 856c4dd880 | (Samantha McVey)++ | 2 files19:33
¦ MoarVM: Make MVM_string_utf16*_decode consistent and use char*19:33
¦ MoarVM:19:33
¦ MoarVM: Fixes a bunch of type warnings.19:33
¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/856c4dd88019:33
¦ MoarVM: 4228ad1b07 | (Samantha McVey)++ | tools/update-changelog.p619:33
¦ MoarVM: Use ANSI bold to make update-changelog.p6 easier to read19:33
¦ MoarVM:19:34
¦ MoarVM: Cosmetic changes.19:34
¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/4228ad1b0719:34
→ ilmari_ joined19:40
→ wictory[m] joined19:40
p6bannerbot set mode: +v19:40
p6bannerbot set mode: +v19:40
→ patrickb joined19:50
p6bannerbot set mode: +v19:51
← domidumont left19:52
← ilmari_ left19:56
← AlexDaniel` left19:56
← wictory[m] left19:56
→ travis-ci joined19:59
p6bannerbot set mode: +v19:59
travis-ci MoarVM build failed. Samantha McVey 'Use ANSI bold to make update-changelog.p6 easier to read19:59
https://travis-ci.org/MoarVM/MoarVM/builds/460426305 https://github.com/MoarVM/MoarVM/compare/88a776d1abf1...4228ad1b079819:59
← travis-ci left19:59
→ AlexDaniel` joined20:05
p6bannerbot set mode: +v20:05
dogbert11 hmm, a sudden SEGV20:11
==19776==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f73a87bc830 bp 0x7fff237b19b0 sp 0x7fff237b19b0 T0)20:13
#0 0x7f73a87bc82f in has_little_endian_bom src/strings/utf16.c:920:13
#1 0x7f73a87bc82f in MVM_string_utf16_decode src/strings/utf16.c:20720:13
samcv i made a typo in my change i think :)20:14
jnthn .oO( Somebody set up us the bom )20:14
dogbert11 pesky typos :)20:15
Geth ¦ MoarVM: c9f4863170 | (Samantha McVey)++ | src/strings/utf16.c20:17
¦ MoarVM: Fix typo causing segfault in utf16 code20:17
¦ MoarVM:20:17
¦ MoarVM: Fixes a typo in 856c4dd880b32514c033371d46d0951398f5c2f020:17
¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/c9f486317020:17
samcv dogbert11, should be fixed now20:17
dogbert11 cool20:18
dogbert11 rebuilds20:18
samcv jnthn, not sure where the best place to put arch conditional ldflags in Configure.pl20:20
dogbert11 works perfectly, samcv++20:20
samcv and make sure it works for cross as well20:20
jnthn samcv: I don't know; I'm pretty weak at build/configure stuff, so most of what we have today has been done by folks who understand that better :)20:21
samcv: This is for the AIX fix?20:22
samcv yeah20:30
something about memory allocation that was reported to me a while ago. that i remembered recently :)20:31
timotimo samcv: stresstest was OK :)20:31
samcv https://github.com/MoarVM/MoarVM/issues/100320:31
timotimo, nice!20:31
→ wictory[m] joined20:32
→ ilmari_ joined20:32
p6bannerbot set mode: +v20:32
p6bannerbot set mode: +v20:32
← lizmat left20:32
→ lizmat joined20:36
p6bannerbot set mode: +v20:36
→ travis-ci joined20:39
p6bannerbot set mode: +v20:39
travis-ci MoarVM build passed. Samantha McVey 'Fix typo causing segfault in utf16 code20:39
https://travis-ci.org/MoarVM/MoarVM/builds/460446200 https://github.com/MoarVM/MoarVM/compare/4228ad1b0798...c9f48631707220:39
← travis-ci left20:39
← zakharyas left20:43
Geth ¦ MoarVM: 60c16e6cbe | (Samantha McVey)++ | docs/ChangeLog21:03
¦ MoarVM: Update ChangeLog for release21:03
¦ MoarVM: review: https://github.com/MoarVM/MoarVM/commit/60c16e6cbe21:03
samcv ok. changelog is pushed. timotimo can you look at the "Other" section and make sure my MAST description is alright?21:04
timotimo samcv: i wouldn't say "MAST class changes", because MAST was pretty much completely removed21:05
a detail i should probably have mentioned is that what used to be in mastcompiler.c is now implemented in nqp code21:05
i gotta go for now, though. food ain't gonna cook itself!21:06
samcv kk21:08
timotimo "expose first entry time in profiler data" is in theret wice21:34
samcv oops21:34
timotimo otherwise i didn't see anything wrong21:36
samcv but say that MAST was mostly removed? and is now implemented in nqp code?21:36
so why does this consume less memory if it's using nqp and not c code? is that just because it's better for other reasons?21:37
timotimo21:50
← patrickb left22:16
timotimo yeah, it's better for other reasons23:15
sorry for answering so late23:15
good foods was had23:15
samcv: still around? i should be able to explain23:16
export EXPLAIN_FRONTEND=noninteractive23:56
so the previous way required all of the QAST to be transformed into a whole bunch of MAST::Blah objects, and there would be one object for each operation and one for each register, or lexical, or label, ...23:58
and since there's only a single op that takes all the MAST objects and turns them into a single mbc compunit, the entire MASTified QAST tree for the whole comp unit has to be in memory at that one point in time, along with the original QAST, because that doesn't get collected just yet23:58
(that single op is masttocu, i believe, which is how mastcompiler.c's code has been invoked so far)23:59

Logs Search ←Prev date Next date→ Channels Documentation