The Glowfic Constellation

[Marri]

GC.start triggers garbage collection, it doesn't release the memory. My concern with setrlimit is that all it seems to do is shut down the worker when you're out of memory, which would spare me needing to keep an eye on our memory usage and restart the server, but it would cause user errors instead of swap memory which seems suboptimal from a user perspective.

[DanielH]

Wait, it shuts down the worker? That seems like a big bug on Ruby’s part, you actually needing more than 256 MB for some requests, or a leak. What it does is make the OS say “no” to Ruby asking for more memory, which should just trigger a more aggressive GC and compaction, and only error out if it really can’t find enough memory for everything.

[Marri]

Ruby has settings to trigger both minor GC and major GC runs, yes.

[DanielH]

My point is that setrlimit shouldn’t cause a crash; it should trigger as aggressive a GC as necessary instead. The only reason calling GC.start is inadvisable in the middle of something major is that it stops the world, right? That’s still preferable to a crash in pretty much any application written in a language like Ruby. Either you really do need more than 256 MB in some requests and the only solution is to disallow big requests or do streaming (or switch to one worker and lose the ability to handle multiple requests at once), or you don’t and triggering the GC if necessary is better than an R14.

That’s why I’m surprised you’re actually seeing crashes. That shouldn’t happen.

What generally does happen when Heroku throws an R14? I thought it killed the worked but that doesn’t sound right given what you’ve said here.

[Marri]

Setrlimit is not the same as triggering a GC run, but I am way too tired and dealing-with-family-drama to keep explaining Ruby for now, so further questions will temporarily need to go to Google.

[jalapeno_dude]

Setrlimit is not the same as triggering a GC run, but I am way too tired and dealing-with-family-drama to keep explaining Ruby for now, so further questions will temporarily need to go to Google.

Relevant links are here for Ruby's Process::setrlimit() and here for unix's setrlimit(2).

[DanielH]

Oh, an R14 is just a warning and swap slowdown; I thought it meant Heroku killed the app.

I failed to find the other answers on Google, perhaps because I don’t know enough about Ruby, but it seems like it doesn’t crash the application but instead throws a NoMemoryError, which I suppose if it happened in the middle of Rails code instead of your own code could cause a crash. Before throwing the error it looks like it does trigger the GC, though; this makes setrlimit equivalent to triggering the GC when it thinks it’s necessary.

Simple experiment

---

Note: this experiment was done without rlimits, but I got similar results with.

Interestingly, in my simple test case (which is hampered by my not knowing Ruby, like, at all), calling GC.start immediately freed the memory back to the OS, as did making a large allocation that there wasn’t room for without garbage collection. This was on Ruby 2.3.

Code: Select all

irb(main):001:0> Process.pid
=> 29875
irb(main):002:0> conf.echo = false
irb(main):003:0> puts `ps --pid 29875 --format rss,vsz`
  RSS    VSZ
10496  50136
irb(main):004:0> myarray = Array.new(2**28)
irb(main):005:0> puts `ps --pid 29875 --format rss,vsz`
  RSS    VSZ
2105292 2147292
irb(main):006:0> myarray = nil
irb(main):007:0> puts `ps --pid 29875 --format rss,vsz`
  RSS    VSZ
2105292 2147292
irb(main):008:0> GC.start
irb(main):009:0> puts `ps --pid 29875 --format rss,vsz`
  RSS    VSZ
 8700  50136
irb(main):010:0> myarray = Array.new(2**29)
NoMemoryError: failed to allocate memory
        from (irb):10:in `initialize'
        from (irb):10:in `new'
        from (irb):10
        from /usr/bin/irb:11:in `<main>'
irb(main):011:0> puts `ps --pid 29875 --format rss,vsz`
  RSS    VSZ
 8888 115672
irb(main):012:0> GC.start
irb(main):013:0> puts `ps --pid 29875 --format rss,vsz`
  RSS    VSZ
 8888 115672
irb(main):014:0> myarray = Array.new(2**28)
irb(main):015:0> myarray = nil
irb(main):016:0> puts `ps --pid 29875 --format rss,vsz`
  RSS    VSZ
2105684 2212828
irb(main):017:0> myarray = Array.new(2**27)
irb(main):018:0> puts `ps --pid 29875 --format rss,vsz`
  RSS    VSZ
1057036 1164252

(I’m not sure why the virtual size grew after failing to allocate something; my guess is that a 50-megabyte library was first loaded in when the exception was thrown or something.)

If I had failed to explicitly set myarray to nil on line 15, the allocation on line 16 would have failed because it tries to allocate the new array before deallocating the old one.

Of course, with multiple processes in the list, it can’t say, “I’m out of memory; I’ll trigger a GC and also ask the other one to do a GC”, so that wouldn’t help as much as I was thinking earlier.

[Marri]

Updates from Throne:
- You can now mass add characters (that do not already have a template) to a template when adding or editing it, or mass remove a template's characters when editing it.
- Top header bar now has your user id in the galleries and characters link.
- Galleries in Add Existing are now sorted by name
- Favoriting a post no longer redirects you back to page=unread, just the correct page number

Updates from me:
- Test coverage is at 81.12%. More details for the curious:
- I realized that the code coverage number and favicon turn orange when you break 80%, so that's delightful. Way less angry red things!
- Our mailers are now 100% tested!
- Our libraries (the post scraper) and our async jobs (...we don't have any) and our presenters (JSON code for the forthcoming API / existing piecemeal Javscript calls) and our concerns (shared code, e.g. posts/replies) were already 100% tested.
- Our application helpers are at 87.14% tested and will remain so for the foreseeable future, as they are used almost entirely for rendering views and those I need to either come up with a sane way to test that in my existing RSpec framework (unlikely) or install proper Cucumber view tests (likely but time consuming).
- Our models are at 96.01% tested and are my current burndown target.
- Our controllers are 71.57% tested, not to mention a good 55% some percent of our code all by themselves, so they go last.

Also someone has been loading A Tale of Two Hells, the post on the Constellation with the most replies, in flat mode today. Twice. My memory consumption jumped 120MB the first time and the second time failed outright (it's... likely to just time out until I get streaming working? Which can't be until I have Rails 4 working? Which can't be until I have more tests? THERE'S A REASON I WROTE MORE TESTS TODAY UGH.) I need to look into precomputing the damn thing.

[jalapeno_dude]

Something seems to have gone wrong with templates pages--the Posts with Template Instances part is working fine, and view=list correctly displays only the characters that are members of the template, but view=icons now seems to display all of an author's characters instead of just the ones with that template, see e.g. here and here.

[Throne3d]

I think Marri's fixed it?