The Glowfic Constellation

[pedromvilar]

One (reasonably rare) use case for per_page = all is when I want to edit all of a character's posts past a certain point (for instance, i'm thinking about forking Jack Slash for one of my threads with Wizard of Woah) and going through pages is a bore. There is still the use case of "reading a whole thread while spending a long time without internet because of e.g. subway or going to visit your grandmother". I've been using the search feature a lot lately but it's still somewhat less good than a ctrl+f everywhere but that's relatively minor and I think overall the search post feature is better.

[Timepoof]

The use case described in the middle sentence is a very important use case.

[jalapeno_dude]

In other news, for the curious, this is our memory consumption:


Starts at my last deploy (cause memory resets at a deploy) and goes until we hit max. The three jumps are two per_page=all and a view=flat :/

Why is memory usage going up permanently because of this? I understand that the memory usage needs to go up while the page is being created/sent to the user/whatever, but why can't it all get deleted/garbage collected right after that? (My experience is with Python and C/C++, so my guess is that this is something Ruby or server-specific, but I'm curious what exactly the obstacle is.)

[Throne3d]

Part of it is that it requests more memory from the OS and then fails to release it afterwards, for … probably complicated reasons that I do not in fact know.

[DanielH]

I still want a wall to get the parts of the thread I have not yet read available for offline use. Something like page=unread&per_page=rest. This is often small enough that it would be a reasonably-sized request. I avoid per_page=all even when that’s easier than calculating what parameters I actually need for that, but it’s when I most want that.

Was this graph made from within Ruby? I thought it only freed stuff to the OS slowly, but the OS numbers are the relevant ones. Is there no way to force it to do the big cleanups it apparently does after the big jumps more often?

Would Ruby handle its memory better if you used ulimit so it knew what the limit was?

[Moriwen]

I try very hard to avoid per_page=all, especially on long threads, but occasionally I know I'm going to lack internet for a long period of time and would like to use that time to finish reading a thread. I still only do it very rarely, but it seems like a potentially common use-case.

If there's some better way to do that, I'll be more than happy to switch to it.

[Ezra]

I don't feel like I get the full picture of why Ruby does what it does with memory. Here's what I think I understand:

• Ruby uses garbage collection to try to stay within the amount of memory it has from the OS
• Sometimes it gets a request that it can't fit into that memory, so it requests another chunk from the OS
• It expects that a request that happens once can happen again, so it holds on to that memory to reuse it next time

What I don't get is why, when it gets another big request, it doesn't just reuse the big chunk of memory it grabbed last time? That was the whole point of keeping it, right?

Would it be meaningful to graph, within that graph of how much memory Ruby has over time, how much of that memory is "in use" vs waiting for Ruby to need it again?

[Marri]

The graph was from my Heroku dashboard. Ezra, your explanation is correct as far as I understand it, but I don't know if it's failing to reuse the big chunk or if it's legitimately just needing even more. (The first page=all was 2k replies and the second was 3k, for example; it's possible it legitimately needed the whole block.) Also please remember that I am running a single server*, but that one server is running two Ruby processes, so it's also possible that one large request went to process A and the second went to process B.

I'll see if I can figure out "load remaining unread" as an option, though I will probably have to disable pagination in some fashion (since what if you haven't read the last 3 of a 100 page thread, I can certainly put in the < 1..4 > links but what 'page' does it think you're on?)

*I am currently running us a Heroku Hobby server which is $7/mo. Upgrading to a Standard Professional server, the next level up, increases the cost to $25/mo, gets us basically nothing I care about and certainly nothing that would affect the server specs, but is required to be able to add additional servers, which of course also cost $25/mo. So to have more than one server, my costs go from $7/mo to $50/mo. Uh. Tip your coder?

[DanielH]

Why do you have two Ruby processes? Does Ruby or Rails not support threads nicely?

I imagine that Ruby doesn’t try very hard to keep within what it already has. I bet that if it did then there wouldn't be a problem.

How big is the whole database? Assuming file distribution were a solved problem (which of course it isn’t if the database is large enough), would you be willing to provide a dump of the non-private threads so that a test setup could have large quantities of data?

[Marri]

I believe my Procfile specifies that we run 2 workers with 1 thread, so that I can serve more than one request at once. While Rails can do threadsafe, it's not the default and it gets messier with every gem you add that may or may not be threadsafe itself.

Ruby's GC and heap management can be tuned, but I don't know memory management well enough to do it myself, and when I tried the automated one the problem got significantly worse.

Our database is currently 400MB. I'd be willing to update the seed file with production data, but I'd have to also sanitize e.g. user emails and honestly I don't care how good my hashing is I'm sanitizing the passwords too, etc etc.