The Glowfic Constellation

[DanielH]

Home wifi, Google public DNS, certificate error just now as a fairy was threatening Minor across a page break. The certificate presented was for *.lexile.com, which I had never heard of before now.

Edit: Fixed just a few minutes later

[Ezra]

Looking up Lexile, I don't know what they could possibly have to do with anything. Is somebody stealing their certificate?

[DanielH]

My guess is that Heroku’s at fault somehow. Is it possible to tell if Lexile is hosted there?

[Marri]

Yeah, I saw the same thing. I agree that this is probably a weird blip thing on a host server's fault, but if people keep seeing it I'll poke more, since I certainly haven't changed anything.

Also there was a brief (couple minutes) outage last night when the Heroku databases refused to connect; sorry about that, no idea what they did, I restarted the app and that seemed to fix it. Which was vaguely sad because restarting makes my memory use reset and I was monitoring it.

On which note: I made it a good 14ish hours yesterday staying below 450MB, then had to reset for the database; remaining large jumps seem to so far all be ?per_page=1000 and view=flat requests. Which will eventually become unfixable and I'll have to turn them off, but for now I'll start on a "search within thread" feature so at least I can minimize the use case.

[pedromvilar]

What exactly about these requests consumes so much memory?

[DanielH]

And does Ruby not have a way to explicitly mark stuff unused afterwards for such requests?

[Marri]

Okay I swear I'm not trying to sound snarky or anything but text tone is hard, sooo: each object I create uses memory, and every reply I load is an object (and used to be 4, before I fixed it) so the more replies I load the more objects I make and the more memory I use.

There's a difference between allocated memory and free memory; Ruby can free the memory fine* to reuse but the memory is still allocated to Ruby. So Ruby says "I need this much total memory right now" and then after that it can free it up for future requests but never gives it back to non-requests, if that makes sense?

*Well, it's worse at it after our 2.1 upgrade, that was the whole problem, but it does in fact do garbage collection

[pedromvilar]

Would it not be possible to, rather than reloading all of a thread's posts whenever someone clicks view=flat, manufacture a persistent "flat-view object" per thread that's just a string with the html code, and whenever it's accessed the code verifies if the thread's been changed since that object was manufactured and, if so and the changes are new posts, merely append the new posts to that string and then present it?

Edit: although I suppose that might in and of itself just consume way too much memory or space on the db or something

[Throne3d]

That sounds rather like caching! It does in fact consume too much memory (in Redis, the cache store), yes, which is why Marri turned off caching for logged out users.

[Marri]

I have kicked around the idea of storing view=flat precomputed in the database, yup; problem is that storing objects longer than 65k characters can get fussy (doable, but fussy) and also we pay by row still. Also also, for that kind of caching, edits are a disaster. More also I'd just append new posts at time of write rather than time of access.

Additional also, view=flat was originally intended (and used) as a way to download and backup your posts, not as a way to just load the whole damn thing at once. So "you click this and I compute it in the background and email it to you" is also an option, though it does leave memory questions for my async servers.

Further also I feel like "search within this thread" is something I should have anyway.

I'm done saying also now I swear.