Daily Reports Chatter

[pedromvilar]

Holy shit, forty-five threads, what the fuck is going on

[Kappa]

GLOWFIC (ﾉ◕ヮ◕)ﾉ*:･ﾟ✧

[Throne3d]

I'm now marking threads that are put on hiatus! With the color #9A534D. (Kappa chose it – thanks, Kappa.)

Example (purposefully altered link):
███ ███ Geometry, Rachel and Sadde in the City of Angles

[Throne3d]

Added "(last post by [username])" to things not updated that day (only applies to Constellation threads).

Example:
███ ███ Lost in reflection, carmines and lotuses in MO (last post by Throne3d)

[Throne3d]

… Apparently I did a thing a while ago, almost as soon as I added the flags, that broke them inside the spoiler boxes? (This was due to how I processed it – it was reprocessing the processed thing and getting blank strings for the flags, hence no flags the second time chapters appeared.)

This has now been fixed, since it wasn't intended behavior, so that the spoiler boxes now contain the flags! If you want me to remove them from there for some reason (which… I don't think people would, since they're used for distinguishing, but nobody's complained about them being missing so perhaps?) feel free to ask and we can have an informal poll or something.

(Also thanks to rockeye for pointing out that I missed a start-bracket for the 'color' BB-code for Teah today. This has been fixed.)

[Throne3d]

I won't be at a computer for most of tomorrow so the report won't be available for most of (maybe all of) the day.

[Throne3d]

… Something broke somehow in how it's scraping, and this has caused it to not update various threads in the listing. I just re-ran it, forcing it to reprocess threads, and it seems to have updated properly now?

I was messing around with some things yesterday (and the day before) which might have caused this, but I'm not really sure why, because it should have either failed to notice any changes at all (which it didn't, because it re-downloaded the right pages) or done it all (which it didn't, because it didn't re-process the right pages).

I'll keep watch on it over the next few days.

[Throne3d]

I'm pretty sure I've found my issue! I did a thing for the report generator so that it process stuff quicker (so it takes about 7 minutes instead of about 30 – apparently reprocessing is slow, as is repeated serialization, and I haven't yet worked out how to mark objects 'dirty' and in need of re-serialization processing neatly! Eugh). This means that it only saves the chapters at the end of downloading all the updated pages.

Issue was: it was "getting" the updated page and flagging the chapter for reprocessing, but it didn't save the flag on the chapter until the end of the whole process. So if it was interrupted part way through (due to an error or by me explicitly stopping it), it would have downloaded the updated pages but not have marked it for reprocessing, so the next time I run it it doesn't think anything has changed. (Yes, design flaw. I wasn't sure of a better way to do it.)

I have temporarily fixed this by having it save on an error if anything's changed, and I think I will fix it slightly better by appending a copy of the data to each chapter in the serialization so it can, when processing, notice that the last page is not in fact what it expects! (Because the page was altered and the chapter list doesn't contain information that this should have happened.) This will unfortunately slow it down some, but it shouldn't be too much, since this will just be a file reading and comparison.

It shouldn't break like this in the future, though, unless I explicitly kill the process, but now I know what the issue is I'll just try to remember to reprocess everything if I do that.

[Throne3d]

It broke like that again! Because the process got killed when the whole virtualbox crashed. Reprocessing, so if you got an email that there was a reply to the daily report thread and can no longer find it, it'll be because I deleted it.

Edit: Reprocessed.

[Marri]

Three days in a row with no Dreamwidth posts... SOON IT WILL ALL BE MINE.