Hugs!
In advance: Excuse the long post. Apparently my tendency to ramble in the IRC has now extended over here. (Apparently I am also summarizing my rambles now, woo.)
I've done
a report for the past ~a month (in regular format, not summary format).
Stuff about my scraper
I updated my scraper to output it neatly like that, so I should now be able to do that relatively easily, so long as it's clear when new threads are announced, who's participating (both authors and the characters), who's the world (if applicable), the name of the world, that sort of thing.
For code-savvy people who desire to see the code, it can be found
here, as usual (same code as the EPUB generator, uses a different command: "repdo_report" instead of "do_[sandbox/constellation/theotrics]" and "output_epub_[theotrics, ...]"). It's written in Ruby, uses a few other things such as ActiveSupport for things (I'm not actually sure if some of that's even necessary), bits of the code are ugly but I'm attempting to clean them up and make them neater, seems to function properly, has a neat README for how to get started, even has an
issue tracker for where I've done things badly or need to improve things.
Discretion advised warnings: frustrating
Posts that are marked "discretion advised" (or "NSFW"?) on dreamwidth will not be automatically processed by the scraper, because I can't work out how to make it use cookies properly so that it can view those posts, and it doesn't seem to be as simple as just going to the 'right' URL that isn't a discretion advised one – the cookie data seems to be gibberish and also journal-specific, as well as browser-specific (?) so I don't think I can just copy them over to Ruby and somehow get it to use those when it fetches it from the applicable journal.
This means I had to do the entry for
don't touch me (
spoiler alert, link to the thread at yesterday's earliest update) manually, so I might have placed it in the wrong position (I don't think I did) or failed to include it somewhere (just "DW Only" and the latest update, I think?). I understand that people use NSFW/discretion advised for valid reasons, so I'm hesitant to just request that you don't use it, but… it would make my life easier to scrape things if you didn't use it? (I mean, you can still do a huge warning at the top of the post, which should be approximately as effective, I think, but I get it if you'd prefer to leave it marked 'NSFW'.)
General "please let me know if it's wrong"
Let me know if anything is incorrect, such as me failing to mark a thread as complete or something, or an incorrect / missing moiety or a typo. If I miss any threads, it'd be good if you could also point that out – whether they're new and I failed to add them to the list, or they're old but I failed to notice they're updating.
Wow
Also: wow. 29 threads updated yesterday? This actually made me go manually check each of the threads it listed as updated yesterday to make sure it wasn't doing something stupid, but apparently there
were indeed 29 threads.
Overall: I hope you guys find the return of the daily reports useful? It shouldn't take me too much time to do each one – I don't know how long it took Adelene to do each one before? I hope you don't mind that I've kinda stolen this job from you? – and they will hopefully continue.
(And if I mysteriously disappear for some reason, which I don't expect, someone else who's Ruby-savvy can probably set it up and run it themselves; it's not too much effort and it's mostly autonomous.)