Daily Reports Chatter
-
MaggieoftheOwls
- Posts: 733
- Joined: Sun Apr 05, 2015 7:39 pm
- Pronouns: she/her/hers
-
PlainDealingVillain
- Posts: 622
- Joined: Tue Apr 01, 2014 10:15 pm
- Pronouns: he etc.
Re: Daily Reports Chatter
More hugs!
And I definitely know the feeling. Most times I move I feel my eating habits screwed up for weeks and half-starve myself because it's too hard to make decisions.
And I definitely know the feeling. Most times I move I feel my eating habits screwed up for weeks and half-starve myself because it's too hard to make decisions.
-
atheistcanuck
- Posts: 198
- Joined: Mon May 04, 2015 9:18 am
- Pronouns: she/her
- Location: LaLa Land
Re: Daily Reports Chatter
<hugs> Adelene
Re: Daily Reports Chatter
Hugs!
In advance: Excuse the long post. Apparently my tendency to ramble in the IRC has now extended over here. (Apparently I am also summarizing my rambles now, woo.)
I've done a report for the past ~a month (in regular format, not summary format).
Stuff about my scraper
I updated my scraper to output it neatly like that, so I should now be able to do that relatively easily, so long as it's clear when new threads are announced, who's participating (both authors and the characters), who's the world (if applicable), the name of the world, that sort of thing.
For code-savvy people who desire to see the code, it can be found here, as usual (same code as the EPUB generator, uses a different command: "repdo_report" instead of "do_[sandbox/constellation/theotrics]" and "output_epub_[theotrics, ...]"). It's written in Ruby, uses a few other things such as ActiveSupport for things (I'm not actually sure if some of that's even necessary), bits of the code are ugly but I'm attempting to clean them up and make them neater, seems to function properly, has a neat README for how to get started, even has an issue tracker for where I've done things badly or need to improve things.
Discretion advised warnings: frustrating
Posts that are marked "discretion advised" (or "NSFW"?) on dreamwidth will not be automatically processed by the scraper, because I can't work out how to make it use cookies properly so that it can view those posts, and it doesn't seem to be as simple as just going to the 'right' URL that isn't a discretion advised one – the cookie data seems to be gibberish and also journal-specific, as well as browser-specific (?) so I don't think I can just copy them over to Ruby and somehow get it to use those when it fetches it from the applicable journal.
This means I had to do the entry for don't touch me (spoiler alert, link to the thread at yesterday's earliest update) manually, so I might have placed it in the wrong position (I don't think I did) or failed to include it somewhere (just "DW Only" and the latest update, I think?). I understand that people use NSFW/discretion advised for valid reasons, so I'm hesitant to just request that you don't use it, but… it would make my life easier to scrape things if you didn't use it? (I mean, you can still do a huge warning at the top of the post, which should be approximately as effective, I think, but I get it if you'd prefer to leave it marked 'NSFW'.)
General "please let me know if it's wrong"
Let me know if anything is incorrect, such as me failing to mark a thread as complete or something, or an incorrect / missing moiety or a typo. If I miss any threads, it'd be good if you could also point that out – whether they're new and I failed to add them to the list, or they're old but I failed to notice they're updating.
Wow
Also: wow. 29 threads updated yesterday? This actually made me go manually check each of the threads it listed as updated yesterday to make sure it wasn't doing something stupid, but apparently there were indeed 29 threads.
Overall: I hope you guys find the return of the daily reports useful? It shouldn't take me too much time to do each one – I don't know how long it took Adelene to do each one before? I hope you don't mind that I've kinda stolen this job from you? – and they will hopefully continue.
(And if I mysteriously disappear for some reason, which I don't expect, someone else who's Ruby-savvy can probably set it up and run it themselves; it's not too much effort and it's mostly autonomous.)
In advance: Excuse the long post. Apparently my tendency to ramble in the IRC has now extended over here. (Apparently I am also summarizing my rambles now, woo.)
I've done a report for the past ~a month (in regular format, not summary format).
Stuff about my scraper
I updated my scraper to output it neatly like that, so I should now be able to do that relatively easily, so long as it's clear when new threads are announced, who's participating (both authors and the characters), who's the world (if applicable), the name of the world, that sort of thing.
For code-savvy people who desire to see the code, it can be found here, as usual (same code as the EPUB generator, uses a different command: "repdo_report" instead of "do_[sandbox/constellation/theotrics]" and "output_epub_[theotrics, ...]"). It's written in Ruby, uses a few other things such as ActiveSupport for things (I'm not actually sure if some of that's even necessary), bits of the code are ugly but I'm attempting to clean them up and make them neater, seems to function properly, has a neat README for how to get started, even has an issue tracker for where I've done things badly or need to improve things.
Discretion advised warnings: frustrating
Posts that are marked "discretion advised" (or "NSFW"?) on dreamwidth will not be automatically processed by the scraper, because I can't work out how to make it use cookies properly so that it can view those posts, and it doesn't seem to be as simple as just going to the 'right' URL that isn't a discretion advised one – the cookie data seems to be gibberish and also journal-specific, as well as browser-specific (?) so I don't think I can just copy them over to Ruby and somehow get it to use those when it fetches it from the applicable journal.
This means I had to do the entry for don't touch me (spoiler alert, link to the thread at yesterday's earliest update) manually, so I might have placed it in the wrong position (I don't think I did) or failed to include it somewhere (just "DW Only" and the latest update, I think?). I understand that people use NSFW/discretion advised for valid reasons, so I'm hesitant to just request that you don't use it, but… it would make my life easier to scrape things if you didn't use it? (I mean, you can still do a huge warning at the top of the post, which should be approximately as effective, I think, but I get it if you'd prefer to leave it marked 'NSFW'.)
General "please let me know if it's wrong"
Let me know if anything is incorrect, such as me failing to mark a thread as complete or something, or an incorrect / missing moiety or a typo. If I miss any threads, it'd be good if you could also point that out – whether they're new and I failed to add them to the list, or they're old but I failed to notice they're updating.
Wow
Also: wow. 29 threads updated yesterday? This actually made me go manually check each of the threads it listed as updated yesterday to make sure it wasn't doing something stupid, but apparently there were indeed 29 threads.
Overall: I hope you guys find the return of the daily reports useful? It shouldn't take me too much time to do each one – I don't know how long it took Adelene to do each one before? I hope you don't mind that I've kinda stolen this job from you? – and they will hopefully continue.
(And if I mysteriously disappear for some reason, which I don't expect, someone else who's Ruby-savvy can probably set it up and run it themselves; it's not too much effort and it's mostly autonomous.)
-
Alicorn
- Site Admin
- Posts: 4226
- Joined: Fri Mar 21, 2014 4:44 pm
- Pronouns: She/her/hers
- Location: The Belltower
- Contact:
Re: Daily Reports Chatter
Reports are useful! I think it would be a little effort to make sure a single continuity was next to itself - Odette and Illia with Elves, f'rinstance.
Re: Daily Reports Chatter
They're currently in order of the first time they were updated on that day (taking a shortcut, for example, at the top of the list, was first updated here at "May 28, 2016 6:19 AM" GMT, and Let Me Tell You What I Wish I'd Known, at the bottom of the list, was updated at "May 27, 2016 1:37 PM" GMT).Alicorn wrote:Reports are useful! I think it would be a little effort to make sure a single continuity was next to itself - Odette and Illia with Elves, f'rinstance.
I could put the single continuities next to each other in another spoiler box, ordered alphabetically by continuity title and then reverse-chronologically by first-update-on-day? I was just doing it this way because that's how Adelene did it previously.
As in, "Apricum" comes before "Opalescence" comes before "Unknown Title", so you'd get something like this (if all six posts were updated on the same day):
Continuities
Re: Daily Reports Chatter
I updated the report with a continuities box, and I've made the scraper able to produce it too.
Looks like this:
Looks like this:
Continuities