Alicornutopia

How do people figure out novel rituals and what ingredients are right for them? What if they get it wrong?

There isn't a whole lot of distinction between discovering and inventing rituals, and instinct can go a long way. That said, in the past most witches were gifted 'cause other people couldn't get a lot of mileage out of magic, so they mostly went with trial and error and relied on the stabilising effects of their gifts (and often covens) to avoid backlash. If they fucked up too much or tried to do a ritual that was too big or didn't have gifts or stuff like that, backlash from a wrong ritual could kill the witch out of unexpected/uncontrolled magical maelstroms. Most of the time, though, wrong rituals range from not doing anything to fucking up in sometimes destructive but not often lethal ways, in chaotic but thematically appropriate ways.

Nowadays people have more knowledge about the most common magical properties of different ingredients and their interactions with each other and different levels of activity of the magic field, there are safety procedures that can be added onto an experimental ritual to isolate potential sources of trouble, and papers are written and published on journals and arxiv about that kind of stuff, good practices, and the interactions of the mental components of the spell. Inventing new rituals is basically analogous to building a machine: there are multiple ways to build a car and pieces that go into it, if you study a lot of mechanical engineering you start getting a feel for it and standing on the growing pile of shared knowledge is absolutely necessary for truly novel work.