Of course, we know that know aesthetic endeavour actually has rules, and we can in theory do anything we want, but should we? It's one of those questions that inevitably comes to me when I run into a particularly incomprehensible piece of modern art, or inaccessible piece of poetry, or apparrently random piece of jazz-fusion guitar (I'm looking at you, Alan Holdsworth. Incidentally, what is it about the north of England, where he and John McLaughlin both hail from, and jazz-fusion? Even I've started playing wrong notes.)
Not that there's anything wrong with any of those things when they're done well, but is there perhaps a difference between recognising that you don't have to do something (such as playing something that vaguely relates to the melody) and deciding that you never should? And is merely being informed of modern approaches to the theory of something necessarily going to ensure that you do it well?
Still, that's enough ranting. What I actually wanted to look at was the idea that fiction, and especially particular genres, does have some rules if you want it to be recognisable as such. There are certain elements that have to be present to produce a Western, or an Urban Fantasy novel, or a Romance, in the same way that there are certain elements that identify a piece of music as blues, or rock, or jazz. (Or jazz fusion. Sorry fusioners, but "you must play squillions of wrong notes and it can't sound like pop" is as much a rule as any other)
I think that one of the most useful things you can do as a writer is to work out what the key elements of your genre or usual story-type are. What are the things that customarily show up in all pieces of this type? What are the things that never show up? What, if anything, is absolutely essential for the piece to be recognised as being in a particular genre? Do it now, if you like. Take your favourite genre and write down the essentials.
Having done that, you're in a position to ask why they're essential. What are they there to do? Are they actually essential? Are they something that you can legitimately change or parody, or would doing so change the whole scope of what you write? I'm not saying that you have to change anything. There's fun to be had using the traditional elements of a style around a good, strong story and some beautiful writing. But at the very least it reminds you that most of these elements are there to achieve something, rather than simply elements that you have to include because everyone does. If you'll permit me yet another musical analogy, it's the same as looking through all those blues licks people learn by heart and changing a few notes here and there to suit yourself. The result is not unlistenable oddness, or bland repetition, but is instead the reinvigouration of a sound that you know works.