Refactoring / AST rewriter in Pydev

Right now, there's a bit of quietness in the Pydev realm... I'm working on the current refactoring engine -- which is right now based on the PEPTIC refactoring engine, BRM and the previous Pydev Extension code.

The idea is to stop using BRM altogether and use the others to make up an improved refactoring engine (as that's a big task, I'm avoiding pushing it until it's not stable).

I decided that the first step to do that would be getting the latest PEPTIC code and making it work with the current tests (I had already done some changes to the current Pydev version, so, it diverged a bit).

So, while starting to fix those tests, I realized that its AST rewriter did not properly support the Python 3.0 grammar, so, this is where I am right now, making a decent rewriter. There are 2 AST-based rewriters right now, one that needs the parsing tokens together with the AST and one that only needs the AST -- needles to say none works the way I think they should, so, I decided to go on a different approach, creating a new rewriter.

This rewriter will work with the tokens if they are available (which will make it able to maintain the comments in the correct places) and will generate tokens if they're not available.

I've decided on the following approach: instead of just parsing and writing things to a string in a single step (as the current rewriters do), it'll traverse the structure and fill an auxiliary document structure with the meta information found (i.e.: token requests at a given position and the actual tokens found -- if available -- plus information on the AST, such as statement start and end, nodes, etc), and in a 2nd step, it will generate the actual python code by traversing this document structure. A problem in the previous approaches is that sometimes it found a token and then saw that a comment should be added before it, or a token would not be in the expected AST (and it was really difficult to go back to fix it) -- so, I hope this won't be a problem anymore :)

Unfortunately, as I said, this is a big task, so, I don't think I'll be able to make a release this month -- but looking on the other way, the current release seems pretty stable :)