[nem-en] Manager class

[NoiseEHC]

Inlined
> This is a bit strange, since LexerString and LexerColorizer allow very
> similar functionality - they just hold a string and position in it. So
> in general if you assign previous position to LexerString, it should
> correctly "back in time", though there are several fields, which
> actually hold the state of lexing - they are used in preprocessor
> directives parsing.
That is what I tried to do some time ago but gave up. The problem is that
LexerString holds a little bit more state than a position and I simply 
could
not figure out which ones are important (some are seems just to hold info
about some recursive stacks) since it is a hand written parser and I am
not qualified for this kind of code... :(
> It is quite interesting how MS implemented this stuff in C# bindings,
> since VS seems to behave quite intelligent with respect to
> preprocessor directives - and in general this requite more than a
> simple int state.
>
If anybody is interested (maybe Vlad and Andrei Faber) VS work like this:
1. It is a COM based stuff (not .NET)
2. The C# language service is written by using the Babel framework (C++ 
and COM)
3. There is a .NET framework I am using but it is a little bit sparsely 
documented
so I use the IronPython integration (which is a sample) as a base. 
Currently in sync
with the 2006 April CTP.
4. VS every time it displays a line calls the language service to lex 
that line to get
the colors for display so it has to be fast.
5. Every other time (brace matching, code completion, etc) it calls the 
language service
to do a full parse of the source then uses the returned data for 
highlighting matching
braces etc. It is up to the language service developer to decide whether 
the parsing
can be done a faster way if the full parse three is not needed.
6. It in default saves an int for every line, to give it to the lexer as 
state but it can be
disabled and then it would be the developer's job to maintain lexer 
state in some
efficient way. That is what I have to do either way, since full syntax 
highlighting requires
at least keeping track of open namespaces (eg we do not want to 
highlight return without
"using Nemerle.Imperative;"). The problem is that it is hard to do (not 
in the IronPython
source and not over documented).
7. Another possibility popped into my mind: always preparse the whole 
file, cache it then
use the data for the lexer. I need some time to think about the 
feasibility...

In the meantime it would be nice if you could implement 
SaveState/RestoreState methods
in LexerString... It would be far easier for you to determine the amount 
of savable data. :)
(I know, I tried...)

> My original idea for LexerColorizer was that it should be fast and it
> should colorize the default code correctly (e.g. assuming that
> standard code macros are loaded, not considering preprocessor
> directives, etc.). Then in the less frequent parsing passes we could
> use LexerString to additionally colorize some new keywords or
> preprocessor stuff. But the problem is that if we want for example
> matching of parenthesis we must use LexerString anyway.
> So it is not clear for me if maintaining two separate implementations
> of Lexer makes any sense... but maybe yes :)
>
Of course it would be cool if we could get rid of LexerColorizer but
for me it is a too big task. I simply cannot comprehend the compiler's
inner structure...
>
> You probably need to initialize the ManagerClass object. Now as we
> have the more object-oriented structure, it should be reasonably easy
> to subclass ManagerClass (or use the Nemerle.Completion.Engine) and
> make it perform only some chosen set of steps (like only initialize
> the library references and then perform lexing) on the Run method.
>
> I'm looking forward for defining some more methods in ManagerClass,
> which could be overridden in subclasses, so we can easily enable /
> disable some compilation steps or simply jump to single step.
> So the idea is to always call some methods defined in ManagerClass and
> avoid directly using other compiler's API.
>
>
Currently I am looking at the completion engine since it seems that it can
use that cursed Manager class. There shall be some reason why Alejandro 
did not
use some derived stuff (can be a historical reason), so I will not dare 
to move that
way before it is cleaned up. I mean when Alejandro will be able to 
remove that
code duplication, I will know that it would be usable for mere mortals 
as me... :)

A little suggestion for your refactoring work: it would be cool if you 
could remove all
the command line baggage from the Manager (compiler) classes and use 
them only in
ncc.exe. I mean: progressbar, colors, etc. So it should work that all 
the text IO handling
would happen interfaces and ncc.exe would use some sink to create the 
text from that
which you put to the console. That way for example msbuild should not 
parse the formatted
output not to mention the completion and future lexer/parser...