[nem-en] Re: [nem-feedback] Visual Studio 2005 VSIP package

[NoiseEHC]

> I guess it would be nice if you had an svn account for our repository.
> Please send me the output of https://nemerle.org/svnaccount.php form
> privately, so I will set it up :)
>
>   
Already tried the write permission. I am hoping that no harm was made.
The patch was incomplete (missing files) but it is updated now...
>> Currently brace matching works but it is not perfect.
>> The problem is I think that LexerColorizer does not create the same tokens
>> as the PreParser.
>>     
>
> That is true, LexerColorizer for ment to be a faster version of normal
> LexerBase and it probably outputs a little bit different tokens.
> And also PreParser is a different story - it changes the token input
> from flat stream, like
>
> 'class' 'A' '{' 'public' 'foo' '(' ')' '{' '}' '}'
> into tree
>
> LooseGroup ('class', 'A', BracesGroup ('public', 'foo', RoundGroup ()))
>   
It took a little time while I realized what it does. Probably we should 
have some comments
in the source. For example what is the "monkey string"?
> You could just use LexerString for analysing flat stream of tokens,
> but it won't make much sense - first you would need to do the same
> work as PreParser to match braces, second it is PreParser which is
> loading and analysing new keywords from macros.
>
> So I think it is good to stick with PreParser, which is doing the
> initial preprocessing.
>   
> PreParser already handle all kinds of braces, also <[ ]>  is changed
> into QuoteGroup.
>
> The interesting part is if we should abandon LexerColorizer completely
> and use LexerString/PreParser also for highlighting or try to use
> LexerColorizer and PreParser at the same time.
> This question is mainly about performance - probably PreParser is too
> slow to use it for syntax highlighting, though only there we have all
> information about keywords. What is your  experience here?
>   
The problem is that for example for the "}" token the LexerString gives 
me a token starting from
the beginning of the line to the closing brace. While the PreParser 
gives me a BraceGroup.
 From this group I can calculate the span of the closing brace (assume 
LastChar-2 to LastChar-2).
Now VS shows the matching opening brace but it behaves slightly 
differently at the left and right
side of that brace what is a bit lame. It is not documented everywhere 
but I think that the cause is
the difference between token spans because the LexerString does not 
handle Whitespace.
Another problem is that it does not handle <[ ]> at all.
So we should fix LexerString or abandon it.
A question popped out: shall we colorize quoted expressions differently?
>> +                               // We cannot optimize by ignoring groups with wrong location early since
>> +                               // for example loosegroup contains only the first token's location.
>> +                               if(token is NC.Token.RoundGroup || token is NC.Token.BracesGroup || token is NC.Token.SquareGroup || token is NC.Token.QuoteGroup) {
>>     
>
> You should be able to look at for example BracesGroup's Location.Line
> and Location.EndLine
>
> This will give you the way to skip the whole matching braces group.
>
>   
It was an old comment since then I realized that we shall not look at 
positions but only add groups to AuthoringSink.