[nem-en] Re: [nem-feedback] Visual Studio 2005 VSIP package

[Kamil Skalski]

06-02-14, NoiseEHC <NoiseEHC at freemail.hu> napisał(a):
> Ok, here is a patch (I hope it works, using TortoiseSVN)

I guess it would be nice if you had an svn account for our repository.
Please send me the output of https://nemerle.org/svnaccount.php form
privately, so I will set it up :)

>
> Currently brace matching works but it is not perfect.
> The problem is I think that LexerColorizer does not create the same tokens
> as the PreParser.

That is true, LexerColorizer for ment to be a faster version of normal
LexerBase and it probably outputs a little bit different tokens.
And also PreParser is a different story - it changes the token input
from flat stream, like

'class' 'A' '{' 'public' 'foo' '(' ')' '{' '}' '}'
into tree

LooseGroup ('class', 'A', BracesGroup ('public', 'foo', RoundGroup ()))

You could just use LexerString for analysing flat stream of tokens,
but it won't make much sense - first you would need to do the same
work as PreParser to match braces, second it is PreParser which is
loading and analysing new keywords from macros.

So I think it is good to stick with PreParser, which is doing the
initial preprocessing.


> It should have to hande for example <[ ]> and
> whitespace (it precedes tokens with the whitespace before them).

PreParser already handle all kinds of braces, also <[ ]>  is changed
into QuoteGroup.

The interesting part is if we should abandon LexerColorizer completely
and use LexerString/PreParser also for highlighting or try to use
LexerColorizer and PreParser at the same time.
This question is mainly about performance - probably PreParser is too
slow to use it for syntax highlighting, though only there we have all
information about keywords. What is your  experience here?

>
> I tried to modify the Nemerle source but with no success. Because it takes
> minutes to create a new nemerle.compiler.dll, I decided to create an
> MSBuild task which can build it in one step. However it encounters an
> internal compiler error and the msbuild task throws an exception (will
> fix that). Kamil, can the compiled compiler (9.2) compile the nemerle
> source in one step? I am really not interested in bootstrapping it.

Yes, compiler can be builded in one step, though you should verify
that the new compiler actually can compile something ;)

Here is the output of stage3 from bootstrap (use ./configure
--disable-quiet to produce it):

out.stage2/ncc.exe -no-stdlib -r:mscorlib -r:System -dowarn:10006
-pedantic-lexer -def:_stage3 -stats -bar+ -def:RUNTIME_MS 
-target-library -out:out.stage3/Nemerle.dll 
../lib/internal-numbered.n ../lib/internal.n ../lib/core.n
../lib/macros.n ../lib/getopt.n ../lib/hashtable.n ../lib/icloneable.n
../lib/icollection.n ../lib/input.n ../lib/linkedlist.n ../lib/list.n
../lib/listenumerator.n ../lib/option.n ../lib/queue.n ../lib/stack.n
../lib/tree.n ../lib/idictionaryenumerator.n ../lib/idictionary.n
../lib/heap.n ../lib/narray.n ../lib/nstring.n ../lib/pair.n
../lib/set.n ../lib/concurrency.n ../lib/AssemblyInfo.n
../lib/LazyValue.n ../lib/PipeReader.n ../lib/PipeWriter.n
../lib/oldapi.n

out.stage2/ncc.exe -no-stdlib -r:mscorlib -r:System -dowarn:10006
-pedantic-lexer -def:_stage3 -stats -bar+ -def:RUNTIME_MS 
-target-library -out:out.stage3/Nemerle.Compiler.dll \
        -reference:out.stage3/Nemerle.dll \
         CompilationOptions.n codedom/NemerleCodeCompiler.n
codedom/NemerleCodeGenerator.n codedom/NemerleCodeProvider.n
external/Codec.n external/InternalTypes.n external/LibrariesLoader.n
generation/HierarchyEmitter.n generation/ILEmitter.n
generation/ImplementsWrapperMaker.n generation/MatchingCompiler.n
generation/DecisionTreeCompiler.n generation/Typer3.n
generation/Typer4.n hierarchy/ExtensionMethod.n
hierarchy/BuiltinMethod.n hierarchy/ClassMembers.n
hierarchy/CustomAttribute.n hierarchy/DelegateClassGen.n
hierarchy/GlobalEnv.n hierarchy/MacroClassGen.n
hierarchy/NamespaceTree.n hierarchy/ScanTypeHierarchy.n
hierarchy/TypeBuilder.n hierarchy/TypeInfo.n hierarchy/TypesManager.n
hierarchy/XmlDump.n misc/AssemblyInfo.n misc/PrettyPrint.n
misc/Stats.n parsing/AST.n parsing/Lexer.n
parsing/MainParser-Extensions.n parsing/MainParser.n
parsing/ParseTree.n parsing/PreParser.n parsing/PreParserIndent.n
parsing/Utility.n passes.n typing/ConstantFolder.n
typing/DecisionTreeBuilder.n typing/ErrorMessage.n
typing/LocalContext.n typing/LocalValue.n typing/MType.n
typing/MacroColorizer.n typing/MacroRegistry.n typing/Macros.n
typing/Messenger.n typing/OverloadPossibility.n
typing/RestartInErrorMode.n typing/Solver.n typing/StaticTyVar.n
typing/Subst.n typing/TyVar.n typing/TyVarEnv.n typing/TypedTree.n
typing/Typer-CallTyper.n typing/Typer-DelayedTyping.n
typing/Typer-OverloadSelection.n typing/Typer-PatternTyper.n
typing/Typer.n typing/Typer2.n completion/CodeCompletionEngine.n
completion/CompletionEngineError.n completion/CompletionEngineTree.n

out.stage2/ncc.exe -no-stdlib -r:mscorlib -r:System -dowarn:10006
-pedantic-lexer -def:_stage3 -stats -bar+ -def:RUNTIME_MS 
-target-library -out:out.stage3/Nemerle.Macros.dll \
        -reference:out.stage3/Nemerle.dll \
        -reference:out.stage3/Nemerle.Compiler.dll \
         ../macros/Profiling.n ../macros/xml.n ../macros/text.n
../macros/io.n ../macros/core.n ../macros/compiler.n
../macros/assertions.n ../macros/operators.n ../macros/concurrency.n
../macros/Util.n ../macros/Nemerle.n ../macros/Memoize.n
../macros/Logging.n ../macros/Internals.n ../macros/English.n
../macros/Data.n ../macros/DesignPatterns.n ../macros/AssemblyInfo.n

out.stage2/ncc.exe -no-stdlib -r:mscorlib -r:System -dowarn:10006
-pedantic-lexer -def:_stage3 -stats -bar+ -def:RUNTIME_MS  -target-exe
-out:out.stage3/ncc.exe \
        -reference:out.stage3/Nemerle.dll \
        -reference:out.stage3/Nemerle.Compiler.dll \
         main.n

if you change the stage2 and stage3 to your custom directories, it should be ok.
The crucial flat here is '-no-stdlib', it is probably the reason why
it crashed for you.

The MSBuild task for building compiler would be nice. If you have it
working post it on the list :)


> +                               // We cannot optimize by ignoring groups with wrong location early since
> +                               // for example loosegroup contains only the first token's location.
> +                               if(token is NC.Token.RoundGroup || token is NC.Token.BracesGroup || token is NC.Token.SquareGroup || token is NC.Token.QuoteGroup) {

You should be able to look at for example BracesGroup's Location.Line
and Location.EndLine

This will give you the way to skip the whole matching braces group.



--
Kamil Skalski
http://nazgul.omega.pl