[nem-en] Metaphor, staging, typed reflection

[Kamil Skalski]

> > This code of course would be generated, but at compile-time on the
> > provider side, not at runtime on consumer side... I feel no sorry for
> > physical code duplication as long as this fact is hidden in every day
> > use -
>
> This has severe consequences for updates due to bug fixes, etc. See below.
>
> > Absolutely no! I'm arguing against pushing such delicate and hard task
> > as generating code to the runtime.
>
> ...and instead you would push generating code to the macro. How is that
> safer? Either your staged/macro-expanded code type checks, or it
> doesn't. Even if it type checks, there is runtime behaviour that the
> type system cannot verify and you will need to resort to testing.
>
> > One of the most insightful opinion
> > about macros, which appeared some time ago on this list is that macros
> > are not hard to write, they are hard to debug. Static multi-staged
> > typing helps with generating correct code, so it helps to solve this
> > problem - but delegating any code generation to the runtime makes it
> > much harder.
>
> Testing is the only recourse in either case. Compile-time macro, or
> multistaged code generation, either way you need unit tests if your type
> system cannot verify the properties you're interested in.

Right, but this is a trivial fact not applicable to our discussion ;)

> I think the
> safety of compile-time macro and multistaging is actually equivalent.

In the same sense as giving a mathematical formula for calculating
measurements to the engineer building bridge a day before your
holidays instead of giving him the ready made computations. ;)  The
difference isn't that big, but in one case you at least made sure that
formula produced some results given the real life data.

>
> > I agree, but it has nothing to do with  compile-time / run-time
> > dispute we are having here.
>
> Let me lay out a concrete scenario:
>
> 1. Let's assume safety achievable between compile-time macro and staged
> code generator is equivalent.
>
> 2. You have written some algorithm with a bug in it.
>
> Do we both agree on the above assumptions? Consequences:
>
> 1. Algorithm as a compile-time macro: macro expansion leads to the buggy
> serialization algorithm dispersed in many different classes, assemblies,
> etc. Updating the macro dll, and restarting the application does not fix
> the bug. Hot-updates (like in ASP.NET) cannot fix the bug unless all
> dependent assemblies are rebuilt and updated.

Well.. if you actually have ANY code at client side, then you need to
make sure that you have the update mechanism for copying new dlls to
the client anyways. So I just ask for a reasonable response, which is
safer:
- giving client a ready code, which will replace the previously stored assembly
- or giving client a ready code generator, which will be executed on
his machine to generate the ready code?

A rule of thumb in doing real business stuff is to minimize the number
of things, which could break. Delegating the code generation to be
made outside of your controlled environment is definitely the
opposite.


>
> 2. Algorithm as a staged computation: the VM JIT is essentially a
> runtime macro expander, and we are assured the generated code at least
> type checks (same assurance as a compile-time macro). The VM can now
> hot-update any running code when it detects the dlls have been updated.
>
> So we have equivalent safety guarantees with staging as with a
> compile-time macro, but we also have additional flexibility. Of course,
> a compile-time macro for algorithms is generally not a good idea, but
> it's just an example of the additional flexibility you gain for no
> appreciable cost.

We have the additional flexibility, but you will never convince me
that JIT engine is more reliable than compiler generating target code
in advance.

>
> > There is CAS, which I guess is quite good at isolating the execution.
>
> I don't want to get into an off-topic security debate, but CAS is
> actually not very good at all. ;-)
>
> > Most of those features as I can understand them (I'm not familiar with
> > all) are also possible to implement as compile-time analysis and code
> > generation.
>
> But "compile-time analysis and code generation" is exactly what
> intensional type analysis and multistaging is, except all the analysis
> can now be expressed in the host language you're compiling, and also
> guided by the developer!

I don't get what you mean. So you agree that those example could be
pushed to compile-time instead of run-time? Indeed, compile-time code
generation is also a multi-staged computation, actually two-staged
one... :)
We could push our debate one level of abstraction higher:

I state that for n-staged program it is safer, more efficient (as seen
by the user's experience) and more convenient during development if
for all stages k < n they are executed in advance at the programmer's
development environment.

Of course if you are a final user of some tool you made for yourself,
then its usage is the n-th stage :-)

>
> > This has the ultimate advantage of giving you ready code
> > to look into, debug, analyze and feed to utility tools (like  IDE,
> > which shows you the class members you generated).
>
> Also achievable by reflection. :-)

Only when I generate the actual code, so the reflection could pick it
up. If during my whole development cycle I designed my system to not
generate (or need to generate) given piece of code, then I couldn't
use it.. :)

>
> > Maybe a little example. In my project I wanted to add the "properties"
> > mechanism to allow configuring its behavior. They are usually just key
> > value pairs of various type (integer, boolean, string). So I created a
> > macro to fetch the key-value pairs from data-base, generate the fields
> > in a specified class, layout them in nested hierarchy (they are
> > organized in tree like "namespaces"). Now I can access them in IDE and
> > code completion allows me to choose the desired key very fast. I can
> > also easily inspect the whole tree of properties.
> > This level of integration is possible just because of compile-time
> > processing.
>
> This is simply stage-0 multistaging though. I believe the same is
> achievable in Metaphor, or at least can be achieved in principle.

Yes, if I turned it into the compile-time processor, that is execute
all the stages.

>
> > I suppose there are many valid examples, where run-time code generation
> > is more powerful, easier, etc. and I think any research to make it
> > more safe is a very good thing. But for me design-time / compile-time
> > generation has numerous advantages and I prefer making my designs go
> > into this direction.
>
> I agree that compile-time should always be preferred, except when it
> limits you in some important way. For example, making assemblies
> dynamically updatable in .NET was done for a reason, and macros can
> hinder that. Garbage collection is necessary for all languages with
> dynamic memory allocation, as region inference and type systems for
> tracking resource use aren't quite sufficient yet, and macros can't help
> you here.
>
> I guess the reasoning is simple in the end:
>
> 1. if we can agree that the safety of multistaging and macros is
> equivalent, and

Macros are actually multi-staged programs... So I agree :-)

>
> 2. if we agree that there are *some* things, even if they are few and
> far between, that simply *must* be done at runtime due to lack of
> information,

Yes, for one reason or the other. JIT compiler in .NET is a good
example: you have abstract program representation and then you
transform it at runtime into native code.

>
> 3. then we must conclude that adding intensional type analysis and
> multistaging yields a safer, more efficient, more expressive language
> overall.
>

I'm not quite sure what do you mean by this sentence... Multi-staging
is a good tool - it gives more abstraction to the programmer and
allows producing more efficient code to be executed by the user. But
it is always safer to have system with less stages - another example
here is comparison of compiler and interpreter, they do actually the
same job for the user, allow running some code and see the results.
But one is MUCH more complex than the other.

-- 
Kamil Skalski
http://nazgul.omega.pl