[nem-en] Metaphor, staging, typed reflection

[Sandro Magi]

Kamil Skalski wrote:
> 2007/1/7, Sandro Magi <smagi at higherlogics.com>:
>> Yup, that's a viable approach, but the whole point of generic/polytypic
>> programming is that you don't need to duplicate all this code.
>> Duplicated code leads to bug proliferation.
> 
> This code of course would be generated, but at compile-time on the
> provider side, not at runtime on consumer side... I feel no sorry for
> physical code duplication as long as this fact is hidden in every day
> use - 

This has severe consequences for updates due to bug fixes, etc. See below.

> Absolutely no! I'm arguing against pushing such delicate and hard task
> as generating code to the runtime.

...and instead you would push generating code to the macro. How is that
safer? Either your staged/macro-expanded code type checks, or it
doesn't. Even if it type checks, there is runtime behaviour that the
type system cannot verify and you will need to resort to testing.

> One of the most insightful opinion
> about macros, which appeared some time ago on this list is that macros
> are not hard to write, they are hard to debug. Static multi-staged
> typing helps with generating correct code, so it helps to solve this
> problem - but delegating any code generation to the runtime makes it
> much harder.

Testing is the only recourse in either case. Compile-time macro, or
multistaged code generation, either way you need unit tests if your type
system cannot verify the properties you're interested in. I think the
safety of compile-time macro and multistaging is actually equivalent.

> I agree, but it has nothing to do with  compile-time / run-time
> dispute we are having here.

Let me lay out a concrete scenario:

1. Let's assume safety achievable between compile-time macro and staged
code generator is equivalent.

2. You have written some algorithm with a bug in it.

Do we both agree on the above assumptions? Consequences:

1. Algorithm as a compile-time macro: macro expansion leads to the buggy
serialization algorithm dispersed in many different classes, assemblies,
etc. Updating the macro dll, and restarting the application does not fix
the bug. Hot-updates (like in ASP.NET) cannot fix the bug unless all
dependent assemblies are rebuilt and updated.

2. Algorithm as a staged computation: the VM JIT is essentially a
runtime macro expander, and we are assured the generated code at least
type checks (same assurance as a compile-time macro). The VM can now
hot-update any running code when it detects the dlls have been updated.

So we have equivalent safety guarantees with staging as with a
compile-time macro, but we also have additional flexibility. Of course,
a compile-time macro for algorithms is generally not a good idea, but
it's just an example of the additional flexibility you gain for no
appreciable cost.

> There is CAS, which I guess is quite good at isolating the execution.

I don't want to get into an off-topic security debate, but CAS is
actually not very good at all. ;-)

> Most of those features as I can understand them (I'm not familiar with
> all) are also possible to implement as compile-time analysis and code
> generation.

But "compile-time analysis and code generation" is exactly what
intensional type analysis and multistaging is, except all the analysis
can now be expressed in the host language you're compiling, and also
guided by the developer!

> This has the ultimate advantage of giving you ready code
> to look into, debug, analyze and feed to utility tools (like  IDE,
> which shows you the class members you generated).

Also achievable by reflection. :-)

> Maybe a little example. In my project I wanted to add the "properties"
> mechanism to allow configuring its behavior. They are usually just key
> value pairs of various type (integer, boolean, string). So I created a
> macro to fetch the key-value pairs from data-base, generate the fields
> in a specified class, layout them in nested hierarchy (they are
> organized in tree like "namespaces"). Now I can access them in IDE and
> code completion allows me to choose the desired key very fast. I can
> also easily inspect the whole tree of properties.
> This level of integration is possible just because of compile-time
> processing.

This is simply stage-0 multistaging though. I believe the same is
achievable in Metaphor, or at least can be achieved in principle.

> I think that this is turning too much into a flame-war... :( 

There's been no name calling yet, so I think we're ok. ;-)

> I suppose there are many valid examples, where run-time code generation
> is more powerful, easier, etc. and I think any research to make it
> more safe is a very good thing. But for me design-time / compile-time
> generation has numerous advantages and I prefer making my designs go
> into this direction.

I agree that compile-time should always be preferred, except when it
limits you in some important way. For example, making assemblies
dynamically updatable in .NET was done for a reason, and macros can
hinder that. Garbage collection is necessary for all languages with
dynamic memory allocation, as region inference and type systems for
tracking resource use aren't quite sufficient yet, and macros can't help
you here.

I guess the reasoning is simple in the end:

1. if we can agree that the safety of multistaging and macros is
equivalent, and

2. if we agree that there are *some* things, even if they are few and
far between, that simply *must* be done at runtime due to lack of
information,

3. then we must conclude that adding intensional type analysis and
multistaging yields a safer, more efficient, more expressive language
overall.

Sandro