On Mon, 26 Oct 2009 22:22:14 GMT, Richard Harter wrote:

Some significant disadvantages:

* Low abstraction level, basically lacking any abstraction. Data flow works
with primitive built-in atomic types forced into value semantics.

* Already mentioned inefficiency because of value semantics (i.e. either
permanent marshaling inputs, or else blocking the producers or the
consumers when data are shared) Normally it is the programmer to choose how
to protect states. With the dataflow it must be the engine to decide.
Basically it is lack of abstraction again. Working with *data* instead of
*states*, that summaries it.

* The networks of wired blocks cannot be encapsulated into reusable opaque
primitives. You can group blocks, that's it. Usual methods of decomposition
do not work, because there is a firewall between "block" and "data". You
cannot merge them (like in OO) into a reusable entity (type = values +
behavior).

* Unmaintainable code. Look at large data flow programs (e.g. in DiaDem,
Simulink, LabView). It is impossible to use reasonable software processes
on them. How to compare two diagrams? When are they semantically
equivalent? How do I validate a diagram? How to search for anything in a
diagram? Another example of this problem is represented by GUI libraries,
which are mostly event controlled. Practically none of them can be easily
used in multitasking environment. It is a horror to debug hangups or
generators in messages routed from one messages loop to another. And the
proposal is to build *everything* this way. Shudder.

* Semantic problems, the barriers of a block that fire the computation of
the block's output is a very primitive model that does not scale in
reality. It almost instantly goes into a mess of deadlocking vs. race
condition choices. And see above, it practicably cannot be debugged except
for trivial cases (without feedbacks, with all inputs synchronous etc) it
cannot be decomposed into simpler tasks...

On Thu, 29 Oct 2009 05:10:37 -0700 (PDT), Nick Keighley wrote:
On 27 Oct, 08:55, "Dmitry A. Kazakov" <mail... at (no spam) dmitry-kazakov.de
wrote:

* Unmaintainable code. Look at large data flow programs (e.g. in DiaDem,
Simulink, LabView). It is impossible to use reasonable software processes
on them. How to compare two diagrams?

convert it to a textual represention then run diff on it. I'm not
saying it's trivial but I don't think its intractable either.

Are pixel positions and sizes of the blocks relevant to the comparison?
()

The very argument that a text form is somewhat better raises a suspicion
why not to use it from the start?

When are they semantically equivalent?

when they are the same.

Yes two programs are equivalent when they are...

Code gives you exactly the same problem.

Sure, but for a programmer it is easier to decide when he deals with text.
"Find 10 differences" is a game known for pictures, no for texts.

(I am not an expert in psychology, but it seems that abstract visual
information cannot be digested at a finely detailed level. Maybe it is a
fundamental problem. Maybe not. But so far text works at best.

Look at the
younger generation, who read virtually nothing. They only watch. They have
got completely new gadgets (like mobile phones), my generation didn't have.
I.e. it was a fresh start. And what we see? SMS - naked texts reborn!)

How do I validate a diagram?

there were tools that could do this in the 80s

There was no LabView that time, and there is no such tools now... ()
AFAIK, engineers designing models in Simulink, LabView etc, validate the
generated code. They do not the diagrams.

How to search for anything in a diagram?

solved in the 80s

Ah, that is maybe because there were only alphanumeric display device back
then? ()

It would be nice if there were
a graphical differences tool- that is one that would didplay the
picture and highlight the differnces. Not undoable I'm sure.

On 29 Oct, 18:15, "Dmitry A. Kazakov" <mail... at (no spam) dmitry-kazakov.de
wrote:
On Thu, 29 Oct 2009 05:10:37 -0700 (PDT), Nick Keighley wrote:
On 27 Oct, 08:55, "Dmitry A. Kazakov" <mail... at (no spam) dmitry-kazakov.de
wrote:

* Unmaintainable code. Look at large data flow programs (e.g. in DiaDem,
Simulink, LabView). It is impossible to use reasonable software processes
on them. How to compare two diagrams?

convert it to a textual represention then run diff on it. I'm not
saying it's trivial but I don't think its intractable either.

Are pixel positions and sizes of the blocks relevant to the comparison?
()

I presume the smiley means you can answer that yourself...

The very argument that a text form is somewhat better raises a suspicion
why not to use it from the start?

where did I say better? They have different strengths and weaknesses.
Programmers I know spend a fair amount of time drawing strange
diagrams on whiteboards. Personnally, I think they should then throw
the diagrams away and code it up but a lot of people like diagrams.
You can have both.

yes, but I fail to see why the graphical form presents any different
problems from the textual form.

Ah, that is maybe because there were only alphanumeric display device back
then? ()

blinking lights and switches. If you can't read the machine code in
binary you aren't a Real programmer. Octal is for quiche eaters.

Nick Keighley <nick_keighley_nospam at (no spam) hotmail.com> writes:

It would be nice if there were
a graphical differences tool- that is one that would didplay the
picture and highlight the differnces. Not undoable I'm sure.

Indeed. Have a look at:
http://www.informatimago.com/develop/pic-merge-diff3/index.html
A trivial patch to this script would highlight the differences.

(Unfortunately, it's only a pixel diff/merge operation, we would have
to use more sophisticated algorithms to detect in a concise form block
transformations; but even in this case, this gimp script is rather
effective, visually).

On 10/26/2009 06:22 PM, Richard Harter wrote:
The antithesis is that it isn't turtles all the way up, or at
least it shouldn't be turtles all the way up. That is, the kinds
of languages and programming technologies that we use in
large scale programming should be quite different from the kind
of languages used for programming in the small.

I think most people would agree that methodologies for small-scale
programs are different from those for large-scale programs, although
there might be considerable disagreement over the dividing line. I
certainly am not going to be used a full-blown OOP paradigm for writing
even something like an automatic code generator; on the other hand, I
shudder at the thought of writing something so complex as an operating
system in a functional paradigm.

An interesting project would be to take a large corpus of various mature
open-source programs of varying size and see if there is a correlation
between size and the use of certain language features or patterns.

One difference between traditional programs and data flow
programs is that traditional programs use LIFO semantics whereas
data flow programs use FIFO semantics. That is, a traditional
program puts data on a stack and gets data back on the same
stack. In data flow programs each element gets data from a queue
and puts data to other queues.

One nit to point out: at this stage, many programs don't follow a
procedural model of programming. OOP is the dominant paradigm, and I
don't see the sequence of data flow as being LIFO. Yet you continually
refer to procedural programming as those that make up `traditional
programs.

Another difference is that the connectivity of traditional
programs is deeply embedded in the code. To pass data from A to
B, A calls B. That is, the caller has to specify where the data
goes. In data flow programs the connectivity can be separate
from the code. A doesn't pass data directly to B; instead it
passes data to the run time system which in turn passes the data
to B. The caller does not have to specify where the data goes.

Two words: function pointers. You can also go with virtual or dynamic
method dispatch, but function pointers is shorter and sounds better.

* Concurrency and parallelism are natural. Code can be
distributed between cores and even across networks. Many of the
problems associated with threads disappear.

They don't disappear, they're pushed into the runtime system. From
practical experience, that means they probably bubble up and annoy you
in the programming language.

* Data flow networks are natural for representing process. This
is particularly true for transaction processing.

Maybe I'm just being a stupid idiot here, but I don't see how data flow
is natural for representing some common processes. For example, the
HTML5 tokenization process. I suppose you can flow current-state output
back around and into itself as next-state input, but that's not exactly
natural.

This also brings up a question of how the language deals with loops in
the dependency graph. The solutions I see either bring back the problems
with threading or could create unreliability.

* Message passing gets rid of the problems associated with shared
memory and locks.

Just as credit default swaps and other financial machinations got rid of
risk .

From what I can tell, similar problems arise, but they're in a
different form (data flow dependency graphs, particularly the fact that
they're typically not acyclic).

* Data flow programs are more extensible than traditional
programs. Elements can be readily composed into composite
elements from the bottom up rather than top down.

Even if you consider good old procedural programming, I would say that
the exact same thing is easily doable in traditional programs. I can
take code that does asynchronous socket reads and turn it into a MIME
decoder by creating a module that calls the asynchronous socket read
module appropriately. It's also an example of your "bottom-up composition."

Some significant disadvantages:

* Using data flow programming in the large pretty much requires
that it be used from the start. That is, converting traditional
programs into data flow programs is difficult because the
structuring is so different.

I don't see it that way. Ultimately, most libraries are merely "pass X
in as input, get Y out as output"--it shouldn't be too hard to make that
an atomic data-flow program block.

On Fri, 30 Oct 2009 12:19:46 +0100, Pascal J. Bourguignon wrote:
Nick Keighley <nick_keighley_nos... at (no spam) hotmail.com> writes:

It would be nice if there were
a graphical differences tool- that is one that would display the
picture and highlight the differnces. Not undoable I'm sure.

Indeed. Have a look at:
http://www.informatimago.com/develop/pic-merge-diff3/index.html
A trivial patch to this script would highlight the differences.

(Unfortunately, it's only a pixel diff/merge operation, we would have
to use more sophisticated algorithms to detect in a concise form block
transformations; but even in this case, this gimp script is rather
effective, visually).

Yes, as someone with image processing, pattern recognition and AI
background I can tell you, there is no such algorithms. You will need a
complex scene analysis in order to get at the level of a diagram. Even this
does not work. Because image segmenting and line detection do not really
do. Such a trivial task, but alas, no algorithm can compete human in image
segmenting. But even if you passed through, got geometric regions, lines
etc. To analyse their connections, shapes (e.g. the scene), that does not
work. Diagrams generated by GUI tools skip these steps, because the tool
places this information into the intermediate file.

The point is that the task is immense and though we have solved it during
our evolution, my guess is that it leaves no free "computational" resources
to analyse the picture at a much detailed level. It wasn't evolutionary
needed too.

On Fri, 30 Oct 2009 03:32:30 -0700 (PDT), Nick Keighley wrote:
On 29 Oct, 18:15, "Dmitry A. Kazakov" <mail... at (no spam) dmitry-kazakov.de
On Thu, 29 Oct 2009 05:10:37 -0700 (PDT), Nick Keighley wrote:
On 27 Oct, 08:55, "Dmitry A. Kazakov" <mail... at (no spam) dmitry-kazakov.de

* Unmaintainable code. Look at large data flow programs (e.g. in DiaDem,
Simulink, LabView). It is impossible to use reasonable software processes
on them. How to compare two diagrams?

convert it to a textual represention then run diff on it. I'm not
saying it's trivial but I don't think its intractable either.

Are pixel positions and sizes of the blocks relevant to the comparison?
()

I presume the smiley means you can answer that yourself...

No, equivalence of two directed graphs is not a simple task.

The very argument that a text form is somewhat better raises a suspicion
why not to use it from the start?

where did I say better? They have different strengths and weaknesses.
Programmers I know spend a fair amount of time drawing strange
diagrams on whiteboards. Personnally, I think they should then throw
the diagrams away and code it up but a lot of people like diagrams.
You can have both.

Yes, physicists draw lots of diagrams too. Nevertheless the language of
physics is differential equations. The role of diagrams is always
supplementary, they cannot serve as a language.

yes, but I fail to see why the graphical form presents any different
problems from the textual form.

It is easier to analyse textual form for both humans and computers.

Ah, that is maybe because there were only alphanumeric display device back
then? ()

blinking lights and switches. If you can't read the machine code in
binary you aren't a Real programmer. Octal is for quiche eaters.

Yep, that was fun. You could immediately see if the program ran in a cycle,
the pattern repeated itself. Blue screen, that's not suckers! Real men read
the program counter of the crash location looking at the front panel LEDs!
()

mike wrote:

I think that most of the problems inherent in any large-scale
programming project result from the inherent 'fragility' of all
programming languages.

If you compare a large computing project with a large engineering
project there are clear similarities, but one very significant
difference is that almost any undetected error in code has the potential
to result, somewhere down the line, in catastrophic outcomes; whereas if
a nail is not quite hammered in as far as specified or if a light
fitting (or even a window) is put in the wrong place then the building
usually remains safe, functional, and fit for purpose.

If someone has some ideas about a programming language/paradigm that is
fault-resistant, not only in the sense that it reduces the number of
actual bugs and/or errors produced, but also in the sense that many bugs
have no significant effect on function and behaviour then large-scale
projects may be a lot easier to manage.

Whether this will ever be a possibility remains, in my opinion, unknown
to date.

In engineering works there are "factors of safety" to take account
of variability of materials, unpredictability of actual loadings
and inaccuracies in construction etc. etc. In computing, analogous
has been done. For results for control of critical (including
in particular potentially dangerous) processes, one employs
double or triple hardware to take care of the possibility of
malfunction of hardware. As far as I know, in order to better
detect programmer errors, one similarly employs different and
independent teams of programmers to do the same job and then with
test cases to compare the results of the resulting programs.
However, if I don't err, this comparision is only done in the design
phase of the software and later only the work of one of the teams
is selected for practical application. A safer way, I think, would
have these presumably equivalent software (resulting from different
teams, employing preferably different programming languages and
environments) in actual production runs to always work parallelly
on multiple hardware (of possibly different types), so as to futher
reduce the risk of errors, since testing of software in the design
phase might not be thorough enough to uncover all errors that may be
present. Of course, errors could not be "absolutely" eradicated,
in accordance with Murphy's Law.

Yes - it is true that paralelling the hardware and software with result

mok-kong.shen wrote:
[snip]
.......... A safer way, I think, would
have these presumably equivalent software (resulting from different
teams, employing preferably different programming languages and
environments) in actual production runs to always work parallelly
on multiple hardware (of possibly different types), so as to futher
reduce the risk of errors, since testing of software in the design
phase might not be thorough enough to uncover all errors that may be
present. Of course, errors could not be "absolutely" eradicated,
in accordance with Murphy's Law.

Yes - it is true that paralelling the hardware and software with result
comparison migh help reduce software errors, but I think it may be a
while before I can load up a [Windows/Linux/Chrome] operating system on
my new [AMD/Intel/Via(?)] chipset...

On 2009-10-28, Pascal J. Bourguignon <pjb at (no spam) informatimago.com> wrote:
Now if you need to write a million of source line, then just don't
do it. Use metaprogramming to generate this million of source lines
from a smaller source. And so on, you can add layers of
metaprogramming all you need to compact your sources and always have
something of manageable size.

So, you're saying that *every* programming problem can be solved in at
most a few tens of thousands of lines of code?

Can you not specify all programming problem in less that a few
thousands of lines of specification?

Well, you can always write more detailed specifications, but I can
assure you that sales peoples will always be able to put the whole
specifications of your software on a 2-page booklet.

Certainly some problems can, but most can't. Metaprogramming is just
a form of compression, and there is no compression system that can
reduce every source below a given size.

Some problems really are irreducibly complex, and demand complex
solutions.

Yes indeed. However, assuming a big ontology (eg. take wikipedia, or
even the whole web), wouldn't it be possible to express the needs for
any software in less than ten thousands lines, and let the
sufficiently smart system develop it, filling in the blanks in the
specifications with all the knowledge it can extract from the web?

On 2009-10-28, Pascal J. Bourguignon <pjb at (no spam) informatimago.com> wrote:
On 2009-10-28, Pascal J. Bourguignon <pjb at (no spam) informatimago.com> wrote:
Now if you need to write a million of source line, then just don't
do it. Use metaprogramming to generate this million of source lines
from a smaller source. And so on, you can add layers of
metaprogramming all you need to compact your sources and always have
something of manageable size.

So, you're saying that *every* programming problem can be solved in at
most a few tens of thousands of lines of code?

Can you not specify all programming problem in less that a few
thousands of lines of specification?

Yes but that is usually a top-down specification, and while implementing
lots of little decisions still have to be made.

So usually, such specifications are not complete.

Well, you can always write more detailed specifications, but I can
assure you that sales peoples will always be able to put the whole
specifications of your software on a 2-page booklet.

Yes, but strictly speaking that is a summary. Not a full specification.

IOW you reduce complexity by removing details, and describe first order
operation only. Not by introducing an abstraction that reduces data.

Certainly some problems can, but most can't. Metaprogramming is just
a form of compression, and there is no compression system that can
reduce every source below a given size.

I like that analogy.

I think a better analogy is that metaprogramming is a form of indexing

Yes indeed. However, assuming a big ontology (eg. take wikipedia,
or even the whole web), wouldn't it be possible to express the needs
for any software in less than ten thousands lines, and let the
sufficiently smart system develop it, filling in the blanks in the
specifications with all the knowledge it can extract from the web?

Or take the problem actually in the other dirrection. Would you
trust any implementation of a system that has orders of magnitude
more than ten thousand lines of specifications?

How can you ensure these specifications are consistent? How can you
ensure that they're effectively implemented?

Wouldn't you be more able to understand and check the specifications
if they were shorter, that is indeed, given the ultimate limits to
compression, if what they specified was less complex or of a more
limited scope?

If you accept that big systems must be decomposed into small
programs,

Then the degree of automatization in the process of translating the
specifications into executable code is only a matter of advancement
of the techniques, while the size of the executable code is only
(roughly) a function of the number of metaprogramming levels used.

On 2009-10-28, Pascal J. Bourguignon <pjb at (no spam) informatimago.com> wrote:
... However, assuming a big ontology (eg. take wikipedia,
or even the whole web), wouldn't it be possible to express the needs
for any software in less than ten thousands lines, and let the
sufficiently smart system develop it, filling in the blanks in the
specifications with all the knowledge it can extract from the web?

You can do that: but now your program's correctness depends upon the
correctness of every detail in Wikipedia, or even the whole web, as
well as that of both the "sufficiently smart" system that extracts
information from it, and the ten thousand lines of specification.

And yet, there will still be programs that cannot be specified in less
than ten thousand lines (unless the lines are of unbounded length).

a form of compression, and there is no compression system that can
reduce every source below a given size.

I like that analogy.

I think a better analogy is that metaprogramming is a form of indexing
or referencing.

A compressed file must contain all the information in the original file
in such a way that the original file can be restored.

But in a high- level or metalanguage I don't need to know all the nuts and
bolts beneath the surface.

So, for example, if a specification requires you to calculate the
Chi-square of a data set, then it may be possible to use a trusted
statistics package to get tre results, without knowing what the source
language is, what hardware it will be processed on or even what Chi-square
is. The code is 'out there' somewhere, but it is not part of your code, so
the statistics package is not compressed into your code but referenced by
it.