                                   Chapter 4

                                   UTILITIES


      "over" and "by"
      'by' and 'over' in J 3.2
      Tree Representation
      J for 386 PCs


*========================================
# "over" and "by"

+------------------
| Rudi Rynders
| <15160010@hpdmd48.boi.hp.com> 10 Jan 91 20:14:02 GMT


 The  "OVER" and  "BY" constructs mentioned in my previous
 note are defined as follows:

    over =.;@({.;}.)&":@,
    by =.(,~"_1' '&;&;)~

[[NB. sws.  These are now (from the Dictionary)

    over=. ({.,.@;}.)@":@,
    by=.   ' '&;@,.@[,.]
]]

  Using these one can neatly produce the result of the APL
  outer product function, as in A jot.+ B for instance.
  It glues the A vector on to the result as the first column
  as well as displaying the B vector accross the top.
  The command is:

    A by B over A+/B

  but how this it work?


+------------------
| Sam Sirlin
| <1991Jan14.030907.17684@jato.jpl.nasa.gov>


In article <15160010@hpdmd48.boi.hp.com> rrr@hpdmd48.boi.hp.com (Rudi Rynders)
writes:
>
> The  "OVER" and  "BY" constructs mentioned in my previous
> note are defined as follows:
>
>    over =.;@({.;}.)&":@,
>    by =.(,~"_1' '&;&;)~
>
>
>  Using these one can neatly produce the result of the APL
..
>
>  but how this it work?
>
To see how they work, I applied the various operators first. For example
   a over b
is
   a u@v&w@x b   for appropriate verbs u-x
which is equivalent to
   u v w (a x b)
or
   ;({.;}.)":(a,b)
which is easy to understand. Applying this idea to by (a little more subtle)
I get
   a by b
equivalt to
   (' ';;a),"_1 b
which is also much easier for me to comprehend.

Sam Sirlin
Jet Propulsion Laboratory         sam@kalessin.jpl.nasa.gov

[[NB. sws.  Shall I? Well ok. Here's the same for the new versions:

   over=. ({.,.@;}.)@":@,

   u=. {. NB. head/take
   v=. ,. NB. ravel items
   w=. ;  NB. link
   x=. }. NB. drop/tail
   y=. ": NB. format
   z=. ,  NB. append items

    NB. this gives:
   over=. (u v @ w x)@y@z f.

    NB. so then
   a over b
    NB. becomes

   a (u v @ w x)@y@z b
   (u v @ w x)@y a z b
   (u v @ w x) y a z b
   (u (v @ w) x) y a z b
   (u (v @ w) x) (y a z b)
   (u y a z b) (v @ w) (x y a z b)
   v (u y a z b) w (x y a z b)
    NB. which is
   ,. ({. ": a , b) ; (}. ": a , b)
    NB. ravelitems (head format a,b) link (tail format a,b)

    NB. similarly, if
   c=. b over a+/b
    NB. a by c becomes
   a (' '&;@,.@[,.]) c
   a ( (' '&;@,.) @[,.]) c
   a ( ( (' '&;@,.) @[ ) ,.]) c
   ( ( (' '&;@,.) @[ )a ) ,. (] c)
   ( ( (' '&;@,.) @[ )a ) ,.  c
   ( (' '&;@,.) [ a ) ,.  c
   ( (' '&;@,.) a ) ,.  c
   (' '&; ,. a ) ,.  c
   (' '; ,. a ) ,.  c

NB. we have a new option now, tacit conversion. Trying this on by, we get

   overa=. ',. ({. ": x. , y.) ; (}. ": x. , y.)' :11
   bya=. '('' ''; ,. x. ),.y.' :11

NB. These new verbs produce the same results as the above, but are different!
]]

+------------------
| Roger Hui
| <1991Jan14.210146.28858@yrloc.ipsa.reuter.COM>


Reply to the following msg re "over" and "by":

> From: rrr@hpdmd48.boi.hp.com (Rudi Rynders)
> Subject: Re:  "over" and "by"
> Message-ID: <15160010@hpdmd48.boi.hp.com>
> Date: 10 Jan 91 20:14:02 GMT
>
> The  "OVER" and  "BY" constructs mentioned in my previous
> note are defined as follows:
>
>    over =.;@({.;}.)&":@,
>    by =.(,~"_1' '&;&;)~
>
> Using these one can neatly produce the result of the APL
> outer product function, as in A jot.+ B for instance.
> It glues the A vector on to the result as the first column
> as well as displaying the B vector accross the top.
> The command is:
>
>   A by B over A+/B
>
> but how this it work?
>
> Rudi Rynders (rrr@hpdmd48.boi.com)

"over" and "by" are verbs defined in tutorial frame "Da" distributed
with J.  They are used to label the result of outer product,
making an annotated table, as indicated in the tutorial.  Thus:
      over=.({.;@;}.)&":@,
      by=.(,~"_1 ' '&;&;)~
      a=.3 1 4 2
      b=.4 2 8 5 7
      a */b
   12 6 24 15 21
    4 2  8  5  7
   16 8 32 20 28
    8 4 16 10 14


      a by b over a */b
   +-+-------------+
   | | 4 2  8  5  7|
   +-+-------------+
   |3|12 6 24 15 21|
   |1| 4 2  8  5  7|
   |4|16 8 32 20 28|
   |2| 8 4 16 10 14|
   +-+-------------+

(Note a:  Rynders transcribed "over" incorrectly.  Note b:
The box drawing chars look better than this on the PC.  Here, I use
standard ASCII chars to avoid problems in mail transmission.)

The question is, how does this work?  Probably, a reader new to the
language would start on easier verbs than these.  KEI was the original
author, and I'd guess that it took even him at least 5 minutes to write
these verbs...  But, since the question is asked, here's how I'd try to
get a handle on them:

(0) As always, a good start in understanding a verb is to experiment
with it.  Try it on various arguments.

(1) If a verb is primitive, try looking it up in the Dictionary.
The Dictionary may be terse, but it is written carefully, and
rewards (and requires?) careful reading.

(2) If a verb is non-primitive, decompose it into smaller objects.
In doing this, I find it helpful to look at the display of a verb.
For example, "over" looks like this:
      over
   +----------------------+-+-+
   |+---------------+-+--+|@|,|
   ||+--+-------+--+|&|":|| | |
   |||{.|+-+-+-+|}.|| |  || | |
   |||  ||;|@|;||  || |  || | |
   |||  |+-+-+-+|  || |  || | |
   ||+--+-------+--+| |  || | |
   |+---------------+-+--+| | |
   +----------------------+-+-+

Some facts one can deduce about verb displays:
(a) At each level of nesting, there are 1, 2, or 3 boxes.
(b) A 1-box display is a primitive.  (Try entering ";" or "}." by itself.)
(c) A 2-box display is either a verb derived from an adverb,
if the second box is an adverb; or an instance of the hook phrasal
form, if not.
(d) A 3-box display is either a verb derived from a conjunction, if the
second box is a conjunction; or an instance of the fork phrasal form,
if not.

I recommend entering various verbs and looking at their displays
to get a feel for them, starting from very simple cases.  Here is a
set of 24 exercises to get you started:
  +        +/        +/\
  +.       +./       +./\
  +&*      +.&*      *&+.
  +/&*     *&+/      *&(+/)
  +%       +%/       (+%)/
  +,-      -,+       +;-
  +/,-     (+/),-    (+/,-)
  -,+/     (-,+)/    -,(+/)
(Some of these verbs don't do anything useful.  The object is to
see how verbs combine.)

The display of "over" tells you that it is of the form
   f2   =. ;  @  ;
   f1   =. {: f2 }:
   f0   =. f1 &  ":
   over =. f0 @  ,
Having these parts in hand, recursively apply steps (0), (1), and (2).



+------------------
| Rudi Rynders
| <15160014@hpdmd48.boi.hp.com>


 Well , this is very interesting ! The definition of "over" I quoted in
 my original query , I did NOT transcribe incorrectly ! It came straight
 from lesson 5 of "TANGIBLE MATH". The "over" mentioned in the previous
 response is different from this. To reiterate:

 As listed in my copy of "TANGIBLE MATH ": over=.;@({.;}.)&":@,
 whereas in the previous response        : over=.({.;@;}.)&":@,

[[NB. sws.  These are now (from the Dictionary)

    over=. ({.,.@;}.)@":@,
    by=.   ' '&;@,.@[,.]
]]

 They both seem to be legal because they achieve the same result.

 My thanks to Sam Sirlin and Leigh Clayton and Roger Hui, for their
 responses.

 One further comment on the issue of whether the "Dictionary" is
 "obscure" or merely "terse". I'll settle for " a bit abstruse".
 It definitely requires some careful study .

 As for my suggestion of a text comparing APL and J: I still
 think it would be a good idea to have one to make the path to "J"
 a bit less thorny. The chances of "J" becoming popular will be a
 lot better with it than without it IMHO.


+------------------
| Peter Biddlecomb
| <1991Jan29.194223.7806@yrloc.ipsa.reuter.COM>

In response to the request from Rudi Rynders, here's an attempt to explain how
<over> works, concentrating on the bits which APL users are likely to have
problems with. It does not attempt to explain terms such as 'item' and 'atom'.
The explanation was arrived at from sevral directions:

(a)  Examining the results of (5!:1 <'over') to get an idea of the work done.
(b)  Writing my own version in " monad :: dyad " form.
(c)  Reading the 'Parsing and Execution' section of the Dictionary of J, and
     the definitions of @ and &.

As there are at least two versions of <over>, let's note the one we're looking
at and what it does:

      over =. ;@({.;}.)&":@,
      z over z+/z=.1+i.3
+-----+
|1 2 3|
+-----+
|2 3 4|
|3 4 5|
|4 5 6|
+-----+
      t=. z+/z    [ right arg. of over ]

We start by inserting the definition of <over> between <z> and <t>:

     z ;@({.;}.)&":@, t

This of course gives the same result as <z over t>.  <over> does exactly the
same job as <iota=.i.> - it gives us a name for a sequence of J primitives.
The causes of bafflement when I try to analyse <over> by eye are:

(i)    large number of primitives - the APL one-liner effect
(ii)   unfamiliar symbols (and possibly different implementations of the
       primitives they represent) if you're used to APL
(iii)  the use of conjunctions - primitives which affect order of execution

Here's <over> divided into meaningful units:

          ;:   'z ;@({.;}.)&":@, t'

+-+-+-+-+--+-+--+-+-+--+-+-+-+
|z|;|@|(|{.|;|}.|)|&|":|@|,|t|
+-+-+-+-+--+-+--+-+-+--+-+-+-+

 n v c <----v----> c v  c v n     [this line is NOT provided by ;:]

The bottom line categorises the contents of each box as Noun, Verb, or
Conjunction.  We can mark the fork ({.;}.) as a verb at this stage.

Now we parse the expression.  The crucial point to note is that conjunctions
are executed before verbs, so we analyze them first to get an expression
which consists of nouns and verbs.
  The other point to note is that "the left argument of [a] conjunction is the
entire verb phrase that precedes it".  So starting with the rightmost
conjunction and using the same notation for its arguments as the dictionary,
we have:

        z ; @ ( {. ; }. ) & ": @ , t

        x <--------u---------> @ v y    [ <u> is the verb phrase... ]

the dictionary tells us that  x u @ v y => u x v y: >--+
                                                       |
        <---------u--------> x v y  <------------------+

        ; @ ( {. ; }. ) & ": z , t

        <------u------> & v  <-y->  >------------------+
                                                       | u & v y => u v y
        <------u------> v  <-y->  <--------------------+

        ; @ ( {. ; }. ) ": z , t

        u @ <----v-------> <-y->    >------------------+
                                                       | u @ v y => u & v y
        u <----v-------> <-y->  <----------------------+

        ; ( {. ; }. ) ": z , t

Now we have an expression which tells us what to do with z and t.  The only
barriers to understanding now are differences in symbols and their
implementation, and working out what the fork ( {. ; }. ) does. In detail, we:

(a) catenate <z> on top of <t> [equivalent to <,[quad-IO]>]
(b) format the result [if we didn't do this, our column headers would be
       misaligned for some values of t.]
(c) catenate the boxed first item (row in this case) onto the boxed remaining
    items [note that take and drop operate on "items" rather than scalars]
(d) convert the result into a one-column matrix

We can write a version of <over> which is easier for me to understand:

   over =. '' :: ';({.;}.)":x.,y.'

Apart from the treatment of the monadic version of <over>, this does just the
same thing.

<by> can be analysed in the same way, but this is left for you to do!


+------------------
| Roger Hui
| <1991Jan29.194223.7806@yrloc.ipsa.reuter.COM>


Just one comment:  as I pointed out in 4836625, if you simply enter the
name of the verb, you get a boxed display of the verb.  This display
tells you J's idea of what squiggle is grouped with what other squiggle,
ie. very much like a tree, ie. the result AFTER parsing.  This removes
word formation and syntax from the picture, leaving only the semantics
to be analyzed (meanings of @, &, ":, etc.).

Of course, this is just another alternative.  The approach taken in your
msg can be no less valuable.



*========================================
# 'by' and 'over' in J 3.2

+------------------
| Tony Vignaux
| <1991Aug19.205300.1061@comp.vuw.ac.nz>

You may have discovered that the 'by' and 'over' functions described in
the original J literature no longer  work in J version 3.2.

L. J. Dickey <ljdickey@watmath.uwaterloo.ca> has sorted it out and
suggests the following changes:

   [ over  =. ;@( {. ;     }. ) & ": @ ,     [' old version'
     over  =.   ( {. ,. @; }. ) & ": @ ,

   [ by    =. (, ~ "_1 ' ' & ; & ;  )~    ['old version'
     by    =. (, ~ "_1 ' ' & ; & ,. ) ~

   prices=. 3 1 4 2
   orders=. 2 0 2 1
   prices by orders over prices */ orders

+-+-------+
| |2 0 2 1|
+-+-------+
|3|6 0 6 3|
|1|2 0 2 1|
|4|8 0 8 4|
|2|4 0 4 2|
+-+-------+


*========================================
# Tree Representation

+------------------
| Roger Hui
| <1991Sep27.064028.20543@yrloc.ipsa.reuter.COM>


NB.  Model of tree representation (5!:4).

ar      =. 5!:1
type    =. 3!:0
boxed   =. 32&=@type
mt      =. 0&e.@$
oarg    =. >@(1&{)
shr     =. |.!.''
shl     =. 1&(|.!.'')
boxc    =. 9!:6 ''
dash    =. 10{boxc

sh      =. (*/@}: , {:)@(1&,)@$ $ ,
rows    =. */\.@}:@$
bl      =. }.@(,&0)@(+/)@(0&=)@(]/ i.@{.@(,&1))
mask    =. 1&,. #&, ,.&0@>:@i.@#
mat     =. mask@bl@rows { ' '&,@sh

extent  =. (+./\ *. +./\.) @ (' '&~:) @: ({."1)
limb1   =. 1&|.@$ 1&~: }. (10 6 0{boxc)&,@($&(9{boxc))
limb    =. -@(i.&1)@[ |. #@[ {. limb1@]
pfx     =. (limb +/)@extent ,. ]
pad     =. [ {. ] ,. dash&=@({:"1)@] {  ' '&,:@($&dash)@(-&{: $)
take    =. pad`({.&(,.' ')@[) @. (mt@])
rc      =. #@>@{."1 ; >./@:({:@$@>)
kernt   =. (0{boxc)&=@shl@[ *. ' '&~:@]
kernb   =. (6{boxc)&=@] *. ' '&~:@shl@[
kern    =. (<0 0)&{&>"2 (kernt +./"1@:+. kernb) (<_1 0)&{&>"2
gap     =. ,&.>"_1 {&((0 1$' ');1 1$' ')@kern
graft   =. (pfx&.>@{. 0} ]) @ (,&.>/) @ gap @ ({@rc take&.> ])

lab     =. ,: @ (2&|.) @ ((' ',dash,dash,' ')&,)
label   =. lab`((,.dash)&[) @. (e.&'0123456789'@{.)
center  =. ((i.&1) -@+ <.@-:@(+/))@] |. #@] {. [
root    =. label@[ center extent@>@{.@]

leaf    =. ,@<@(((,:dash,' ')&[ center $&1@#) ,. ])@mat@":

trx     =. >@{. (root ; ]) graft@:(tr@>)@oarg
trgl    =. >@{. (root ; ]) graft@:(trx@>@{. , tr @>@}.)@oarg
trgr    =. >@{. (root ; ]) graft@:(tr @>@{. , trx@>@}.)@oarg
trg     =. trgr`trgl`trx @. (i.&(<,'`')@oarg)
trtil   =. trx`(leaf@(,&' ')@oarg@>@{.@oarg) @. ((<,'0')&=@{.@>@{.@oarg)
trcase  =. (leaf@oarg)`trgl`trgl`trg`trtil`trx @. ((;:'0@.`:4~')&i.@{.)
tr      =. leaf`trcase @. boxed

rep     =. ([.(`({"0 1)))`(].(,&)("0))\
west    =. (e.&(9{boxc) *. shr"1@(e.&dash)) rep (5{boxc)
cross   =. (e.&(5{boxc) *. shl"1@(e.&dash)) rep (4{boxc)
east    =. (e.&(9{boxc) *. shl"1@(e.&dash)) rep (3{boxc)
north   =. (e.&(6{boxc) *. shr"1@(e.&dash)) rep (7{boxc)
connect =. north @ east @ cross @ west

tree    =. connect @ > @ (,.&.>/) @ (> (root ; ]) tr@>@ar)

________________________________________________________________

   tree <'tree'
                         +- connect
                   +- @ -+- >
             +- @ -+                +- <
             |     +- / --- @ ------+     +- ,.
             |                      +- & -+- >
- tree -- @ -+     +- >
             |     |     +- root
             |     +-----+- ;
             +-----+     +- |
                   |                +- tr
                   |     +- @ ------+- >
                   +- @ -+- ar

5!:4 is another in the "representation" series of verbs.
   5!:1  atomic (gerundial)
   5!:2  display
   5!:3  string
   5!:4  tree

5!:4 will be available in the next version (3.5) of J.  The argument
to 5!:4 is a boxed named; the result is a literal table of the tree
representation of the named object.

The model is divided into groups of defns.  Unless otherwise noted,
the defns are verbs.

The first group are utilities:
ar    - atomic representation
type  - internal "type"
boxed - 1 iff boxed
mt    - 1 iff empty
oarg  - open the second element of the list argument
shr   - shift right
shl   - shift left
boxc  - (noun) box drawing characters
dash  - (noun) the "dash" in the set of box drawing characters

"mat" is the main verb of the next group of definition.  The argument
is a literal array; the result "looks like" the argument, but is a table.

The other defns manipulate arrays of a type that I call a "generational
tree": a list of boxed literal tables having the same number of rows,
such that nodes at the same depth are in the same box.  For example,
the GT for "tree" is:

+--------+-----+------+------+-----------+------+-----+
|        |     |      |      |+- connect |      |     |
|        |     |      |+- @ -|+- >       |      |     |
|        |     |+- @ -||     |           |+- <  |     |
|        |     ||     |+- / -|-- @ ------||     |+- ,.|
|        |     ||     |      |           |+- & -|+- > |
|- tree -|- @ -||     |+- >  |           |      |     |
|        |     ||     ||     |+- root    |      |     |
|        |     ||     ||-----||- ;       |      |     |
|        |     |+-----||     |+- |       |      |     |
|        |     |      ||     |           |+- tr |     |
|        |     |      ||     |+- @ ------|+- >  |     |
|        |     |      |+- @ -|+- ar      |      |     |
+--------+-----+------+------+-----------+------+-----+

"graft" is the main verb in the next group of defns.  The argument
is a table whose rows are GTs for the nodes at the same depth.
The result is a GT.

"root" accepts a string left argument and a GT right argument.
The result is a literal matrix with the string centered relative
to the GT.

"leaf" computes a unitary (single-element) GT from its argument.

"tr" accepts an argument which is the opened atomic representation
of an object, and computes a GT therefrom.  The verbs with the "tr"
prefix embody logic to effect "nice" displays for various special cases.
Thus, in "trcase", the items in the agenda are:
   ID                      Agenda
  ----         ---------------------------------------------
   0            leaf@oarg      noun (leaf)
   @.           trgl           the left subtree is a gerund
   `:           trgl           the left subtree is a gerund
   4            trg            bonded (curried) conjunction
   ~            trtil          possible instance of evoke

"rep" is a conjunction whose left argument is a proposition p and
whose right argument is a single literal c, deriving a verb such that
the phrase  p rep c y  replaces with c all the positions in y marked by
p y.

"tree" accepts a boxed name and computes the tree representation of
the named object.  The appearance of the result is improved on systems
with better box drawing characters (such as the PC).  "tree" is a model
of 5!:4, and is slower by a factor of about 30.



*========================================
# J for 386 PCs

+------------------
| Roger Hui
| <1991May28.120527.21336@yrloc.ipsa.reuter.COM>

J currently does not exploit features specific to the 386.  It runs in
"real" mode, and workspaces are at most 640K.

Obviously, it is desirable to remove these restrictions.  The main
obstacle seems to be finding a run-time memory extender which has no
hefty per-copy royalty charge (impossible for a shareware product).

Note that J is available for the Interactive 386ix (System V flavor
UNIX for the 386), and it does not have these restrictions.


+------------------
| Sam Sirlin
| <1991May28.193736.12020@jato.jpl.nasa.gov>

In article <1991May28.120527.21336@yrloc.ipsa.reuter.COM>, hui@yrloc.ipsa.
reuter.COM (Roger Hui) writes:

]> The main
]> obstacle seems to be finding a run-time memory extender which has no
]> hefty per-copy royalty charge (impossible for a shareware product).
]>

Note that GNU gcc is available for msdos, which uses protected mode and
available memory. You can get it from

128.153.13.196     grape.ecs.clarkson.edu

[[NB. sws. I believe the current address to use is

128.153.28.12      barnacle.erc.clarkson.edu

but there are many other locations as well that have it now]]

+------------------
| Sam Sirlin
| <1991May28.193736.12020@jato.jpl.nasa.gov>

In article <1991May28.120527.21336@yrloc.ipsa.reuter.COM>, hui@yrloc.ipsa.reuter.COM (Roger Hui) writes:

|> The main
|> obstacle seems to be finding a run-time memory extender which has no
|> hefty per-copy royalty charge (impossible for a shareware product).
|>

Note that GNU gcc is available for msdos, which uses protected mode and
available memory. You can get it from

128.153.13.196     grape.ecs.clarkson.edu

[[NB. sws. The address to use is now
128.153.28.12      barnacle.erc.clarkson.edu
]]


 