Reply to topic  [ 12 posts ] 
Major changes and bug fixes to OXY package 
Author Message
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post Major changes and bug fixes to OXY package
I've made a couple of major changes to the OXY package, in response to the discussion in a previous post here: http://yorick.sourceforge.net/phpBB3/viewtopic.php?f=7&t=357. I don't think this will break existing code, but I'm not sure. Please report any breakage in this thread, so we can discuss how to handle it.

First: I fixed a major memory management bug that caused any OXY group object with more than 8 members to overflow its buffer -- they were never getting reallocated! So if you've been frustrated by a lot of SIGSEGV in object code, this should fix most of it.

Second, after the discussion in the above thread, I realized that there were two major features missing from the original OXY design:

1. There was no good way to invoke one method of an object from a caller that was itself invoked as a method of the object. That is, one method could not easily invoke another method. The problem only appeared when you modified the values of object members -- but of course, that happens very often!

2. There was no way to invoke a "friend" function (in C++ parlance) in the context of an object without making it a member of the object. Perhaps the most important problem this created was that there was no way for a method in a derived class to call the method it shadowed in its base class -- again, an untenable situation.

Both of these problems are addressed -- I hope solved -- in the new implementation. I've tried to explain the changes in the help,oxy and help,use comments. Here is a very quick summary:

I've added a "use_method" function, which is similar to use as a function, but permits you to invoke methods as subroutines, and will throw an error if its first argument is not a function. This gives you a proper means for invoking a second method of an object from within the code of a method, something you want to do all the time.

The major change I made is to the use function in its subroutine form: It now does *not* restore any "use-d" variables until the return of the *outermost* method -- the one which was not invoked by use_method or use as a function. Instead, all calls to use inside the outermost method, wither in the method itself or other methods it invokes, are cumulative. Thus, the "used" members of the object remain unpacked in the global namespace until the outermost method returns. This is much more nearly the behavior you "expect". Also, "use-ing" the same variable multiple times is now a no-op, so you don't have to worry about whether your caller has already used something -- it you use it, go ahead and declare it with use.

However, it is also the place where there are potential conflicts with the original design: You need to be much more careful about mixing used variables with direct references to the context object (via use(data) or save or restore). The "standard practice" rule becomes: Always declare context data objects with "use,data;" then treat them like extern variables, and never declare context method objects with use -- invoke methods with use_method. Again, report any breakage I caused for you here, especially if you think it was clearer in the original semantics.

Finally, obj(func_expression,arg1,arg2,...) now invokes func_expression in the context of obj, as if it were a method. Therefore, use_method(base_class(method),arg1,arg2,...) lets you invoke a base class method. In the direct obj call, the function expression, maybe just noop(function), lets you invoke "friend" functions.

I'll post more on this later, and in responses to other recent threads.


Thu Mar 15, 2012 7:57 am
Profile
Yorick Master

Joined: Tue Mar 07, 2006 10:31 pm
Posts: 125
Location: Meudon, France
Post Re: Major changes and bug fixes to OXY package
Dear Dave,

The save() functions seems broken in your last commit. Some ways to call save() fail I try to create an object from a method. (They all succeed if I call the "method" as a function, but fail in the context of an object):

The two following snippets fails:
Code:
func method(void) { return save(foo="bar"); }
obj = save(method);
obj(method,);

Code:
func method(void) { obj=save(); save, obj,foo="bar"; return obj; }
obj = save(method);
obj(method,);

with this output:
Code:
ERROR (method) unrecognized member specifier in save


Whereas the snippets below works as expected:
Code:
> obj = save(foo="bar")
> info, obj
object with 1 members:
   foo = array(string)

Code:
func method(void) { obj=save(); obj,foo="bar"; return obj; }
obj = save(method);
obj(method,);

Code:
func method(void) { foo="bar"; obj=save(foo); return obj; }
obj = save(method);
obj(method,);

Code:
func method(void) { return save(foo="bar"); }
obj = save(method);
obj.method();

Code:
func method(void) { obj=save(); save, obj,foo="bar"; return obj; }
obj = save(method);
obj.method();


Thu Mar 15, 2012 9:20 am
Profile WWW
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post Re: Major changes and bug fixes to OXY package
I can't get these to fail on my 11.10 Ubuntu Linux box:

Code:
func method(void) { return save(foo="bar"); }
> obj = save(method);
> obj(method,)(*,)
["foo"]
> obj(method,)(foo)
"bar"


Code:
> func method(void) { obj=save(); save, obj,foo="bar"; return obj; }
> obj = save(method);
> obj(method,)(*,)
["foo"]
> obj(method,)(foo)
"bar"


I wonder if I still have a buffer overrun or some other memory management problem? Can you get these to fail in a freshly started yorick or in a code built with "-g -O0" so we can try to track this down?


Fri Mar 16, 2012 6:58 am
Profile
Yorick Master

Joined: Tue Mar 07, 2006 10:31 pm
Posts: 125
Location: Meudon, France
Post Re: Major changes and bug fixes to OXY package
It's getting tricky: it depends on the complier.
The bug stung on:
  • Mac OS 10.6.8, cc --version: i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5666) (dot 3)
  • Debian SID (up to date), cc --version: cc (Debian 4.6.3-1) 4.6.3
(On both machines, I got a fresh clone of the git repo, compiled with "-g -O0", ran yorick, copied your code. It failed miserably with the same error message:
Code:
ERROR (method) unrecognized member specifier in save
WARNING source code unavailable (try dbdis function)
now at pc= 2 (of 12), failed at pc= 8

But the bug disappeared when compiling on Mac OS with MacPort's gcc --version: gcc-mp-4.4 (GCC) 4.4.5.

I also checked: the bug was introduced (or awoke) in commit 099320090720daf0c7bb9b1a4a195ca3d5940325: it did not sting in commit 19d574c091250e32d4001c3b4df252879a188663.


Fri Mar 16, 2012 8:42 am
Profile WWW
Yorick Master

Joined: Tue Mar 07, 2006 10:31 pm
Posts: 125
Location: Meudon, France
Post Re: Major changes and bug fixes to OXY package
Hi,

I checked the other versions of GCC available in Debian. They all trigger the bug:
  • Debian 4.4.7-1
  • Debian 4.5.3-12
  • Debian 4.6.3-1

Apple and Debian both patch GCC heavily. I'll try under Debian with a stock compiler.

Regards, Thibaut.


Fri Mar 16, 2012 9:57 am
Profile WWW
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post Re: Major changes and bug fixes to OXY package
Thibaut,

Try it now, commit bd411ce fixes an uninitialized value bug in oxy.c that I found with valgrind.

Dave


Fri Mar 16, 2012 6:58 pm
Profile
Yorick Master

Joined: Tue Mar 07, 2006 10:31 pm
Posts: 125
Location: Meudon, France
Post Re: Major changes and bug fixes to OXY package
Hi Dave,

With those changes, you are very strongly discouraging people from calling use as a function or using save, use,.... But there are cases where you really need to do this ( I have real-life need for the former, and I see the latter coming really fast):
  • when you need to pass the entire object as an argument to a (legacy?) function, or to a method of another object (consider cross-connecting a parent and child object);
  • when you need to add a member to the object (but there's no guarantee this member doesn't exist already.

What would you advise in those cases? Would the following make sense?

Code:
self = use();
self = restore(self); // store back external values of the used members
legacy_function, self;
restore, self; // put back members into the context


EDIT: I tried it (more seriously than I did before posting) and it doesn't work at all, since it sets any members that haven't been "used" or "restored" in the current context to [].

Code:
if (anyof(use(*,)=="new_member") {use, new_member; new_member=new_value;}
else { save, use, new_member=new_value; use, new_member; }


The point is that with "use, var", we actually have two copies of certain members: one in the context object, another in the context namespace. It's impossible for the programmer to know whether they are in sync, or when they have been or will be. With your recent changes, you have made the choice that the copy of the members made available through "use, var" should be the central repository for information, whereas I would find it much more robust to declare the context object as universal reference. In any case, there is a need for a very clear mechanism for the programmer to explicitly synchronize the two copies, even without knowing which members may have been changed or even exist.


Mon Mar 19, 2012 1:32 am
Profile WWW
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post Re: Major changes and bug fixes to OXY package
I agree that the problem is that there are two copies -- one in the context object, and a second "unpacked" by use in the global symbol table. This was true with the original semantics as well, but there was the additional problem (pointed out in the thread referenced above) that it was very difficult to have one method call another method of the same object. That need arises so often that in my judgement the semantics I've settled on is much better. You are indeed correct that you should now almost never extract data members of an object with use(data_member), or employ the save,use and restore,use special forms. But I think that is actually a good thing and not a bad thing.

The seriousness of the memory management bug (which guarantees SIGSEGV crashes whenever any object has more than 8 members) convinces me that there must be very little, if any, legacy code. Please post a reply here if you have a serious legacy code problem; since I caused it by a poor initial design, I will try to help you upgrade your legacy code to the new semantics.

To answer your two usage questions:

1. I agree that there is a serious problem if a method of an object needs to pass the object itself to another object. However, that method must surely know which data members the non-member function will need, and whether it will modify them. Thus, the method function can invoke save,use before the call (or the whole object, or for particular members) to make sure that any altered members (the method will likely know what may have been altered as well) before passing use() to the non-member object. After the non-member function returns, the method would invoke restore,use to unpack any members the non-member function may have altered. This is ugly and imperfect, but I believe it is actually a rare usage case. With the original semantics, as pointed out in the previous thread, you have a similar ugliness in the far more common case of one method invoking a second method of the same object. I think the new semantics is the correct choice, given I can't get both things to work perfectly with the simple two-copy implementation I've chosen.

2. The is_obj function is designed to permit you to check for the existence of object members (it returns a different error code for "does not exist" than for "is not an object"). The code is something like:
Code:
if (is_obj(use(),new_member)==-1) save,use, new_member;
use, new_member;
new_member = new_value;

Again, I agree this is slightly ugly, but surely this is a pretty rare use case, considering adding members at runtime is impossible in most object oriented languages.

I think the following two simple rules give you clean, clear code 95+% of the time, when you are writing method functions:

1. Always call use as a subroutine to "declare" any data members of your context object which your method will access (either to read or write). Never make any context object member local to a method (thinking of the "use" call as a declaration does this automatically).

2. Always call use as a function to invoke any method members of your context object. I added a new "use_method" function as syntactic sugar if you need to invoke a method member as a subroutine.

If you do both these things, I believe everything will work exactly as you expect in almost all situations. The first problem you raise -- of a method passing its own context object to a non-method as an argument is, I admit, a problem -- especially if that non-method function itself invokes methods of the context object. A second problem is invoking a method of an object which is itself a member of the context object -- I'm not sure that type of "recursion" will behave properly. Both of these are hard to fix. The question is whether they arise often enough to be a serious concern. I think, for example, to write the planned GTK+ wrapper for yorick, you'll never need to do either of those things. Like the rest of yorick, this is supposed to be a very informal, simple extension that allows you to adopt an object oriented programming style for many problems. I fully expect there will be things that OXY can't handle -- my attitude is that I'll fix what I can fix and work around what I can't, and we'll see how far we can push it.


Fri Mar 23, 2012 9:52 pm
Profile
Yorick Guru

Joined: Thu May 10, 2007 12:07 pm
Posts: 62
Post Re: Major changes and bug fixes to OXY package
I confess, I am a bit lost in the discussion, with the way I used the object so far I never get into the difficulties you mentioned.

All my object's methods are like this, for instance :
Code:
fund _myObj_set_method (varname, something) {
  myObj =  use();
  if (is_void(myObj)) error, "Function _myObj_set_method  called out of a object context";

  /*  Here do what ever I need. I can call other methods without problems  */
   /* for instance: */
  if (!am_subroutine()) myObj = myObj(copy,);
  if (myObj(keys, varname)) error, "the var \""+varname+"\" already exists "; /* key being a method of the object */
 
  /* set something  */   
   return myObj_set (myObj, varname, something);
}
fund _myObj_get_method (varname) {
  myObj =  use();
  if (is_void(myObj)) error, "Function _myObj_get_method  called out of a object context";
  if (is_myObj(myObj)) error, "expecting an object of type myObj";
  if (!myObj(keys, varname)) error, "the var \""+varname+"\" does not exists ";

  return myObj(data)(noop(varname));
}

In short I am using this=use() like I would use the "this" in other languages more object oriented. So, is there something wrong doing that ?
When you say :
" it was very difficult to have one method call another method of the same object."
"there is a serious problem if a method of an object needs to pass the object itself to another object."
I do not see the difficulty here with the way I wrote my methods, so I probably do something wrong or inefficient.

Sylvain.


Wed Mar 28, 2012 8:25 am
Profile
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post Re: Major changes and bug fixes to OXY package
The point of the use subroutine was to make object members look and feel like extern variables in ordinary yorick programs: To refer to a member of the context object, you just use its variable (say x) in expressions; to set it, you just say x=something. The only difference was supposed to be that you are forced to "declare" such member variables with "use,x". I haven't quite succeeded, in that to invoke a sibling method, you have to say "use(sibmeth,args)" instead of "use,sibmeth;...sibmeth(args)...". Other than that, and the subtler problems Thibaut pointed out involving mixing direct save and restore to the context with members extracted by the use function, I think it now works pretty well.

It looks to me like the "this=use()" style you developed was a necessary workaround to my original buggy implementation. That style should continue to work, but I think you will find it unnecessary with the current code. The two rules I stated above will make cleaner code (to my eye anyway). I think you are being unnecessarily cautious to insert the error detection testing whether use() is nil; it's going to fail soon enough without the test, and if your goal is to make it difficult to invoke methods out of an object context, a better strategy is to hide all copies of the method inside objects, so the method function is effectively hidden from casual users.

I really need one or two object oriented utilities in the i/ library, to provide examples of programming styles that work. Because I agree, it just isn't obvious -- a chapter in the manual would be appropriate as well, huh?


Thu Mar 29, 2012 8:14 pm
Profile
Yorick Guru

Joined: Thu May 10, 2007 12:07 pm
Posts: 62
Post Re: Major changes and bug fixes to OXY package
Yes I see, I guess I handled use to much as an Object but it is more general and more powerful than that.
Maybe a good illustration, it help me to understand the philosophy, for use is a collection of quantities and laws (via function) in different systems (e.i. SI, CGS, ...)
I copy past an example :
Code:
func bbPower( lambda, T) {
  /* Return the black body radiation  Power */
  use, c, h, k;
  return 2*h*c^2/lambda^5 / ( exp( c*h/(k*lambda*T))  - 1);
}
func bbTotalPower(T) {
  /* Return the Stefan-Boltzmann law */
  use, sigma; 
  return sigma*T^4;
}

system = save(bbPower , bbTotalPower /*, etc ... */);

SI = save( [], system,
          c = 2.99792453e+08, /* light speed in vacuum */
          h = 6.62606876e-34, /* Plank Constant        */
          k = 1.3806503e-23, /*  Boltzmann constant    */
          sigma = 5.670400e-8 /* Stefan–Boltzmann constant */
           /* etc */
          );
CGS = save([], system,
          c = 2.99792453e+10, /* light speed in vacuum */
          h = 6.62606876e-27, /* Plank Constant        */
          k = 1.3806503e-16 , /*  Boltzmann constant    */
          sigma     = 5.670400e-5 /* Stefan–Boltzmann constant */
         
          );


You'll understand the objects are used like this :
Code:
SI (bbRad, 1e-6, 4000);
CGS (bbRad, 1e-6, 4000);


==================================================================================================

Something else. A thought. When one use "use, k1, k2" in a function it will overwrite every keywords that are defined in the function with the same name (here k1, k2, k3, ...).
Would it be possible to create also a built-in function (lets call it use_as_default ), called in the same way than use it will set the variables only if they are not defined in the context of the function.
The idea is that often we have a set of functions with the same keywords and we want to set a default for al of them depending on what is inside the object.
I have created a interpreted function to do that I am not completely satisfied, but it work, maybe if you think it is useful one can create a clean built-in function to do that.
Code:
func use_as_default (args) {
  obj = args(1);
  if (is_void(obj)) return [] /* do nothing if not in oxy context */
  if (!is_obj(obj)) error, "bad use of "use_as_default", expecting an oxy object has first argument";
  Nargs = args(0);
  for (i=2; i<=Nargs; i++) {
    if (is_void(args(i)) && args(*,i) && obj(*, args(*,i))) args, i, obj(args(*,i)); 
  }
}
wrap_args, use_as_default;


It it used like this :
Code:

func line (y, x, color=, width=) {
  use_as_default, use(), color, width;

  plg, y, x, color=color, width=width;
}

func mark( y, x, color=, marker=) {
   use_as_default, use(), color, marker;

   plmk, y, x, color=color, marker=marker;
}

myPlot = save(line, mark,  color="blue", width=3, marker=3);
myPlot, line, random(10), random(10) ;  /*  Will plot lines in blue  which is the default of myPlot */
myPlot, line, random(10), random(10), color="red" /* Will plot lines in red */
myPlot, mark, random(10), random(10), marker= 7   /* Plot markers 7 with blue color */


This function will save me a lot of code lines like if (is_void(color) && use(*,"color")) use, color ........ and my function will stay backward compatible.

Cheers, have a good weekend.


Fri Mar 30, 2012 1:30 pm
Profile
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post Re: Major changes and bug fixes to OXY package
These are interesting examples. They expose some holes in the yorick language that are only partly related to OXY. For example, the problem of supplying defaults for keywords is not really related to objects, though the context object adds an interesting twist. Similarly, you cannot use a keyword as an output argument to a function (f(&x) doesn't work with keywords), though you'd often like to do that.

The underlying problems have to do with the fact that yorick makes it difficult to "look behind" a variable to see its value in a scope outside the current function. Once something is a local variable, yorick provides no means to discover what its value may have been in its extern scope -- in fact, that's buried on the stack, and difficult to get to. The OXY interface provides a means that I could expose those shadowed extern values (I could make an object whose members were the chunk of the stack that will be swapped back into place when the current interpreted function returns). The question is, should I provide that function?

I think the answer is no -- yorick is already moving toward a perl-like collection of hacks (I like perl, but it is unusually difficult to understand). The thing you are trying to do with default keyword values is to limit the scope of a changed parameter to the function call -- otherwise you would have simply made it an extern variable. I still think the pldefault function for supplying defaults to all the plotting functions is the most straightforward interface to provide that functionality; it's become very easy to implement that style interface with an OXY object to hold the default values.


Sat Mar 31, 2012 10:27 am
Profile
Display posts from previous:  Sort by  
Reply to topic   [ 12 posts ] 

Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by STSoftware for PTF.