Reply to topic  [ 10 posts ] 
yapi.h / C extension questions 
Author Message
Yorick Master

Joined: Wed Jun 01, 2005 11:34 am
Posts: 112
Post yapi.h / C extension questions
I'm working my way through trying to learn how to use the stuff in yapi.h and have come across some things I had questions on. Rather than creating a bunch of threads, I figured it'd be better to post them all in one thread. I'll add future similar questions here as I have them.

Question 1: How can I protect calls to yarg_true?

The comments indicate that yarg_true returns 1 if "if(x)" is true and 0 otherwise. However, there are some inputs that are not valid for yarg_true (because they are also not valid for "if(x)"). A good example of this is array input, which throws a "non-scalar operand to BranchFalse" error. What's the best way to test to see if a value is suitable for passing to yarg_true? (I suppose this is actually more a general Yorick question than a yapi question, since it applies to if(x) as well.) Would "yrank(iarg) <= 0" suffice? Or are there non-array inputs that are also invalid?

Question 2: How do I populate ypush_p?

I was able to figure out how to work with pointers passed in by the user. I figured out how to create new pointers as output, but I can't figure out how to populate them. Suppose I wanted a very simple C function that provided the equivalent functionality to this Yorick function:

Code:
func test_pointer() {
  result = array(pointer, 3);
  result(1) = &[1,2,3];
  tmp = 4.2;
  result(3) = &tmp;
  return result;
}


How would I code that in C? Here's what I've managed so far:
Code:
void Y_test_pointer(int nArgs)
{
  // ignore any parameters
  long dims[Y_DIMSIZE], i;
  for(i = 0; i < Y_DIMSIZE; i++) dims[i] = 0;
  dims[0] = 1;
  dims[1] = 3;
  ypointer_t *result = ypush_p(dims);
  // now what?
}


The above returns array(pointer, 3) with all null pointers. How do I populate indexes 1 and 3?

Question 3: What functions let me work with structs?

The documentation in yapi.h references structs a few times, but I don't see any tools for working with them. Am I overlooking them? Should I look in another include file? Are they something we're not supposed to interact with via compiled Yorick extensions?

Notably, yapi.h makes these two defines:

#define Y_STRUCT 9
#define Y_STRUCTDEF 15

However, I can't find any instances of either of those names being used in any of the *.h or *.c files.


Sun Mar 03, 2013 6:26 am
Profile
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post Re: yapi.h / C extension questions
1. I think arrays are the only things illegal in if(x) statements. Thus, !yarg_rank(iarg) && yarg_true(iarg) ought to do what you want. But the hard failure of yarg_true and the other yapi.h functions like ygets_q is supposed to be a feature, not a bug. The idea is to permit you to simply call these functions without requiring you to detect and handle errors involving argument type mismatches. So if you feel you need to "protect" yarg_true, you might want to consider just letting it blow up -- my thought is that is what you'd usually want to do. When your function does different things depending on the kind of argument passed, this can get a little tricky, but you can usually reorder the various yarg... and yget... calls to make it work fairly smoothly.

2. You populate ypush_p pointer arrays with other yorick arrays you build on the stack with ypush_X calls. After pushing your array of pointers onto the stack, you'd call, for example, ypush_d to push an array of doubles onto the stack. You have to call yget_use when you store the pointer in your pointer array, then you call yarg_drop to pop the copy on the stack off; you don't have to dispose of the handle returned by yget_use, since you have given your use to yorick by storing it in the array of yorick pointers. Note that a pointer to a scalar double, long, or int, like your interpreted example, is a special case requiring more work. For a scalar of one of those three types, you have to call yget_d (or yget_l or yget_i) after you call yget_use, to undo yorick's scalar performance optimization.

Code:
void **ptrs = ypush_p(dims);
double *d = ypush_d(dims2);
yget_use(0);
if (!dims2 || !dims2[0]) d = ygeta_d(0, 0, 0);
ptrs[0] = d;
yarg_drop(1);
/* repeat ypush_X, yget_use, yarg_drop cycle as often as required to fill ptrs */


3. The yapi.h macros duplicate the older ydata.h macros used in most of the code; the ones you want are called T_STRUCT and T_STRUCTDEF in the (old) yorick source; you should use the yapi.h versions in new package code, as they will be the ones I carry forward into the future. I don't have a complete way to deal with interpreted structs in yapi.h; it would need to be a very complicated interface. I don't like structs in interpreted code; they are present in order for yorick to interface to compiled C code. I recommend the oxy object interface for interpreted code, which is more in the spirit of the yorick language, and which does feature a complete (I think) interface in yapi.h. That said, I believe you can use ygeta_any to retrieve a struct array argument. You'd need to build an interpreted wrapper function to guarantee that the argument actually had the type you wanted; your compiled function would just have to take it on faith that it had been passed a sensible argument (as functions always do in an ordinary C program). The ygeta_any call does enable you to retrieve the dimensions of such an array.

I appreciate that the oxy object support is not yet complete, since you cannot yet save them in a file, while you can save struct arrays. That will change, at which time there won't be a compelling reason to use structs in interpreted code, except as a bridge to compiled code structs.


Mon Mar 04, 2013 5:00 pm
Profile
Yorick Master

Joined: Wed Jun 01, 2005 11:34 am
Posts: 112
Post Re: yapi.h / C extension questions
1. You are right, I think I normally would prefer the hard_failure. The question came up in what I suppose is an unusual scenario. I wrote a function that iterates over each entry in the stack and prints out information using the various introspective yapi.h functions (yarg_typeid, yarg_nil, yarg_rank, etc.). Mostly as a learning exercise to let me verify that I understood how Yorick was thinking about things. I was including yarg_true for completeness, but didn't want to have errors if I passed in invalid input.

2. That's just what I needed, my test code is now working properly!

3. You were correct. This did what I wanted:
Code:
struct SAMPLE *data = (struct SAMPLE *)ygeta_any(0, &ntot, dims, &the_typeid);

I had to define the struct twice, though: once in C and once in Yorick. I couldn't figure out a way to make Yorick aware of the C version of the struct. Is defining it in both places the correct solution, or is there a way to make Yorick aware of the C version?

Thank you as always for the thorough explanations! :)


Wed Mar 06, 2013 9:06 pm
Profile
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post Re: yapi.h / C extension questions
There isn't any way to "make yorick aware" of the C struct. In principle, you could write a perl or python script that scanned C source code for a struct definition in .c and .h files, then translated it to the equivalent yorick code, recursively descending to define as many structs as necessary. Your package Makefile could invoke such a script to ensure that your yorick struct definitions in .i files were kept up to date with the C definitions, so that the .i file was built as an output file. But I wouldn't recommend that.

The truth is, structs are not very useful in interpreted code. The only real reason to have them is to call compiled functions for which you don't want to write proper wrappers. It isn't worthwhile to try to get too fancy in what should be an unusual construct in yorick interpreted code.


Thu Mar 14, 2013 3:42 pm
Profile
Yorick Master

Joined: Wed Jun 01, 2005 11:34 am
Posts: 112
Post Re: yapi.h / C extension questions
For better or worse, we use structs extensively in our code (much of it written before oxy was introduced).

New question:

4. Can I call Yorick functions from C code?

There are several kinds of functions so I suppose I might as well ask with respect to each. But I'm most interested at the moment in calling built-in Yorick functions, such as sort (Y_sort).

I thought I might be able to fudge it by making sure the value I wanted to sort was on top of the stack and then call Y_sort(1). But that didn't give the results I expected. On exploration, it appears that Y_sort is using "sp" which I'm inferring is a pointer to the start of the stack at the point when Yorick calls my Y_ function. I was nervous enough about trying to call Y_sort directly (since I have no way of knowing how many extra items it might put in the stack; I could figure that out by reading the code, but no guarantees it wouldn't change). However, touching sp seems like an even worse idea. So I'm pretty sure that whole direction is completely bad, bad, bad.

So is there a way for me to use Yorick built-in functions? And since it's so closely related, how about Yorick interpreted functions, wrap_args functions, and closure functions?


Fri Mar 15, 2013 11:10 am
Profile
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post Re: yapi.h / C extension questions
If you've been using the interpreted stack heavily, you need to be sure you've called ypush_check. If you've pushed anything on to the stack yourself and you're being paranoid, and you refuse to read the source code of the built-in function you're calling, a call to ypush_check(8) before calling a built-in is guaranteed to be sufficient. Yorick does not check that there is sufficient stack space on each call to push something, a minor speed optimization that forces you to be at least minimally aware. I deemed eight slots as a threshhold you shouldn't have to think about much.

Other than that, what you did ought to work: You push n arguments onto the stack and call Y_built_in_function(n).

The one other thing you need to do is to clean up the stack, of course. The built-in will leave its result on top of the stack, but it may leave a bunch of temporaries under it. As you know since you're writing them, the built-in is not responsible for cleaning up the stack. Look at the bottom of fnctn.c:EvalBI for a clue how to do that. There is no official yapi.h stack clean-up function, so the interface may change out from under you at my whim. I'll consider adding one. The complicating feature is that the entire stack may have moved if the built-in needed more than eight slots. By reading the code, you can almost always figure out what state the built-in has left the stack and clean it up with the yapi.h swap and drop functions. In the case of Y_sort, a quick look in std1.c suggests that Y_sort simply pushes its result on top of its argument, so that a swap-and-drop ought to leave the stack as it would be after an interpreted call to sort, having consumed the argument and replaced it by the result.

My original position was that it makes no sense to call interpreted code from compiled code, since it runs several hundred times slower. You can usually reframe your compiled code so that an interpreted wrapper would call the required interpreted function. As usual, rethinking your task as an event driven automaton is the key: You create a new object type to hold the state of your algorithm, with interpreted functions (or just one function, or make it an oxy object with methods -- you have lots of choices) to modify the automaton state. The interpreted wrapper then creates the automaton, loops through its states performing the required interpreted tasks between, and destroys the automaton when finished.

That said, if you read task.c, you will find that there is an undocumented (and way outside the safety of the yapi.h interface) means for setting up and running an interpreted task (mostly near the Y_include function, which does it). I strongly recommend the seemingly more complex automaton strategy. For example, what happens when the interpreted code you are running blows up with an error? Are you sure you don't have any unprotected temporaries or other critical state information that can survive an unexpected longjump around the rest of your code? With the automaton strategy, you don't have to worry about any of that -- each of your operations is self-contained and well-defined, and you can leave the interpreter to sort out the global stuff. It's also much closer to the way your code is going to perform. Literally, when you drop into interpreted code you are entering a different reality, just like the human timescales associated with typing a new command line are a different reality from your compiled code.

I keep planning to write an ODE integrator for interpreted functions that would demonstrate this approach, but converting the canned integrators to the automaton model is pretty challenging -- they all assume you will simply pass a function pointer, and store all their state information in very subtle ways in their automatic variables. So I feel your pain. Nevertheless, that's my recommendation for now. Rethinking your task as an automaton sometimes gives you interesting insights as well.


Sat Mar 16, 2013 8:34 am
Profile
Yorick Master

Joined: Wed Jun 01, 2005 11:34 am
Posts: 112
Post Re: yapi.h / C extension questions
5. Is there a "right" way to open a file handle?

I'm pretty sure if I just use fopen on its own, I open up the possibility of someone CTRL-C during execution and then ending up with an open file handle left in memory. But I don't see any API methods that look designed to address that issue.

Here's my current approach:

Code:
void close_f(void *tmp)
{
  fclose(*((FILE **)tmp));
}

void Y_some_func(int nArgs)
{
  char *fn = NULL;
  FILE *f = NULL;
  FILE **fptr = NULL;
  // Skipping code that populates *fn
  f = fopen(fn, "rb");
  if(!f) y_error("error opening file");
  fptr = (FILE **)ypush_scratch(sizeof(FILE **), close_f);
  *fptr = f;
  // Skipping remaining code that actually uses *f
  // When finished, do not call fclose(f) since that is done when Yorick frees **fptr
}


That seems a bit convoluted to me, but it seems to basically work. Is there a better way?


Fri Sep 20, 2013 8:20 am
Profile
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post Re: yapi.h / C extension questions
That doesn't look convoluted to me. What you are trying to protect against is the thing that is convoluted, so I don't have a problem with the solution being a bit non-obvious.

The idea of protecting against asynchronous interruption of your algorithm requires you to provide two things: (1) Some sort of state object that keeps a record of what you have to unwind, and (2) Some sort of callback function that uses that state object to recover. Even a try/except construct in python or C++ is doing that -- it's just that the state object is arranged to be the whole stack frame with all its local variables, and the callback is an inline chunk of code.

In your case, your state object is fptr, and your unwind callback is close_f. If you wanted to make it clearer what was going on, you could have written:
Code:
extern FILE *protected_fopen(char *name, char *mode);
static void protected_fclose(void *obj);

FILE *
protected_fopen(char *name, char *mode)
{
  FILE **fobject = ypush_scratch(sizeof(FILE**), protected_fclose);
  return fobject[0] = fopen(name, mode);
}

static void
protected_fclose(void *obj)
{
  FILE **fobject = obj;
  if (fobject[0]) fclose(fobject[0]);
  fobject[0] = 0;
}


If you used these two API functions, your code would look perfectly clean. Because protected_fclose can be called twice without any problems, you can even put the matching call to protected_fopen in your function if you like. If you were going to publish this API, however, you might be better off not providing protected_fclose as part of the API (hence the static scope), and merely noting that the fclose happens whenever the interpreted stack frame vanishes, whether from the ordinary function return, or from the unwind after an exception. On the other hand, if you thought is was important to be able to close the file before your function returns, maybe you should publish protected_fclose -- although you then are wasting a stack element on an unused scratch object... Note that if you enter debug mode, the file will still be open, which is probably what you want. Even the protected_f functions themselves, when you isolate them like this, are pretty clear.


Sun Oct 06, 2013 8:58 am
Profile
Yorick Guru

Joined: Wed Nov 24, 2004 12:51 pm
Posts: 97
Location: Observatoire de Lyon (France)
Post Re: yapi.h / C extension questions
To simplify testing and development, I am really interested in calling Yorick functions (interpreted and builtin ones) from C-code itself called by Yorick. For instance, I have numerical optimization routines which expect to get pointer to a function and I want to write this function in Yorick code. In the past I used to apply "reverse communaication" rules which involves splitting the code around function calls and saving/restoring local variable in/from some workspace. This is feasible but cumbersome and imply heavy modifications of complex but otherwise well tested code.

I first achieved this with yexec_include: I have a small interpreted wrapper (_yc_worker) which retrieves the function and its arguments (a single argument to simplify) from external variables (here _yc_f and _yc_x respectively) and stores the result into another external variable (_yc_fx).
Code:
plug_in, "ycall";

extern _yc_bisection; /* link to a built-in *private* function */

func _yc_worker {
  extern _yc_fx;
  _yc_fx = _yc_f(_yc_x);
}

func yc_bisection(f, a, b, rtol)
/* DOCUMENT c = yc_bisection(f, a, b, rtol);
     Find a root of f(x) between A and B with a relative precision RTOL. */
{
  local _yc_fx, _yc_f, _yc_x, _yc_code;
  eq_nocopy, _yc_f, f;
  _yc_code = strchar("_yc_worker;"); // the piece of code to interpret and execute with yeec_include
  return _yc_bisection(a, b, rtol);
}

In the C-part of the plugin, I have a wrapper with prototype
Code:
double f(void*data, double x);
which will be called by the C routine which search the root of a user defined function (DATA is anything needed by the user defined function and X is the function argument). The C-code of the plug-in is as follows (to simplify, I omitted the code for the bissection function):
Code:
#include <yapi.h>

extern double bisection(double (*f)(void* ctx, double x), void* ctx,
                        double a, double b, double rtol);

/* Indexes into Yorick symbol table (will be initialized by first call). */
static long f_index = -1L;
static long fx_index = -1L;
static long x_index = -1L;
static long code_index = -1L;

static double wrapper(void* ctx, double x)
{
  double result;

  ypush_double(x);          // push the argument
  yput_global(x_index, 0);  // define the external variable _yc_x
  ypush_global(code_index); // get code stored into external variable _yc_code
  yexec_include(0, 1);      // "execute" the code (actually just calling _yc_worker)
  ypush_global(fx_index);   // push current value of external variable _yc_fx on top of stack
  result = ygets_d(0);      // get contents of topmost stack item (that's the function return value)
  yarg_drop(3);             // clean-up stack
  return result;
}

void Y__yc_bisection(int argc)
{
  double a, b, c, rtol;

  if (argc != 3) y_error("bad number of arguments");

  if (code_index < 0L) {
    /* Initialize indexes. */
    f_index = yget_global("_yc_f", 0);
    fx_index = yget_global("_yc_fx", 0);
    x_index = yget_global("_yc_x", 0);
    code_index = yget_global("_yc_code", 0);
  }

  /* Get remaining arguments (function f has already been taken into account). */
  a = ygets_d(2);
  b = ygets_d(1);
  rtol = ygets_d(0);

  /* Call root-finding code and push result. */
  c =  bisection(wrapper, NULL, a, b, rtol);
  ypush_double(c);
}
This seems a bit complicated but it works...

However, I suspected that using yexec_include which involves interpreting and then executing some piece of code (even if it is very small) has some non-negligible overhead (I measured something aroung 1.6 microsecond per call of the wrapper which is several thousand cycles on my machine!). So I tried to avoid that and directly call the "Eval" member of Yorick objects -- going back to the old pre-yapi times ;-). To that end: I push 2 stack symbols (first the function object then its argument), prepare an Operand structure, call op->ops.Eval(op) and, then, my expectation was to simply get the result from the top of the stack and, after cleaning the stack, just return this value. But, except for built-in functions, after the call to Eval, the topmost stack item is a returnSym with some virtual machine code which is not the result I expected. So my question is how do I execute the VM code? YRun() is my guess but it is only called by DoTask() and there are a lot of housekeeping around which I am sure is not optional...

Note: according to my timings (and of course with a built-in function) directly call the Eval method is more than 10 times faster...


Tue Dec 10, 2013 9:53 am
Profile WWW
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post Re: yapi.h / C extension questions
I have long deprecated this sort of thing, because interpreted code is so slow relative to compiled code, you are unlikely to be satisfied with the result. That is why there is no interface in yapi.h, although as you note, you can abuse the yexec_include function to get what you want. Of course, when you do that, the overhead includes the time to parse the string you passed to yexec_include. That could slow it down quite a bit if the interpreted function is very tiny -- I don't have a clear idea where the breakpoint would be.

If you insist on doing this, you need to build a sequence of yorick VM instructions by hand, set the interpreted program counter pc by hand (of course saving its previous value), then call YRun. The correct sequence of instructions is in task.c:y_on_idle, near the top in the taskCodeInit block that sets taskCode. You will set taskCode[1].count to your argument count; otherwise it will look just like taskCode[]. Set pc like when task.c:DoTask sets pc=taskCode and call YRun. --->

Oops. That won't work, because YHalt changes the critical ym_state flag, which is static in task.c. So you'll need to write your own version of YHalt, which simply omits the line that modifies ym_state. And you'll need to write your own version of YRun, which doesn't call p_abort when p_signalling is set. But that should be pretty easy -- they're both only a few lines long. In fact, forget YRun -- all you need is the two line while loop, and a custom YHalt that just sets p_signalling to -1. Be sure you save the number of stack elements and drop anything left after you break out of your while loop, and set p_signalling back to 0. If you get p_signalling anything other than -1, you have to clean up your mess and re-instate the original pc and stack state before you call p_abort.

God help you if anything goes wrong inside your interpreted function -- almost certainly that will cause yorick to die a violent death. But as long as you write perfect bug free code, it's got a chance. By the way, that's the other reason I don't support this. If you get an interpreted error inside your function, you'll hose the interpreted stack when it aborts and longjumps out. So don't get any errors.


Fri Dec 13, 2013 9:55 pm
Profile
Display posts from previous:  Sort by  
Reply to topic   [ 10 posts ] 

Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by STSoftware for PTF.