Reply to topic  [ 11 posts ] 
plug_in with C++ source 
Author Message
Yorick Guru

Joined: Sat Jan 22, 2005 2:44 pm
Posts: 86
Location: Pasadena, CA
Post plug_in with C++ source
Trying to produce a module from C++ code (blitz).

With mods to "make.i", "Make.cfg", and "Makepkg", I can produce a shared object file "*.so". Unfortunately I cannot load it from yorick, even though it is at the correct location (Error: "unable to find dynamic library file")

Any ideas?


PS: It would seem that the easiest way to call C++ from C would be to "wrap" the C main into a C++ main... However, I don't know how I would do this with Yorick....


extern "C" void addbz(double* a, double* b, long n);


void addbz(double* a, double* b, long n)
Array<double,2> A(a, shape(n, n), neverDeleteData, fortranArray);
Array<double,2> B(b, shape(n, n), neverDeleteData, fortranArray);

A = A + B;

Thu Apr 21, 2005 4:27 pm
Profile YIM
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
I'm not too surprised. One suggestion is to build all of yorick with
C++ instead of C. You might have to make a few minor adjustments to
the source, since C++ has hundreds more reserved words than C, and
I may have inadvertently used a few (e.g. "new" or "class"). Let me know if
that is the case (and give me a list). Something like
env CC=g++ make config
(or CC or cccc or c++ or whatever) ought to do the trick.

The other thing you might not have experimented with is building a custom
yorick instead of attempting to load your package dynamically. If you do that,
you may need to compile yorick's main (play/unix/pmain.c) using C++ (but I
hope not); you'll certainly need to override the load command in your package
Makefile to use C++ instead of C. Start by changing to TGT=exe near the top of your
Makefile; override the load macro LD_EXE below the three include lines.

Assuming the above workarounds all fail, here is what you need to concentrate
on: Can you build any C++ library which can be called by a C main program?
(In other words, is C++ a general purpose programming language, or are C++
libraries only callable by C++ programs?) The way to answer this question is to
start with a "Hello world" library written in C++. I can tell you immediately that the
biggest problem is going to be C++ programs which invoke initializers that run
before main() (or before any routines in your shared library). That is the biggest
difference between C and C++: no part of a C program runs before main(), while most
(or very many) C++ programs run a lot of intializer code before main() is ever called.
Anyway, at minimum your C++ "Hello world" ought to invoke an initializer before
anything is called. The C program which calls C++ "Hello world" needs to consist of
at least two files: a trivial main (like play/unix/pmain.c), and a second file which
actually invokes the C++ function. There are a couple of scenarios for you to try
out: First, can you compile the main with C, provided you perform the load with C++?
That would be really great, but if it proves impossible, that's still not a great blow,
since I'm fully prepared to force you to recompile main.c with your C++ compiler in
order to build a custom yorick. What HAS to work is that your C worker file needs
to be compilable with C.

(By the way, I think your example program is both too complex and too simple - too
complex because it invokes some non-standard array library, and too simple because
it doesn't contain any static intiializers, unless the library does. You may need to
prepare an example that does not rely on pre-main initializers, since one possible
outcome of this study is that the absence of such initializer calls is a prerequisite for
C++ code which can be dynamically loaded into yorick.)

Once you've got all that working for the statically loaded case, you need to move on
to the infiintiely more complex dynamically loaded case. (Note that at this point you
will already know how to build a custom version of yorick which includes your C++
package, so you will constantly be asking yourself, why am I doing this?) Now you
need to modify the C part of your test code, the one that calls C++ "Hello world", to
call "dlopen", then use "dlsym" to get the address of the "Hello world" interface routine.
This time, the rules change: you now must both compile and load your main program
with plain C, whilie you compile and load the "Hello world" shared library with C++.

Actually, come to think of it, the first thing to do is to build everything with C++
and see if you can get "Hello world" working at all under C++ as a dynamically
loaded library using "dlopen/dlsym". You will probably find out that doesn't work
immediately either, and that you need to take some special actions to make it work.
There's a pretty good chance that once you can make dlopen/dlsym work at all in a
C++ program, that you will know how you have to construct your "Hello world"
program in order to make it operable using dlopen/dlsym. Once you can make this
20 lines of test code work, you will probably be home free -- just fix up your actual
yorick package so that you can dynamically load it, like you learned to do with your C++
"Hello world".

At this point, you can modify your C++ package code and your package Makefile
(you certainly need to modify your package Makefile, and I'm betting you need to
modify your C++ code), and you'll be in business. I'll be interested to here how you get
on, but only a few trivial improvements to the yorick source are likely to result from all
your effort so far. The problem is, that your Makefile modifications most likely will not
be correct for platforms other than the one you are working on, because special
compiler switches and dynamic loading systems have no standards and in fact differ
across platforms. It is also quite possible that any modifications you need to make in
your C++ code to make it dlopen/dlsym callable will not be portable. To be really useful
to me, you need to repeat the exercise across all paltforms where yorick runs.

The platforms which have totally unrelated dlopen/dlsym (in fact those are just the Linux
function names) are Linux, MacOS X, AIX, HPUX, and MS Windows. It would be
surprising if the fixes you need on any one of these operating systems bear any
relation to the fixes you need on any other, so I'm afraid you need to start fresh on each.
(By the way, dynamic loading in yorick itself may not work under HPUX, because I no
longer have access to any HPUX machines.) If you do all that, I can actually make the
changes in yorick necessary to support dynamic loading of C++ packages, but until
then it's pointless to think about modifying yorick itself. The main problem with dynamic
loading is that there is no standard; that's why I've been so reluctant to add it to yorick.
It took almost a year of mindnumbing labor to get dynamic loading working on just the
majority of platforms we commonly use here at Livermore. I didn't feel the need for the
additional challenge of making it all work for C++ programs as well as for C programs...

A more modest goal is to learn how to build simple, statically loaded, custom versions
of yorick with C++ packages. That will be possible by simple modifications to your
package Makefile, which are likely to be fairly portable.

Unless we get some additional comments here indicating strong general interest in this
topic, I suggest we take this up by email until we have some answers to report here. So
if anyone wants to see this discussion blow by blow, or to contribute (especially if you
commonly use a platform other than Linux or MacOS X), post something here or send
Thierry and me an email.


Mon Apr 25, 2005 9:41 am
Yorick Guru

Joined: Sat Jan 22, 2005 2:44 pm
Posts: 86
Location: Pasadena, CA

Thank you for your reply. I hope to be able to report on some kind of progress, but it will take "some" time.

I had tried to build Yorick with CC/g++, but need to pursue this further. I will also try to link the module statically. (That should have been the first thing I tried.)

I am not able to call C++ from a C main in general. "hello world" works, but I in the "blitz" case I am missing a few symbols (4, all having to do with I/O, though there are no IO calls in the C++ code.) It may have to do with initialization of some static classes... A three lines wrap of the C main does the trick.

I will look at each suggestion as carefully as I can. And, yes, sticking to C for modules should be all that is ever needed. Being able to embed a blitz module is just a shortcut that's probably not worth the trouble.

BTW, I moved from static modules to plugins effortlessly on SGI and Linux (OS X next), and It was a breeze to install the plugins from this site: many thanks to all. The modules I have unfortunately are not of general interest (old f77 geophysics stuff, convenience modules for specific databases ...)

Long live Yorick,

Mon Apr 25, 2005 10:32 am
Profile YIM
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post The Fix
Well, after my diatribe, I'm ashamed to say that the solution to the problem of C++
in plugins seems to be quite simple. Thierry has tried this on MacOS X and SGI,
and it seems to work there as well as under Linux. Someday I'll get around to checking

The fix is to add three lines to your package Makefile, if your source includes C++ files,
In addition to adding the corresponding .o files to the OBJS= line. These are:


The LD_DLL and LD_EXE lines must come after the include lines in order to override
the definitions in Makepkg. The CXXFLAGS line can go anywhere. The CXXFLAGS
line assumes that the C++ compiler accepts the same optimization and other options as
the C compiler (-fPIC is the critical one for plugins under Linux). The LD lines ensure
the the plugin or executable are loaded using C++ instead of C, both in order to
automatically get any C++-specific libraries, and to generate special C++ code to
handle static constructors.

No doubt this is not a complete solution, but it ought to suffice for most people. If the
dynamic library does not work, there is still a better chance that
make TGT=exe
will build a functioning custom executable.

I will modify future versions of make.i to recognize .cc source files and insert these lines
in the automatically generated Makefile, so users won't have to puzzle them out for
themselves. For now, you'll need to insert them by hand.


Fri Apr 29, 2005 7:24 pm

Joined: Thu Feb 16, 2006 12:42 pm
Posts: 4
Location: Yale University
Are those lines all that are necessary to compile cpp code? I've been mucking about with this trying to use some CERN ROOT libraries, and I can't even get a "hello world" program to work. It will compile and link, but when I actually call the code I get an "undefined symbol" error and yorick exits completely. I'm using linux by the way.

Thu Mar 30, 2006 7:24 am
Profile WWW
Yorick Master

Joined: Tue Mar 07, 2006 10:31 pm
Posts: 125
Location: Meudon, France

I guess a lot of progress as been made since these messages were posted, because I was able to make a working C++ plugin fairly easily. The Makefile created by make.i contained a few special lines for C++ code, which is really great. A sample plugin in C++ is online.

I was wondering if someone had some experience with C++ plugins by now and had some tips on how to design such a plugin in a sane way.

In particular, I am afraid of memory leaks. When designing a C plugin, it is good practice to avoid any malloc and let Yorick do the memory management, but I'm not sure how to do this for a C++ plugin.

The way I've designed my C++ plugin is very simple: I create my objects using "new" and store a pointer to these objects in a Yorick long. Of course, that means that if I forget to "delete" my objects before the Yorick variable holding its address is destroyed...

Has anyone come up with a reasonable strategy to take care of this issue? Is there a way to tell Yorick to call a destructor when such or such variable is destroyed? I could certainly store a table with all these pointers somewhere and have a routine to just destroy them all when needed, but that looks somewhat desperate and of little interest compared to just restarting Yorick...

Best regards, Thibaut.

Wed Nov 19, 2008 10:43 am
Profile WWW
Yorick Guru

Joined: Sat Jan 22, 2005 2:44 pm
Posts: 86
Location: Pasadena, CA
I have not pursued this. Even with an "old" library like blitz++, I ran into some portability issues (pure cpp/blitz will not compile on some of the boxes I need to work with.) As you know, the pointer destruction issue you are describing also concerns C. I have a similar situation with FFTW "plan" pointers which I store in yorick longs, but that need to be "manualy" deleted using the provided libray function. The hdf5 plugin has similar pointers.


Fri Nov 21, 2008 12:04 pm
Profile YIM
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Boy, you guys have lost me completely.

The only persistent objects (in C or C++ plugins), that is, objects which last longer than the call to your plugin function, would be pointed to from an opaque object you created and returned (or set as a side effect). Since the interface for opaque objects in yapi.h almost forces you to provide a destructor for any opaque object you create, obviously the destructor you provide should properly clean up whatever you built. Whether your cleanup involves calling the matching flavor of free to the malloc you used to create your object, or the matching delte for the new that created your object makes no difference at all. Yorick could care less how you manage your own memory. You certainly do not have to use the yorick p_malloc and p_free for your objects -- yorick is completely indifferent and neutral to the memory management scheme you choose for your plugin. (Everything from X11 to libpng does heavy memory management internally, which yorick knows nothing about.)

To the best of my knowledge, memory management and memory leaks under yorick are precisely the same as they would be if you linked your plugin into a native C or C++ code rather than into yorick. Your plugin creates an object, and you supply a destructor function to clean it up -- there is no difference at all from what you do if you were designing a library to be called in any other way. What am I missing here?

Sat Nov 29, 2008 11:00 am
Yorick Master

Joined: Tue Mar 07, 2006 10:31 pm
Posts: 125
Location: Meudon, France
munro wrote:
Boy, you guys have lost me completely.

Well, indeed I guess we lost track somewhere...

The problem is that I don't know of a beginner's guide to YAPI, so the learning curve for doing anything beyond what's documented in the Yorick manual is pretty steep.

I'll be looking more thoroughly at yapi.h.

Regards, Thibaut.

Wed Dec 03, 2008 7:30 am
Profile WWW
Yorick Master

Joined: Tue Mar 07, 2006 10:31 pm
Posts: 125
Location: Meudon, France

Just a message for my fellow would-be-plugin-authors who don't dare reading yapi.h:

I confess that I was a little bit impressed by yapi.h and thought using it would be very complex. But finally I have found out that it's not that hard, and that the added complexity is very localized: I can still reuse most of my code (read on to see why).

I've looked more carefully at yapi.h and I have read again this post.

Indeed with some concentration I was able to rewrite some of my functions which used to return a pointer as a Yorick long so that they now return an opaque object.

I have provided a destructor, so no home-keeping required anymore.

The nice thing is that I didn't have to change all the functions which use this pointer: I wrote a simple on_eval function which returns the pointer as a long. So whenever I used to call a function like this:
I now call it like that:

So indeed just converted two functions which used to return the original pointer, and add "()" a few times. That's all. Not overly complex in the end.

Wed Dec 03, 2008 10:35 am
Profile WWW
Yorick Guru

Joined: Sat Jan 22, 2005 2:44 pm
Posts: 86
Location: Pasadena, CA
I also wonder if a C wrapper is absolutely necessary, or if it might be possible to use YAPI directly from within the C++ code -- since "yapi.h" has the appropriate "extern C," it should be possible. So far I have not been able to do so... but I am not knowledgeable enough to know for sure. When I try
including "yapi.h" from source compiled as C++, the compiler does not see the yapi prototypes.

Wed Apr 15, 2009 9:16 am
Profile YIM
Display posts from previous:  Sort by  
Reply to topic   [ 11 posts ] 

Who is online

Users browsing this forum: No registered users and 1 guest

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by STSoftware for PTF.