Reply to topic  [ 6 posts ] 
Linking under OsX 
Author Message
Yorick Master

Joined: Sun Sep 26, 2004 10:33 am
Posts: 150
Location: Australia
Post Linking under OsX
Some of us have had problems recently with linking yao. The issue arised after I removed some functions from yao, as they already were in imutil. But that means now yao becomes a special case: it is the only plugin (I know of) that actually link against *another* plugin. This is not problem at all under linux, but appears to be a problem under OsX. This is an issue for yao right now, but will be an issue for anyone wanting to create general purpose yorick plugin libraires, so I guess we ought to solve it.
I have tried many different things, including the regular:
Code:
yorick-yao-4.8.2 $ cc -O   -bundle -bundle_loader /Users/frigaut/yorick-2.2/relocate/bin/yorick -o yao.so aoSimulUtils.o utils.o yao_fast.o ywrap.o -L/Users/frigaut/fftw/lib -lfftw3f -lm imutil.so
ld: in imutil.so, can't link with bundle (MH_BUNDLE) only dylibs (MH_DYLIB)
collect2: ld returned 1 exit status
which fails as indicated. I have tried to create dylibs with the --dynamiclib flag to gcc when creating the imutil, but that fails too.

The only trick I found to make it work is the following: Together with the bundle shared library (.so), create a static library (.a). E.g. in the problem of interest, when building and linking imutil, do that after the make:
Code:
libtool -static -o imutil.a bilinear.o imutil.o insort.o sedgesort.o spline2.o

Then copy it in the regular plugin library directory (should be Y_HOME/lib, get Y_HOME from within yorick). We should eventually modify the Makefile for that, I'll keep that for a future job.

Then when you link yao, instead of the regular:
Code:
cc -O -bundle -bundle_loader ...

use:
Code:
cc -O   -bundle -bundle_loader /Users/frigaut/yorick-2.2/relocate/bin/yorick -o yao.so aoSimulUtils.o utils.o yao_fast.o ywrap.o -L/Users/frigaut/fftw/lib -lfftw3f -lm /Users/frigaut/yorick-2.2/relocate/lib/imutil.a

This will produce a significantly bigger (in my case 1MB instead of 20kB) yao.so, but at least it works, waiting for a more elegant solution.


Tue Dec 21, 2010 8:00 am
Profile WWW
Yorick Guru

Joined: Sat Jan 22, 2005 2:44 pm
Posts: 86
Location: Pasadena, CA
Post Linking under OsX
I have thought of doing something similar, but my understanding is that, in such cases, it would be a better solution to create a common C library called by both plugins independently. Package distribution would still be a bit different...

$0.02
Thierry


Wed Dec 22, 2010 9:48 am
Profile YIM
Yorick Master

Joined: Sun Sep 26, 2004 10:33 am
Posts: 150
Location: Australia
Post 
Thierry,

Not sure I understand. The "common C library" would itself be a plugin, right? So this is almost exactly what we are dealing with here, and what I was describing.


Wed Dec 29, 2010 5:02 am
Profile WWW
Yorick Guru

Joined: Sat Jan 22, 2005 2:44 pm
Posts: 86
Location: Pasadena, CA
Post 
I am not sure that I fully understand the problem... Several plugins may link to the same library. Think of the math libraries for example (-lm.) It might be a solution to write a common (typically C) library containing the required functions from "imutil" and "yao." That library can then be independently linked by both plugins. Off course that assumes that there is little interpreted code used as wrapper, because that will still have to be "duplicated" between packages. Another example would be an "fftw" package, and some other plugins that would require that functionality. It is always possible to rewrite a common C-library which minimizes wrapper code duplication in both plugins.

... just a thought.


Wed Dec 29, 2010 10:57 am
Profile YIM
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post 
There is a big difference between a dynamic library which is loaded at runtime by the equivalent of dlopen, and a dynamic library which is loaded at execute time by the equivalent of ldso. In the former case, the linker never saw the library when the dlopen caller was linked; in the latter case, the shared library was known to the linker when the code was built. Hence ldso does only part of the task that dlopen must do, since part of dlopen's job has already been done at link time.

The MacOSX "dlopen" equivalent is a 5-sigma case bearing no relation to any other UNIX dynamic loading scheme. In fact, it has been evolving steadily, especially in the early MacOSX releases, 10.1, 10.2, and 10.3. There are in fact several different choices for how to make a dynamically loadable library under MacOSX. The one I chose for yorick plugins -- called a "bundle" -- has since been deprecated by Apple. The favored scheme is called a "dylib" (as opposed to a "bundle", but beware that "bundle" has a second, more common, unrelated meaning relating to application packaging).

The problem with a "dylib" is, that it is nearly impossible to set up a "dylib" in such a way that it can call functions in the executable which calls it. If you think about it, this is totally backwards from the dependency a library is supposed to have -- for example, a libm function like cos(x) will NEVER call a function in the executable which calls it, while all yorick plugins MUST call many functions (for example, in yapi.h) in yorick, the executable which links to the plugin.

This is a serious problem with all plugin architectures. You can amuse yourself by learning how, for example, firefox handles this same issue. The answer is not very helpful for yorick, because it is neither simple nor easy to maintain across multiple architectures.

I had almost given up on plugins for MacOSX before I ran across the "bundle". If Apple has now made good on their implicit promise to break all code using "bundles", that is bad news for yorick plugins.

There are several possible workarounds, but they all involve a total rewrite of the entire yorick plugin system, probably on all architectures to try to avoid a completely separate yorick source on MacOSX. The thing the UNIX guys want you to do is to make all of yorick into a shared library -- that is, libyor.so instead of libyor.a. Then all plugins can link against yorick as a shared library, and they can naturally call the yapi.h interface, blithely unaware that yorick itself is calling them. This is extremely difficult to do in a portable way, because the various UNIX flavors differ tremendously in exactly where and how shared libraries must be installed in order for the executable to start. UNIX has its own brand of "dll-hell" which invovles users who get their LD_LIBRARY_PATH environment variable (it has other names on some systems) set incorrectly...

You have a pretty simple workaround, which I encourage you to try before we make the big investment in a new plugin system (almost certainly requiring a rewrite of every existing plugin, to fix their Makefiles):

It sounds like the only plugin which is called by other plugins is imutil. If so, when you build imutil, simply do this:
Code:
make TGT=exe
make install

This will build a static version (.a) of imutil, and a version of yorick which has imutil built into it as if it were a part of libyor.a. Whenever you run this version of yorick as "yorick -batch make.i" to build the Makefile for some other profile, it will arrange to load your imutil static library.

You can repeat this as often as you like -- in other words, your imutil-yorick can statically build another plugin with TGT=exe, and all future plugin builds with that version of yorick will load both the imutil .a library and the new .a library.

You should only build plugins with TGT=exe in cases which you know multiple plugins will call functions in that plugin. In this case, it sounds like you only need to use TGT=exe when you build imutil. And of course, you only need to do it on new Macs for which bundles are broken...


Mon Jan 03, 2011 9:26 pm
Profile
Yorick Master

Joined: Tue Mar 07, 2006 10:31 pm
Posts: 125
Location: Meudon, France
Post Re: Linking under OsX
Hi guys,

I, too, am building a pair of plug-ins where one of the plug-ins (call it the client) has to call symbols from the other one (the server) and I have found a way to do it properly.

I have seen somewhere on this forum a suggestion by which the server would be compiled as a static library built into the client. In my case, this does not work because I use y_userobj_t objects: both the client and server can create and manage the same kind of objects (the client is an extension of the server). However, if I ypush an object from the client and yget it in the server, Yorick won't see that this really is the same kind of object, because it uses a pointer to a static variable to do that, and we have two copies of this variable.

My trick is to manually do what the linker ought to do : export the symbols from the server and import them in the client. Really, it is much simpler to do than it appears (in fact, I spent less time implementing it than I take explaining it here). Basically, my server has two methods to export:
Code:
  ypush_MyObject();
  yget_MyObject(int iarg);

(but I could have as many as needed).

In the server, I initialize a static variable which is a structure containing pointers to these functions. In pseudo code:

Code:
static struct MyAPI {
      function * ypush_MyObjet;
      function * yget_MyObject;
}

The server also provides a function which can be called _from the interpreter_ to retrieve a pointer to MyAPI:
Code:
void Y___exportMyAPI(int argc ){
MyAPI.ypush_MyObject = &ypush_MyObject;
MyAPI.yget_MyObject = &yget_MyObject;
ypush_long(&MyAPI);
}

(This function will also initialize MyAPI by setting each member to the respective function pointer, which, AFAIK, cannot be done at compile time).

On the client side, I also have a static variable which will be a pointer to MyAPI and which will be initialized from the interpreter:

Code:
static MyAPI_t * LocalAPI;
void Y__importLocalAPI(int argc) {
LocalAPI = ygets_l(0);
}


from the include file of the client (client.i) I just have to do:
Code:
extern __importLocalAPI;
__importLocalAPI, __exportMyAPI;


Now, in the client, instead of calling ypush_MyObject or yget_MyObject directly, I need to call LocalAPI->ypush_MyObject and LocalAPI->yget_MyObject. To ease this task, I can have to preprocessor macros, defined in server.h (which will need to be included in client.c anyway):
Code:
#define ypush_MyObject LocalAPI->ypush_MyObject
#define yget_MyObject LocalAPI->yget_MyObject


Actually, my server plug-in provides two files: a server.h which contains the API and the above #defines, and a server_import_API.c which implements the "import" procedure. Thanks to a neat preprocessor macro game, you don't have to modify it for each client: each client copy just compiles it with a macro defined and links against it, a bit like that:
Code:
gcc -o client_import.o -DAPI_IMPORTER=ClientImporter /path/to/yorick/include/server_import_API.c

This way, the name of the import function will be something like "Y___ClientImporter", different for each client.

If people want more explanations or a real-life example, feel free to ask (or look at https://github.com/gyoto/Gyoto/blob/mas ... k/ygyoto.h).

Regards, Thibaut.


Sat Mar 19, 2011 12:36 am
Profile WWW
Display posts from previous:  Sort by  
Reply to topic   [ 6 posts ] 

Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by STSoftware for PTF.