Reply to topic  [ 15 posts ] 
Yorick Wrappers for C or C++ files 
Author Message
Yorick Padawan

Joined: Tue Jun 07, 2011 1:46 pm
Posts: 32
Location: UNH
Post Yorick Wrappers for C or C++ files
Hi,

I was interested in creating a wrapper for GDAL and libLAS to use in Yorick and I looked at the plugin SOY as an example. I was just curious, to create a plugin wrapper, do I just need one C or C++ file with the same name as the plugin .i file with a huge list of extern commands for each C/C++ function?

Also, in those libraries, there are 100s of .c or .cpp files. How do I include all of them?

Thank you,


Wed Sep 12, 2012 7:45 am
Profile
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post Re: Yorick Wrappers for C or C++ files
Read http://dhmunro.github.com/yorick-doc/ma ... #Embedding and study the extend/ example subdirectory in the yorick distribution. The short version is that you use yorick -batch make.i to create a package Makefile template (after you've got a first cut of all your source code). You need at least one "package" .i file (which will go in the i0/ installation directory). The extern declarations in that file define the yorick interface to your compiled routines. If you have a big library you've gotten elsewhere, you probably want to either put it by itself in a subdirectory of your package directory, or better yet omit it entirely and assume it is properly installed in your system /usr/include and /usr/lib. I can't answer any questions about how to manage your third party libraries; that's exceedingly difficult in general and beyond the scope of the yorick packaging system. Typically, you will want to provide a reduced yorick interface designed for interpreted use of your libraries. If you do decide to wrap every single C API, you still need to declare them all in your .i file. There is no tool to do that automatically; you also have to write all the code required to check correctness of arguments passed to library functions.


Sun Sep 16, 2012 3:44 pm
Profile
Yorick Padawan

Joined: Tue Jun 07, 2011 1:46 pm
Posts: 32
Location: UNH
Post Re: Yorick Wrappers for C or C++ files
Hi munro,

I read the article you linked. Just to make sure I understand this. Is this the correct method to port a 3rd party library?

  1. Prerequisites:
    1. It has to be written in ANSI C (possibly C or C++ but may have some problems)
  2. Compile 3rd party library. In my case, I installed liblas1, liblas-dev, liblas-bin on Ubuntu
  3. Create a directory for mypkg
  4. Create mypkg.i file in the the /mypkg directory with the following code.
    Code:
    plug_in, "mypkg";

    extern, function1;
    /* PROTOTYPE
       double function1(double array input, long length)
    */
    extern, function2;
    /* PROTOTYPE
       double function1(double array input, long length)
    */
    ...
    extern, function3000;
    /* PROTOTYPE
       double function1(double array input, long length)
    */
  5. Place all other .i, .c, .h (.cpp?) in the /mypkg directory
  6. Run the following command in the terminal in the subdirectory to create a makefile template
    Code:
    yorick -batch make.i
  7. Edit makefile to add loader flags (-l -L) for dependencies and compiler flags (-I) to find header files.
  8. Run
    Code:
    make install

Few questions:
  1. I wasn't able to find the extend/ example subdirectory. Where is that located?
  2. Does this work with .cpp files? or do I need to convert all code to ANSI C?
  3. The PROTOTYPE keyword is used for C functions that do not start with "Y_", correct?
  4. Could you further explain
    Code:
    reshape, my_global, datatype;
  5. Could you further explain the addition of the loader and compiler flags into the make file?
  6. The article didn't mention any /i0 directory. Do I make a copy of the mypkg.i file (with the extern statements) in that directory?
  7. I don't really understand the whole yapi.h method. Would it be better to write a C wrapper or use yapi.h to port a library? I want to be able to port this as easily as possible.

R


Fri Sep 28, 2012 2:09 pm
Profile
Yorick Padawan

Joined: Tue Jun 07, 2011 1:46 pm
Posts: 32
Location: UNH
Post Re: Yorick Wrappers for C or C++ files
I also followed the embedding example.

rtleg.i
Code:
plug_in, "rtleg";
func rtleg(r, x)
{
  /* convert r to double, broadcast it to result shape */
  r = double(r) + array(0.0, dimsof(r, x));
  /* broadcast x to result shape, blow up if not conformable */
  x += 0.*r;
  _crtleg, r, x, numberof(r);
  return r;
}
extern _crtleg;
/* PROTOTYPE
   void crtleg(double array r, double array x, long n)
*/


rtleg.c
Code:
#include <math.h>
extern void crtleg(double *r, double *x, long n);
void
crtleg(double *r, double *x, long n)
{
  double y;
  while (n-- > 0) {
    y = *(x++) / r[0];
    *(r++) *= sqrt(1. - y*y);  /* result overwrites r input */
  }
}


Ran:
Code:
yorick -batch make.i
sudo make install


and it created Makefile, rtleg.o, rtleg.so, ywrap.c, and ywrap.o. What do I do with these files to be able to run crtleg inside yorick?

R


Fri Sep 28, 2012 2:16 pm
Profile
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post Re: Yorick Wrappers for C or C++ files
I'm sorry that I don't have time to lead you through this in detail. Here's a few quick hints:

The extend/ directory is in the yorick source code. You probably want to obtain the whole yorick source tree from http://github.com/dhmunro/yorick, and follow the instructions in the comments of the extend/ directory after you've built the code. However, you can also just copy the extend/ directory and use your installed yorick.

If you are installing into your system /usr/lib (with sudo make install in your example), you have made your package available to anyone running the system yorick. Just start yorick, include,"rtleg.i", and start using your rtleg function. If you specify a separate autoload .i file to be installed in the i-start/ directory (it's name is one of the Makefile macros), you won't need the explicit include -- it will happen automatically when you reference rtleg. Look in the installed i-start/ directory (a child of Y_SITE, which is the name of a yorick variable you can print) for examples of autoload .i files.

The automated make.i will not find .cpp files; you will need to add those by hand. More importantly:

Calling C++ functions from a C program is exceedingly difficult on most platforms, because C++ programs have a whole bunch of stuff which runs before your main() entry point. I warn you it is depressingly platform-dependent as well, so that what you learn about how to do it on Linux isn't likely to help you on Mac OSX or Windows. You are on your own if you try to write a C++ package for yorick. You will need a deep understanding of both C and C++ to do it, and you will need to heavily customize yorick's Makefile system (read the Makepkg file in the top level of the yorick source tree). It may be possible for a C program to dlopen (a system call) a dynamic library loaded with a C++ compiler, although that will be a highly platform and compiler dependent operation. If that's not possible on your platform, you will need to build yourself a custom version of yorick itself which is loaded using a C++ compiler (prebuilt binaries will all be loaded with a C compiler). I simply don't have the time to lead you through such an effort myself; personally, I've always decided that C++ is just not worth the effort.

The PROTOTYPE comments are for automatically generating wrapper functions. You can omit them if you are willing to write your own wrapper using the yapi.h interface. If you use PROTOTYPE comments only, you never need to use yapi.h; instead, you will need to write interpreted wrapper functions to implement argument type checking and generate array length arguments.

Read the comments in the Makepkg file (top level of the yorick source tree at github), and the comments in the Makefile automatically generated by "yorick -batch make.i", for complete descriptions of how to pass additional flags to the loader. This may not be easy, you need to understand the UNIX make utility fairly thoroughly, at the level of the user manual for GNU make, for example. I don't have time to lead you through that; you'll need to experiment to get the flags you need. Third party library dependencies, in particular, can be very tricky, especially if you want your source code to build on multiple platforms. You will probably need to write a configure script in that case. See the yorick-z package source code at github for an example.


Sun Sep 30, 2012 4:05 pm
Profile
Yorick Padawan

Joined: Tue Jun 07, 2011 1:46 pm
Posts: 32
Location: UNH
Post Re: Yorick Wrappers for C or C++ files
I didn't realize how difficult a task this would be. I thought of a quick work around. How about just installing the liblas and gdal packages natively in Linux and then call their functions using:
Code:
system, "lasinfo -i test.las"

and I'm not quite sure how to save output in Yorick to a variable but for now I was thinking
Code:
system, "lasinfo -i test.las >> info.txt"
...
some F I/O and string parsing functions to get the data I desire

Would this be better/easier than trying to write a wrapper (albeit slower)?

R


Tue Oct 02, 2012 3:06 pm
Profile
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post Re: Yorick Wrappers for C or C++ files
You can use the spawn function or the popen function to capture the stdout from a command.

The LAS file format is well-documented at http://www.liblas.org/development/specifications.html and is relatively simple. It would be fairly easy to train yorick to read LAS files without using liblas at all (somewhat harder if the data is compressed). See netcdf.i (help,nc_open) or idlsave.i (help,idl_open but if it has been removed from your binary distribution, look in https://github/com/dhmunro/yorick/tree/master/i) for examples of how to train yorick to understand binary file formats.

But it looks to me like the liblas software includes a C API (interface liblas.h, e.g.- in debian liblas-dev package), which would be easy to link into a yorick package. There are a couple of hundred functions in this API; surely you don't really want all of that -- presumably you just want to read a few arrays out of the file. The easiest thing to do is to write a few functions in C to open the file, perhaps query the contents, extract the arrays you want, and close the file, calling the C library -- this is one .c file. You then write a .i file with one extern statement and an associated PROTOTYPE comment for each function in your simplified interface. Alternatively, you can dispense with the PROTOTYPE comment and write your own yorick wrapper functions, probably in a second .c file, using the yapi.h interface. If you use the PROTOTYPE comments, you will almost certainly need to write interpreted wrappers to the raw compiled interface functions declared by the extern statements in your .i file. There are many complex examples of this in the hex/ and drat/ packages at the top level of the yorick source tree https://github.com/dhmunro/yorick.

As long as you stay away from the C++ API, it shouldn't present any serious challenges.


Wed Oct 03, 2012 8:06 pm
Profile
Yorick Padawan

Joined: Tue Jun 07, 2011 1:46 pm
Posts: 32
Location: UNH
Post Re: Yorick Wrappers for C or C++ files
I'm sorry to keep dragging this but we found another way to do this that seems simpler but I'm having trouble setting it up. Trying to use this link as a reference:

http://www.maumae.net/yorick/doc/plugins.php
Quote:
A package is a set of interpreted yorick include files. A plugin includes compiled C (or C++, or fortran) objects, to extend the capabillities of yorick (e.g. wrap libraries).

As of yorick-1.6, Yorick has the capabilities to load packages that include dynamic libraries (plugins). This has several big advantages:

- it solves a distribution problem. One can now distribute packages in binary form, and they can be loaded by the current yorick executable,
- although it is still possible, it is not anymore necessary to build a special version of yorick when linking to custom C libraries,
- an arbitrary number and combination of packages can be loaded in a given yorick session.

Linux and MacOsX:

- You can use pkg_mngr to install plugins and packages (#include "pkg_mngr.i"). This makes for painless install, the main advantage being that it includes external libraries (statically linked).
- You can also install plugins from our debian/fedora/mandriva repositories (recommended).

The installed file is located at /usr/lib/liblas.so.1.0.0
Judging from the quoted paragraph, could I just copy that file into the yorick/lib directory, include pkg_mngr.i, and install liblas.so.1.0.0 as a package and have yorick easily call liblas's compiled functions?

I made a liblas.i file with the following:
Code:
#include "pkg_mngr.i"
plug_in, "liblas"
func yk_liblas
{
   return "liblas"
}
//pkg_setup
pkg_install,"liblas"

but it's saying to run pkg_setup and when I do, it asks for a server to connect to.
Will the pkg_mngr link the .so file to the yorick code if I already have the .so file in the yorick/lib directory from the original installation of liblas?

Also when I include the liblas.i file, this is my output:
Quote:
> include,"liblas.i"
Use "pkg_setup" to set up pkg_mngr
ERROR (*main*) plug_in: dynamic library missing yk_<pkgname> function
WARNING detailed line number information unavailable
now at pc= 1 (of 11), failed at pc= 5
LINE: 2 FILE: /opt/eaarl/yorick/lib/liblas.i
To enter debug mode, type <RETURN> now (then dbexit to get out)

R


Fri Oct 12, 2012 1:45 pm
Profile
Yorick Padawan

Joined: Tue Jun 07, 2011 1:46 pm
Posts: 32
Location: UNH
Post Re: Yorick Wrappers for C or C++ files
Also, we work a lot with LAS files for remote sensing and it's very possible that all the functions contained within liblas will be used if we can integrate it seamlessly into Yorick.


Fri Oct 12, 2012 1:46 pm
Profile
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post Re: Yorick Wrappers for C or C++ files
You can only use the plug_in command to load libraries which you have created using the yorick interfaces described in the user manual. In particular, there has to be a yorick wrapper for each individual function you wish to call (declared with an extern statement in the package .i file, as described in the manual). There also has to be a function (generated automatically by the yorick package Makefile you create with make.i) which initializes the yorick package when it is loaded, telling yorick the name of the package .i file to include, among other things.

The only shared libraries you can load into yorick are its own plug-ins, build using the yorick Makefile/Makepkg build system. This is exactly the same as, for example, a plug-in for firefox or another browser -- you can't just tell firefox to load liblas.so and expect your browser to be able to read LAS files, no matter whether you copy liblas.so into the same directory as all your other firefox plug-ins or not!

So if you insist you need all 300 liblas APIs, you need to design a yorick interpreted interface with 300 built-in functions callable from the interpreter. There is no automatic way to do all that design work. The only partly automated step is if you have a function with a very simple interface, which accepts arrays of numbers or strings and returns arrays of numbers (returning strings is harder), then you can automatically generate wrappers using PROTOTYPE comments, as described in the user manual. You still need to design the PROTOTYPE comments, and ususally you will need to also provide an interpreted wrapper around the compiled wrapper to do sanity checking on the arguments you pass in, to prevent stepping out of array bounds accidentally, for example, which will almost certainly completely wipe out yorick in a way that its usual error recovery cannot handle.

This is exactly the same as the problem you face wrapping your library with python. Check out the existing python liblas package to get a feeling for the scale of the effort you will need to wrap the entire library for use within yorick. The python package involves exactly the same computer science you need for yorick: There will be special compiled code wrappers which invoke the liblas.so functions, created partly by hand and partly automatically, there will be special interpreted code wrapping those compiled functions, and the python interpreter will not directly call any of the functions in liblas.so -- instead, there will be a python-based shared library that serves as the python plug-in, which calls liblas.so.

I have no idea what kind of functionality you want out of this library. I also have no idea how complicated the individual LAS files you care about are. If they contain thousands of separate images with complicated interrelationships among them, then it will take considerable thought to figure out how you want to present that to a yorick programmer -- roughly the same amount of work as it took the liblas developers to create their python API, which differs considerably from their C API because of the differences between python and C. If, on the contrary, the LAS files you are interested in just contain a few arrays with straightforward meaning so that you understand what they are without needing to analyze a bunch of metadata, then you can have a very simple yorick interface -- just present the arrays of numbers to your users and you're done. Designing an interpreted API which gives you the functionality you need is simple in the latter case, but requires a lot of thought in the former case.


Mon Oct 15, 2012 8:04 pm
Profile
Yorick Padawan

Joined: Tue Jun 07, 2011 1:46 pm
Posts: 32
Location: UNH
Post Re: Yorick Wrappers for C or C++ files
Hi again,

I think I have a grip on this now. I am trying to include a couple of simple functions from liblas and one is very basic called usage. This function will just return the latest version number of las2txt ( LAS_GetVersion() ) and print out statements ( usage() ) on how to use it.

usage()
Code:
void usage()
{
    fprintf(stderr,"----------------------------------------------------------\n");
    fprintf(stderr,"    las2txt (version %s) usage:\n", LAS_GetVersion());
    fprintf(stderr,"----------------------------------------------------------\n");
    // other print statements
}

las2txt plugin yorick code
Code:
plug_in, "las2txt";

extern usage;
/* PROTOTYPE
   void usage(void)
*/

extern LAS_GetVersion;
/* PROTOTYPE
   void LAS_GetVersion(void)
*/

LibLAS file requirements: liblas.h, las_config.h, las_version.h, and liblas/export.hpp

And compiled it successfully
Code:
yorick -batch make.i
sudo make
sudo make install

When I run yorick, include the las2txt.i file from my /i0/ folder, and run usage, this is my output:
Code:
/opt/eaarl/yorick/bin/yorick: symbol lookup error: /opt/eaarl/yorick/lib/las2txt.so: undefined symbol: LAS_GetVersion

I did define it but I suppose that it needs to compile the original definition of LAS_GetVersion so I found it in a file called c_api.cpp which unfortunately is written in C++ code and is expressed in the docs that C++ would be a nightmare to create wrappers for.
c_api.cpp
Code:
LAS_DLL char* LAS_GetVersion() {
    return LASCopyString(liblas::GetVersion().c_str());
}

Does this mean, I look through LASCopyString and liblas::GetVersion.c_str() and convert them to C in order to create the necessary wrappers to use them? or is there a better way to do this?


Mon Jan 28, 2013 11:13 am
Profile
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post Re: Yorick Wrappers for C or C++ files
You are really, really confused about what is going on here. What you did was to create a Y_LAS_GetVersion wrapper that called a function LAS_GetVersion. Since you are linking dynamically, the linker (compiler) presumed that the missing function would be found in some other dynamically linked library at runtime. But you never told it which library or how to find it, hence the failure to find the symbol at runtime. To do that, you need to supply a -llas compiler option, and possibly a -L/directory/containing/liblas/ option, and possibly other more complex options depending on how your operating system locates dynamic libraries. (This is especially difficult for C++ libraries -- you may need to redefine the CC make macro or take other heroic action in order to be able to link a C program like yorick to a dynamic C++ library. You may need to write a small example code that does this to figure out what is necessary on your operating system and compilers -- they are all different; there are no applicable standards.) To get the various extra compiler options, you need to edit the Makefile that yorick -batch make.i created for you. That Makefile has comments about what the various macros do. The one you need to set is PKG_DEPLIBS as shown in the comment above that line -- your package DEPENDS on the liblas library, and this is where you need to tell the yorick build system that.

In practice, on many operating systems, it is exceedingly difficult to figure out the compiler flags you need in order to link a library that will be loaded at runtime (that is, a plugin library) against a dynamic library, say liblas.so, which is intended to be linked directly to an executable or to other compile-time-loaded dynamic libraries. So there's a good chance that even after you get the basic dependency options right, your package still won't work. The simplest workaround if this happens to you is to link your package against the static library liblas.a. If the dynamic library is installed, it may be difficult to convince the compiler to use an adjacent static library (more compiler options to figure out). The simplest way is usually to simply put the path to the static library onto the load command line instead of -L or -l options:

PKG_DEPLIBS=/path/to/liblas.a

The yorick documentation (in the user manual and in comments in the Makefile and source code) does explain all this, but admittedly requires very close reading and study to figure it out. The procedure for creating a python package is nearly identical and is much better documented. If you're lost, it might be easier to first create your plugin as a python plugin, then follow all the same steps to translate it into a yorick plugin.


Wed Feb 06, 2013 9:54 am
Profile
Yorick Padawan

Joined: Tue Jun 07, 2011 1:46 pm
Posts: 32
Location: UNH
Post Re: Yorick Wrappers for C or C++ files
You're right, I was very confused. Your suggestion was perfect. I set the following in the Makefile and the simple usage function worked as it should. Thank you very much.

PKG_CFLAGS=-fPIC
PKG_DEPLIBS=/usr/lib/liblas.so

I'm now trying to break the las2txt "main" function into smaller functions.


Fri Feb 15, 2013 1:20 pm
Profile
Yorick Padawan

Joined: Tue Jun 07, 2011 1:46 pm
Posts: 32
Location: UNH
Post Re: Yorick Wrappers for C or C++ files
I'm assuming the "main" function cannot be simply wrapped and I would have to break it up into smaller functions to be able to use it properly. Is this a correct assumption?

Also, I'm having trouble finding the Python package documentation within Yorick. Could you post a link?


Fri Feb 15, 2013 1:41 pm
Profile
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post Re: Yorick Wrappers for C or C++ files
You should not need the -fPIC flag. The yorick Makefile system figures that one out during (yorick's) configuration and inserts it where it is required -- and more importantly, omits it where it is not required.

All things python are at http://python.org. There are many references under the "extending and embedding" link on the main documentation page. You want "extending". There is no way to "embed" yorick, so that material won't give you any clues for extending yorick. Bear in mind that extending yorick is completely different in detail from extending python; what I am saying is that it is conceptually very similar -- you need to carry out all the analogous steps in both cases.

I do not understand your question about a "main" function. There's no such thing. You are simply writing a set of functions which you can invoke from yorick interpreted code -- just like the "cos" or "exp" functions. Your functions may, in turn call the functions in the third party library you are linking against. Any "main" program will be written in interpreted code, and is unknown to you when you are creating your package, just as whoever wrote your third party library had no idea what your "main" program would look like. Using PROTOTYPE comments, you can arrange for codger (yorick's code generator during the package build process) to generate simple wrappers that directly invoke third party library functions, as you did for LAS_GetVersion. (Your PROTOTYPE comment for LAS_GetVersion renders it a totally useless function, since any result I assume the function should return will be discarded.) As I said before, it is usually a very bad idea to directly wrap third party functions. Almost always you are better off writing a set of higher level C functions which call common sequences of third party functions, and wrap the third party data structures in more convenient objects for interpreted programs to manipulate. What you want is a yorick API to the third party library, not its C API, which was not designed to be used by interpreted code. In particular, if there are any persistent objects in the C API which the library has to allocate and free (e.g.- file handles, or any other pointer to a data structure for which the C API provides a free or destroy or close function), there is no escape from explicitly writing C code to create and manage corresponding objects for use by a yorick interpreted program.


Fri Feb 22, 2013 9:54 am
Profile
Display posts from previous:  Sort by  
Reply to topic   [ 15 posts ] 

Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by STSoftware for PTF.