Linux - Is it possible to build a shared library that calls other shared libraries without having to do a static link to them?

[Michel_Gauvin]

Hello,

Operating System = Linux (Ubuntu 14.04 LTS 32 bits)

I'm building a shared library *.so.  This *.so is to be used by LabVIEW.  This *.so is calling functions from other shared libraries.  In my case the other share library is libxml2 and libz.  The only way I was able to compile my shared library and make it work was by doing a static link of libxml2.a and libz.a.  Is there a way to do a dynamic link?

Below is the makefile I'm using.  The target "TestValidateSchema" compiles the shared library without the static link and this shared library is called by a test "c" application and it's working.  Now, to make it work in LabVIEW I need to do a static link as you can see in the target "install:" otherwise the "call library node" function is throwing me a unresolve symbol....

Is this the only way to do so, and if yes why?

Thanks,

--------------------------------

CC=gcc

all: TestValidateSchema

TestValidateSchema: testValidateSchema.c
$(CC) -o Release/validateschema.o -c -Wall -Werror -fpic Src/validateschema.c -I./Include -I/usr/include/libxml2
$(CC) -shared -o ./Release/libvalidateschema.so ./Release/validateschema.o
$(CC) -Wall -o ./Release/testvalidateschema testValidateSchema.c ./Release/validateschema.o -lxml2

clean:
$(RM) $(OBJS) ./Release/testvalidateschema ./Release/libvalidateschema.so ./Release/validateschema.o /usr/include/validateschema.h /usr/lib/libvalidateschema.so

install:
$(CC) -o Release/validateschema.o -c -Wall -Werror -fpic Src/validateschema.c -I./Include -I/usr/include/libxml2
$(CC) -shared -o ./Release/libvalidateschema.so ./Release/validateschema.o Src/libxml2.a Src/libz.a

--------------------------------------

[mikerophone]

Michel,

These links might be helpful for accomplishing your goal.  You can use a Call by Reference Node to dynamically call VIs.  You will need to include these VIs in your source file for creating the shared library.

https://decibel.ni.com/content/docs/DOC-15556

http://zone.ni.com/reference/en-XX/help/371361H-01/lvconcepts/dynamic_loadcall_vis/

http://digital.ni.com/public.nsf/allkb/DE6B3E00C1C1223486257B480046337C

[Michel_Gauvin]

Hello mikerophone,

Thanks for your reply but none of the link applies to my problem.  So here's another way to describe my problem:

A shared library has the *.so extension under Linux and is the equivalent of a *.dll under Windows.  So, I'm programming in "c" language the *.so under Linux, let's call it "hello.so".  This "hello.so" is calling functions from other shared libraries (*.so), which are  "libxml2.so" and "libz.so".

Now, LabVIEW comes into play and wants to call the "hello.so" with the "call library node".  The only way that the LabVIEW call is succefful is when I perform a "static link" of "libxml2.so" and "libz.so", as describe by the gcc compiler command:

$(CC) -shared -o ./Release/libvalidateschema.so ./Release/validateschema.o Src/libxml2.a Src/libz.a

I want to know if it's possible to compile "hello.so" by dynamically linking "libxml2.so" and "libz.so" and have a succefful call in LabVIEW?  The gcc compiler command is:

$(CC) -shared -o ./Release/libvalidateschema.so ./Release/validateschema.o

Definition:

In Static Binding (linking), the code associated with a given procedure is known at the time of compilation. In Dynamic Binding (linking) the code associated with a given procedure is known at the time of running the program. 

Maybe I need to put something special in the *.c or *.h files... or maybe it's not possible at all.

Thanks for the help,

Michel

[smithd]

Hey Michel,

First, I'd like to clarify a potential point of confusion. LabVIEW is not doing anything magical -- it is just calling the load command for the system you are on:

http://digital.ni.com/public.nsf/allkb/EBAE870D564CBBE78625788B004E76D3

For linux, this is dlopen.

So, the question I would ask is what error are you getting? If it is one of the errors related to loading, then it sounds like your compiler configuration is generating an invalid .so file. If its some other error, maybe its actually executing the code but something else is going wrong.

Finally, I am really *not* an expert on any of this stuff -- but looking at this forum it sounds like that command is *not* static linking:

http://www.linuxquestions.org/questions/linux-newbie-8/forcing-static-linking-of-shared-libraries-69...

It says including the flag "-static" will tell the compiler to not use the .so files but to instead look at the .a files. I would assume the reverse it true? Since you have the keyword -shared, wouldn't the compiler try to find the appropriate *.so files on your system rather than using the *.a files you specified?

I was also taking a look through this (http://www.yolinux.com/TUTORIALS/LibraryArchives-StaticAndDynamic.html) and it seemed like you still have to reference the *.so files in that last step even for dynamic linking. If so, have you tried compiling with this line instead:

$(CC) -shared -o ./Release/libvalidateschema.so ./Release/validateschema.o Src/libxml2.so Src/libz.so

Anyway, hopefully some of this helps at least a little bit.

-Daniel

[rolfk · Accepted solution]

In addition to what Daniel said about specifying any external shared library in the link stage of your shared library build, together with the -shared option, there is one more thing to possibly consider:

Your hello.so library then contains references to functions in the external shared libraries libxml.so and zlib.so. When LabVIEW calls dlopen("/path/dir/hello.so", ...) it basically hands of all control to the Linux run-time linker, ld.so which handles the rest. ld.so then attempts to load the shared library recognizing that it has external references and attempting to resolve them.

First it tries to match the missing external references to any symbols that have been previously loaded into the current process already. If that isn't successful for a particular system it queries the ldcache for a library and symbol. ldcache is updated through the ldconfig program. When you add a new shared library to the system that you want the ld.so runtime linker to be able to find you have to run ldconfig.

NOTE: Running ldconfig isn't normally necessary for shared libraries that you directly access through the Call Library node, since you specify there the entire path to your custom shared library already, so ld.so will immediately fail if it can't find the specified library at the given absolute position and won't attempt to query ldcache at all.

If you install packages through your platform package manager updating of ldcache for any publically accessible shared libraries in those packages is usually done automatically for you as part of the package installer script. If however you install libraries manually you have to run ldconfig yourself. In your case you could add the libxml.so and zlib.so to /usr/lib and simply run ldconfig from the commandline as it by default searches /lib and /usr/lib for shared libraries and rebuilds the ldcache from that. If you add your new libraries to some other directory on your machine you have to run ldconfig with the -n option telling it in which directory the new shared libraries were installed.

[Michel_Gauvin]

Hello rolfk and Daniel,

First thank you so much for the insightful comments, and I just want to say that I'm not an expert either on this stuff.

I was convince that the libxml2.so and libz.so was located in the /usr/lib directory but they were not.  This is the reason I had to statically link the *.a file to my shared library in order to be able to load and execute the library function in LabVIEW.  I was performing the static linking with the following:

$(CC) -shared -o ./Release/libvalidateschema.so ./Release/validateschema.o Src/libxml2.a Src/libz.a  (This was producing a 1.6MB libvalidateschema.so file, so I'm guessing the static linking worked and the compiler put the code of both library in the libvalidateschema.so)

Now, here's what I'm using to perform dynamic linking:

$(CC) -shared -o ./Release/libvalidateschema.so ./Release/validateschema.o /usr/lib/i386-linux-gnu/libxml2.so /lib/i386-linux-gnu/libz.so.1 (This produce a 12.2KB libvalidateschema.so file and it's working, so I'm a happy customer)

I have another questions related to this for an for in-depth understanding, when building my "c" application to test the libvalidateschema.so I don't need to specify the path to the libxml2.so and libz.so library.  When I'm calling the compiled "c" application from a terminal it's working and the program is able to find where both shared libraries are located.  What would be the difference between this call and the LabVIEW call, because in the case of LabVIEW I get an unresolve symbols...?

Thanks,

Michel

[rolfk]

What does ldconfig -v -N tell you on your machine? Unless the zlib.so and libxml.so do show up in there it likely won't work in LabVIEW. As to why it may work in your executable: most likely you have your own shared library and all your dependencies in the same directory as your executable (or alternatively in the current directory in which you were when launching your executable).

But it's definitely not a good idea to start putting dependencies in the same directory as labview.exe.

[Michel_Gauvin]

Alright, I finaly figured out how to make the appropriate makefile in order for the *.so to find all it's dependencies while everything is now dynamically linked.

Thanks Rolf.

FYI here's the makefile for the test application and the libvalidateschema.so shared library:

----------------------------------

CC=gcc

all: TestValidateSchema

TestValidateSchema: testValidateSchema.c
$(CC) -o Release/validateschema.o -c -Wall -Werror -fpic Src/validateschema.c -I./Include -I/usr/include/libxml2
$(CC) -shared -o ./Release/libvalidateschema.so ./Release/validateschema.o -lxml2
cp ./Release/libvalidateschema.so /usr/lib/libvalidateschema.so
ldconfig
$(CC) -Wall -o ./Release/testvalidateschema testValidateSchema.c -lvalidateschema -I./Include -I/usr/include/libxml2
$(RM) $(OBJS) ./Release/libvalidateschema.so ./Release/validateschema.o

install:
$(CC) -o Release/validateschema.o -c -Wall -Werror -fpic Src/validateschema.c -I./Include -I/usr/include/libxml2
$(CC) -shared -o ./Release/libvalidateschema.so ./Release/validateschema.o -lxml2
cp ./Release/libvalidateschema.so /usr/lib/libvalidateschema.so
ldconfig
$(RM) $(OBJS) ./Release/libvalidateschema.so ./Release/validateschema.o

clean:
$(RM) $(OBJS) ./Release/testvalidateschema ./Release/libvalidateschema.so ./Release/validateschema.o /usr/lib/libvalidateschema.so

----------------------------------