[C++] How to know size of off_t in external C library

[bepaald]

Hi!

I am writing a program which uses libcdio for accessing audio tracks on a cd. After making some changes to the code, the program started to misbehave. It did not crash but exhibited some strange (and unwanted) behaviour. It took me half of the day to figure out what was going on.

It turned out libcdio is compiled with large file support, which causes the off_t type to be 8 bytes. In my program (compiled without -D_FILE_OFFSET_BITS=64) an off_t is only 4 bytes. At one point in the program I call a libcdio function taking an off_t and an int as arguments. However because of the mismatch in off_t sizes, my program would call the function with 8 bytes of arguments (4 for off_t, 4 for int). The function would take all 8 bytes and interpret them as a single off_t, and the int would have some random(?) garbage value. Here is a minimal example showing this problem:

[~] $ cat function.h
#ifndef FUNCTION_H_
#define FUNCTION_H_

#include <stdio.h>

extern void function(off_t a, int b);

#endif
[~] $ cat function.c
#include "function.h"

void function(off_t a, int b)
{
  if (a == 1 && b == 2)
    printf("Succes!\n");
  else
    printf("Fail...\na: %lld\nb: %d\n", a, b);
}
[~] $ cat main.cc
extern "C"
{
#include "function.h"
}

int main()
{
  off_t a = 1;
  int b = 2;

  function(a, b);

  return 0;
}
[~] $ gcc -Wall -c -D_FILE_OFFSET_BITS=64 function.c
[~] $ ar srv libtest.a function.o
ar: creating libtest.a
a - function.o
[~] $ g++ -Wall main.cc -L. -ltest
[~] $ ./a.out
Fail...
a: 8589934593
b: 0
[~] $

So I know I can fix this by compiling my own program with large file support, or by defining off_t to some 64 bit int in my sources. But then the program would break in a similar fashion if it were ever linked against a libcdio built without large file support.

So the question is, is there any way to find out (preferably at compile time) the size of an off_t in the external library? Just having to guess, compile (always works without even a warning), run and then see if the program behaves properly is obviously not an elegant solution.

thanks,
bepaald

[David Becker]

sizeof is your friend.

David

[bepaald]

Thanks for your answer David. Sizeof() is indeed a good friend of mine, but unfortunately even your best friends can't always help you.

Maybe I wasn't clear in my first post, but the library I'm linking against (libcdio) is a third party library, which comes from the repository pre-compiled into a shared library. Using sizeof(off_t) in any of my own code would return the size of an off_t in my code which is not necessarily the same size as the size of an off_t in the external library. The problem is that I need to make sure somehow that my off_t's do have the same size as the ones in the library.

Even if I would have access to the library, or could get the writers of libcdio to add a function 'int our_off_t_size() {return sizeof(off_t);}', how would that help get the size of the type at compile time?

bepaald

[fpmurphy]

There is no way of finding out the size of off_t in a compiled library other than writing a simple test program and checking the result. Much like the configure utility does.

[bepaald]

There is no way of finding out the size of off_t in a compiled library other than writing a simple test program and checking the result. Much like the configure utility does.

OK, thanks, I thought as much. But wouldn't this still require the library to provide some function(s) to allow a simple test program? As far as I know libcdio doesn't have any such function. The function that fails for me is used to seek to a sector on an audio cd. The smallest program that would do that would still be pretty big. Also it would require a CD in the drive. And it doesn't actually return any useful information, the only way to know that it failed is to start playing the audio and determine that it is not playing the sectors you were expecting if the seek had succeeded. I don't think I can get a configure script to do that. Maybe it's time to send an email to the libcdio-devel list.

thanks again,
bepaald

[David Becker]

fmurphy is correct in pointing out that you're going to have to run some test. What's going on between your program and the library in question is very onbepaald.

You may also consider adapting your own program to the particular target, but I guess you've already been considering that. I would think that having your program to be portable in the manner I think you're intending to do would have other repercussions besides knowing the sizeof an off_t. And yes, a little more cooperation on libcdio's side would probably go a long way.

David

[stevea]

Slow down there -
off_t used as a file offset and defined in unistd.h/types.h is a system issue since the underlying lseek is a system call. There is only one valid type PER SYSTEM.

The POSIX correct way to determine the size of off_t is"
At compile time test the macros in unistd.h, specifically

_POSIX_V6_ILP32_OFF32
_POSIX_V6_ILP32_OFFBIG
_POSIX_V6_LP64_OFF64
_POSIX_V6_LPBIG_OFFBIG

at runtime use the sysconf call.


If the library has been built for the specific target kernel/system (like the fedora repo libraries) - then it should have been compiled with the correct off_t definition.

There is the complication that 64bit Linux kernels support 32bit syscalls and libraries - but there is no possibility of linking 32 & 64 bit binaries together.

---

The OP's example code runs correctly here *but* generated a warning about prining and off_t with an %ld format.

You should NOT be modifying _FILE_OFFSET_BITS unless you are building for another system.

[fpmurphy]

POSIX.1 is a strictly a source API standard. It is silent on the issue of binary incompatibilities such as the OP is asking for help with.

[jpollard]

Slow down there -
off_t used as a file offset and defined in unistd.h/types.h is a system issue since the underlying lseek is a system call. There is only one valid type PER SYSTEM.

Not quite. If you are running 32 bit apps, and developing 32 bit apps in a 64 bit environment, off_t will be 32 bits when doing
32 bit work, with 32 bit libraries. It will be 64 bits if you are using
the 64 bit runtime and include files. Therefore there are two possible values, depending on your environment.

This is why there are two sets of runtime, and the 32bit compatibility layer that converts 32 bit off_t to 64 bit.

[bepaald]

OK, thanks guys. I forgot to mention that I am running a 32 bit machine and developing only for 32 bit machines at this point.

Stevea:
I am a bit puzzled as to why the code works for you (on 32-bit machine?). Unless of course you left out the -D_FILE_OFFSET_BITS=64 option when compiling the c-function. Then obviously, the code works here as well, issuing the same warning. I specifically put that option there to mimic the behaviour I'm seeing with libcdio (32bit version, from the official repo), which also sets the symbol to 64. Because of

You should NOT be modifying _FILE_OFFSET_BITS unless you are building for another system.

Could I then conclude this is a bug in libcdio, because it does in fact modify this symbol?

Just to make sure I now understand the problem (and whose problem it is) correctly, and before I make a fool of myself on the libcdio mailing list, is the following a correct summary of the bug?:
'When compiled with large-file support, the size of an off_t is changed from system default through the _FILE_OFFSET_BITS symbol which breaks binary compatibility on 32 bit systems. Instead it should change the prototype of functions using off_t to use off64_t in this case.' (Like lseek64, the large-file variant of lseek, does, as I just read in its man page.)

[fpmurphy]

Could I then conclude this is a bug in libcdio, because it does in fact modify this symbol?

No, you cannot.

Some background. The LFS (Large File Summit) extensions were agreed upon by various major OSVs and ISVs back in 1994 or 1995 and submitted for standardization in 1996. The purpose of the LFS extensions is to enable applications build for the IPL32 data (programming) model (i.e. 32-bit platforms) access data files > 2Gb. Read here for more information if interested.

If the binary library that you want to link against expects a 64-bit offset_t, then you need to pass it a 64-bit offset_t. You do that by building your application with -D_FILE_OFFSET_BITS=64

[David Becker]

I honestly would have expected some kind of error at compile time or run time. If you're using static libraries, then a(n incompatibility) compile time error is expected. If you're running a program against a shared library then a (symbol lookup failure) run time error is expected. Unless something esoteric is taking place, I wouldn't expect the run time system to simply transform the function call's parameters from 2x 4 bytes data type into a single 8 byte data type while forgetting about the 2nd parameter.

David

[jpollard]

That would be link time rather than compile.

The runtime doesn't, the 32 bit compatibility layer is done at system call time.

When using object libraries the only time the linker can detect errors is by symbol name typing. off_t doesn't exist at this point as it is strictly a compiler symbol for a type. This type name doesn't exist as it isn't associated with a global symbol.

A symbol, if memory serves, is only for relocatable types, and only includes a length. The only name is for the item being assigned a memory location - and not a true data type. This exists in the ELF file format structure for object files. Parameters to functions don't exist, and don't have anything other than addressing offsets used in an argument list. The values associated with those offsets can be anything.

This is why function prototypes need to be complete. It is only the prototype that can give the compiler the hints on how the parameters are supposed to be matched up/used at runtime.

[David Becker]

That would be link time rather than compile.

If we're splitting hairs then I'd have to say compile time linkage vs run time linkage.

The runtime doesn't, the 32 bit compatibility layer is done at system call time.

I'm not sure what you mean here. The library and the program (of the OP) would seem to be 32bit even though the off_t (in the library) is effectively off64_t.

When using object libraries the only time the linker can detect errors is by symbol name typing. off_t doesn't exist at this point as it is strictly a compiler symbol for a type. This type name doesn't exist as it isn't associated with a global symbol.

A symbol, if memory serves, is only for relocatable types, and only includes a length. The only name is for the item being assigned a memory location - and not a true data type. This exists in the ELF file format structure for object files. Parameters to functions don't exist, and don't have anything other than addressing offsets used in an argument list. The values associated with those offsets can be anything.

This is why function prototypes need to be complete. It is only the prototype that can give the compiler the hints on how the parameters are supposed to be matched up/used at runtime.

Correct. I was under the impression that the OP was using c++ libraries while the provided examples involve c-style libraries with a c++ program.

I'm not sure what the OP is doing when compiling (and thus linking) the program with the g++ compiler when there's also a c++ library for cdio. Maybe that would help the OP with the issue?

David

[jpollard]

If we're splitting hairs then I'd have to say compile time linkage vs run time linkage.

There is no such thing as compile time linkage except in a JIT interpreter.
Runtime linkage uses ld.linux.so, as part of the dlopen/dlclose/... functions to combine shared libraries together with a application at runtime. This linker can no more handle compile time definitions than ld can.

c++/g++/cc always chain to ld to perform the linking.

The closest thing to compile time linking (in C++) is the use of nested internal object functions. But final resolution of the addressing is still up to the linker and not the compiler. And this doesn't happen with modules/object used through a library as the object templates don't have the actual addresses until the library symbols are resolved by ld.

The system call compatibility layer interprets 32 bit system calls when running applications compiled and linked for a 32 bit environment. It is what allows i386/i686 binaries to run on x86_64 processors.

g++ is the GNU C++ compiler. Linking for both is done by ld to ensure standard executable file formats. There are multiple C++ compilers from different vendors that could be used.

I assumed the library might have been compiled by a different compiler than the one available to the OP.