Comments on: openais: an alternative to clvm with cman

By: Olly

Olly — Thu, 17 Jun 2010 08:37:45 +0000

thx for your howto, but i have a problem to compile the clvm package. I got this error message

make[3]: Leaving directory `/usr/src/lvm2-2.02.39/debian/build/build-deb/daemons/clvmd’
make[3]: Entering directory `/usr/src/lvm2-2.02.39/debian/build/build-deb/daemons/clvmd’
gcc -c -I../../include -I/usr/src/lvm2-2.02.39/debian/build/build-deb//include -DUSE_OPENAIS -D_REENTRANT -DHAVE_CONFIG_H -g -O2 -g -O2 -fPIC -Wall -Wundef -Wshadow -Wcast-align -Wwrite-strings -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -Winline -Wmissing-noreturn -Wformat-security -g -O2 -fPIC -Wall -Wundef -Wshadow -Wcast-align -Wwrite-strings -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -Winline -Wmissing-noreturn -Wformat-security -fno-strict-aliasing -g -O2 -fPIC -Wall -Wundef -Wshadow -Wcast-align -Wwrite-strings -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -Winline -Wmissing-noreturn -Wformat-security clvmd-command.c -o clvmd-command.o
In file included from clvmd-command.c:74:
clvmd-comms.h:79:29: error: openais/saAis.h: No such file or directory
clvmd-comms.h:80:35: error: openais/totem/totem.h: No such file or directory
In file included from clvmd-command.c:76:
clvmd.h:32: error: ‘SA_MAX_NAME_LENGTH’ undeclared here (not in a function)
make[3]: *** [clvmd-command.o] Error 1
make[3]: Leaving directory `/usr/src/lvm2-2.02.39/debian/build/build-deb/daemons/clvmd’
make[2]: *** [clvmd] Error 2
make[2]: Leaving directory `/usr/src/lvm2-2.02.39/debian/build/build-deb/daemons’
make[1]: *** [daemons] Error 2
make[1]: Leaving directory `/usr/src/lvm2-2.02.39/debian/build/build-deb’
make: *** [debian/stamps/build-deb] Error 2
dpkg-buildpackage: failure: debian/rules build gave error exit status 2

Any ideas about that?
regards
olly

By: Rico

Rico — Fri, 09 Apr 2010 21:01:27 +0000

I compiled your how-to and the cluster works fine. But there is a problem with clvmd… Look at this log when I launch it in debug mode :

# clvmd -d 1
CLVMD[3e5cd770]: Apr 9 13:56:41 CLVMD started
CLVMD[3e5cd770]: Apr 9 13:56:41 Our local node id is -1062714761
CLVMD[3e5cd770]: Apr 9 13:56:41 Add_internal_client, fd = 7
CLVMD[3e5cd770]: Apr 9 13:56:41 Connected to OpenAIS
CLVMD[3e5cd770]: Apr 9 13:56:41 Cluster ready, doing some more initialisation
CLVMD[3e5cd770]: Apr 9 13:56:41 starting LVM thread
CLVMD[40800950]: Apr 9 13:56:41 LVM thread function started
CLVMD[3e5cd770]: Apr 9 13:56:41 clvmd ready for work
CLVMD[3e5cd770]: Apr 9 13:56:41 Using timeout of 60 seconds
CLVMD[3e5cd770]: Apr 9 13:56:41 confchg callback. 1 joined, 0 left, 2 members
File descriptor 4 left open
File descriptor 5 left open
File descriptor 6 left open
WARNING: Locking disabled. Be careful! This could corrupt your metadata.
CLVMD[40800950]: Apr 9 13:56:41 LVM thread waiting for work

Did you see the WARNING ? The locking does not work, and I have no idea why ! my /etc/lvm/lvm.conf is modified to use internal locking, I can see lock/unlock requests in openais logs, but clvmd just doesn’t care about that !!!

Any help will be appreciated :)

By: Xen Fan

Xen Fan — Wed, 10 Mar 2010 19:30:36 +0000

Regardless of userid or the use of fakeroot, the dpkg-buildpackage gives a rules error.

With fakeroot as root:

fakeroot debian/rules clean
/usr/bin/fakeroot: line 164: debian/rules: Permission denied
dpkg-buildpackage: failure: fakeroot debian/rules clean gave error exit status 126

Without fakeroot as root:

debian/rules clean
Can’t exec “debian/rules”: Permission denied at /usr/bin/dpkg-buildpackage line 475.
dpkg-buildpackage: failure: debian/rules clean failed with unknown exit code -1

With fakeroot as regular user:
fakeroot debian/rules clean
/usr/bin/fakeroot: line 164: debian/rules: Permission denied
dpkg-buildpackage: failure: fakeroot debian/rules clean gave error exit status 126

I appreciate your help and apologize for the annoyance, but not being a debian package guy I’ve been banging my head on this for a while. I can’t even figure out what it’s trying to do.

By: jcl

jcl — Wed, 10 Mar 2010 19:22:38 +0000

Xen Fan,

If you are already root, you dont need to use -rfakeroot at all.

By: Xen Fan

Xen Fan — Wed, 10 Mar 2010 19:14:10 +0000

Yeah, I was confused because you did the same thing above:

root@host:/usr/src/lm2-2.02.39# dpkg-buildpackage -rfakeroot -uc -b

I’m not clear on where I need to become another user. A regular user won’t be able to create the build dir in /usr/src.

By: jcl

jcl — Wed, 10 Mar 2010 18:46:26 +0000

Xen Fan,

Looks like youre trying to use fakeroot when youre already root.

By: Xen Fan

Xen Fan — Wed, 10 Mar 2010 18:43:04 +0000

Thanks for the great writeup, however I’m having difficulty building this on lenny. At the dpkg-buildpackage stage I get this error.

/usr/src/lvm2-2.02.39# dpkg-buildpackage -rfakeroot -uc -b
dpkg-buildpackage: warning: using a gain-root-command while being root
dpkg-buildpackage: set CFLAGS to default value: -g -O2
dpkg-buildpackage: set CPPFLAGS to default value:
dpkg-buildpackage: set LDFLAGS to default value:
dpkg-buildpackage: set FFLAGS to default value: -g -O2
dpkg-buildpackage: set CXXFLAGS to default value: -g -O2
dpkg-buildpackage: source package lvm2
dpkg-buildpackage: source version 2.02.39-7.3
dpkg-buildpackage: source changed by root
dpkg-buildpackage: host architecture amd64
fakeroot debian/rules clean
/usr/bin/fakeroot: line 164: debian/rules: Permission denied
dpkg-buildpackage: failure: fakeroot debian/rules clean gave error exit status 126

Any ideas? Do you have a package anywhere for download? If I could get clvm going without all the other nonsense it would be amazing. Thanks!

By: /root.eu » Klastrowy LVM i Xen

/root.eu » Klastrowy LVM i Xen — Mon, 23 Nov 2009 19:19:50 +0000

[...] już chyba temat na odrębny wpis. Ku czemu innemu zmierzam. Mianowicie bardzo zainteresował mnie ten wpis. Okazuje się, że jest możliwe zmuszenie CLVM do działania bez skonfigurowanego RHCS! [...]

By: jcl

jcl — Fri, 23 Oct 2009 11:42:14 +0000

ezio,

/var/log/openais is indeed empty on my installations. im not sure if you have to enable logging somehow in openais or not.

there might be some way to configure openais/clvm timeout periods… i havent looked into that. its also possible that clvm assumes certain things about cman that arent true about openais and that is causing the problem. perhaps there have been improvements in the latest source code for both packages.

another thing you could try, though its dangerous if you do modifications to the cluster lvm metadata from different nodes at the same time, would be to put each openais instance into its own ring. that would trick clvm into thinking it had a quorate cluster since each node is, in effect, the only node in the cluster.

- jcl

By: ezio

ezio — Fri, 23 Oct 2009 10:31:47 +0000

Hi jcl,

my SAN is not in a stable status. I could not configure the multipath service against my active-passive controllers yet, since I am waiting for a multipath compatible firmware upgrade. so my logs are full of errors because the nodes try to connect through FC connections to redundant devices which are not actually reachable. The fall of performance could by huge. Nevertheless I decided to test your environment, because I felt that your solution was excellent for our aims.

Since the problem described is more frequent in my case, and my nodes are quite full of syslog errors due to the problems above, this could mean that the problem is related to timeout problems, performance of the SAN and ways to configure timeouts in the clvm and openais packages.

Is /var/log/openais empty In your installation too ?

I ordered 2 little AoE’s too and am studying your article about it.

Thanks.
ezio