I’m trying to make a distributed compilation set up working, using icecream, and I’ve run into some rather odd issues that I have no clue how to solve.
First of all, the situation is like this: the distributed compilation cluster is composed of 2 nodes - the target, an i686 device using Gentoo, and the host, using openSUSE. The target is very slow, the host is very fast. Since the architectures don’t match, I need to set up cross-compilation.
So far, what I did is install icecream on both devices, then installed cross-i386-gcc47-icecream-backend on the host. Then I copied the /usr/share/icecream-envs/cross-i386-gcc47-icecream-backend_x86_64.tar.gz file to the target. After that, I used icecream-create-env on the target to get a tarball of the native compilation toolchain there. The first problem is that it didn’t succeed completely - it somehow missed a few dependencies in /usr/lib. So I manually added those dependencies into the tarball. After that, I named the native toolchain tarball host-i686-dist-i686.tar.gz. Then I moved both tarballs to ~, and set ICECC_VERSION to “x86_64:~/cross-i386-gcc47-icecream-backend_x86_64.tar.gz,i686:~/host-i686-dist-i686.tar.gz”. Then, on the host, I started the scheduler and iceccd. Then, on the target, I set up $PATH to point to icecc first, and started iceccd.
So here’s the problem: for some reason, when starting iceccd on the target, even though I have ICECC_VERSION set, it still insists on creating a new tarball with the local toolchain (which is once again broken), and when it sends that over to the scheduler, the scheduler dies. (the target iceccd says “scheduler dead ?!” as if it wasn’t its fault to begin with! 
) Here’s the trace: SUSE Paste
Now, if I start the scheduler on the target device instead, or if by off chance the target iceccd doesn’t generate a new tarball, the scheduler somehow decides that all the tasks should only be done locally. icemon shows that the host iceccd connected just fine and has open slots, but the scheduler just doesn’t send anything to it! If the target iceccd is started with “-m 0”, the scheduler adds the task to the target’s “job list” (and waits indefinitely afterwards).
There is also the part where the openSUSE wiki suggests that for interfacing with Gentoo, the tarball should not contain any processor-specific optimisations (if I understood the wiki correctly, it’s extremely confusing). That could also be the cause of the crashes, I guess, but then the tarball that gets generated is incomplete anyway.
So the main question is - how do I tell the target iceccd to not create a tarball, but use one that is already provided? Did I miss something in the ICECC_VERSION variable? And is there any way to tell why the scheduler refuses to use the host iceccd?