DISCLAIMER: This is a continuing series detailing the painful story of a DIY render farm build. It is terribly technics and
somewhat frustrating. Those who are unprepared for such “entertainment” are advised to ignore these posts.
As you may know, I have, at this juncture, made an Primary Domain Controller (PDC) out of an old PPC Mac Mini. It will dole out the render tasks to the higher performance machines, a bank of Mac Pro 1,1s now hacked to run Mac OSX 10.9.5.
Job submission, however, was still an issue that was not quite clear to me. By my understanding, I should be able to use the X-Quartz front-end (drqman) to submit a job by a simple button. The files to be rendered must be uploaded somewhere – one of the shared directories? And then one of the shared directories would also be the location of the rendered files? It was all a bit foggy, but I figured I could get a better feel for it once I had at least one slave in the network.
The “job” could be, for example, a simple Lux .lxs file for Luxrender. And what of other files? Houdini and After Effects projects need shared assets – and those can be quite big. AEFX renders on a 4K project could involve a lot of network traffic if each slave must grab that info from a shared directory. Is it better if these assets are copied to local drives? If not, than does the farm crawl as each slave looks for the same stuff on the shared drive? Does my crap network become a huge liability then?
It seems I forgot Maxwell Render, which I’d like to try – the look from Maxwell is so good. But Dr. Queue does not do Maxwell, at least not without writing the job scripts yourself. So that would be out unless I found some script for it or I could manage one on my own… how difficult is it to write a script? I had no idea. Dr. Queue had been widely lauded as a solution that allows maximum flexibility. In theory any script can be sent to each slave. So if one could work out the syntax for a Maxwell render, then one could submit all one liked.
Basically, Lux will be the most effective right now, and will make a good test of the system. I will have to test each app so that I know the workflow is perfect before cloning each slave. Closer inspection of the Dr. Queue information reveals that the job script could be written so that it first copies all assets to each local drive. At that point the network will crawl as each slave tries to access the shared assets. But it will do so only momentarily, after which it should run much faster.
Even better! In the drqueue/contrib folder I found a shell script to setup drqueue demons every time you restart! I was going to do this in an uncomfortably complicated way. This script could be a real time-saver.
I investigated other Open Source render engines that might also serve my purposes, but many of those are less robust compared to Lux. I’ve done just about all I can at home. I return to school next week, at which point I’ll see what happens when I plug the new slave into the network.