ewtroan

36" of snow

2009-12-23T13:41:08Z

I spent the weekend, and Monday, and some of Tuesday, in the Virginia mountains. It's only about 3500' above sea level and really doesn't get that much snow. This weeks storm was a complete exception though. We made it up after a harrowing drive through driving snow (plows and salt were no match for it), and woke up to about two feet of snow on the ground. The snow kept falling until the early evening, for a total of around three feet. That is a lot of snow.

The driving wind kept up for a few more hours, and we woke up to clear skies on Sunday morning. We also woke up to massive snow drifts. Think six foot high snow drifts about four feet thick. My driveway was completely drifted in (which mattered very little given the condition of the roads). Fortunately, most of the people I was with were skiers and had easy slope access (where easy means trudging through waist high drifts for about ten feet).

Eventually we had to get to work on the driveway though. The driveway is heated, so the parts which weren't covered by drifts stayed somewhat clear. Here's a picture of what the rest looked like (the angle makes the drifts look a little smaller than they were; the tops were easily above my Mom's head).

Amazon mp3 downloader, redu

2009-12-11T18:37:16Z

A little while ago I built the amazon mp3 downloader for Foresight. Well, amazon decided to have some more free albums today, and my wife (and kids) like Christmas music, so I went to grab them. Doing so made it blindingly obvious that my wrapper script should have passed the command line arguments onto the application itself.

A "conary update amazonmp3" will get you a version which is better integrated with firefox, and a "conary update amazonmp3=oot.rpath.org@fl:2" will get it for you if you haven't already installed it.

Conary Capsules

2009-11-30T14:34:47Z

Today is a pretty big day for us here at rPath. We've just announced support for Red Hat Enterprise Linux in our release management product offering. For the past year we've gotten significant interest from large organizations who want to use Conary's version control capabilities to manage their system deployment. Red Hat's incredible success has made their enterprise offerings a requirement for those businesses though, so they've been trying to find a way to get the best of Red Hat along with the best of rPath.

After scratching our heads for a while, we realized that what those folks want out of Conary isn't package installation, it's version control. They're looking for a way to organize sets of RPMs, define systems using versioned groups, and reliably update those systems once they're deployed. While we have previously imported operating systems (such as Novell SLES, CentOS, and Scientific Linux) we changed the packaging as part of that; doing that would (understandably) concern this set up of users who really do want RPM. But RPM does package installation, not version control, so why not marry Conary version control with RPM package installation to get the best of both worlds?

This line of thought led to capsules in Conary. They're really just a way of delegating the physical package management to another packaging system (RPM in this case) while using Conary to do the high level version orchestration. So Conary decides what gets installed, and RPM does the installation.

The packages themselves are represented as normal Conary components (rpmname:rpm), but the RPM itself gets stored in the repository instead of the contents of the individual files. When a changeset is generated, the entire RPM is sent down the wire and the conary client then passes the RPM package to RPM itself to do the install. If there are multiple RPMs they all go off to RPM simultaneously to make sure RPM can order everything properly.

What's this all mean to Red Hat Enterprise Linux users? It means that you can use Conary to define groups, rBuilder to build images, and Conary to apply updates. Strong version control, package repositories, and our deep dependency resolution are all preserved. It also means you can use RPM to query the system, install packages outside of Conary, and perform RPM system verifications. After all the installation path is identical to anaconda's, with a single RPM transaction installing the entire system for you. It's not a system that looks like Red Hat Enterprise Linux, it is Red Hat Enterprise Linux. After the system is installed the only difference is a Conary binary which can be used for package version orchestration.

Amazon MP3 downloader

2009-11-07T14:12:29Z

I spent a bit of time this morning packaging up the amazon mp3 downloader for Foresight systems. It's a binary only (amazon doesn't release sources), and I mined Fedora 6 and Fedora 10 to get the dependencies right.

If you're interested:

conary update amazonmp3=oot.rpath.org@fl:2

Pearls

2009-10-26T20:34:42Z

Saw this at the local children's museum yesterday.

Pearls from clams. Who knew?

hfield wi-fire adapter

2009-09-19T02:28:30Z

A couple of weeks ago I was killing some time while watching a long commit run by browsing either Engadget or Gizmodo. They mentioned a device called the Wi-Fire from hfield, which is an external USB wifi adapter which was supposed to work at long distances. They were incredibly skeptical about the lightweight, cheap feeling adapter until they tried it. Once they gave it a run they couldn't say enough good things about the $59 gizmo.

I immediately thought about my mountain house. There is no connectivity where I am despite ample connectivity in the resort's buildings. I could see a connection (most of the time) but never get an address, and there are no broadband providers. I figured for $59, what the heck.

Got it, it worked under Linux, and it definitely showed more networks at my primary house than the built in Intel stuff did. It even let me connect to the one unsecured network I saw (other than my own!). Not bad, so I put it aside until my next mountain weekend.

Well, it's the weekend now and I'm in the mountains. The first thing I did (much to my wife's annoyance!) was turn on the laptop, plug in the Wi-fire, and point it toward to log. To my pleasure I got an IP address and I'm writing this blog post through the device. I'm happy to say this is one inexpensive device the works as advertised. Nice job hfield! I'm only getting 700kbps, but I'm almost positive that's the fault of the lodge's wifi, not the Wi-Fire. I'll try and test in the lodge sometime to verify that, but it's never been all that quick.

Oh, between getting the device and getting to the house to try it Verizon sent me a bulk mailing saying that broadband is now available at the mountain house. 1Mb for $20/month. Oh well.

Stupid. Stupid.

2009-07-15T19:41:16Z

In a fundraising appeal titled "Hillarycare revisited," the RNC warned about "Obamacare" and said the government "already runs car companies, banks and mortgage companies. Republicans believe that the last thing the American people want is government telling them when and where — or even whether — they can get medical treatment for their families."

The American people much prefer their credit card companies telling them whether they can get medical treatment for their families.

All this argument is about how to ration, not whether to ration. Like every other good in our economy, health care. Today it's rationed by ability to pay; that may or may not be fair. Arguing that it shouldn't be rationed is either utopian, idiotic, or duplicitous.

Clouds as terrorist targets?

2009-07-06T14:14:49Z

As a rule I don't repost from other blogs; I figure there isn't much of a need to just parrot at a distance. I can't help myself this time though.

Thanks Bruce.

Conary 2.0.44

2009-07-04T13:06:31Z

Yesterday we released Conary 2.0.44 with a rather dry change list. Hidden in there are some significant performance improvements for update, updateall, migrate, and group cooks.

For the various forms of update, the time before Conary has built it's update plan is much shorter, especially if multiple passes of dependency resolution are required. For updateall --info on my laptop, the time went from twelve minutes to four thanks to a whole bunch of changes. Some of those changes were in the client side dependency solver, whose iterative resolving is significantly more efficient. So much more so that those changes were enough to speed up the build time of the Foresight groups by about 30%.

Now you know what I do on airplanes!

Amazon On Demand

2009-06-05T19:37:59Z

I've spent the last couple of weeks upgrading my home theater set up. One of those things where one failure (a subwoofer) cause another (the receiver) and the to do list just snowballed (switch to hdmi [finally], use the RF feature on the harmony remote, and so on). One of the items was to hook my TV up to an ethernet cable now that Panasonic VIERAcast has support for Amazon On Demand.

It all seems to work really well, but here's the rub: I haven't bought anything. The shows on there are just too expensive. One of the most egregious example I've found is Jimmy Neutron season one (can you tell I have kids?). They charge $1.99 for a 30 minute episode. That's $4/hour, or about the same as a first run movie at the theater! For further comparison, the "best of season one" DVD is $27 for 17 episodes, one of which is double length. So 9 hours of "entertainment" for $27, or $3/hour. Why would I pay 33% more for on demand, which I can't watch in my car, on a plane, or during a vacation. I can even get the DVD two days for free later thanks to Amazon on demand. Not even my six year old thought Jimmy Neutron was worth $4/hour.

There is a similar oddity if you compare HD shows to DVD purchases. They tend to cost more than the DVD (though the on demand for standard def costs less than the DVD). Given that it's not real HD (due to the reduced bitstream and bandwidth) it's hardly a compelling purchase. So much for Amazon on demand.

Don't get me started on the pricing for current episodes compared to Hulu.

Secession

2009-04-17T16:08:09Z

Sit back and imagine how Rush Limbaugh and his compatriots would have reacted if the Democratic governor of New York had talked about seceding in 2002.

Can you even imagine the deafening cries of unamericanism? Of a lack of patriotism? Of unfitness for office?

What a bunch of hypocrites.

A version to rule them all

2009-04-16T22:43:21Z

I've been working on Conary for a long time now. For those who don't know, Conary is basically a version control system for deployed systems. We all use Subversion or whatever for our sources (and probably anything else that looks like source if you squint the slightest bit) for years now, but for some reason we don't use them for deployed systems.

At best, we use a loose collection of versions. Tools like dpkg and rpm (and apt and yum) have integrated the versions of software components into peoples minds pretty well. Questions like "which version of vi are you running" are easy to answer in the Linux world. Simple commands like "rpm -qp /bin/vi" will give you reasonable answers.

What that leaves is systems defined by, oh, 500 versions or so. Oh, and those versions probably don't include version information for your actual application or third party software products. Add those to the count.

What all this means is that those 10,000 servers you're running don't have a version each, they have 500 versions each. You can't go look at two of them and easily see if they're the same. You can't make two of them match without going through backflips. You can't ask what version a server was running last week because the question doesn't make sense; you have to ask what 500 versions it was running last week. It's like using RCS to manage all of your source files. You might get a tag to get some kind of consistency, but aren't git versions better? Finally a single version to describe the state of your source tree. A single way ("hg parent" in my world) to know what the heck is there.

Conary provides the same capability to running systems. Define your systems as Conary groups and have a single version. You need another system like that one? Just install the same version of the same group? You want to know what it was running last week? Look at what version of the group was on there last week (the rollback stack or /var/log/conary will both tell you). Do you need to downgrade? No problem.

I simply don't know how system management can scale without a version associated with a system. Not a piece of a system, but the entire system.

Fun and games with a Kindle

2009-03-16T00:03:15Z

Right after the Kindle 2 started shipping, I bought one (that Monday). I figured if I could get FAA approach plates on the thing (like ReaderPlates does for the Sony) I'd save enough money to make the thing free within about 18 months. Oh, and it was a toy.

Funny enough, I've discovered that I kind of like reading books on the thing. Especially big, fat books where I lose track of things and having the search is great. But really any book I don't feel the need to own forever I'm happy to read on the device. It's light, the UI is good, the battery lasts forever, and you really disappear into a good book just like you do with paper. Nice job Amazon.

That aside, I've also enjoyed hacking up a couple of web apps which generate ebooks on the fly for the thing. I have one which I can list a couple of airports on and get an ebook with all of the FAA plates I need to fly in instrument conditions. The plates are date stamped, so I know when I'm current. Michael was a huge help in getting all of that done. It gave me a chance to play with django and genshi too, which I needed to learn more about anyway.

Today I decided to play with the TripIt APIs and see what I could make happen. After 240 lines of python code and about the same amount of genshi templates (with a huge dependency on xobj!), I have a web app which lists my itineraries in TripIt, let's me choose one, and creates an ebook summary of that itinerary. Now all the intregration I've done with email filters for TripIt pays off not only in iCal, but in a book format. Alright, so it's not clear why I need my itinerary on my phone's calendar and in my kindle, but it is cool to have hacked that up so quickly (almost all of it while a chicken was roasting). The authentication isn't real (it uses basic auth, not oauth), but it's real enough for me!

Airline TV payment model

2009-02-18T05:33:49Z

I'm writing this at 30,000 feet (I'll post it later) on a flight from New York to San Diego. A long flight. It's scheduled for six and a half hours nonstop; it's about as far as you can go without leaving the continental United States.

I spent some time hacking on a side project I'm working on with mkj, wrote about a third of a white paper he and I are working on with Jake, read the latest Economist for a while, ate dinner (eh), had dessert (yummy, but I shouldn't have done it), tried not to watch Gary Unmarried, watched all of Big Bang Theory, watched most of a How I met Your Mother, mostly ignored a movie about a teenage girl (I don't identify), played mine sweeper a bit out of pure desperation, and now I'm waching the video screen loop.

I have an hour and a half left.

As my eyes flicked back up to Gary Unmarried I realized it was a repeat. A loop. This flight is so long they ran out of material. Get me out of here.

I was actually tempted to watch it this time, but managed to resist and started thinking about what shows I do watch. The list is really pretty short. In order or priority they are:

30 Rock
My Name is Earl
Scrubs
some animated stuff I record but don't seem to actually watch

What struck me about this list is that I started watching 30 Rock on airplanes and My Name is Earl in hotel rooms (yes, I travel a bit). Thinking about this list I realized that I also watch Big Bang Theory and Everybody Loves Raymond when I'm traveling because I've seen it on airplanes and they can both make the time pass. That's about all I watch save the Family Guy which seems to have eaten TNT.

Ignoring Family Guy (which is little more than background noise), and I've started watching 60% of my shows on airplanes, and another 20% when I'm traveling. My wife got into Scrubs, so of the shows I've picked out to watch over the last few years, I've started watching all of them when I'm traveling, and most of them on airplanes!

This surprised me a bit, and has an interesting implication. I've always wondered how the partnership between airlines and the networks work (American partners with CBS and Delta with NBC; notice I don't watch anything from ABC). I had always assumed the airlines paid the networks for content, but based on a sample size of one the highly captive audience on an airplane is incredibly valuable because it has a high potential of leading to new viewers. I wonder if the networks pay the airline to show content to that audience. If they don't, they probably ought to.

80 minutes left. I'm back to being board.

Google Maps Fail

2009-02-04T20:47:09Z

Too really get this, click on the links...

Say you're staying at the Doubletree hotel in downtown boston. You look to see where it is in google maps. Realizing that you are renting a car, but there is no reason to keep it for the next day, you search for Avis locations nearby.

Look at that, there is an Avis location right next to the hotel at 1 Bennet Street! Fantastic! So call up Avis (you're out on a Sunday and you need the car for the next day), get the pickup at Logan, and the drop off at 1 Bennet Street. All set.

Now it turns out that the Charles Hotel at 1 Bennett Street (the hotel with the Avis, and note the extra t) is in Cambridge, not Boston. That's miles away, and not at all useful. There is a 1 Bennet Street in Boston though, right next to the hotel you're staying at. Apparently that and Cambridge being kind of like Boston were enough to confuse google maps, and you.

Toll roads

2009-01-12T16:57:05Z

Here is a concise, well written justification of toll roads. My wife regularly makes fun of me for suggesting that privatizing the interstate system would be a good idea. It's nice to see some sort of economic argument about how to fix road overcrowding.

Now, if someone would explain to me it makes sense to highly subsidize automobiles, moderatley subsidize air travel, and barely subsidize train travel I'd appreciate it.

XObj, Part 2

2008-12-26T02:24:41Z

giles7777 posted some comments about the XObj announcement I posted here on Tuesday. He raised some points Brett and I talked about quite a bit when we put this thing together, so I wanted to respond to them. As an aside, giles7777 and I did more extreme programming together (at the time, we called it finding the damn bugs, but now it has an actual name!) than I've done with anyone else, so I hate ignoring his thoughts!

He asked if we had thought about parsing the schema instead of the XML instance in order to generate types. This approach is quite popular in the Java world, for instance. It's quite sane for a non-dynamic language where items need to be strongly typed from the beginning, and there are project in the python world for it as well. Brett and I talked long and hard about starting with the schema, and it was the approach I favored in the beginning. There were a few reasons we didn't take that approach.

One important reason was we wanted a framework which works with XML documents without schemas. While there are proper schemas for many types of documents, the informal XML formats we use internally at rPath, as well as in many of our APIs have never had a schema written. You can certainly argue that we're lazy not to do so, but I strongly suspect the vast majority of XML documents do not have a schema associated with them.

Another reason is that schemas can be poorly defined, requiring augmentation to be really useful. OVF is an example of this, unfortunately. The types use attributes which should rightfully be defined as IDREF types, but aren't. Having a way to augment a formal schema was necessary for us, and leaving the schema behind was an easy way of achieving this. (The python XObj implementation currently resolved ID/IDREF pairs for both serializing and parsing, automatically referencing a single object in both places; the AS3 implementation is a little behind but will get this feature soon)

The third major reason we wound up ignoring schemas was forward compatibility. The approach we took will parse any XML, let the application modify the pieces it understands, and output the XML without losing the unknown parts. We can represent both the known and the unknown consistently, allowing us high levels of compatibility as the document content gets enhanced. You can argue this is a misfeature as it has a high tolerance for documents which are not compliant with their own schema, and the Python implementation does allow the caller to validate a document against a schema as part of the parsing process.

The final reason, which may not be as important of a consideration as the other two, is we wanted a learning curve for this library which was flat. A lot of the existing XML parsing solutions take some getting used to, and coders who aren't steeped in XML take time to get productive. By ignoring the schema, not having to have code generation steps, and generating native objects XObj is easy to use. A single line turns XML into an object hierarchy which is easy to use, modify, and spit back out. The downside of course is that it's easy to create documents which violate a schema when you do so.

Introducing XObj

2008-12-23T22:14:19Z

A couple of weeks ago, I was talking with Brett Adam about XML decoding. We're looking at how to handle OVF in python, and I was talking about the approach the open-ovf had taken and my desire to get real python classes around the OVF objects instead of a python layer around the XML. Something which turns the XML into an artifcat of the object model instead of an object model which allows access to the XML.

Brett told be about the approach to XML he'd used for the ActionScript implementation of the rPath Management Console, and I thought it was a good approach. He enhanced the XML parser provided with ActionScript by making it type aware; elements would get parsed into classes of the right type which could then be serialized on the output. This provides a natural place to hang methods as well as a nice way of providing typing hints into the parser.

I decided to code up a similar approach for python. As I worked on it, I kept walking into Brett's office with corner cases, XML oddities, and things I just didn't understand. We decided to work together on a common approach to XML parsing in the two languages, and release the work as the XObj open source project under an MIT license. The code hasn't been released yet, but it is available via mercurial at http://hg.rpath.com/xobj/".

You should really think of what we've built as an object reflector. It's goal is to either take the objects described by an XML document and turn those into a set of classes (either python or AS3) consistent with that document, or to take a set of objects and serialize those in XML. It allows type information for both elements and attributes, the caller to specify which objects to use where, and places to put serialization hints.

One of this projects primary goals is to be easy to get started with and let the programmer use the more complicated features as he finds the need. Let me show you a couple of examples from python.

>>> from xobj import xobj
>>> class Foo(object):
...     pass
... 
>>> class Bar(object):
...     def sum(self):
...         return self.first + self.second
... 
>>> foo = Foo()
>>> foo.val = "value"
>>> foo.bar = Bar()
>>> foo.bar.first = 1
>>> foo.bar.second = 2
>>> print xobj.toxml(foo, 'foo', xml_declaration = False)
<foo>
  <bar>
    <first></first>
    <second>2</second>
  </bar>
  <<val>value</val>
</foo>

See? Nice and simple. Now, let's say that same hunk of XML is stored in the string varaible xml. Here's how you turn that into a python object tree:

>>> doc = xobj.parse(xml)

The object which is returned, doc, represents the entire document. If includes some housekeeping elements which make generation cleaner, as well as objects which repesent the XML document. The top level element in the XML was called foo, so that's where the element representing the top level element was stored.

>>> doc.foo.val
'value'
>>> doc.foo.bar.first
'1'
>>> doc.foo.bar.second
'2'

Simple in and out object serialization with XML in between. Nothing fancy, and there are certainly other ways of doing this. Let's look at what else we can do now though. We have lost the class information for Bar though since we didn't tell the parser what object to use for the bar element. We can fix that though.

>>> doc = xobj.parse(xml, typeMap = {'bar' : Bar})
>>> doc.foo.bar.sum()
'12'

Okay, so maybe that's not quite what we wanted. We need to tell the parser that first and second are integers, not strings.

>>> doc = xobj.parse(xml, typeMap = {'bar' : Bar, 'first' : int, 'second' : int})
>>> doc.foo.bar.sum()
3

That typeMap specifies the type for bar elements and it's elements first and second. To avoid those maps getting overly complicated, here is another way of doing the same thing:

>>> class Bar2(Bar):
...     first = int
...     second = int
>>> doc = xobj.parse(xml, typeMap = {'bar' : Bar2})
>>> doc.foo.bar.sum()
3

Here we're using class variables as form of prototyping. We could carry this even further, and specify a class for foo which tells what bar</tt> should be. >>> class Foo2(Foo): ... bar = Bar2 ... >>> doc = xobj.parse(xml, typeMap = {'foo' : Foo2}) >>> doc.foo.bar.sum() 3 </pre>

There are lots of other things you can do with prototypes like this. The last I'm going to show here (as it is 5pm two days before Christmas!) shows how you can force an item to be a list.

>>> doc = xobj.parse(xml, typeMap = {'bar' : [ Bar2 ] } )
>>> doc.foo.bar[0].sum()
3

By making mapping the bar element to a list of </tt>Bar</tt> classes, we've told the parser to always use a list (normally an element creates a list of objects only if that element appears more than once. (Note that I used a typeMap here instead of the prototype in the Foo class, but both methods are identical).

There are lots more things XObj can do, all of which are demonstrated in the python test case. Give it a try, and tell me what you think!

Conary and Version Control

2008-11-25T14:20:39Z

When we got started on Conary, one of the original goals was to bring the strengths of the version control tools used by software developers to the systems tools used by system administrators. We explored those ideas in the first paper we presented on Conary, and it's been an important guideline ever since.

I was talking to an architect at a well known financial institution last week, and he had a good analogy for the power of Conary. He compares the rpm/dpkg/yum/apt methodology to rcs. Each file is separately managed, with separate versioning and little to no coordination between the files. Conary, according to him, was much more like subversion (I'd have preferred mercurial, but nevertheless!) where the entire archive was treated and versioned as a whole (thanks to Conary's groups). That makes it much easier to keep sets of things in sync and operate at a system level instead of an individual package level.

I thought this was a pretty good way of thinking about how Conary is different from the older approaches, and provides some insight into why groups are so powerful.

iPhone NDA, redux

2008-10-01T16:47:24Z

Kudo's to Apple for changing their mind on the draconian NDA they've imposed. Now, if we could just get some progress on opening up the platform a bit we'd all feel good.

I guess my wife's MacBook is feeling safe!

iPhone Apps

2008-09-26T12:58:22Z

Some people still wonder why I'm not comfortable with an iPhone. I think engadget summarizes it best:

Sections 5.1, 5.2, and 5.3: This SDK legal agreement is confidential. Engadget shouldn't even be writing about it. Also confidential: any knowledge we share with you about how to develop iPhone apps. So if you were thinking about doing a blog or a book about developing for the iPhone, or simply open-sourcing your app, then think again. Knowledge is a dangerous thing -- why do you think we did a commercial themed after Orwell's 1984?

Hopefully most folks know me well enough to know I'm not going to be particularly supportive of a platform which makes open source apps a contract violation. Ironic for a phone which is based on a pile of open source, isn't it? Makes me think about taking my wife's MacBook away.

No frills airlines

2008-09-14T02:25:09Z

Remember when Southwest was the no-frills, discount airline?

Now that so many of the other airlines charge for checked bags, aisle row seats, or calling them for a reservation it seems like US Air, American, and friends ought to be called the discount airlines. It's a shame they charge more than Southwest most of the time.

One size fits all

2008-08-29T17:04:15Z

Apparently Red Hat has a problem with perl. It's slow. Really slow. To fix it, you just need a recompile. Of course, recompiling breaks our support agreement for perl, but never mind that.

I'm sure Red Hat will fix this soon. It's only been two years.

Isn't it time to work with a Linux partner who thinks you should change the things that bother you. A partner who realizes they won't get everything right? A partner who thinks that open source should be about, well, the source?

USB Power

2008-05-22T20:49:24Z

I'm sitting at Houston Hobby airport waiting for a flight home, and while I was looking for power I saw these pedestals between a lot of the seats at the Southwest gates.

Not only does it have one 120V outlet between each seat, but it has one powered USB plug between each seat! Time for all of those manufacturers who think proprietary power plugs still make sense to grow up and realize custom wall warts are not a feature.

Recipe Factories

2008-03-03T15:56:48Z

For a while now I've been struggling with how to get recipes out of source components. In many cases, our superclasses are now smart enough to eliminate the need for them altogether, leaving just a stub recipe. For binary tarballs, a single addArchive rule is all that's needed in the recipe (you end up putting a version in there as well, but it's almost always encoded in the filename itself making even that spurious). I've been hoping to get to the point where you can check in (say) a single binary tarball and cook without having to have a recipe at all.

For this to work well, the cook process would need to look in the repository for the right cook rules, just like it does for superclasses already. We'd also want to be able to leverage the work which has already been done with superclasses, so whatever we added needs to be compatible with those. We came up with the idea of a recipe factory, and I just finished up a working implementation for the next version of Conary 2.

The basic idea is that a source component can now have a type, and that type tells the cook process to go look in the repository for help building that source component. For example, if testtar:source has type archive, Conary will look for a factory-archive:source component, and use the recipe in that component as the starting point for building the testtar package.

The factory-archive:source component itself must be of type factory, and it is not a normal recipe. It instead provides a recipe factory, which is a class derived from the Factory superclass. Conary instantiates an object of that class and calls that object's getRecipeClass method. That method returns a Recipe class (not an object!), often descended from the PackageRecipe superclass.

If the original source component, testtar:source in this case, does not provide it's own recipe, the recipe class returned by the recipe factory is used to build the package. All of the normal recipe rules apply for that class; it must have the right name attribute, it must specify a version string, and so on. From this point on, it's just a normal build. If the source component did include it's own recipe, the class returned by the factory gets called FactoryRecipeClass and the source component's own recipe is then loaded. The source component's recipe file will then create it's own recipe class using FactoryRecipeClass as it's superclass, which will be used to drive the build.

One last point: the factory knows a bit about the recipe it's supposed to build. The name of the package being built (testtar in this case) is available as self.name and the path to all of the source files is self.sources. It can also open any of the source files by using self.openSourceFile(path), so it can inspect the files in the source component to decide how to proceed.

An example of a very simplistic recipe factory is in oot.rpath.org as factory-archive:source, and a source component which uses it is in the same repository as testtar:source. Factories could be used in a large number of different ways, and I think it will take some experimentation to find out what works well and what doesn't.

Get all of this? I know it's a bit complicated, but it's really very powerful.