The author of the piece linked above has scratched this itch with Maatkit, a suite of replication-focused MySQL tools which include some for checksumming tables.

This can run with standalone instances but most appealing is the way it runs across a replication chain: the statements to perform the table checksums are themselves introduced into the replication and store their results in a table alongside the master’s checksums. You can then compare and contrast on each slave: neat. All straightforward to roll into Nagios or other monitoring.

The first firewalls were on trains. Coal-powered trains had a large furnace in the engine room, along with a pile of coal. The engineer would shovel coal into the engine. This process created coal dust, which was highly flammable. Occasionally the coal dust would catch fire, causing an engine fire that sometimes spread into the passenger cars. Since dead passengers reduced revenue, train engines were built with iron walls right behind the engine compartment. This stopped fires from spreading into the passenger cars, but didn’t protect the engineer between the coal pile and the furnace. (There’s a lesson for sysadmins in this somewhere.)

(My emphasis)

Particulary cool is that they will let you at the source of your network. You can’t then wander off and run it elsewhere, it sits atop of their core web framework that you can’t see. With the source (which is encouragingly well written PHP) you can do perform all manner of modifications to bend the template network to your will.

Ning provide their own authentication system, and there’s no API to hook in someone else’s which is a hassle if you’re trying to build a social network alongside another site: maintaining a login for each site is going to annoy the users and be a nightmare to manage. Nil points.

One solution is to build a single sign-on system around what Ning already provide, which is robust, tested and better presented than anything we could achieve! I’d sketched out such a system for a proposal years ago but never had the opportunity to build it. A current project provided the perfect excuse to try this out.

Layout

Operation
The numbers here match those in the diagram.

1. User visits www.example.com (which for us happens to be a Rails) site.
2. The PageController notices the browser supplied no cookie and must therefore log on to Ning before proceeding. The site returns a redirect to Ning’s authentication page.
3. The browser follows this redirect to Ning.
4. Ning authenticates the user. The login code is modified from the original Ning behaviour. Here, it issues a redirect to sso.example.com with some parameters, including the user’s Ning ID and a salted hash to prevent spoofing. In this redirect Ning sets a range of cookies, including one that identifies the user to Ning.
5. The browser follows this redirect to the SSO server (which happens to be a Merb site - I wanted to try it out!)
6. The SSO app checks the provided hash against its own idea of what it should be. Assuming they match it considers the Ning ID to be valid. Finally, the SSO app issues another redirect along with cookies for just .example.com. This cookie identifies the user to the external site
7. In passing, the SSO app keeps track of users it has seen, and if this is a new user it will make an API request to Ning to fetch that user’s profile data and create a matching user on the www.example.com site.
8. The browser follows this last redirect back to Ning. It could be back to www.example.com or Ning depending on the situation.
9. Ning knows who the user is by merit of its cookies.
10. As does our external site.

Notes

• Having the SSO app different from the www.example.com site is perhaps a bit baroque. It works because the SSO app issues a cookie for .example.com which the browser will offer to both sso.example.com and www.example.com. In favour of this approach is that the SSO app is simple, and thus less likely to fail during development iterations than the nascent www.example.com site. A failure in the SSO app is bad, becaise it locks people out of Ning too. That Rails and Merb can share session data (and a database) is cool.
• The SSO app’s fetching of Ning profile data allows us to maintain a local version of a user’s profile to avoid the need to fetch it from Ning every time we need. There’s a nuance of Ning API authentication that meant I had to write a custom widget (Ning site component) to handle that.

Specifics
I was about to paste bits of modified Ning code, but I’ll need to check if I can under the various blurbs you agree too when signing up! Anyhow, the information above should help answer some of the requests for details from the Ning developers forum.

• There’s no invention here, I believe this setup is very common.
• High availability isn’t the same thing as load balanced. There is nothing here to intelligently shared load across the frontend servers, and one backend server is essentially idle all the time.
• This cluster is built with a bunch of open-source software on non-fancy kit. As such it doesn’t have the enormous capacity of clusters built upon commercial shared-storage products, SAN kit, layer 7 web switches etc. Its ambition is to run a few busy Rails sites well whilst coping with hardware failure gracefully.

Layout

Operation

• Web traffic is spread across the managed frontend interfaces by multiple A records in the DNS.
• Wackamole uses a Spread messaging network to ensure these multiple A record IPs are always present across the frontend. It achieves this by managing the hosts’ interfaces when it detects hosts joining or leaving the cluster.
• A pair of MySQL servers run in master:master configuration on the backend hosts
• The backend hosts use DRBD to maintain a mirrored block device between them.
• These block devices back a NFS filesystem.
• Heartbeat runs on the backend hosts to do several tasks:
  1. Manage which host is the DRBD primary and therefore can be written to.
  2. Manage which host has the DRBD filesystem mounted and exported with NFS.
  3. Manage the IP through which the frontend mounts the filesystem and talks to MySQL.
• With all this in place, Nginx accepts web connections and serves static assets off the NFS mount and passess other requests to Mongrel, a HTTP server that’s well suited to running a Rails instance.

Notes

• One of the main hazards of MySQL master:master setups is primary key collision if an INSERT occurs on both hosts at once. We avoid that here by letting Hearbeat manage the IP that the frontends connect to.
• I’ve built two of these clusters to date. The second one is now four servers wide on the frontend.

Future work

• DRBD can now run in dual-primary mode, allowing both hosts to accept writes. This makes it a candidate for filesystems like GFS that use shared storage to present a filesystem that can be written to on multiple hosts. More here.
• To add some load balancing I’m considering using HAProxy or LVS to actively distribute traffic across the frontends.
• HA aside, there’s also some cool things like evented Mongrel that it would be interesting to try.

Extra credit for assigning a keyword (eg ‘ktx’) in the bookmark properties, allowing you to just type, eg, ‘ktx 1234′ in the Location bar to achieve the same.

This is a rework of a similar hack for a much older ticketer, PTS, which is amazingly still in use at one of my previous workplaces. You can gauge its age by the fact that it was ported to PHP3 and was pretty open to most injection attacks!

However the magic doesn’t touch everything. I was so bold as to move Dictionary.app from its default home in /Applications into the Utilities subdirectory. This broke the Ctrl-Cmd-D system-wide (Cocoa-wide?) lookup box. I don’t use it that much but that irked me nonetheless.

Console logs included this:

com.apple.launchd[335] (com.apple.DictionaryPanelAgent[490]): posix_spawn(”/Applications/Dictionary.app/Contents/SharedSupport/DictionaryPanel.app/Contents/MacOS/DictionaryPanel”, …): No such file or directory

Clearly something thinks it knows where the nested DictionaryPanel application should live. This thing is a launchd-managed process used by the keyboard shortcut, configured out of /System/Library/LaunchAgents/com.apple.DictionaryPanelAgent.plist.

The hoojah to tweak this does of course involve XML:


$ launchctl unload /System/Library/LaunchAgents/com.apple.DictionaryPanelAgent.plist

$ plutil -convert xml1 -o - /System/Library/LaunchAgents/com.apple.DictionaryPanelAgent.plist | perl -pe 's|/Applications/Dictionary.app/|/Applications/Utilities/Dictionary.app/|;' > /var/tmp/com.apple.DictionaryPanelAgent.plist

$ plutil -lint /var/tmp/com.apple.DictionaryPanelAgent.plist
/var/tmp/com.apple.DictionaryPanelAgent.plist: OK

$ plutil -convert binary1 /var/tmp/com.apple.DictionaryPanelAgent.plist

$ mkdir /Library/LaunchAgents-orig

$ sudo mv /System/Library/LaunchAgents/com.apple.DictionaryPanelAgent.plist /Library/LaunchAgents-orig && sudo cp /var/tmp/com.apple.DictionaryPanelAgent.plist /System/Library/LaunchAgents/com.apple.DictionaryPanelAgent.plist

$ launchctl load /System/Library/LaunchAgents/com.apple.DictionaryPanelAgent.plist

The last step was intended to make the change work this session, but it didn’t work. The DictionaryPanelAgent still loads but didn’t do anything until logged out and in again. I think this is something to do with launchctl domains / sessiontype. Blunder factor is high.

This violates the “don’t frig with stuff in /System” principle but I don’t know how else to solve it. The modified plist could go in /Library/LaunchAgents of course, but I’d still need to disable the system version (with launchctl unload -w) which is equally naughty. I think.

The self-wire kits I found have the RJ45 connections fixed but let you decide the DB9 pin assigments. The Avocent needs a crossover which means the wiring isn’t one to one. It’s detailed here.

So I don’t lose it again:

DB9 pin 9 is unused. No idea how standard the colour / RJ45 assignment is. The merging of brown and white wires, and the splitting of the orange into two pins is the fudge: admittedly I thought I was building straight-through connectors when I set out!

mail.server.default.fetch_by_chunks to false

From here. Unsure if the break is local or on the mailserver, a reasonably normal Courier IMAP deal.

Happily “the vm_kern.c.2 patch” (it’s well known!) sorted things. I stressed ZFS-backed IO massively without any further wobbles.

Now I can micro-quota all my jail(8)s with obsession!

Nov 7 14:03:45 localhost imapd: Failed to connect to socket /tmp/fam--
Nov 7 14:03:45 localhost imapd: Failed to create cache file: maildirwatch (lemon@example.com)
Nov 7 14:03:45 localhost imapd: Error: Input/output error

So we’re left with either rebuilding Courier to not link against it (this is a non-starter, no SRPMs available for this version and I can’t wholesale replace it owing to “control panel” sillyness), or installing FAM itself. For which there are no RPMs, and the source RPMs break owing to a g++ change that occurred ages ago.


/usr/lib/gcc/i386-redhat-linux/4.1.1/../../../../include/c++/4.1.1/backward/backward_warning.h:32:2: warning: #warning This file includes at least one deprecated or antiquated header. Please consider using one of the 32 headers found in section 17.4.1.2 of the C++ standard. Examples include substituting the header for the header for C++ includes, or instead of the deprecated header . To disable this warning use -Wno-deprecated.
../include/BTree.h:240: error: expected constructor, destructor, or type conversion before ‘BTree’
../include/BTree.h:352: error: expected constructor, destructor, or type conversion before ‘BTree’
../include/BTree.h:408: error: expected constructor, destructor, or type conversion before ‘BTree’
../include/BTree.h:503: error: expected constructor, destructor, or type conversion before ‘BTree’
../include/BTree.h:561: error: expected constructor, destructor, or type conversion before ‘BTree’
../include/BTree.h:591: error: expected constructor, destructor, or type conversion before ‘BTree’
make[2]: *** [Client.lo] Error 1

Struggling to remember C++ resulted in this patch:

--- fam-2.6.10.orig/fam/SmallTable.h    2003-04-15 05:21:43.000000000 +0100
+++ fam-2.6.10/fam/SmallTable.h 2007-11-07 13:50:14.000000000 +0000
@@ -98,7 +98,7 @@
 }

 template <class Tkey, class Tvalue>
-SmallTable<Tkey, Tvalue>::Closure
+typename SmallTable<Tkey, Tvalue>::Closure
 SmallTable<Tkey, Tvalue>::position(const Tkey& key) const
 {
     unsigned l = 0, r = n;
Only in fam-2.6.10/: fam-2.6.10.tar.gz
diff -ur fam-2.6.10.orig/include/BTree.h fam-2.6.10/include/BTree.h
--- fam-2.6.10.orig/include/BTree.h     2003-04-15 05:21:19.000000000 +0100
+++ fam-2.6.10/include/BTree.h  2007-11-07 13:55:15.000000000 +0000
@@ -236,7 +236,7 @@
 //  to the right and returns them.

 template <class Key, class Value>
-BTree<Key, Value>::Closure
+typename BTree<Key, Value>::Closure
 BTree<Key, Value>::Node::remove(unsigned j)
 {
     Key k = key[j];
@@ -348,7 +348,7 @@
 }

 template <class Key, class Value>
-BTree<Key, Value>::Closure
+typename BTree<Key, Value>::Closure
 BTree<Key, Value>::Node::next(const Key& pred) const
 {
     if (!this)
@@ -404,7 +404,7 @@
 //  nodes as necessary on the way back.

 template <class Key, class Value>
-BTree<Key, Value>::Closure
+typename BTree<Key, Value>::Closure
 BTree<Key, Value>::insert(Node *p, const Key& key, const Value& value)
 {
     if (!p) return Closure(key, value, NULL);
@@ -499,7 +499,7 @@
 //  Returns UNDER if node p is too small afterward, OK otherwise.

 template <class Key, class Value>
-BTree<Key, Value>::Status
+typename BTree<Key, Value>::Status
 BTree<Key, Value>::underflow(Node *p, unsigned i)
 {
     assert(p);
@@ -557,7 +557,7 @@

 template <class Key, class Value>
-BTree<Key, Value>::Closure
+typename BTree<Key, Value>::Closure
 BTree<Key, Value>::remove_rightmost(Node *p)
 {
     int i = p->n;
@@ -587,7 +587,7 @@
 //  back up.

 template <class Key, class Value>
-BTree<Key, Value>::Status
+typename BTree<Key, Value>::Status
 BTree<Key, Value>::remove(Node *p, const Key& key)
 {
     if (!p)
diff -ur fam-2.6.10.orig/libfam/Client.c++ fam-2.6.10/libfam/Client.c++
--- fam-2.6.10.orig/libfam/Client.c++   2003-04-15 05:21:25.000000000 +0100
+++ fam-2.6.10/libfam/Client.c++        2007-11-07 13:24:28.000000000 +0000
@@ -34,7 +34,7 @@
 #include <syslog.h>
 #include <errno.h>

-#include <iostream.h>
+#include <iostream>

 #include "fam.h"
 #include "Client.h"

Well, that was fun.