The need arose today to issue a bulk set of HTTP DELETE operations against our applicaiton's REST-based API.

It took me a little digging to find out how to use curl to do this, so here it is:

    curl -X DELETE http://localhost/example/    

Loving curl!

It is quickly becoming one of my favourite command line tools.

After reading a lot about node.js recently, I decided to give it a whirl.

Having just built a clean desktop install of Ubuntu 10.4 I had the ideal environment.

Here are the steps to get it up and running.

First, start a super user console session

    sudo -i -- and enter password to elevate privileges

Now, setup the pre-requisites for compilation and test execution:

    apt-get install g++ curl libssl-dev apache2-utils

Next, download, unpack and build:

    wget http://nodejs.org/dist/node-v0.1.96.tar.gz
    gunzip node-v0.1.96.tar.gz
    tar -xf node-v0.1.96.tar
    cd node-v0.1.96/
    ./configure
    make
    make install

Finally, to run the tests:

    make test

That's it, you're done. Happy noding!

If you are doing LDAP integration work with Active Directory and you want to update a user's password via an LDIF script you need to specify the password attribute in a very specific way.

The sequence of steps is this:

• Surround the password with speech marks, "like this"
• Covert to bytes using the Unicode character encoding - a two byte code page which pads the low ASCII characters with 0
• Encode using Base 64

So something like password becomes IgBzAGUAYwByAGUAdAAiAA==

This all sounds well and good, here's a code sample:

    Public Function FormatPassword(ByVal password As String) As String

        If password Is Nothing Then Throw New ArgumentNullException("password", "password cannot be nothing")

        ' enclose in speech marks
        password = String.Format("""{0}""", password)

        ' convert to bytes with unicode encoding
        Dim bytes() As Byte = Encoding.Unicode.GetBytes(password)

        ' convert to base 64
        Dim base64 As String = Convert.ToBase64String(bytes)
        Return base64

    End Function

Some sample passwords:

    admin     IgBhAGQAbQBpAG4AIgA=
    root      IgByAG8AbwB0ACIA
    password  IgBwAGEAcwBzAHcAbwByAGQAIgA=
    apple     IgBhAHAAcABsAGUAIgA=
    banana    IgBiAGEAbgBhAG4AYQAiAA==
    pencel    IgBwAGUAbgBjAGkAbAAiAA==

LDIF Set Password Script

To use this encoded password in an LDIF script, you will need to create LDIF fragments like this:

    dn: cn=timhastings,ou=Users,dc=example,dc=com
    changetype: modify
    replace: unicodePwd
    unicodePwd:: IgBwAGEAcwBzAHcAbwByAGQAIgA=

The double-colon is required on the unicodePwd attribute as it identifies that the data which follows is Base64 encoded.

To import this into Active Directory, you will need to use the ldifde command line too.

Password Policy

You must make sure your passwords meet the strength and history requirements of the systems password policy, otherwise you will receive the following error message:

    The server side error is: 0x52d Unable to update the password.
    The value provided for the new password does not meet the length, complexity, or history requirements of the domain.
    The extended server error is: 0000052D: SvcErr: DSID-031A11E5, problem 5003 (WILL_NOT_PERFORM), data 0

ldifde Secure Connection

When settings passwords, Active Directory insists you use a secure connection otherwise you will get an Unwilling To Perform error like this:

    Add error on entry starting on line 1: Unwilling To Perform
    The server side error is: 0x1f A device attached to the system is not functioning.
    The extended server error is:            
    0000001F: SvcErr: DSID-031A11E5, problem 5003 (WILL_NOT_PERFORM), data 0

To specify a secure connection to LDAP, use ldifde like this:

    ldifde -i -f -h -v myscript.ldf

I hope this helps!

A number of times I have found myself unable to access a file because some other process is writing to it.

Usually lock files which are used to prevent multiple instances of the same program from running at the same time, or to guard against multiple operations running at the same time that would interfere with each other.

The Linux command which comes to the rescue is called lsof which I guess is short for "list of open files"

It can be used in several different ways:

List All Open Files

    lsof

This will list all the God damn files open on your system. There will be lots of other interesting things listed which are not files, things like pipes and network sockets. This is because on Linux, everything is abstracted to a file, even the keyboard.

The output from lsof can be piped to grep to search for bits of file names.

    lsof | grep pony

This will perform a very crude search for any file name, path or process with anything to do with 'pony'

List All Open Files for a Given Process (by PID)

For example:

    lsof -p 12345

Lists all file handles currently held by process 12345

lsof headings

By default, the lsof output looks like this (without headings)

mysqld    30353     mysql  219u      REG        8,2 5976322292    6652117 /mnt/mysql/tagwalk/msg.MYD
mysqld    30353     mysql  220u      REG        8,2  672237472    6652170 /mnt/mysql/tagwalk/tag.MYD
mysqld    30353     mysql  221u      REG        8,2  672237472    6652170 /mnt/mysql/tagwalk/tag.MYD
mysqld    30353     mysql  222u      REG        8,2 5976322292    6652117 /mnt/mysql/tagwalk/msg.MYD
(cmd)     (pid)     (user) (handle)  (type) (device)(size/offset) (node)  (path)

From left-to-right, the columns are:

• Command (first 9 characters)
• Process ID (PID)
• User
• File Descriptor and Access Mode: r=read, w=write, u=read/write
• Type: REG=Regular File, PIPE=Pipe, IPv4=TCP/IP socket (everything's a file remember!)
• Device: In this case, I'm not sure what "8,2" means
• File Size or Offset (if relevant)
• Node: The node number of this file (an internal file system thing)
• Name: The file name and path of the open file

Hope this helps!

This week, the Mono team released version 2.2 of the framework (release notes).

I have created a new Amazon EC2 image with Mono 2.2 installed on Amazon's Fedora 8 image.

This new image supersedes the previous versions I created for 1.9.1 and 2.0.

Here's is a link to this image's page in the EC2 resource centre.

The EC2 AMI and manifest path are:

ami-dc8760b5
nonhostile-mono2-2-i386/image.manifest.xml

Once you start this instance, you can browse to its public DNS name to see the XSP test sites.

This instance is intended to serve web applications or run console based applications or services; it does not have any kind of graphical interface.

• Amazon’s Fedora 8 base image (ami-5647a33f v1.08)
• Latest updates via yum
• Apache 2.2.9 installed via yum
• Apache and mod_mono are configured to serve the XSP 'test' files from the web site's root in /etc/httpd/conf/httpd.conf (at the bottom)

The following have been compiled and installed from the Mono 2.2 stable sources with the --prefix=/usr

• mono-2.2
• mono-basic-2.2
• mod_mono-2.2
• libgdiplus-2.2
• xsp-2.2

The sample ASP.Net files for 1.0 and 2.0 are installed at this root:

/usr/lib/xsp/test

The Apache config (/etc/httpd/conf/httpd.conf) file has been configured to serve an ASP.Net application from this location.

You can edit the Apache configuration file using vim. Use Ctrl-D in vim to page-scroll down.

vim /etc/httpd/conf/httpd.conf

# Set mono as the handler
SetHandler mono

# Configure a 'root' web application to run from root
MonoApplications root "/:/usr/lib/xsp/test"
MonoServerPath root /usr/bin/mod-mono-server2
<Location />
  MonoSetServerAlias root
</Location>

To serve your own ASP.Net applications from this instance, you need to store the files somewhere, for example, /mnt/MyApp, then modify the MonoApplications directive at the bottom of the Apache configuration file.

MonoApplications root "/:/mnt/MyApp"

Good luck and happy Mono developing!

• Mono AMI: documentation at Amazon community directory
• Mono 2.2 Release Notes
• Mono: The Mono homestead sporting a new look
• mod_mono: Documentation on configuring mod_mono
• VIM Shortcuts: For help using vim
• Twitter: My twitter link if you want to keep in touch

bookmark this on del.icio.us

From time to time, my web host changes some aspect of the server that this weblog is hosted on. The code that runs this site is written in classic ASP which means it is hosted on IIS. Recently another configuration change took place, presumably one of Microsoft's many hot fixes or an upgrade to a new version of Windows or IIS.

The following error started to appear tagged onto the bottom of every web page:

    msxml3.dll error '80004001'
    Not implemented

Behind the scenes, this site use XML and XSL to separate the site's data from its layout, in the middle somewhere is a line of code which performs an XSLT transformation on the XML DOM into the HTML output which gets squirted out into the response.

    ' method to transform the response (with errors)
    public sub TransformToResponse_Old(xml, xsl)
        xml.transformNodeToObject xsl, Response
    end sub

Somewhere in the depths of ASP or IIS, a change has occurred which has dropped the IStream support of the classic ASP response object (I presume looking at the symptoms).

The fix introduces an intermediate stream to receive the XSLT transformation, and then send that to the response. This method also sets the response code page to be UTF8, which this site uses.

    ' method to transform the response (without errors)
    public sub TransformToResponse_New(xml, xsl)

        ' By Tim Hastings, www.nonhostile.com
        ' prepare stream to receive transformation
        dim outputStream
        Set outputStream = Server.CreateObject("ADODB.Stream")
        outputStream.Open 
        outputStream.Charset = "UTF-8"
        outputStream.Type = 1 ' adTypeBinary

        ' transform and output to stream
        xml.transformNodeToObject xsl, outputStream
        
        ' set character-set
        Response.CharSet = "UTF-8"
        Response.ContentType = "text/html"        
        Session.CodePage = 65001
        
        ' rewind the stream and send to response
        outputStream.Position = 0
        Response.BinaryWrite outputStream.Read()
                                
        ' finished with the stream
        outputStream.Close
        set outputStream = Nothing

    end sub

Hope this helps!

On Monday, Mono 2.0 was released by the Mono Development team. This release has been in-progress for a couple of years and is a major step forward in the mission to run .Net on Linux and other platforms. If you check out the release notes you can see there's a bus load of new features supported; including Linq!

To make it as easy to use Mono 2.0 in Amazon EC2 cloud I have built a new version of the image I created with Mono 1.9.1 a couple of months ago.

Update: a Mono 2.2 AMI is available

The EC2 AMI and manifest path are:

ami-b627c3df
nonhostile-mono2-i386/image.manifest.xml

Once you start this instance, you can browse to its public DNS name to see the XSP test sites.

This instance is intended to serve web applications or run console based applications or services; it does not have any kind of graphics interface.

• Amazon’s Fedora 8 base image (ami-2b5fba42 v1.07)
• Latest updates via yum
• Apache 2.2.6 installed via yum
• Apache and mod_mono are configured to serve the XSP 'test' files from the web site's root in /etc/httpd/conf/httpd.conf (at the bottom)

The following have been compiled and installed from the Mono 2.0 stable sources with the --prefix=/usr

• mono-2.0
• mono-basic-2.0
• mod_mono-2.0
• libgdiplus-2.0
• xsp-2.0

The sample ASP.Net files for 1.0 and 2.0 are installed at this root:

/usr/lib/xsp/test

The Apache config (/etc/httpd/conf/httpd.conf) file has been configured to serve an ASP.Net application from this location.

You can edit the Apache configuration file using vim. Use Ctrl-D in vim to page-scroll down.

vim /etc/httpd/conf/httpd.conf

# Set mono as the handler
SetHandler mono

# Configure a 'root' web application to run from root
MonoApplications root "/:/usr/lib/xsp/test"
MonoServerPath root /usr/bin/mod-mono-server2
<Location />
  MonoSetServerAlias root
</Location>

To serve your own ASP.Net applications from this instance, you need to store the files somewhere, for example, /mnt/MyApp, then modify the MonoApplications directive at the bottom of the Apache configuration file.

MonoApplications root "/:/mnt/MyApp"

Good luck and happy Mono developing!

• Mono AMI: documentation at Amazon community directory
• Mono 2.0 Release Notes
• Mono: The Mono homestead sporting a new look
• mod_mono: Documentation on configuring mod_mono
• VIM Shortcuts: For help using vim
• Twitter: My twitter link if you want to keep in touch

bookmark this on del.icio.us

The article discusses how to setup MySQL Replication between two Amazon EC2 instances.

It walks you though setting up replication for an empty database server. Adding a slave to a server already full of data is a different article.

It is assumed that you already know the basics of starting EC2 instances, connecting to them via SSH and editing files in Linux using vi/vim etc.

For this tutorial, I am using the Amazon built machine image ami-2b5fba42 which is Fedora 8 base image.

In this tutorial, we will:

• Launch two EC2 instances, a Master and a Slave
• Install MySQL Server and tools onto each machine (must be the same version of MySQL on both)
• Configure MySQL on each so that each has a unique server ID and keeps its data in an EC2 friendly place
• Create a user on the master for replication and configure the slave to use it.
• Synchronize the master/slave replication logs
• Test it all works

The steps for configuring the master are as follows:

• Launch an EC2 instance using your favourite method (I like the ElasticFox Firefox extension)
• Install MySQL Server and Client Tools (via Yum)
• Edit /etc/my.cnf to alter the data folder and edit replication settings
• Configure MySQL to run at boot (in case we restart the instance)
• Start MySQL and configure to run at boot
• Create a MySQL user for replication

First, launch an instance of the Fedora machine, and connect using SSH.

This gives us a base Fedora instance.

Next, install MySQL and configure it to start when the machine boots (in case we decide to restart it)

Install MySQL Server and tools:

yum install -y mysql mysql-server

Rename the old config file (if you're interested in keeping it) and edit the /etc/my.cnf:

mv /etc/my.cnf /etc/my.cnf.old
vi /etc/my.cnf

To look something like this:

[mysqld]

# replication settings for MASTER
server-id               = 1

# data folder
datadir                 = /mnt/mysql

# switch on binary logging (required for replication)
log-bin                 = mysql-bin

# system stuff
user                    = mysql
socket                  = /var/lib/mysql/mysql.sock

[mysqld_safe]
log-error               = /var/log/mysqld.log
pid-file                = /var/run/mysqld/mysqld.pid

Now we can configure MySQL to start at boot, start it now and connect.

chkconfig --level 2345 mysqld on
service mysqld start
mysql

Now we can create a replication user account:

GRANT REPLICATION SLAVE ON *.* TO 'ReplicationUser' IDENTIFIED BY 'ReplicationPassword';
FLUSH PRIVILEGES;

We're done. Next, the slave...

This is an almost identical sequence of steps, with a couple of minor differences.

• Start another Amazon instance in the same availability zone (if you want speed and no bandwidth cost) or in a separate zone if you are more concerned about availability.
• Install MySQL Server and Client Tools (via Yum)
• Edit /etc/my.cnf to alter the data folder and edit replication settings
• Configure MySQL to run at boot (in case we restart the instance)
• Start MySQL and configure to run at boot
• Tell the MySQL Slave who/where the Master is (using internal DNS name)
• Start replication on the slave

Once the slave instance is running, connect using SSH and install MySQL and tools (just like the master):

yum install -y mysql mysql-server

Rename the old config file (if you're so inclined) and edit the /etc/my.cnf:

mv /etc/my.cnf /etc/my.cnf.old
vi /etc/my.cnf

To look like this:

[mysqld]

# replication settings for SLAVE
server-id               = 2

# data folder
datadir                 = /mnt/mysql

# switch on binary logging (not essential but handy if you want to replicate from this slave in the future)
log-bin                 = mysql-bin

# system stuff
user                    = mysql
socket                  = /var/lib/mysql/mysql.sock

[mysqld_safe]
log-error               = /var/log/mysqld.log
pid-file                = /var/run/mysqld/mysqld.pid

Now we can configure MySQL to start at boot, start the service going and connect:

chkconfig --level 2345 mysqld on
service mysqld start
mysql

On the Master, we need to determine the binary log's starting position. This is the byte offset that our Slave should start reading from. On the master, type the following:

mysql> SHOW MASTER STATUS;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000003 |      319 |              |                  | 
+------------------+----------+--------------+------------------+
1 row in set (0.00 sec)

Next, we want to connect the SLAVE to the MASTER. For this, we need the replication user details we created earlier and the internal (or private) DNS name for the MASTER, this should be available from ec2-dim or the ElasticFox instances list. We also need the file and position we get from the master in the step above. This is done with the MySQL CHANGE MASTER command:

CHANGE MASTER TO
   MASTER_HOST='domU-11-22-33-44-55-66.compute-1.internal',
   MASTER_USER='ReplicationUser',
   MASTER_PASSWORD='ReplicationPassword',
   MASTER_LOG_FILE='mysql-bin.000003',
   MASTER_LOG_POS=319;

START SLAVE;

Now replication is working, we can issue commands against the Master, and check to see if the Slave replicates the result.

To do this, use two terminals side by side, one connected to the Master and one to the Slave.

mysql> CREATE DATABASE HelloWorld;
Query OK, 1 row affected (0.00 sec)

mysql> use HelloWorld
Database changed
mysql> CREATE TABLE Message (Content VARCHAR(100) NOT NULL PRIMARY KEY);
Query OK, 0 rows affected (0.01 sec)

mysql> INSERT INTO Message (Content) VALUES ('Howdy slave');
Query OK, 1 row affected (0.00 sec)

On the Slave, we can now verify if the changes made to Master have been replicated:

mysql> SELECT * FROM HelloWorld.Message;
+-------------+
| Content     |
+-------------+
| Howdy slave | 
+-------------+
1 row in set (0.00 sec)

Congratulations! You have a working replication.

There are a number of things you can use your slave for:

• Taking backups without locking the Master: If you cannot afford the downtime on your Master to take a backup (backups must lock tables/databases for consistency) you can use STOP SLAVE to pause replication while you take a backup from the Slave. Resuming replication will pickup where it left off. This prevents the master from slowing down or having to lock tables while a consistent backup is taken.

• As part of a scale out strategy: You can modify your application to read data from one of a number of available slaves. This is suitable for read intensive applications with slow-changing data.
• As a replication master: You can daisy-chain slaves off other slaves in either a long bus configuration or in a tree style hierarchy. Just because you can seems to be a good enough reason to me.
• Run tests against like live data: If you break the replication link, you have a complete copy of your live databases to test your latest version against.
• As a stunt-double Should disaster strike, and something bad happen to your master, you can change you application to use the Slave instead. If you enabled binary logging on the slave, then you are already in a position to attach new slaves to that and promote it to the new master. How easily your application can switch over to the new master is a design issue you must consider when encountering problems connecting to the master.

• MySQL Replication Documentation
• Amazon EC2
• Elasticfox Firefox Extension
• Twitter: My twitter link if you want to keep in touch

bookmark this on del.icio.us

Here's another round-up of quotes, wisdom and snippets...

On succeeding

• Whether you think you can or think you can't - you are right. [Henry Ford]
• Anything worth having is worth fighting for.
• Your most unhappy customers are your greatest source of learning. [Bill Gates]
• If you’re trying to go south and you find yourself walking north, it’s always best to turn around. “We’ve walked this far already” isn’t a good enough reason to continue heading in that direction.
• Around the time of Caesar, there was a European tribe that, when the assembly horn blew, always killed the last warrior to reach his assigned place, and no one enjoyed fighting this tribe.
• My wife would rather have a ticket for one fur coat, than a ticket that gave her two or nothing. [Warren Buffet]
• Son, you can do whatever you set your mind to. There's no shame in doing any job, the best you can hope for is that you enjoy it.
• Defender's dilema: The guard must keep watch continuously, while the prisoner only needs to escape once.
• Fall seven times, stand up eight. [Japanese proverb]
• Let a hundred flowers blossom. [Chairman Mao]
• Everyone chooses more or less what kind of events will happen to them by their conscious patterns of blocking and yielding.
• Optimism makes you plan for success, pessimism makes you plan for failure. [Tim Hastings]

On the creative process

• The first draft of anything is shit. [Hemingway]
• Easy reading is damn hard writing. [Hawthorne]
• No one has to see your failures unless you add vanity to folly and exhibit them. [Robert McKee]
• The key to great writing is to leave out the boring bits that people skip. [Elmore Leonard]

On getting things done

• A good plan, violently executed now, is better than a perfect plan next week.
• Only put off until tomorrow what you are willing to die having left undone.
• Worry often gives a small things a big shadow. [Swedish proverb]
• Nothing of value comes easy.

On economics

• Markets can remain irrational longer than you can remain solvent. [John Maynard Keynes]
• If you owe the bank $100 that's your problem. If you owe the bank $100 million, that's the bank's problem. [JP Getty]
• It is generally agreed that casinos should, in the public interest, be inaccessible and expensive. And perhaps the same is true of Stock Exchanges. [John Maynard Keynes]
• OCTOBER: This is one of the peculiarly dangerous months to speculate in stocks in. The others are July, January, September, April, November, May, March, June, December, August, and February. [Mark Twain]
• Save a little money each month and at the end of the year you'll be surprised at how little you have. [Ernest Haskins]
• The safe way to double your money is to fold it over once and put it in your pocket. [Frank Hubbard]

On technology

• Any sufficiently advanced technology is indistinguishable from magic. [Arthur C. Clarke]
• It's a poor sort of memory that only works backwards. [White Queen, Through the Looking-Glass]

On software development

• You can be Date Driven or Feature Driven but not both.
• Normalize until it hurts, denormalize until it works.
• I have wasted so much of my life making futile web apps that nobody really cares about. [Paras Chopra, Are you guilt of not solving the world's problems?]
• But user content usually follows the 80/20/1 rule, 80% browse, 20% interact (comment), and 1% contribute (add albums). [Dan McGrady, 7 Reasons Why My Social Music Site Never Took Off]
• Good judgement comes from experience, and experience comes from bad judgement.
• Premature optimization is the root of all evil. [Knuth] ...but... Belated pessimization is the leaf of no good. [Lattanzi]
• Tesler's Law of Conservation of Complexity: You cannot reduce the complexity of a given task beyond a certain point. Once you've reached that point, you can only shift the burden around. [Larry Tesler]
• Jakob’s Law of the Internet User Experience: Users spend most of their time on other sites. This means that users prefer your site to work the same way as all the other sites they already know.

On everything else

• Everything you will ever need to know about medicine: Air goes in and out. Blood goes round and round. Oxygen is good. [from What we learn from the dying]
• Better to keep your mouth shut and appear a fool than to open it and remove all doubt [Mark Twain]
• You should not be judged by your looks, but you will.

Here are some previous quote dumps.
Enjoy!

• 266Mhz ARM Intel XScale CPU
• 32MB SDRAM
• 8MB Flash
• 10/100 RJ-45 Ethernet port
• 2 USB 2.0 ports
• 5V DC power consumption (solar power?)

    echo slug > /etc/hostname

    # The primary network interface
    allow-hotplug eth0
    iface eth0 inet dhcp
        hostname slug

• NSLU2 Linux: the homestead of all Linux things for the NSLU2
• Debian Installer Firmware: to bootstrap the installation
• Installing Debian on the Linksys NSLU2
• WinSCP and PuTTY: The essential SSH tools for Windows

ami-0abe5a63
nonhostile-mono-i386/image.manifest.xml

mono hello-world.exe

• Amazon’s Fedora 8 image (ami-f51aff9c v1.06)
• Apache 2.2.6 installed via yum
• Mono 1.9.1 binary installation
• mod_mono 1.9 compiled and installed from source
• Apache and mod_mono are configured to serve the XSP 'test' files from the web site's root in /etc/httpd/conf/httpd.conf (at the bottom)

ln -s /usr/lib/libgdiplus.so.0 /usr/lib/libgdiplus.so

• Mono: The Mono homestead
• mod_mono: Documentation on configuring mod_mono
• Amazon EC2: The mother ship
• Elasticfox: a Firefox Extension for Amazon EC2
• WinSCP and PuTTY: The essential SSH tools for Windows
• Twitter: My twitter link if you want to keep in touch

bookmark this on del.icio.us