Monday, May 31, 2010

My $1 Plastic Toy Car has a Metal Flywheel and a Plastic Gearbox

One of the greatest things about being a parent is the ability to relive your own childhood – often vicariously as you watch your own children experience life with fresh eyes. But with toys, we get to experience childhood again firsthand. I believe the electric train and R/C car toy markets exist principally to cater to the fathers rather than the children. I know my dad spent much more time setting up and working the electric train table than us kids. But this is hardly the point of this blog post.

tractor_withtop

Rather, I glimpsed inside a broken toy tonight, and was intrigued by what I could see. I was interested enough to break it open and see exactly how it was put together. It turns out there is some good mechanical engineering principles at work within even the cheapest of toys.

Yellow Tractor from the $1 Bin

In this case, it is a yellow tractor purchased at Target last year for $1. It is the kind of vehicle that you can “rev” up by running it across the table with your hand, and then release and let it move under its own power. My daughter had dismantled pieces from the top, which allowed me to peek inside.

I will provide the model number (Ankyo234041829F16819779-08/09) just in case it is a rare toy that is now worth a million dollars. How about giving me $200,000 for a busted one? As you would expect, you also find Made in China stamped to the bottom.

Inside the toy are the following components:

  • Plastic frame; holes in the plastic serve as bearings for the axles
  • 4 plastic wheels (inside of the fake outer tracks)
  • Metal axles for the wheels, with a spline in the middle to drive the first cog in the drivetrain
  • A gearbox of 3 cogs; the middle cog is actually 2 cogs mated together (small and big). Metal shafts hold the cogs in place.
  • A big metal flywheel (probably lead, to provide for better inertia)

I was surprised at how much mechanical advantage was provided by the gearbox. Moving the wheels just a fraction of a turn spun the big flywheel all the way around. Nice job, Made in China guys!

tractor_gearbox_flywheel

Note: that little green plastic tab is only there to make noise. It rests on top of the flywheel input cog, and gives the gearbox a little clickity click sound as it rakes over the teeth.

The Gearbox

I have time boxed this blog post to less than 45 minutes, because, well, this isn’t something I should be spending a lot of time on. But I did want to understand how much mechanical advantage was being provided by the gearbox. I decided that counting teeth on each wheel was going to take too long, so I am just making some estimates here. Also, I am not rechecking my high school physics text book, so I could be doing this all wrong. Like it matters.

But I estimate the advantage to be 125:1. Meaning, every rotation of the wheel will result in 125 rotations of the flywheel. Imagine what happens when you rapidly move the car across, say 2 feet, of table top. Assuming the wheel circumference is 2 inches, that means the the wheels will spin 12 times, causing the flywheel to spin 1500 times. Holy shrapnel-if-it-disintegrates!

Once again, I am timeboxed. So this diagram is the best I can come up with to illustrate the gearbox design:

tractor_sequence

I came up with my 125:1 figure by estimating three places of mechanical advantage of 5:1. If my physics are right, we multiply those together to get 125. Sorry for wasting 3 minutes of your life if that isn’t right, hey we are all doing the best we can.

Conclusion

Is there one?

Perhaps this is interesting stuff to you, as it was to me. I didn’t think a toy from the $1 bin at Target would reveal a lesson in mechanical advantage interesting enough for me to write about. I was wrong. Sorry, Made in China guys, for underestimating you.

(elapsed time in researching and writing this post – 51 minutes)

Technorati Tags: ,,,

Sunday, May 16, 2010

Browsing your SnapLogic SQLite Configuration Database

This blog post explains how to browse the configuration database of your SnapLogic server. I found this helpful when I was modifying some SnapLogic server code, and messed something up. I thought I had corrupted something in the database (I hadn’t), so wanted to open it up and take a look. I thought I’d share what I learned.

It turns out the SnapLogic configuration repository is really simple. There is only one table you really care about, and it has but a few columns.

Setting Up Your SQLite Browser

The configuration database for SnapLogic is a standard SQLite database. There are various tools you could use to browse it, but I like the Firefox SQLite Manager plugin. It installs into a tool you probably already use (Firefox) and does everything you need.

Steps to get running:

  • Install the SQLite Manager plugin and restart Firefox
  • Navigate to Tools->SQLite Manager in the Firefox menu
    • A new window will popup with the manager in view
  • Navigate to Database->Connect Database in the menu
  • When the file browser pops up, choose your SnapLogic repository DB
    • Choose [SNAPLOGIC_HOME]/repository/repository.db
    • You will have to disable file extension filtering; make sure “All Files” is selected

You should now have the SnapLogic database open for browsing.

Inside the Resource Table

The table of interest is the resource table. It contains the definitions of all of your resources, including component instances and pipelines. Almost everything you do in the SnapLogic Designer UI is stored in this table.

The table has the following structure:

Column Purpose
uri The logical URI given to the resource in the Designer. As in: /SnapLogic/Tutorial/Exercise_5/Resources/SortProspects
guid A nice globally unique id for the instance. For migration/backups, this is very nice to have.
gen_id A version number. Incremented each time you click Save on the resource in the Designer.
object The actual definition of the resource. This is a large field, and is written in a readable format (JSON?).
validated Has this resource been validated? 0 for false, 1 for true. See caveat below.

Overall, it is pretty self explanatory as far as I can tell. Perhaps there are hidden nuances, but none that I have detected.

The only surprise was the use of the validated column. I thought that would be updated when you hit the “Validate…” button in the Designer. Not so, at least in SnapLogic 2.1.0 (beta). This column seems to be updated upon a successful run of a Preview operation.

I hope this blog post helps you in your SnapLogic development. Post a comment if so!

Technorati Tags: ,

Friday, May 14, 2010

Upgrade to Ubuntu 10.04 broke my Postgres – “Could not Connect to Server” in Pgadmin

I upgraded last week to Ubuntu 10.04, and just noticed today that my development environment was broken (yeah yeah, it took a week to notice, give me a hard time about not spending enough time coding). I looked into it, and found a weird problem caused by the upgrade – perhaps if you are reading this you have that problem too. My Postgres database was apparently not operating.

Postgres 8.4 Upgrade

I did notice that the Ubuntu 10.04 upgrade was taking me from Postgresql 8.3 to 8.4. I figured that should be fine. And it is, but it caused some downstream problems.

First off, put away any notion that the upgrade does an automatic migration of your databases from 8.3 to 8.4. Ubuntu is good, but not that good. You will have to manually migrate your databases yourself. I imagine you should dump all your databases before you do your upgrade, as the upgrade does remove 8.3 from /usr/lib/postgresql. Since I could rebuild my databases, I didn’t have to worry about migration – but do some research ahead of time if you do.

This was one issue – my dev databases were not migrated into 8.4. I needed to rebuild them, and recreate our dev user in 8.4. But before I could do that…

Pgadmin Could Not Connect

A major roadblock was that I could not connect to Postgres using Pgadmin, even as the ‘postgres’ user. In fact, I couldn’t seem to get network connectivity to Postgres at all. What is going on?

Server doesn't listen
The server doesn't accept connections: the connection library reports could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432?

To troubleshoot this, I made sure that the processes were running:

sudo /etc/init.d/postgresql-8.4 restart

and then made sure it stayed started:

ps –ef | grep postgres

and also checked the logs:

/var/log/postgresql$ cat postgresql-8.4-main.log

But everything seemed to be kosher. I was also able to use psql from the command line. What is going on, I still can’t connect from Pgadmin? An invocation of netstat is what solved it:

netstat -ln | grep 5432

That should have returned some lines like:

tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN    
tcp6       0      0 ::1:5432                :::*                    LISTEN    
unix  2      [ ACC ]     STREAM     LISTENING     5623     /var/run/postgresql/.s.PGSQL.5432

but it returned nothing. Eh? 5432 is the default port for postgresql, and the port I use in my Pgadmin connections. It should be there.

I then looked at the postgres server configuration in  /etc/postgresql/8.4/main/postgresql.conf and found the problem:

port = 5433

For some reason the upgrader installed 8.4 on port 5433 instead of 5432. During the upgrade it perhaps needs to run 8.3 and 8.4 at the same time, and therefore puts them on different ports. But after upgrade, shouldn’t it switch 8.4 back to the default port?

Its an easy fix:

sudo gedit postgresql.conf   (change 5433 to 5432)

sudo /etc/init.d/postgresql-8.4 stop

sudo /etc/init.d/postgresql-8.4 start

Note that I first tried just restart, but found that the port change did not take effect. I had to do a stop and start explicitly.

I am now back up and running again with Postgres 8.4. Joy.

Technorati Tags: ,

Thursday, May 13, 2010

Installing SnapLogic Community Edition 2.1.0 on Ubuntu 8.10 in 5 minutes

If you read my last post, you would know that I hit a snag running the community edition of SnapLogic 2.2. There appears to be a hard dependency on a license check (see ticket 2449). Since I have no idea how long that will take to resolve, I decided to find a way around the problem.

I chose to roll back to version 2.1.0. But in doing so, I had some struggles. SnapLogic 2.1.0 is based on Python 2.5.x and related components. My Ubuntu 10.04 install uses Python 2.6.x, and the 2.5.x stuff has been removed from the Ubuntu repositories.

After trying some minor hackery to get P2.5.x on U10.04 (building stuff from source), I gave up at trying to get the dependencies resolved. Instead, I just plopped down a new Ubuntu 8.10 VM on my machine, and got things going pretty quickly.

These are my notes. By the way, the 5 minutes I reference in the title is just for the SnapLogic install. Installing and updating Ubuntu takes about an hour, depending on network speeds.

  • Install 8.10
    • Download a VDI and install into VirtualBox, or
    • Do a bare metal install (e.g. dual boot)
  • Update your OS
    • 368 updates and counting as of today
    • Make sure not to upgrade Python beyond 2.5.x
    • Reboot
  • Download SnapLogic 2.1.0
  • Launch the SnapLogic installer
    • chmod u+x [blahblah.bin]
    • It will inform you what is missing from prereqs
  • Install the missing prereqs
    • Launch Synaptic Package Manager, and install:
    • python2.5-dev
    • libmysqlclient15-dev
    • libxml2-dev
    • libxslt1-dev
  • Rerun SnapLogic installer
    • Accept all defaults
    • Password is password
    • Elect to start up the servers at the end
  • Install Flash Plugin for Firefox
  • Navigate to the Designer UI
    • http://[hostname]:8081/
    • Open the Leads_to_Prospects Pipeline
    • Select the Run tab
    • Run the pipeline (this is where SL 2.2 fails for me)
    • Success!

 

SLDesigner

Technorati Tags: ,,

Building and Installing SnapLogic 2.2 from Source on Ubuntu 10.04

This blog post walks you through building and installing SnapLogic 2.2 on a Ubuntu 10.04 box. These instructions will lead you through building the product from source. There is also a handy installer available, but that is only for the Pro version. The Pro version has a 14 day trial, and then you’ll have to fork over money. Since I plan to work on my prototype over a longer period of time, I have decided to build the community version from source.

Note: there is currently a blocking issue that prevents this from working. Be sure to read through to the bottom of this page for details before proceeding on this yourself.

The official build and installation instructions live on the SnapLogic wiki, but I am posting here so I can provide more details as I go. Also, there are a couple of changes from the official instructions presumably due to 10.04. Since it seems only Ubuntu 9.x is supported for SnapLogic 2.2, updating the official wiki would probably not be welcome. Items marked with DELTA below are different relative to the official docs.

Caveat: I am running 10.04, but my laptop has been a Ubuntu machine since 9.04. I did an in place upgrade to 9.10 and then 10.04, and have installed a lot of things along the way. Your mileage may therefore vary with these instructions.

Step 1: Install Prerequisite Packages (DELTA)

Execute the following command in a terminal window:

sudo apt-get install python-mysqldb python-pysqlite2 python-cherrypy3 python-feedparser python-simplejson python-lxml python-beautifulsoup python-paste python-genshi

Note: make sure to specificy python-cherrypy3, otherwise you get 2.3.0-3, which will not work with SnapLogic 2.2.

Step 2: Download the Source

Go to the community site and download the source for both the server and the designer:

Step 3: Extract the Source into a Directory

cd ~/dev
mkdir snaplogic
cd snaplogic/
unzip /home/plaird/Desktop/downloads/snaplogic-src-2.2.0.zip
 

Step 4: Create a Runtime Directory

cd ~/dev/snaplogic/
mkdir runtime

Step 5: Setup your Shell

Ultimately, you want to define these in ~/.bashrc, but we will do it inline for now.

export SnapPackageDir=/home/plaird/dev/snaplogic
export SnapRuntimeDir=/home/plaird/dev/snaplogic/runtime
export PYTHONPATH=$PYTHONPATH:${SnapPackageDir}

Step 6: Build out your Runtime Directories

mkdir ${SnapRuntimeDir}
mkdir ${SnapRuntimeDir}/bin
mkdir ${SnapRuntimeDir}/repository
mkdir ${SnapRuntimeDir}/designer
mkdir ${SnapRuntimeDir}/cache_dir
mkdir ${SnapRuntimeDir}/compute
mkdir ${SnapRuntimeDir}/static
mkdir ${SnapRuntimeDir}/logs
mkdir ${SnapRuntimeDir}/config

Step 7: Copy in Runtime Files (DELTA)

The official instructions inform you that you need to install icon files at this point. That no longer seems necessary, as the Designer installation in a later step now takes care of that.

However, I found that you do need to install the components and tutorial into the runtime directory.

cd ${SnapRuntimeDir}
cp -r ../snaplogic/components/ .
cp –r ../snaplogic/example/tutorial .
cd tutorial
mkdir data
mv *.csv data

Step 8: Launch SnapAdmin

cd ${SnapRuntimeDir}
python ${SnapPackageDir}/snaplogic/tools/snapadmin.py

Step 9: Create your Repository in the Runtime Directory

repository create -t sqlite /home/plaird/dev/snaplogic/runtime/repository/repository.db

Step 10: Create a Server Configuration File

Download the template configuration file from the website (link is on this page) and copy it into your ${SnapRuntimeDir}/config directory. Make the following modifications:

  • Replace ${SnapRuntimeDir} with absolute path
  • Change YOURHOST.EXAMPLE.COM to the fully qualified domain name of your host (three locations)
  • Optionally change the ports from 8088, 8081 and 8089

Step 11: Install the Authentication Files

These files give you a starting solution for authentication:

cp ${SnapPackageDir}/snaplogic/server/auth/simple_snapaccess.conf ${SnapRuntimeDir}/config/
cp ${SnapPackageDir}/snaplogic/server/auth/simple_passwords ${SnapRuntimeDir}/config/

Step 12: Install the Designer GUI

cd ${SnapRuntimeDir}unzip /home/plaird/Desktop/downloads/snaplogic-designer-2.2.0.zip

Step 13: Create the Server Start Script (DELTA)

The default control script is not included in the community edition. Create a file called startServers.sh in your runtime directory. Its contents should be as follows (replacing localhost:8088 with your server URL):

echo "Starting data server"
python ${SnapPackageDir}/snaplogic/server/server.py  --config=${SnapRuntimeDir}/config/snaplogic_server.conf &
echo "Starting management server"
python ${SnapPackageDir}/snaplogic/mgmtserver/mgmt_server.py --config=${SnapRuntimeDir}/config/snaplogic_server.conf &
echo "Starting component server"
python ${SnapPackageDir}/snaplogic/cc/component_container.py --name=cc1 --server=
http://localhost:8088 &

Step 14: Correct the Bad Import on Ubuntu 10.04 (DELTA)

If you are running with Ubuntu 10.04, you probably have package python-simplejson at version 2.0.9-1build1. SnapLogic 2.2 was certified against version 1.7.1. You will need to remove an import statement from a SnapLogic file:

gedit /home/plaird/dev/snaplogic/snaplogic/rp/streaming_json_rp.py

Edit out this line:

from simplejson.decoder import JSONScanner

Step 15: Start the Servers and View the UI (DELTA)

chmod u+x startServers.sh
./startServers.sh
firefox
http://<your_machine_name>:8081 &

Step 16: Walk through Tutorial

You can at this point start with the tutorials.

Step 17: ROAD BLOCK

Sadly, at this point I have hit a road block. The community edition of the product apparently has a bug in that it depends on an license file for a commercial version of the product (pro or core). I have filed defect 2449 – hope to get a fix shortly!

In the meantime, see my follow on post on how to install SnapLogic 2.1.0 instead. While it is an earlier version, the upside is the 2.1.0 version has a community installer.

 

Technorati Tags: ,,

Thursday, May 6, 2010

Example Linux startup service for Python, Java, Ruby, Perl, or Script as a daemon process in init.d

This blog post provides sample Bash code for converting a Python, Java, Ruby, Perl, C++, script, or any other program as a Linux service that will start running at system startup. It is a script that is deployed in /etc/init.d like many other common service programs.

Note that there may be language specific ways of doing what you want. For example, there is an evolving Python package that is supposed to do this for you. At this writing, that package seems to be still in progress.

This solution has three components:

  • The actual program that needs to run as a service. In this example, it is a python (.py) program. But it really can be anything, including Java, Ruby, C++, or whatever
  • A wrapper Bash script that simply invokes the actual program. The only value add is that it writes the process identifier of the actual program to a file. You could possibly folder this functionality into the daemon itself, but I haven’t bothered to do that.
  • The daemon script. It is what is located in /etc/init.d and ultimately is responsible for starting and stopping the service.

Actual Program – echoHW.py

For this example, it does nothing real. It just spins the process:

import time

while 1 < 2:
    time.sleep(10)

Wrapper Script – echoHW.sh

This script simply invokes the actual program. It does also write out the process identifier to the file.

# Execute the actual python script
# The "$@" passes in any parameters into the python exectuable
# The '&' puts the process into background (as a daemon)
# The 'echo $! > mydaemon.pid'  write the process id to a file

python echoHW.py "$@" &
echo $! > mydaemon.pid

Daemon Script – mydaemon

This file must be copied into /etc/init.d.

#!/bin/bash
#
# mydaemon          Start/Stop any shell script
#
# chkconfig: 345 95 65
# description: mydaemon 
# processname: mydaemond
#

# ENVIRONMENT
# Edit these for your configuration

# Name for the service, used in logging
NAME=mydaemon

# Name of the user to be used to execute the service
SCRIPT_USER=plaird

# Example of how to pass paramters into the command
PARAM2=whatever

# In which directory is the shell script that this service will execute
MYDAEMON_SCRIPTS_DIR=/home/plaird/dev/linux

# Construct the command the will cd into the right directory, and invoke the script
MYDAEMON_COMMAND="cd $MYDAEMON_SCRIPTS_DIR; ./echoHW.sh 'param1' $PARAM2"

# How can the script be identified if it appears in a 'ps' command via grep?
#  Examples to use are 'java', 'python' etc.
MYDAEMON_PROCESS_TYPE=python

# Where to write the log file?
MYDAEMON_SVC_LOG_FILE=$MYDAEMON_SCRIPTS_DIR/mydaemon.log

# Where to write the process identifier - this is used to track if the service is already running
# Note: the script noted in the COMMAND must actually write this file
PID_FILE=$MYDAEMON_SCRIPTS_DIR/mydaemon.pid

# Load system specific optional arguments
# Create and populate this file with machine specific settings
if [ -f /etc/sysconfig/mydaemond ]; then
    . /etc/sysconfig/mydaemond
fi

# Is the service already running? If so, capture the process id
if [ -f $PID_FILE ]; then
    PID=`cat $PID_FILE`
else
    PID=""
fi

# SERVICE ENTRY POINTS (START/STOP)

# Start Command
start() {
    if [ "${PID}" != "" ]; then
        # Check to see if the /proc dir for this process exists
        if [ -a /proc/${PID} ]; then
            # check to make sure this is likely the running service
            ps aux | grep ${PID} | grep $MYDAEMON_PROCESS_TYPE >> /dev/null
            # If it is a process of the right type assume that it is mydaemon and just exit
            # otherwise remove the subsys lock file and start mydaemon
            if [ "$?" = "0" ]; then
                exit 1
            else
                echo "mydaemon lock file still exists, removing..."
                rm /var/lock/mydaemond
            fi
        else
            # The process running as pid $PID is not a process of the right type, remove subsys
            # lock and start mydaemon
            echo "mydaemon lock file still exists, removing..."
            rm /var/lock/mydaemond
        fi
    fi
    echo -n "Starting mydaemon: "   
    su - $SCRIPT_USER -c "/bin/sh -c \"$MYDAEMON_COMMAND > $MYDAEMON_SVC_LOG_FILE 2>&1\"" & RETVAL=$?
    sleep 3
    touch /var/lock/mydaemond
    exit 0
}

stop() {

    echo -n $"Stopping mydaemon: "
    if [ "${PID}" != "" ]; then
        echo -n "killing " $PID
        kill ${PID}
        for i in {1..30}
        do
            if [ -n "`ps aux | grep $MYDAEMON_PROCESS_TYPE | grep mydaemon `" ]; then
                sleep 1 # Still running, wait a second.
                echo -n .
            else
                # stopped
                rm -f /var/lock/mydaemond
                rm -f $PID_FILE
                echo
                exit 0
            fi
        done
    else
        echo "$NAME is not running"
        exit 1
    fi
    echo "Failed to stop in 30 seconds."
    kill -QUIT ${PID} # Request a thread dump so we can diagnose a hung shutdown
    exit 1
}

case "$1" in
  start)
      start
    ;;
  stop)
      stop
    ;;
  *)
    echo $"Usage: $0 {start|stop}"
    exit 1
esac

Installation

1. Start by copying the actual program and wrapper script somewhere on your machine in a permanent location.

2. Next, edit the mydaemon script to reflect the location where the program was installed. You may want to rename the file, and update the name of mydaemon inside the file.

3. Copy the mydameon script to /etc/init.d. As in:

sudo cp mydaemon /etc/init.d

4. Install the service. Installation will vary depending on your flavor of Linux. If you have chkconfig installed, it will be:

sudo chkconfig –-add mydaemon

If you are on a distro like Ubuntu, use update-rc.d, as in:

sudo update-rc.d mydaemon defaults

Service Usage

The service should now start on startup the computer. But you can invoke it manually as root:

sudo service mydaemon start

or:

sudo service mydaemon stop

Technorati Tags: ,,