Showing posts with label snaplogic. Show all posts
Showing posts with label snaplogic. Show all posts

Sunday, May 16, 2010

Browsing your SnapLogic SQLite Configuration Database

This blog post explains how to browse the configuration database of your SnapLogic server. I found this helpful when I was modifying some SnapLogic server code, and messed something up. I thought I had corrupted something in the database (I hadn’t), so wanted to open it up and take a look. I thought I’d share what I learned.

It turns out the SnapLogic configuration repository is really simple. There is only one table you really care about, and it has but a few columns.

Setting Up Your SQLite Browser

The configuration database for SnapLogic is a standard SQLite database. There are various tools you could use to browse it, but I like the Firefox SQLite Manager plugin. It installs into a tool you probably already use (Firefox) and does everything you need.

Steps to get running:

  • Install the SQLite Manager plugin and restart Firefox
  • Navigate to Tools->SQLite Manager in the Firefox menu
    • A new window will popup with the manager in view
  • Navigate to Database->Connect Database in the menu
  • When the file browser pops up, choose your SnapLogic repository DB
    • Choose [SNAPLOGIC_HOME]/repository/repository.db
    • You will have to disable file extension filtering; make sure “All Files” is selected

You should now have the SnapLogic database open for browsing.

Inside the Resource Table

The table of interest is the resource table. It contains the definitions of all of your resources, including component instances and pipelines. Almost everything you do in the SnapLogic Designer UI is stored in this table.

The table has the following structure:

Column Purpose
uri The logical URI given to the resource in the Designer. As in: /SnapLogic/Tutorial/Exercise_5/Resources/SortProspects
guid A nice globally unique id for the instance. For migration/backups, this is very nice to have.
gen_id A version number. Incremented each time you click Save on the resource in the Designer.
object The actual definition of the resource. This is a large field, and is written in a readable format (JSON?).
validated Has this resource been validated? 0 for false, 1 for true. See caveat below.

Overall, it is pretty self explanatory as far as I can tell. Perhaps there are hidden nuances, but none that I have detected.

The only surprise was the use of the validated column. I thought that would be updated when you hit the “Validate…” button in the Designer. Not so, at least in SnapLogic 2.1.0 (beta). This column seems to be updated upon a successful run of a Preview operation.

I hope this blog post helps you in your SnapLogic development. Post a comment if so!

Technorati Tags: ,

Thursday, May 13, 2010

Installing SnapLogic Community Edition 2.1.0 on Ubuntu 8.10 in 5 minutes

If you read my last post, you would know that I hit a snag running the community edition of SnapLogic 2.2. There appears to be a hard dependency on a license check (see ticket 2449). Since I have no idea how long that will take to resolve, I decided to find a way around the problem.

I chose to roll back to version 2.1.0. But in doing so, I had some struggles. SnapLogic 2.1.0 is based on Python 2.5.x and related components. My Ubuntu 10.04 install uses Python 2.6.x, and the 2.5.x stuff has been removed from the Ubuntu repositories.

After trying some minor hackery to get P2.5.x on U10.04 (building stuff from source), I gave up at trying to get the dependencies resolved. Instead, I just plopped down a new Ubuntu 8.10 VM on my machine, and got things going pretty quickly.

These are my notes. By the way, the 5 minutes I reference in the title is just for the SnapLogic install. Installing and updating Ubuntu takes about an hour, depending on network speeds.

  • Install 8.10
    • Download a VDI and install into VirtualBox, or
    • Do a bare metal install (e.g. dual boot)
  • Update your OS
    • 368 updates and counting as of today
    • Make sure not to upgrade Python beyond 2.5.x
    • Reboot
  • Download SnapLogic 2.1.0
  • Launch the SnapLogic installer
    • chmod u+x [blahblah.bin]
    • It will inform you what is missing from prereqs
  • Install the missing prereqs
    • Launch Synaptic Package Manager, and install:
    • python2.5-dev
    • libmysqlclient15-dev
    • libxml2-dev
    • libxslt1-dev
  • Rerun SnapLogic installer
    • Accept all defaults
    • Password is password
    • Elect to start up the servers at the end
  • Install Flash Plugin for Firefox
  • Navigate to the Designer UI
    • http://[hostname]:8081/
    • Open the Leads_to_Prospects Pipeline
    • Select the Run tab
    • Run the pipeline (this is where SL 2.2 fails for me)
    • Success!

 

SLDesigner

Technorati Tags: ,,

Building and Installing SnapLogic 2.2 from Source on Ubuntu 10.04

This blog post walks you through building and installing SnapLogic 2.2 on a Ubuntu 10.04 box. These instructions will lead you through building the product from source. There is also a handy installer available, but that is only for the Pro version. The Pro version has a 14 day trial, and then you’ll have to fork over money. Since I plan to work on my prototype over a longer period of time, I have decided to build the community version from source.

Note: there is currently a blocking issue that prevents this from working. Be sure to read through to the bottom of this page for details before proceeding on this yourself.

The official build and installation instructions live on the SnapLogic wiki, but I am posting here so I can provide more details as I go. Also, there are a couple of changes from the official instructions presumably due to 10.04. Since it seems only Ubuntu 9.x is supported for SnapLogic 2.2, updating the official wiki would probably not be welcome. Items marked with DELTA below are different relative to the official docs.

Caveat: I am running 10.04, but my laptop has been a Ubuntu machine since 9.04. I did an in place upgrade to 9.10 and then 10.04, and have installed a lot of things along the way. Your mileage may therefore vary with these instructions.

Step 1: Install Prerequisite Packages (DELTA)

Execute the following command in a terminal window:

sudo apt-get install python-mysqldb python-pysqlite2 python-cherrypy3 python-feedparser python-simplejson python-lxml python-beautifulsoup python-paste python-genshi

Note: make sure to specificy python-cherrypy3, otherwise you get 2.3.0-3, which will not work with SnapLogic 2.2.

Step 2: Download the Source

Go to the community site and download the source for both the server and the designer:

Step 3: Extract the Source into a Directory

cd ~/dev
mkdir snaplogic
cd snaplogic/
unzip /home/plaird/Desktop/downloads/snaplogic-src-2.2.0.zip
 

Step 4: Create a Runtime Directory

cd ~/dev/snaplogic/
mkdir runtime

Step 5: Setup your Shell

Ultimately, you want to define these in ~/.bashrc, but we will do it inline for now.

export SnapPackageDir=/home/plaird/dev/snaplogic
export SnapRuntimeDir=/home/plaird/dev/snaplogic/runtime
export PYTHONPATH=$PYTHONPATH:${SnapPackageDir}

Step 6: Build out your Runtime Directories

mkdir ${SnapRuntimeDir}
mkdir ${SnapRuntimeDir}/bin
mkdir ${SnapRuntimeDir}/repository
mkdir ${SnapRuntimeDir}/designer
mkdir ${SnapRuntimeDir}/cache_dir
mkdir ${SnapRuntimeDir}/compute
mkdir ${SnapRuntimeDir}/static
mkdir ${SnapRuntimeDir}/logs
mkdir ${SnapRuntimeDir}/config

Step 7: Copy in Runtime Files (DELTA)

The official instructions inform you that you need to install icon files at this point. That no longer seems necessary, as the Designer installation in a later step now takes care of that.

However, I found that you do need to install the components and tutorial into the runtime directory.

cd ${SnapRuntimeDir}
cp -r ../snaplogic/components/ .
cp –r ../snaplogic/example/tutorial .
cd tutorial
mkdir data
mv *.csv data

Step 8: Launch SnapAdmin

cd ${SnapRuntimeDir}
python ${SnapPackageDir}/snaplogic/tools/snapadmin.py

Step 9: Create your Repository in the Runtime Directory

repository create -t sqlite /home/plaird/dev/snaplogic/runtime/repository/repository.db

Step 10: Create a Server Configuration File

Download the template configuration file from the website (link is on this page) and copy it into your ${SnapRuntimeDir}/config directory. Make the following modifications:

  • Replace ${SnapRuntimeDir} with absolute path
  • Change YOURHOST.EXAMPLE.COM to the fully qualified domain name of your host (three locations)
  • Optionally change the ports from 8088, 8081 and 8089

Step 11: Install the Authentication Files

These files give you a starting solution for authentication:

cp ${SnapPackageDir}/snaplogic/server/auth/simple_snapaccess.conf ${SnapRuntimeDir}/config/
cp ${SnapPackageDir}/snaplogic/server/auth/simple_passwords ${SnapRuntimeDir}/config/

Step 12: Install the Designer GUI

cd ${SnapRuntimeDir}unzip /home/plaird/Desktop/downloads/snaplogic-designer-2.2.0.zip

Step 13: Create the Server Start Script (DELTA)

The default control script is not included in the community edition. Create a file called startServers.sh in your runtime directory. Its contents should be as follows (replacing localhost:8088 with your server URL):

echo "Starting data server"
python ${SnapPackageDir}/snaplogic/server/server.py  --config=${SnapRuntimeDir}/config/snaplogic_server.conf &
echo "Starting management server"
python ${SnapPackageDir}/snaplogic/mgmtserver/mgmt_server.py --config=${SnapRuntimeDir}/config/snaplogic_server.conf &
echo "Starting component server"
python ${SnapPackageDir}/snaplogic/cc/component_container.py --name=cc1 --server=
http://localhost:8088 &

Step 14: Correct the Bad Import on Ubuntu 10.04 (DELTA)

If you are running with Ubuntu 10.04, you probably have package python-simplejson at version 2.0.9-1build1. SnapLogic 2.2 was certified against version 1.7.1. You will need to remove an import statement from a SnapLogic file:

gedit /home/plaird/dev/snaplogic/snaplogic/rp/streaming_json_rp.py

Edit out this line:

from simplejson.decoder import JSONScanner

Step 15: Start the Servers and View the UI (DELTA)

chmod u+x startServers.sh
./startServers.sh
firefox
http://<your_machine_name>:8081 &

Step 16: Walk through Tutorial

You can at this point start with the tutorials.

Step 17: ROAD BLOCK

Sadly, at this point I have hit a road block. The community edition of the product apparently has a bug in that it depends on an license file for a commercial version of the product (pro or core). I have filed defect 2449 – hope to get a fix shortly!

In the meantime, see my follow on post on how to install SnapLogic 2.1.0 instead. While it is an earlier version, the upside is the 2.1.0 version has a community installer.

 

Technorati Tags: ,,