Upgrade to Ubuntu 10.04 broke my Postgres – “Could not Connect to Server” in Pgadmin

I upgraded last week to Ubuntu 10.04, and just noticed today that my development environment was broken (yeah yeah, it took a week to notice, give me a hard time about not spending enough time coding). I looked into it, and found a weird problem caused by the upgrade – perhaps if you are reading this you have that problem too. My Postgres database was apparently not operating.

Postgres 8.4 Upgrade

I did notice that the Ubuntu 10.04 upgrade was taking me from Postgresql 8.3 to 8.4. I figured that should be fine. And it is, but it caused some downstream problems.

First off, put away any notion that the upgrade does an automatic migration of your databases from 8.3 to 8.4. Ubuntu is good, but not that good. You will have to manually migrate your databases yourself. I imagine you should dump all your databases before you do your upgrade, as the upgrade does remove 8.3 from /usr/lib/postgresql. Since I could rebuild my databases, I didn’t have to worry about migration – but do some research ahead of time if you do.

This was one issue – my dev databases were not migrated into 8.4. I needed to rebuild them, and recreate our dev user in 8.4. But before I could do that…

Pgadmin Could Not Connect

A major roadblock was that I could not connect to Postgres using Pgadmin, even as the ‘postgres’ user. In fact, I couldn’t seem to get network connectivity to Postgres at all. What is going on?

Server doesn't listen
The server doesn't accept connections: the connection library reports could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432?

To troubleshoot this, I made sure that the processes were running:

sudo /etc/init.d/postgresql-8.4 restart

and then made sure it stayed started:

ps –ef | grep postgres

and also checked the logs:

/var/log/postgresql$ cat postgresql-8.4-main.log

But everything seemed to be kosher. I was also able to use psql from the command line. What is going on, I still can’t connect from Pgadmin? An invocation of netstat is what solved it:

netstat -ln | grep 5432

That should have returned some lines like:

tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN    
tcp6       0      0 ::1:5432                :::*                    LISTEN    
unix  2      [ ACC ]     STREAM     LISTENING     5623     /var/run/postgresql/.s.PGSQL.5432

but it returned nothing. Eh? 5432 is the default port for postgresql, and the port I use in my Pgadmin connections. It should be there.

I then looked at the postgres server configuration in  /etc/postgresql/8.4/main/postgresql.conf and found the problem:

port = 5433

For some reason the upgrader installed 8.4 on port 5433 instead of 5432. During the upgrade it perhaps needs to run 8.3 and 8.4 at the same time, and therefore puts them on different ports. But after upgrade, shouldn’t it switch 8.4 back to the default port?

Its an easy fix:

sudo gedit postgresql.conf   (change 5433 to 5432)

sudo /etc/init.d/postgresql-8.4 stop

sudo /etc/init.d/postgresql-8.4 start

Note that I first tried just restart, but found that the port change did not take effect. I had to do a stop and start explicitly.

I am now back up and running again with Postgres 8.4. Joy.

Technorati Tags: ubuntu,postgresql

Installing SnapLogic Community Edition 2.1.0 on Ubuntu 8.10 in 5 minutes

If you read my last post, you would know that I hit a snag running the community edition of SnapLogic 2.2. There appears to be a hard dependency on a license check (see ticket 2449). Since I have no idea how long that will take to resolve, I decided to find a way around the problem.

I chose to roll back to version 2.1.0. But in doing so, I had some struggles. SnapLogic 2.1.0 is based on Python 2.5.x and related components. My Ubuntu 10.04 install uses Python 2.6.x, and the 2.5.x stuff has been removed from the Ubuntu repositories.

After trying some minor hackery to get P2.5.x on U10.04 (building stuff from source), I gave up at trying to get the dependencies resolved. Instead, I just plopped down a new Ubuntu 8.10 VM on my machine, and got things going pretty quickly.

These are my notes. By the way, the 5 minutes I reference in the title is just for the SnapLogic install. Installing and updating Ubuntu takes about an hour, depending on network speeds.

• Install 8.10
• Download a VDI and install into VirtualBox, or
• Do a bare metal install (e.g. dual boot)
• Update your OS
• 368 updates and counting as of today
• Make sure not to upgrade Python beyond 2.5.x
• Reboot
• Download SnapLogic 2.1.0
• http://sourceforge.net/projects/snaplogic/files/
• You want the linux .bin of course
• Launch the SnapLogic installer
• chmod u+x [blahblah.bin]
• It will inform you what is missing from prereqs
• Install the missing prereqs
• Launch Synaptic Package Manager, and install:
• python2.5-dev
• libmysqlclient15-dev
• libxml2-dev
• libxslt1-dev
• Rerun SnapLogic installer
• Accept all defaults
• Password is password
• Elect to start up the servers at the end
• Install Flash Plugin for Firefox
• Navigate to http://get.adobe.com/flashplayer/
• Navigate to the Designer UI
• http://[hostname]:8081/
• Open the Leads_to_Prospects Pipeline
• Select the Run tab
• Run the pipeline (this is where SL 2.2 fails for me)
• Success!

 

Technorati Tags: snaplogic,ubuntu,virtualbox

Building and Installing SnapLogic 2.2 from Source on Ubuntu 10.04

This blog post walks you through building and installing SnapLogic 2.2 on a Ubuntu 10.04 box. These instructions will lead you through building the product from source. There is also a handy installer available, but that is only for the Pro version. The Pro version has a 14 day trial, and then you’ll have to fork over money. Since I plan to work on my prototype over a longer period of time, I have decided to build the community version from source.

Note: there is currently a blocking issue that prevents this from working. Be sure to read through to the bottom of this page for details before proceeding on this yourself.

The official build and installation instructions live on the SnapLogic wiki, but I am posting here so I can provide more details as I go. Also, there are a couple of changes from the official instructions presumably due to 10.04. Since it seems only Ubuntu 9.x is supported for SnapLogic 2.2, updating the official wiki would probably not be welcome. Items marked with DELTA below are different relative to the official docs.

Caveat: I am running 10.04, but my laptop has been a Ubuntu machine since 9.04. I did an in place upgrade to 9.10 and then 10.04, and have installed a lot of things along the way. Your mileage may therefore vary with these instructions.

Step 1: Install Prerequisite Packages (DELTA)

Execute the following command in a terminal window:

sudo apt-get install python-mysqldb python-pysqlite2 python-cherrypy3 python-feedparser python-simplejson python-lxml python-beautifulsoup python-paste python-genshi

Note: make sure to specificy python-cherrypy3, otherwise you get 2.3.0-3, which will not work with SnapLogic 2.2.

Step 2: Download the Source

Go to the community site and download the source for both the server and the designer:

• SnapLogic Server
• SnapLogic Designer

Step 3: Extract the Source into a Directory

cd ~/dev
mkdir snaplogic
cd snaplogic/
unzip /home/plaird/Desktop/downloads/snaplogic-src-2.2.0.zip 

Step 4: Create a Runtime Directory

cd ~/dev/snaplogic/
mkdir runtime

Step 5: Setup your Shell

Ultimately, you want to define these in ~/.bashrc, but we will do it inline for now.

export SnapPackageDir=/home/plaird/dev/snaplogic
export SnapRuntimeDir=/home/plaird/dev/snaplogic/runtime
export PYTHONPATH=$PYTHONPATH:${SnapPackageDir}

Step 6: Build out your Runtime Directories

mkdir ${SnapRuntimeDir}
mkdir ${SnapRuntimeDir}/bin
mkdir ${SnapRuntimeDir}/repository
mkdir ${SnapRuntimeDir}/designer
mkdir ${SnapRuntimeDir}/cache_dir
mkdir ${SnapRuntimeDir}/compute
mkdir ${SnapRuntimeDir}/static
mkdir ${SnapRuntimeDir}/logs
mkdir ${SnapRuntimeDir}/config

Step 7: Copy in Runtime Files (DELTA)

The official instructions inform you that you need to install icon files at this point. That no longer seems necessary, as the Designer installation in a later step now takes care of that.

However, I found that you do need to install the components and tutorial into the runtime directory.

cd ${SnapRuntimeDir}
cp -r ../snaplogic/components/ .
cp –r ../snaplogic/example/tutorial .
cd tutorial
mkdir data
mv *.csv data

Step 8: Launch SnapAdmin

cd ${SnapRuntimeDir}
python ${SnapPackageDir}/snaplogic/tools/snapadmin.py

Step 9: Create your Repository in the Runtime Directory

repository create -t sqlite /home/plaird/dev/snaplogic/runtime/repository/repository.db

Step 10: Create a Server Configuration File

Download the template configuration file from the website (link is on this page) and copy it into your ${SnapRuntimeDir}/config directory. Make the following modifications:

• Replace ${SnapRuntimeDir} with absolute path
• Change YOURHOST.EXAMPLE.COM to the fully qualified domain name of your host (three locations)
• Optionally change the ports from 8088, 8081 and 8089

Step 11: Install the Authentication Files

These files give you a starting solution for authentication:

cp ${SnapPackageDir}/snaplogic/server/auth/simple_snapaccess.conf ${SnapRuntimeDir}/config/
cp ${SnapPackageDir}/snaplogic/server/auth/simple_passwords ${SnapRuntimeDir}/config/

Step 12: Install the Designer GUI

cd ${SnapRuntimeDir}unzip /home/plaird/Desktop/downloads/snaplogic-designer-2.2.0.zip

Step 13: Create the Server Start Script (DELTA)

The default control script is not included in the community edition. Create a file called startServers.sh in your runtime directory. Its contents should be as follows (replacing localhost:8088 with your server URL):

echo "Starting data server"
python ${SnapPackageDir}/snaplogic/server/server.py  --config=${SnapRuntimeDir}/config/snaplogic_server.conf &
echo "Starting management server"
python ${SnapPackageDir}/snaplogic/mgmtserver/mgmt_server.py --config=${SnapRuntimeDir}/config/snaplogic_server.conf &
echo "Starting component server"
python ${SnapPackageDir}/snaplogic/cc/component_container.py --name=cc1 --server=http://localhost:8088 &

Step 14: Correct the Bad Import on Ubuntu 10.04 (DELTA)

If you are running with Ubuntu 10.04, you probably have package python-simplejson at version 2.0.9-1build1. SnapLogic 2.2 was certified against version 1.7.1. You will need to remove an import statement from a SnapLogic file:

gedit /home/plaird/dev/snaplogic/snaplogic/rp/streaming_json_rp.py

Edit out this line:

from simplejson.decoder import JSONScanner

Step 15: Start the Servers and View the UI (DELTA)

chmod u+x startServers.sh
./startServers.sh
firefox http://<your_machine_name>:8081 &

Step 16: Walk through Tutorial

You can at this point start with the tutorials.

Step 17: ROAD BLOCK

Sadly, at this point I have hit a road block. The community edition of the product apparently has a bug in that it depends on an license file for a commercial version of the product (pro or core). I have filed defect 2449 – hope to get a fix shortly!

In the meantime, see my follow on post on how to install SnapLogic 2.1.0 instead. While it is an earlier version, the upside is the 2.1.0 version has a community installer.

 

Technorati Tags: snaplogic,ubuntu,rest

Solution: Mouse disappears when using Skype Share Desktop in Windows VirtualBox VM on Ubuntu Linux Host

That’s a mouthful. But I have an easy solution to this annoying problem. It happens when I run Skype in a Windows VM, and then opt to share desktop. The mouse will disappear, which makes for a tough time navigating around.

This is my configuration:

• Host OS: Ubuntu Linux 9.10 (with all updates)
• Guest OS: Windows XP SP3 (with all updates)
• Guest additions have been installed
• Hypervisor: VirtualBox 3.1.4
• Skype: Windows version 4.2.0.155
• Hardware: Dell Latitude E6400

Solution

Is simple. When I installed

guest additions, it enabled a feature called “Mouse Integration”. It is enabled by default. However, with Skype it causes this mouse problem.

For the duration of your Skype call, simply disable Mouse Integration and this problem goes away. Do this from the VirtualBox menu bar for your VM:

• Machine –> Disable Mouse Integration

Technorati Tags: linux,ubuntu,virtualbox

Solutions to VPN Problems with Ubuntu Linux – vpnc, network-manager, “terminated by dead peer”, WICD

I have been successfully using the Linux vpnc client to connect into the corporate network for months. However, I recently gained access to a more secure VPN network at the company, and I had nothing but trouble. This blog post explains the solution for that, as well as some other random network issues I was having.

Will this blog post solve MY problem?

I don’t know if I can help you, but here are the symptoms I was having on my machine:

• I could establish VPN, but something truncated my connection every 5 minutes
• After establishing VPN from certain locations, I could not ssh to my target instance; it never resolved the IP address
• As I moved around the building (particularly up and down stairs), I had trouble with re-establishing network
• I found the network hanging every 2 minutes for a few seconds
  

Troubleshooting

The key to solving these problems is to look into /var/log/syslog. Do you see either of these entries?

"connection terminated by dead peer detection”

This will be noted when something terminates your VPN connection"

“Roamed from BSSID 00:0F:61:1B:A4:32 (ESSID) to (none) ((none))”, followed by “Roamed from BSSID (none) ((none)) to 00:0F:61:1B:A4:32 (ESSID)”

This will happen every 2 minutes, and will be associated with a pause in your network traffic.

network-manager was the problem for me

I believe the source of all the trouble was the second error noted above. Based on Google searches, this indicates that the network card is scanning for new wireless networks every 2 minutes. During this time (which lasts 6 seconds) no other network traffic can be sent. It makes sense that this would cause issues with a picky VPN.

See this Ubuntu bug report for a full explanation (and confusion as to the proper fix) related to this nasty problem. Some folks indicated the network-manager component to be the source of the problem. network-manager is the default network manager for GNOME Linux distros (like Ubuntu). The problem reported in the bug report does not affect everybody – only some network cards. I run on a Dell Latitude E6400 with an Dell 1510 network card.

Sadly, the network-manager folks didn’t agree that it was their problem. So I had to look elsewhere for a solution.

WICD was the solution for me

Following guidance from others posting on the bug report, I uninstalled network-manager and installed an alterative called WICD (pronounced “wicked”). I have been running WICD for a week, and I am sold. All of the issues that I reported are now gone.

WICD has much of the same functionality of network-manager, but is a bit simpler which perhaps is its strength. Here is a screen shot of the WICD applet:

If you wish to go this route (highly recommended!), issue the following command. It will install wicd and remove network-manager:

sudo aptitude install wicd

Note that wicd doesn't currently (as of 1.6.1) have menu driven support for VPNs. It is supposed to come with version 2.0 though, so keep checking in on this. It simply means that to start a VPN connection, you will have to do it from the command line instead of from the applet. If you are using vpnc, that means you have to do this from a terminal:

sudo vpnc

WICD Pro Tip

One thing that should be the default in WICD, but isn’t, is to always show the wired network connection in the list. Choose this by checking the box in the UI here:

• Preferences->General->Always show wired interface

Technorati Tags: linux,ubuntu,networking,wireless,VPN

Installing Android-x86 as a guest on Ubuntu on a Dell E6400 using VirtualBox

Its funny how rabbit trails start.

After getting the kids to bed, I browsed my Twitter feed. I saw someone tweet about 10+ free, fast-booting Linux distros that aren't Chrome OS. The list includes Puppy and Jolicloud, both distros I have been meaning to try.

I then started the walk down the rabbit trail. I proceeded to install Puppy as a VirtualBox guest to see just how fast it would boot. I actually am not that impressed with the results, but that’s another story.

As I was playing with Puppy, I remembered that I wanted to play with Android, not only because its a popular topic on the web but also because we have built an Android app at work. Moving further along that trail…

I looked around for an Android ISO that I could load into VirtualBox. I found what I was looking for in LiveAndroid. It is a port for Android on x86 platforms. I downloaded the ISO image, and built up a virtual machine in VirtualBox. I won’t provide any more details of this process because in the end it didn’t work. I couldn’t get the Android UI to recognize the keyboard or mouse, and without inputs you aren’t doing much. It was weird – I could drop into the Linux prompt in the VM and the keyboard worked fine. I poked around a bit, and then found mention in the forums that it is likely an abandoned project, and linked to a different Android port. Continuing down that trail…

I moved on to the active port, called Android-x86. It looked promising, so I downloaded the ISO. I followed the instructions and created an Andriod VM for VirtualBox. In short, I got it running and it works pretty well. I even got the Tendril Android app running.

I just wanted to post to document a few things I had to do:

• Use a good x86 Android distro – Android-x86 works for me
• The VM How-To page doesn’t explicitly mention that you can uncheck the Boot Hard Disk option when creating the VirtualBox VM. If you do this, you will select your ISO image when you boot the image for the first time.
• Instead of worrying about the advanced persistence steps, consider just using the Snapshot feature of VirtualBox to capture the state of your runnning VM. Restore it later to regain state.
• If your graphics are all screwy on boot of the VM, try the VESA menu option in GRUB
• You will need to register on AndAppStore before using the App Store in the image.
• To emulate an SD card (necessary for installing dev Apps), use the simple RAM based emulator. At the GRUB menu, hit TAB, and then on the command line make an entry for “SDCARD=premount”

The other part of it was figuring out the user experience/key bindings. Some hints:

• I couldn’t figure out how to go with just my laptop keyboard. I have a Windows friendly USB keyboard with additional keys that works better.
• Windows key (lower left) – is used to bring you back to the desktop
• Menu key (lower right) – is used to show options within an app
• Arrow keys, enter key – do what you think
• Esc – back button
• After you download an app in the app store, you have to install it. To do this, you drag the upper menu bar down with the mouse+mouse click. This seems really odd, but perhaps makes sense on a mobile device?

Enjoy your free Android!

Technorati Tags: android,virtualbox,ubuntu