Spring Data Infinite Loop in HashMap in PreferredConstructor and CustomConversions

Spring Data Infinite Loops in Production

This blog post is related to an issue we have started seeing in production at the company where I work. I am trying to figure out where to take this next. I have posted this report to the Spring Data forum here:

• http://forum.spring.io/forum/spring-projects/data/750072-spring-data-infinite-loop-in-hashmap-in-preferredconstructor-and-customconversions

It started happening a few weeks ago. We would see a box CPU suddenly go to 100% indefinitely until the app was redeployed. Fortunately, we did take thread dumps, and in one case a heap dump. We were able to see that there is a recurring infinite loop in two places in Spring Data.

This has happened a total of 3 times now. Given the number of nodes we run in PROD, INT and QA (some number > 30), and the length of time we have been in production on this stack, we would characterize it as a rare occurrence. But it worries us.

One could ask "what changed in the application to make this start happening?". Ack. We are in our busy season right now, so a lot of commits went into our application for the weeks prior to the first occurrence. And given the rare nature of the problem, it would be difficult to really identify even the time range of commits that we would need to evaluate.

Root Cause

In all 3 cases, the infinite loop is coming from concurrent updates to the thread-unsafe HashMap class. We see it happening within two Spring Data classes:

• org.springframework.data.mapping.PreferredConstructor
• org.springframework.data.mongodb.core.convert.CustomConversions

I could explain why HashMap is getting stuck in infinite loops, but several others have already done a great job doing it, so I refer you there:

• http://mailinator.blogspot.dk/2009/06/beautiful-race-condition.html
• https://code.google.com/p/javasimon/issues/detail?id=115

Why us, not everyone else using Spring Data?

Looking at the PreferredConstructor and CustomConversions source code reveals that indeed they are indeed using raw HashMap instead of ConcurrentHashMap. Does that mean those classes do not expect to be accessed by more than one thread? 

But why are we the first to see this? I see no other complaints about for this issue with respect to Spring Data on the web. Three possibilities:

• We are using Spring Data incorrectly
• This a problem in Spring Data, and
• Other people have hit this, but no one has traced it to Spring Data yet
• We are very unlucky to be the first/only people to have hit it

As for point one, we have reviewed our application and Spring Data docs. We have looked and looked, and cannot see anything amiss in our implementation.

More Often, Occurs Soon After a Deployment

In 2 cases out of 3, this problem happened soon after a redeploy of the application. That suggests that this problem happens during initialization/first run of a code path. But we can't explain the 3rd outage though.

Stack Trace Snippet for PreferredConstructor

This stack trace was taken from a thread dump during one event. A more complete thread dump can be found here: 

• http://pastebin.com/c5erB5XF

"http-bio-8080-exec-235" daemon prio=10 tid=0x00007fb19c0a9800 nid=0x5337 runnable [0x00007fb228953000]

   java.lang.Thread.State: RUNNABLE

at java.util.HashMap.getEntry(HashMap.java:469)

at java.util.HashMap.get(HashMap.java:421)

at org.springframework.data.mapping.PreferredConstructor.isConstructorParameter(PreferredConstructor.java:120)

at org.springframework.data.mapping.model.BasicPersistentEntity.isConstructorArgument(BasicPersistentEntity.java:98)

at org.springframework.data.mongodb.core.convert.MappingMongoConverter$1.doWithPersistentProperty(MappingMongoConverter.java:252)

at org.springframework.data.mongodb.core.convert.MappingMongoConverter$1.doWithPersistentProperty(MappingMongoConverter.java:249)

at org.springframework.data.mapping.model.BasicPersistentEntity.doWithProperties(BasicPersistentEntity.java:257)

at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:249)

at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:223)

at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:187)

at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:183)

at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:77)

at org.springframework.data.mongodb.repository.support.SpringDataMongodbQuery$1.apply(SpringDataMongodbQuery.java:55)

at org.springframework.data.mongodb.repository.support.SpringDataMongodbQuery$1.apply(SpringDataMongodbQuery.java:53)

at com.mysema.query.mongodb.MongodbQuery.list(MongodbQuery.java:270)

at org.springframework.data.mongodb.repository.support.QueryDslMongoRepository.findAll(QueryDslMongoRepository.java:114)

at sun.reflect.GeneratedMethodAccessor356.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.executeMethodOn(RepositoryFactorySupport.java:344)

at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:329)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)

at com.sun.proxy.$Proxy161.findAll(Unknown Source)

at com.silverchalice.scamp.bizniz.v3.media.MediaServiceImpl.findByFilter(MediaServiceImpl.java:293)

Stack Trace Snippet for CustomConversions

This stack trace was taken from a thread dump during one event. A more complete thread dump can be found here:

• http://pastebin.com/gYMGPEkL

"http-bio-8080-exec-75" daemon prio=10 tid=0x00007f3688018000 nid=0x6c2e runnable [0x00007f376e9ef000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.getEntry(HashMap.java:469)
at java.util.HashMap.get(HashMap.java:421)
at org.springframework.data.mongodb.core.convert.CustomConversions.getCustomReadTarget(CustomConversions.java:317)
at org.springframework.data.mongodb.core.convert.CustomConversions.hasCustomReadTarget(CustomConversions.java:281)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:200)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.readCollectionOrArray(MappingMongoConverter.java:791)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.readValue(MappingMongoConverter.java:1048)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.access$100(MappingMongoConverter.java:77)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter$MongoDbPropertyValueProvider.getPropertyValue(MappingMongoConverter.java:999)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.getValueInternal(MappingMongoConverter.java:755)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter$1.doWithPersistentProperty(MappingMongoConverter.java:256)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter$1.doWithPersistentProperty(MappingMongoConverter.java:249)
at org.springframework.data.mapping.model.BasicPersistentEntity.doWithProperties(BasicPersistentEntity.java:257)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:249)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:223)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.readValue(MappingMongoConverter.java:1050)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.access$100(MappingMongoConverter.java:77)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter$MongoDbPropertyValueProvider.getPropertyValue(MappingMongoConverter.java:999)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.getValueInternal(MappingMongoConverter.java:755)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter$1.doWithPersistentProperty(MappingMongoConverter.java:256)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter$1.doWithPersistentProperty(MappingMongoConverter.java:249)
at org.springframework.data.mapping.model.BasicPersistentEntity.doWithProperties(BasicPersistentEntity.java:257)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:249)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:223)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.readCollectionOrArray(MappingMongoConverter.java:791)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.readValue(MappingMongoConverter.java:1048)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.access$100(MappingMongoConverter.java:77)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter$MongoDbPropertyValueProvider.getPropertyValue(MappingMongoConverter.java:999)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.getValueInternal(MappingMongoConverter.java:755)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter$1.doWithPersistentProperty(MappingMongoConverter.java:256)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter$1.doWithPersistentProperty(MappingMongoConverter.java:249)
at org.springframework.data.mapping.model.BasicPersistentEntity.doWithProperties(BasicPersistentEntity.java:257)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:249)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:223)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:187)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:183)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:77)
at org.springframework.data.mongodb.repository.support.SpringDataMongodbQuery$1.apply(SpringDataMongodbQuery.java:55)
at org.springframework.data.mongodb.repository.support.SpringDataMongodbQuery$1.apply(SpringDataMongodbQuery.java:53)
at com.mysema.query.mongodb.MongodbQuery.list(MongodbQuery.java:270)
at org.springframework.data.mongodb.repository.support.QueryDslMongoRepository.findAll(QueryDslMongoRepository.java:114)
at sun.reflect.GeneratedMethodAccessor470.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.executeMethodOn(RepositoryFactorySupport.java:344)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:329)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy124.findAll(Unknown Source)
at com.silverchalice.scamp.bizniz.v3.events.EventServiceImpl.findByLiveStatsAssetId(EventServiceImpl.java:141)

Application Details

Here is a bullet list of details about our application:

• Web application
• We have ~40 Repository definitions (PagingAndSortingRepository)
• We use Spring Java Config
• We use QueryDSL for querying in almost all cases
• Our app is very read heavy, something like 99.9% reads
• Our app ranges from 100 to 10,000 requests per minute on a box. These events took place during times of not-so-high load (say, a few hundred requests per minute)

Stack Details

Here is a bullet list of details about our environment:

• Java webapp hosted as an Amazon EC2 Elasticbeanstalk app
• JVM: java version “1.7.0_51"   Java(TM) SE Runtime Environment (build 1.7.0_51-b13)   Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
• OS:  Amazon Linux 3.2.12-3.2.4.amzn1.x86_64
• MongoDB:  2.4.6 (hosted on another box)

POM

Here are some of the managed dependencies from our POM:

0.2.8

2.2.2

14.0.1

1.8

3.2.1

1.1

1.7

2.1

3.2.4

4.2.5

1.7.1

1.5

1.7.2

3.2.3.RELEASE        1.6.2.RELEASE

1.4.2.RELEASE

1.3.2.RELEASE

3.1.4.RELEASE

2.11.3

1.0.13

How to Replace Volvo XC70 H11 Headlight Bulb 2004 2005 2006 2007 2008 SOLVED

Welcome, you are about to embark on a difficult journey.   :)

You probably just got the indicator on your XC70 dash that says you have a burned out headlight. "No problem, I'll just pop a new one in" you say to yourself, as I did when just faced with this. This is where it gets weird.

Believe it or not, Volvo engineers didn't really provide space for this when they designed the engine compartment layout. Depending on your year of Volvo XC70, you will perhaps find yourself cursing when you open the hood to size up what should be an easy repair.

Volvo Headlight Engineer

"What? I have to get my hands in there?" Since you landed on this page, I assume you have just uttered that quote and then ran to Google to help you figure out what to do next. This blog post will hopefully help you on the next part of journey.

My Story - a 2006 Volvo XC70 with H11 Lights

First: my story.

I owned a 2002 Volvo XC70 until a few weeks ago when someone totaled it by ramming me in a parking lot. Sad. No worries, I just bought a replacement 2006 XC70.

During the first week in my 2006 the driver side low beam went out, but I wasn't worried. Changing the bulbs (H7 type) in my 2002 was not a big deal and I expected the same for my 2006. But then I uttered the quote above. After Googling around to find the easy way to replace the bulb, and finding none, I gave up on the driver side low beam. I took it to my neighborhood shop and watched the technician there replace it.

A few days later when the passenger low beam went out, I decided to replace it myself. The notes below relate to that project as I figured out how to do this myself.

The Main Issues with XC70 Headlight Bulb Replacement

First issue: you have probably already discovered that access to the headlights is very much obstructed by other auto parts. I can't tell you which model years are affected - I just know that my 2006 is definitely this way, while my old 2002 was not.

Second issue: because of issue #1 you can't look into the headlight ports from inside the engine compartment. Which means you can't visually see how to take out the old bulb. Your hand will have to operate blindly.

Third issue: What type of bulb do you have? Hard to know due to issue #2. You may have already read about some posts where people are talking about Bi-Xenon headlights. Do you have those, the H11 standard halogens, or the older H7 halogens? Bi-Xenon costs about $130 per bulb, apparently. H11's and H7's around $15. Its a big deal. I was afraid I just bought into a $130 repair. Fortunately I later learned that I am a H11 (also a Virgo, but that's irrelevant).

Fourth issue: once you determine your bulb type, how do you blindly extract the old bulb?

Fifth issue: once the old bulb is out, how to you put the new one back in? Blindly.

I could stop writing here, but then I would be a jerk. I have just told you all the problems, but none of the answers. OK, we are now getting to the good part. The answers.

Issue 1 : Obstructed Access

The first issue is you cannot easily access the headlight ports from inside the engine compartment. The driver side is worse (although some posters say that passenger side is worse, I don't get that). It may depend on your model year, or options. Dunno.

As you Google around, some folks recommend pulling out the air filter assembly (driver side) or ECU (passenger side). I didn't see a super easy way to do that. The factory instructions apparently tell the technician to remove the front bumper (!) to get access to the headlights. Don't do that.

My answer to the obstruction issue is "just deal with it". For the driver side, I saw the technician do it at the shop by working his skinny arms around the obstructions. I did the passenger side without removing the ECU box. Though I did take the cover off the ECU to make myself feel like I tried; it didn't actually help me at all.

Issue 2 : Can't See Into the Headlight to Know How to Take out the Bulb

The rest of this blog post will help you deal with this issue. I will tell you how to do this without being able to see into the headlight.

Issue 3: What Type of Bulb Do You Have? H11? Bi-Xenon?

I am not an expert on this at all. I tried searching Google Images to help me sort out what I had. You may need to do the auto shop the first time to figure this out. That's how I resolved the question. If it helps, this is my H11 headlight picture below:

Volvo XC70 H11 Headlight

My H11 has a shroud around it with a gray cap cylinder. There is also a small light bulb (parking light?) several inches away from the main bulb.

Issue 4: How to Remove the Old H11 Bulb?

From here on out, my instructions are specific to H11 bulbs. Maybe it is of some use to Bi-Xenon folks, but maybe not that much.

You need to remove the rubber cover (about 5" diameter) that covers the low beam port inside the engine compartment. This is the easiest part of the whole ordeal. If you struggle with this, please slowly back away from the car, reach for those car keys, and drive your helpless soul to the nearest auto shop. The rest is much harder.

The next step is to somehow reach your hand into the headlight port (easier said than done!). Its going to be a tight fit, especially on the driver side. If you can't do it, see references above to removing the air filter or ECU to make this easier. If you have big arms, that's probably your only option.

The next step was really hard for me because I didn't know what to do with my hand once I got it into the port. The answer is actually pretty easy. You need to twist the old H11 bulb in the socket so that the tabs around the metal base unlock from the socket. See here:

See those three tabs sticking out of the base? They are what locks the bulb into the socket. When you turn the bulb, it will somewhat easily unlock and you can pull the bulb out of the headlight port.

Wait! Its even easier than that. If you look into the headlight from the front (in front of the car, like a deer would) while you are twisting it you can actually see the tabs turning in the socket so you can see when they are disengaged. I don't remember if you turn clockwise or counter, but it shouldn't be too hard because you can watch them turn.

At this point, you should have pulled the bulb assembly out of the port. The wire harness is still attached at this point.

To unlock the old bulb from the harness, you just need to disengage the plastic tabs on either side of the H11 bulb connection. It sounds hard, but once you are in there it is obvious. Use a flat head screwdriver to gently pry open the tabs if you must. After that, you should have a dangling wire harness and one old H11 bulb in the trash can.

Issue 5: Putting in the New Bulb

This is more or less self explanatory, now that you know how the thing is supposed to fit together. The first wrinkle I will mention specifically is you need to have a glove on, preferably something thin like a latex. The reason is you don't want to soil the new bulb with your skin grease. When you are putting in the new bulb, you likely get your hands on the glass bulb, which apparently is not a good thing.

I am tempted to write "and the rest is left as an exercise for the reader". Once you pull the old bulb out, you should have a really good idea on how to reverse the process. But I do need to tell you about two more things:

You need to be looking into the headlight (like a deer, remember) to make sure the new bulb finds itself inside that gray shroud thing inside the headlight. The first time I did it, I had the bulb in cockeyed and the bulb was outside the shroud.

Second, look into the headlight to help you engage the tabs and make sure you are turning it correctly. You want to double check that the tabs are engaged.

Finally, don't forget to replace the rubber cover over the headlight port inside the engine compartment.

DONE!

If you found this post useful (or to the contrary) please add a comment below. Thanks! Please include your model year so we can track what models are difficult.

X10 – Noise and Phase Coupling Issues – Part 7

This is part 7 of my X10 blog series.

Parts 1 through 6 of this series has documented my initial installation and configuration of my X10 system and devices. I have been covering these topics and issues in the order in which I have encountered them. For anyone wishing to follow this same path in their own implementation, this order is important.

First, the Good News

In the order of events, at this point you should have a working system with a couple of modules if your experience is like mine. This is good.

Now, the Bad News

Likewise, if your experience is like mine, now is about the time you need to build an awareness of two systemic issues with X10 systems. Since I am still learning myself, I cannot say for certain that I have been afflicted by them. But the evidence is strong.

First, try to understand the issues at a high level:

• Phase Coupling
• Noise

In a nutshell, both contribute to signal strength issues. When in play, both will unpredictably affect how well your modules can hear the controller signals. Either it will be an utter failure (which, I would prefer), or will cause random (and frustrating) failures.

Wimp out for now

Now that you have a high level awareness under your belt, I would recommend strategies to avoid them until we progress (you and me both!) further in our implementation before we eventually diagnose and implement solutions for them.

Strategies I am currently using to keep things simple:

• Pay attention to what household appliances are operating while you test. Vacuum cleaner? Dryer? Try to test during “quiet” electrical periods.
• Locate your module under test in the same room as your controller while testing. I know this isn’t realistic for actual deployment, but its good to eliminate noise as much as possible.
• Rinse and repeat. When something works, try it several more times before declaring success. Wait for 15 minutes, and then try again.
• Be anal in your methodology. Its no fun to waste minutes pointlessly clicking buttons in ActiveHomePro only to find out a light bulb is burned out. Test your device prior to integrating with a module.
• Steer clear of advanced use cases, such as CFL or 220V appliances.

Trouble

So that we can compare notes, here are the issues I have hit. I haven’t done a full troubleshooting exercise yet, so these are unconfirmed diagnosis.

As a basis of understanding, my “test room” is my home office in my basement. My controller, ActiveHomePro computer, and modules are first deployed here and tested. I have no issues currently getting any module to work inside the room. Outside of the room….

• Phase Coupling Issue?
• When I moved a SocketRocket to an upstairs lamp, it stopped responding completely.
• Noise Issue?
• When I moved the SocketRocket to a light socket outside of the test room in the basement, it sometimes works, sometimes not.

 

Technorati Tags: x10

X10 – Compact Fluorescents (CFLs) Support and X10 Modules – Part 6

This is part 6 of my X10 blog series.

I have already covered a couple of lighting topics in my X10 series. In my last post, I talked through the steps of setting up an X10 SocketRocket using ActiveHomePro. In another previous post, I discussed how to setup up a Lamp Module. But I haven’t yet specifically talked about CFLs and X10, and how they play together (and don’t).

The key thing is that X10 supports CFLs if you have the right gear. For me, its been a bit of a research project to figure out what I need, and also what I might be able to get away with.

Why CFLs are Important for X10 Applications

As I started to think about what lights I wanted to automate with my X10 system, I quickly realized that the lights needing automation were these types of lights:

• Security/outdoor lights that need to be on at night
• Indoor hallway lights that I want to conveniently turn off with one button press before going to bed
• Indoor basement lights that I want to turn off proactively in case someone left them on  

In all cases, these are lights that will be on for extended periods of time. As such, they are ideal candidates for CFLs to save energy. And I think my use cases are common, so CFL support is critical to X10.

Support for CFLs with X10 Modules

At this point in my journey, I do not have definitive answers about this topic. There is the official guidance and official support statements, and then user forum postings that sometimes claim something different. I apologize for not having definitive answers, but I wanted to post this to record this as an important topic in the learning process. I hope to follow up later with more information if it differs from what is below.

In general, I assume a module does not support CFL unless explicitly stated otherwise. In fact, modules will explicitly deny support for CFLs unless they do, which is helpful.

For example, modules that explicitly do not support CFL:

• Lamp Module LM465
  • Label disclaimer: “For indoor incandescent lamps only
• Light switches WS467 and WS12A
  • X10 Wiki: “(WS467 and WS12A) should NOT be used”
• SocketRocket LM15a
  • Printed instructions disclaimer: “Do NOT use with fluorescent lamps!”
  • But, my own quick test indicates it works with my brand of CFL (Bright Effects L13T6)
  • But, forums claiming they work:
    • Topic: How Can I Control CFL Bulbs (Inexpensively!)?
    • X10 -- Socket Rocket w/ CFL

And then there are those that explicitly support CFL:

• Appliance Module AM466
  • X10.com Adware: “This simple plug-n-play module lets you control normal lamps, CFL lamps, fans, radios, coffee pots…”
  • Use instead of LM465
• Light switches WS13a and XPS3
  • X10 Wiki: “To control fluorescent fixtures use the X10 Non-dimming Decorator Wall Switch (WS13A) or X10 Pro Non-dimming Wall Switch (XPS3)”
• Socket module
  • There does not appear to be an inline-socket solution, probably due to the 2 wire nature of a socket (see next section)

Engineering Explanation

The X10 wiki provides a good explanation of the reason why CFL support with X10 is an issue:

“The main reason why standard X10 wall switch modules cannot be used to control a fluorescent fixture is because it is a two-wire device and is therefore wired in series with the load. The control signals which are transmitted onto your house wiring have to pass through the incandescent lamp (load). However, a fluorescent light contains a ballast which acts as a block to the X10 signals and makes it impossible to control the switch remotely. Also, this ballast is an inductive load (a coil), and controlling such a load could permanently damage the wall switch module. This is why the standard X10 wall switches (WS467 and WS12A) should NOT be used to control any kind of appliance (such as a ceiling fan).”

Conclusion

The bottom line is that CFLs can cause issues with any two wire module, but may not always. I suppose it is up to the homeowner to decide whether to use the two wire modules and see what happens.

Technorati Tags: x10

X10 Programming a SocketRocket using ActiveHomePro and Controller CM15a Remote Chime – Part 5

This is part 5 of my X10 blog series.

In the previous entry, I explained how I blew up an old radio by using a Lamp Module instead of an Appliance Module (oops). In this entry, I will explain how to easily program a SocketRocket. The details unfortunately were not forthcoming in its documentation.

The Theory

This is how the SocketRocket picks up new programming:

1. Find a lamp that you wish to control over X10
2. Switch it on so that the lightbulb is lit, just to test it works
3. Switch off the light
4. Unscrew the lightbulb
5. Screw in the SocketRocket, and then the lightbulb into the SR
6. Switch on the lamp. The lightbulb will probably not be lit
7. For the next 30 seconds the SocketRocket is waiting for its signal to program itself
8. After 30 seconds, the SR will either be programmed or left in this unknown state

Item 7 is the key part. What is the SR waiting for? It is waiting to see the same X10 command 3 times in a row. If it sees that, it will assume it is targeted at the SocketRocket and will persist that address. That address will persist even if there is a power failure.

The Practice – How to do it with Controller CM15a and ActiveHomePro

The printed instructions didn’t explain how to program the device with the CM15a. I had to google it to find the best way:

• Open ActiveHomePro
• Click the Other link in the left pane
• Drag a “Remote Chime” to whatever room you want
• Set the address of the Chime to be whatever you wish the lamp to listen on (e.g. A14)
• Power on the lamp with SocketRocket
• In the next 30 seconds, hit the Remote Chime control 3 times in ActiveHomePro
• The lightbulb should now be lit
• Now add a SocketRocket Module to the same room.
• Change the new SR module address to match the Remote Chime
• Delete the Remote Chime
• Manually power off the lamp with the SocketRocket, wait 10 seconds
• Manually power on the lamp
• Test again to make sure the settings stuck

As with any X10 device, this procedure may fail if there are connectivity issues between the controller and module. Follow my advice in this future post if you are at this point and cannot get your gear working.

 

Technorati Tags: x10

X10 Do Not Plug an Appliance into a Lamp Module – Part 4

This is part 4 of my X10 blog series.

This is an immediate follow up to my previous post about getting my first X10 up and running: X10 Installing My First Device – Part 3

My Radio as a Lamp

At the end of that post, I proudly declared success in getting a device wired up via X10 so that I could do home automation. I plugged an old radio of mine into a Lamp Module (LM465) and was happy to see it work. Not that I am not planning to actually automate my old radio, I just needed a device that could be used for testing.

And it is probably fortunate that I did, because I learned a very important lesson.

On the bottom of the Lamp Module, it says clearly:

• Lamps Only! No Appliances!!!

Like all engineers, I tend to ignore those types of warnings if I think I understand the core issue behind it and feel like it does not apply. I thought it was worried about power draw. I figured it was making sure someone wouldn’t plug in a 1500W heater into this small 300W relay.

My 12W radio, no problem.

The Radio is dead. I repeat, the Radio is Dead.

I don’t think its just about the power. About 10 minutes after plugging in my 15 year old trusty radio and hearing it play music, it made a quick dying gasp and then went silent. I can’t revive it by any means, even now an hour after the event. Its dead, dead. Trash bin dead.

OK, maybe that warning is something I should have paid attention to. Perhaps its not JUST about the power. After playing with the Lamp Module some more with actual lamps (genius!) I think there may be something else behind that warning.

The Lamp Module LM 465 has a dimmer function on it. I wonder if something in that circuitry caused the radio to freak out. I don’t know why it would after 10 minutes (on startup, ok), but what else could it be? It could be just some crazy coincidence that the radio died now, but after 15 years of solid service its hard to believe.

Cheap Lesson Learned

The old radio was worth about $2 if I tried to sell at garage sale. Not a loss. But now I think I will adhere to the Lamps Only warning sticker before I try plugging in something of more value. I will stick to the Appliance Modules AM466 for now on.

X10 Installing My First Device – Part 3

This is part 3 of my X10 blog series.

In Part 2, I talked about installing ActiveHomePro and the CM15a controller. In this part, I will talk about getting my first device running over X10.

Step 1: Pick an X10 Lamp Module LM465

For your first device you could choose any of your modules, but the Lamp Module seems to be the obvious choice. Even ActiveHomePro (AHP) guides you to pick the LM465 as the first one to try.

Step 2: Find a Guinea Pig LM465 Device

The next step in testing out your X10 system is to find a good guinea pig device. It should have the following characteristics:

• Two prong power form factor
• 300W max consumption
• Visually obvious if it turns on/off
• Probably should be a lamp (see later post why!)
• If it is a lamp, must not have a CFL bulb in it (see later post)

I observed the first three, and ignored the fourth (to my detriment). For my purposes, an old radio was the easiest thing to muster and so I pressed it into service.

Step 3: Find a Suitable Test Outlet

Not only do you need a device, you also need an outlet;

• Plugged into an outlet in same room as you and CM15a
• Is powered on
  • Circuit breaker is not tripped
  • If wall switched, switch is on

Step 4: Make sure device is Switched On and outlet is Powered On

Once you have your device, make sure it is switched on by plugging it directly into the chosen outlet. This is obvious, but I can imagine someone spending time wrestling with X10 only to find that the device is switched off or the outlet is powered off.

Step 5: Plug device into Lamp Module LM465

Now, plug the LM465 into the wall, and the device into the LM465. The device is likely powered off at this moment. Also, the LM465 should be set to A1 on its dials.

Step 6: Try to Power Device On/Off via ActiveHomePro

Go back to AHP. By default, there is a room called My Room and it has a lamp named My Lamp configured as A1. You should now be able to power the lamp on and off by clicking on the switch in AHP. Did it work? It did for me.

Step 7: Enjoy Success (or not)

While I have done this task a million times with ZigBee gear, this was my first X10 device. Glad to see it actually work!

Not working for you? Follow my advice in this future post for some ideas on how to simplify your environment to get some success early.

Footnote: Switched vs. Powered

In my notes above, you will notice I used two words to talk about the state of the device.

• Switched on/off – means how the device was configured to operate using the device’s controls. For a lamp, this is the switch located on the lamp. For an outlet attached to a wall switch, it means that the wall switch is on/off.
• Powered on/off – means whether the device was receiving power. When plugged directly into an active outlet, it is always powered on. When plugged into an X10 module, it is powered on when the module has been commanded to turn the device on.

X10 Installing ActiveHomePro on Ubuntu Linux – Part 2

This is part 2 of my X10 blog series.

In Part 1, I talked about the X10 kit I purchased. In this part, I will talk about getting the software up and running.

Windows VirtualBox Virtual Machine

The title is a teaser if you are a Linux user. No, I didn’t actually try to use Wine to run ActiveHomePro natively on Linux. I can’t tell you if that works (it probably does?). I have a Windows VM for VirtualBox that I use for all Windows stuff, as I did for ActiveHomePro.

Installation

After you purchase your kit, they send you a link for the software download and the license key. Use the link to download the software to Windows, and use the license key at startup to register your wares.

You do need to have your CM15a plugged in via USB (and plugged into the wall) during installation. Apparently it guides the installer to lay down the right device driver. I didn’t test without it, and maybe it just works anyway?

Note: for VirtualBox users, make sure you allow the VM to access the X10 USB device:

• Devices –> USB –> X10 USB Interface

Installation Trouble

I hit my first hiccup after getting into the ActiveHomePro (AHP) UI. Everything seemed ok, but I tried the Intro Wizard:

• Help –> Run Intro Wizard…

and it told me that it could not communicate with the CM15a. I opened the System Device Manager

• Windows Start –> Control Panel –> System –> Hardware

and noticed there was an exclamation point next to the USB device. I tried updating the driver with the one installed here:

• C:\Program Files\Common Files\X10\DriverInstall\Wireless Transceivers

but that didn’t do anything. I noticed VirtualBox’s USB support was in a weird state (USB submenu wouldn’t render anymore) so did a full reboot of the Windows VM and CM15a.

Just my luck, everything came up ok. So I am off and running.

X10 Order has Arrived – Part 1

This is part 1 of my X10 blog series.

My X10 order arrived without issue. I hated their website, and had trouble weeding through the garbage to compare the different kits they had available. It looks like they just did a full redesign of their site, so perhaps future visits won’t be as painful.

I finally decided on the following package for $99:

Part	Description	Qty
AM466	3-pin Appliance Module	2
CM15a	ActiveHome Pro Computer Interface and USB cable	1
KR19a	SlimFire Pocket Remote	1
LM15A	Screw-in Lamp Module	2
LM465	Lamp Module	2
SS13A	Slimline Switch Decorator White	2

 

The key piece is the CM15a. It is the brains behind the system. The rest just seemed like bare minimum for useful pieces to have in the kit. You can go with less.

The order arrived swiftly and had the right stuff. My only complaint about the order process is that it opens your email account to a barrage of marketing emails. The count is 14 emails in less than a month. I just opted out, so I expect that solves it. Who would want that crap?

Adventures with X10 – the Start

This is the first blog post about my installation and use of X10 home automation technology. I am hoping to discover interesting things about both X10 as a technology and home automation in general.

my background

I actually work for a company (Tendril) that engineers and sells home automation technology. However, you would never hear us call it that because the phrase home automation is normally used by vendors who focus on coordination of home devices for the purpose of comfort, convenience, and safety. Technologies such as X10 and Insteon are common in this arena.

Tendril’s home automation is instead focused on energy efficiency and energy grid optimization. Meaning, we build gear that helps consumers monitor and moderate their energy use. Also, in cases where consumers have opted into a program, utilities can make use of this automation to curtail demand during periods of critical grid load. This isn’t a blog for my employer, so I will end my explanation here - more details are on our website if you are interested.

The reason my employment is relevant is that we use a wireless technology called ZigBee to coordinate the activities of home devices. ZigBee is used heavily in the energy space, so it is the natural choice for our purposes. But from time to time other technologies are discussed, from wireless (e.g. Z-wave, wifi) to wireline (e.g. HomePlug). The firmware engineers at Tendril are well versed in these technologies, but me as an enterprise software platform engineer have been so far left out of that party.

The Journey

I decided to start trying other home automation technologies just to see how things go. Additionally, I wanted to play with traditional home automation outside of the energy space to see what I have been missing.

Which, long winded, arrives at the point of this blog post. I am embarking on such a journey with X10 equipment. I have purchased my gear, and will start posting notes as I make progress (or lack thereof). No surprises if you see a lack of activity with this hobby. I fully expect to pick this up in waves, just like everything else in my busy household.

My $1 Plastic Toy Car has a Metal Flywheel and a Plastic Gearbox

One of the greatest things about being a parent is the ability to relive your own childhood – often vicariously as you watch your own children experience life with fresh eyes. But with toys, we get to experience childhood again firsthand. I believe the electric train and R/C car toy markets exist principally to cater to the fathers rather than the children. I know my dad spent much more time setting up and working the electric train table than us kids. But this is hardly the point of this blog post.

Rather, I glimpsed inside a broken toy tonight, and was intrigued by what I could see. I was interested enough to break it open and see exactly how it was put together. It turns out there is some good mechanical engineering principles at work within even the cheapest of toys.

Yellow Tractor from the $1 Bin

In this case, it is a yellow tractor purchased at Target last year for $1. It is the kind of vehicle that you can “rev” up by running it across the table with your hand, and then release and let it move under its own power. My daughter had dismantled pieces from the top, which allowed me to peek inside.

I will provide the model number (Ankyo234041829F16819779-08/09) just in case it is a rare toy that is now worth a million dollars. How about giving me $200,000 for a busted one? As you would expect, you also find Made in China stamped to the bottom.

Inside the toy are the following components:

• Plastic frame; holes in the plastic serve as bearings for the axles
• 4 plastic wheels (inside of the fake outer tracks)
• Metal axles for the wheels, with a spline in the middle to drive the first cog in the drivetrain
• A gearbox of 3 cogs; the middle cog is actually 2 cogs mated together (small and big). Metal shafts hold the cogs in place.
• A big metal flywheel (probably lead, to provide for better inertia)

I was surprised at how much mechanical advantage was provided by the gearbox. Moving the wheels just a fraction of a turn spun the big flywheel all the way around. Nice job, Made in China guys!

Note: that little green plastic tab is only there to make noise. It rests on top of the flywheel input cog, and gives the gearbox a little clickity click sound as it rakes over the teeth.

The Gearbox

I have time boxed this blog post to less than 45 minutes, because, well, this isn’t something I should be spending a lot of time on. But I did want to understand how much mechanical advantage was being provided by the gearbox. I decided that counting teeth on each wheel was going to take too long, so I am just making some estimates here. Also, I am not rechecking my high school physics text book, so I could be doing this all wrong. Like it matters.

But I estimate the advantage to be 125:1. Meaning, every rotation of the wheel will result in 125 rotations of the flywheel. Imagine what happens when you rapidly move the car across, say 2 feet, of table top. Assuming the wheel circumference is 2 inches, that means the the wheels will spin 12 times, causing the flywheel to spin 1500 times. Holy shrapnel-if-it-disintegrates!

Once again, I am timeboxed. So this diagram is the best I can come up with to illustrate the gearbox design:

I came up with my 125:1 figure by estimating three places of mechanical advantage of 5:1. If my physics are right, we multiply those together to get 125. Sorry for wasting 3 minutes of your life if that isn’t right, hey we are all doing the best we can.

Conclusion

Is there one?

Perhaps this is interesting stuff to you, as it was to me. I didn’t think a toy from the $1 bin at Target would reveal a lesson in mechanical advantage interesting enough for me to write about. I was wrong. Sorry, Made in China guys, for underestimating you.

(elapsed time in researching and writing this post – 51 minutes)

Technorati Tags: toy,car,flywheel,timewaster

Browsing your SnapLogic SQLite Configuration Database

This blog post explains how to browse the configuration database of your SnapLogic server. I found this helpful when I was modifying some SnapLogic server code, and messed something up. I thought I had corrupted something in the database (I hadn’t), so wanted to open it up and take a look. I thought I’d share what I learned.

It turns out the SnapLogic configuration repository is really simple. There is only one table you really care about, and it has but a few columns.

Setting Up Your SQLite Browser

The configuration database for SnapLogic is a standard SQLite database. There are various tools you could use to browse it, but I like the Firefox SQLite Manager plugin. It installs into a tool you probably already use (Firefox) and does everything you need.

Steps to get running:

• Install the SQLite Manager plugin and restart Firefox
• Navigate to Tools->SQLite Manager in the Firefox menu
• A new window will popup with the manager in view
• Navigate to Database->Connect Database in the menu
• When the file browser pops up, choose your SnapLogic repository DB
• Choose [SNAPLOGIC_HOME]/repository/repository.db
• You will have to disable file extension filtering; make sure “All Files” is selected

You should now have the SnapLogic database open for browsing.

Inside the Resource Table

The table of interest is the resource table. It contains the definitions of all of your resources, including component instances and pipelines. Almost everything you do in the SnapLogic Designer UI is stored in this table.

The table has the following structure:

Column	Purpose
uri	The logical URI given to the resource in the Designer. As in: /SnapLogic/Tutorial/Exercise_5/Resources/SortProspects
guid	A nice globally unique id for the instance. For migration/backups, this is very nice to have.
gen_id	A version number. Incremented each time you click Save on the resource in the Designer.
object	The actual definition of the resource. This is a large field, and is written in a readable format (JSON?).
validated	Has this resource been validated? 0 for false, 1 for true. See caveat below.

Overall, it is pretty self explanatory as far as I can tell. Perhaps there are hidden nuances, but none that I have detected.

The only surprise was the use of the validated column. I thought that would be updated when you hit the “Validate…” button in the Designer. Not so, at least in SnapLogic 2.1.0 (beta). This column seems to be updated upon a successful run of a Preview operation.

I hope this blog post helps you in your SnapLogic development. Post a comment if so!

Technorati Tags: snaplogic,sqlite

Upgrade to Ubuntu 10.04 broke my Postgres – “Could not Connect to Server” in Pgadmin

I upgraded last week to Ubuntu 10.04, and just noticed today that my development environment was broken (yeah yeah, it took a week to notice, give me a hard time about not spending enough time coding). I looked into it, and found a weird problem caused by the upgrade – perhaps if you are reading this you have that problem too. My Postgres database was apparently not operating.

Postgres 8.4 Upgrade

I did notice that the Ubuntu 10.04 upgrade was taking me from Postgresql 8.3 to 8.4. I figured that should be fine. And it is, but it caused some downstream problems.

First off, put away any notion that the upgrade does an automatic migration of your databases from 8.3 to 8.4. Ubuntu is good, but not that good. You will have to manually migrate your databases yourself. I imagine you should dump all your databases before you do your upgrade, as the upgrade does remove 8.3 from /usr/lib/postgresql. Since I could rebuild my databases, I didn’t have to worry about migration – but do some research ahead of time if you do.

This was one issue – my dev databases were not migrated into 8.4. I needed to rebuild them, and recreate our dev user in 8.4. But before I could do that…

Pgadmin Could Not Connect

A major roadblock was that I could not connect to Postgres using Pgadmin, even as the ‘postgres’ user. In fact, I couldn’t seem to get network connectivity to Postgres at all. What is going on?

Server doesn't listen
The server doesn't accept connections: the connection library reports could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432?

To troubleshoot this, I made sure that the processes were running:

sudo /etc/init.d/postgresql-8.4 restart

and then made sure it stayed started:

ps –ef | grep postgres

and also checked the logs:

/var/log/postgresql$ cat postgresql-8.4-main.log

But everything seemed to be kosher. I was also able to use psql from the command line. What is going on, I still can’t connect from Pgadmin? An invocation of netstat is what solved it:

netstat -ln | grep 5432

That should have returned some lines like:

tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN    
tcp6       0      0 ::1:5432                :::*                    LISTEN    
unix  2      [ ACC ]     STREAM     LISTENING     5623     /var/run/postgresql/.s.PGSQL.5432

but it returned nothing. Eh? 5432 is the default port for postgresql, and the port I use in my Pgadmin connections. It should be there.

I then looked at the postgres server configuration in  /etc/postgresql/8.4/main/postgresql.conf and found the problem:

port = 5433

For some reason the upgrader installed 8.4 on port 5433 instead of 5432. During the upgrade it perhaps needs to run 8.3 and 8.4 at the same time, and therefore puts them on different ports. But after upgrade, shouldn’t it switch 8.4 back to the default port?

Its an easy fix:

sudo gedit postgresql.conf   (change 5433 to 5432)

sudo /etc/init.d/postgresql-8.4 stop

sudo /etc/init.d/postgresql-8.4 start

Note that I first tried just restart, but found that the port change did not take effect. I had to do a stop and start explicitly.

I am now back up and running again with Postgres 8.4. Joy.

Technorati Tags: ubuntu,postgresql

Installing SnapLogic Community Edition 2.1.0 on Ubuntu 8.10 in 5 minutes

If you read my last post, you would know that I hit a snag running the community edition of SnapLogic 2.2. There appears to be a hard dependency on a license check (see ticket 2449). Since I have no idea how long that will take to resolve, I decided to find a way around the problem.

I chose to roll back to version 2.1.0. But in doing so, I had some struggles. SnapLogic 2.1.0 is based on Python 2.5.x and related components. My Ubuntu 10.04 install uses Python 2.6.x, and the 2.5.x stuff has been removed from the Ubuntu repositories.

After trying some minor hackery to get P2.5.x on U10.04 (building stuff from source), I gave up at trying to get the dependencies resolved. Instead, I just plopped down a new Ubuntu 8.10 VM on my machine, and got things going pretty quickly.

These are my notes. By the way, the 5 minutes I reference in the title is just for the SnapLogic install. Installing and updating Ubuntu takes about an hour, depending on network speeds.

• Install 8.10
• Download a VDI and install into VirtualBox, or
• Do a bare metal install (e.g. dual boot)
• Update your OS
• 368 updates and counting as of today
• Make sure not to upgrade Python beyond 2.5.x
• Reboot
• Download SnapLogic 2.1.0
• http://sourceforge.net/projects/snaplogic/files/
• You want the linux .bin of course
• Launch the SnapLogic installer
• chmod u+x [blahblah.bin]
• It will inform you what is missing from prereqs
• Install the missing prereqs
• Launch Synaptic Package Manager, and install:
• python2.5-dev
• libmysqlclient15-dev
• libxml2-dev
• libxslt1-dev
• Rerun SnapLogic installer
• Accept all defaults
• Password is password
• Elect to start up the servers at the end
• Install Flash Plugin for Firefox
• Navigate to http://get.adobe.com/flashplayer/
• Navigate to the Designer UI
• http://[hostname]:8081/
• Open the Leads_to_Prospects Pipeline
• Select the Run tab
• Run the pipeline (this is where SL 2.2 fails for me)
• Success!

 

Technorati Tags: snaplogic,ubuntu,virtualbox

Building and Installing SnapLogic 2.2 from Source on Ubuntu 10.04

This blog post walks you through building and installing SnapLogic 2.2 on a Ubuntu 10.04 box. These instructions will lead you through building the product from source. There is also a handy installer available, but that is only for the Pro version. The Pro version has a 14 day trial, and then you’ll have to fork over money. Since I plan to work on my prototype over a longer period of time, I have decided to build the community version from source.

Note: there is currently a blocking issue that prevents this from working. Be sure to read through to the bottom of this page for details before proceeding on this yourself.

The official build and installation instructions live on the SnapLogic wiki, but I am posting here so I can provide more details as I go. Also, there are a couple of changes from the official instructions presumably due to 10.04. Since it seems only Ubuntu 9.x is supported for SnapLogic 2.2, updating the official wiki would probably not be welcome. Items marked with DELTA below are different relative to the official docs.

Caveat: I am running 10.04, but my laptop has been a Ubuntu machine since 9.04. I did an in place upgrade to 9.10 and then 10.04, and have installed a lot of things along the way. Your mileage may therefore vary with these instructions.

Step 1: Install Prerequisite Packages (DELTA)

Execute the following command in a terminal window:

sudo apt-get install python-mysqldb python-pysqlite2 python-cherrypy3 python-feedparser python-simplejson python-lxml python-beautifulsoup python-paste python-genshi

Note: make sure to specificy python-cherrypy3, otherwise you get 2.3.0-3, which will not work with SnapLogic 2.2.

Step 2: Download the Source

Go to the community site and download the source for both the server and the designer:

• SnapLogic Server
• SnapLogic Designer

Step 3: Extract the Source into a Directory

cd ~/dev
mkdir snaplogic
cd snaplogic/
unzip /home/plaird/Desktop/downloads/snaplogic-src-2.2.0.zip 

Step 4: Create a Runtime Directory

cd ~/dev/snaplogic/
mkdir runtime

Step 5: Setup your Shell

Ultimately, you want to define these in ~/.bashrc, but we will do it inline for now.

export SnapPackageDir=/home/plaird/dev/snaplogic
export SnapRuntimeDir=/home/plaird/dev/snaplogic/runtime
export PYTHONPATH=$PYTHONPATH:${SnapPackageDir}

Step 6: Build out your Runtime Directories

mkdir ${SnapRuntimeDir}
mkdir ${SnapRuntimeDir}/bin
mkdir ${SnapRuntimeDir}/repository
mkdir ${SnapRuntimeDir}/designer
mkdir ${SnapRuntimeDir}/cache_dir
mkdir ${SnapRuntimeDir}/compute
mkdir ${SnapRuntimeDir}/static
mkdir ${SnapRuntimeDir}/logs
mkdir ${SnapRuntimeDir}/config

Step 7: Copy in Runtime Files (DELTA)

The official instructions inform you that you need to install icon files at this point. That no longer seems necessary, as the Designer installation in a later step now takes care of that.

However, I found that you do need to install the components and tutorial into the runtime directory.

cd ${SnapRuntimeDir}
cp -r ../snaplogic/components/ .
cp –r ../snaplogic/example/tutorial .
cd tutorial
mkdir data
mv *.csv data

Step 8: Launch SnapAdmin

cd ${SnapRuntimeDir}
python ${SnapPackageDir}/snaplogic/tools/snapadmin.py

Step 9: Create your Repository in the Runtime Directory

repository create -t sqlite /home/plaird/dev/snaplogic/runtime/repository/repository.db

Step 10: Create a Server Configuration File

Download the template configuration file from the website (link is on this page) and copy it into your ${SnapRuntimeDir}/config directory. Make the following modifications:

• Replace ${SnapRuntimeDir} with absolute path
• Change YOURHOST.EXAMPLE.COM to the fully qualified domain name of your host (three locations)
• Optionally change the ports from 8088, 8081 and 8089

Step 11: Install the Authentication Files

These files give you a starting solution for authentication:

cp ${SnapPackageDir}/snaplogic/server/auth/simple_snapaccess.conf ${SnapRuntimeDir}/config/
cp ${SnapPackageDir}/snaplogic/server/auth/simple_passwords ${SnapRuntimeDir}/config/

Step 12: Install the Designer GUI

cd ${SnapRuntimeDir}unzip /home/plaird/Desktop/downloads/snaplogic-designer-2.2.0.zip

Step 13: Create the Server Start Script (DELTA)

The default control script is not included in the community edition. Create a file called startServers.sh in your runtime directory. Its contents should be as follows (replacing localhost:8088 with your server URL):

echo "Starting data server"
python ${SnapPackageDir}/snaplogic/server/server.py  --config=${SnapRuntimeDir}/config/snaplogic_server.conf &
echo "Starting management server"
python ${SnapPackageDir}/snaplogic/mgmtserver/mgmt_server.py --config=${SnapRuntimeDir}/config/snaplogic_server.conf &
echo "Starting component server"
python ${SnapPackageDir}/snaplogic/cc/component_container.py --name=cc1 --server=http://localhost:8088 &

Step 14: Correct the Bad Import on Ubuntu 10.04 (DELTA)

If you are running with Ubuntu 10.04, you probably have package python-simplejson at version 2.0.9-1build1. SnapLogic 2.2 was certified against version 1.7.1. You will need to remove an import statement from a SnapLogic file:

gedit /home/plaird/dev/snaplogic/snaplogic/rp/streaming_json_rp.py

Edit out this line:

from simplejson.decoder import JSONScanner

Step 15: Start the Servers and View the UI (DELTA)

chmod u+x startServers.sh
./startServers.sh
firefox http://<your_machine_name>:8081 &

Step 16: Walk through Tutorial

You can at this point start with the tutorials.

Step 17: ROAD BLOCK

Sadly, at this point I have hit a road block. The community edition of the product apparently has a bug in that it depends on an license file for a commercial version of the product (pro or core). I have filed defect 2449 – hope to get a fix shortly!

In the meantime, see my follow on post on how to install SnapLogic 2.1.0 instead. While it is an earlier version, the upside is the 2.1.0 version has a community installer.

 

Technorati Tags: snaplogic,ubuntu,rest

Example Linux startup service for Python, Java, Ruby, Perl, or Script as a daemon process in init.d

This blog post provides sample Bash code for converting a Python, Java, Ruby, Perl, C++, script, or any other program as a Linux service that will start running at system startup. It is a script that is deployed in /etc/init.d like many other common service programs.

Note that there may be language specific ways of doing what you want. For example, there is an evolving Python package that is supposed to do this for you. At this writing, that package seems to be still in progress.

This solution has three components:

• The actual program that needs to run as a service. In this example, it is a python (.py) program. But it really can be anything, including Java, Ruby, C++, or whatever
• A wrapper Bash script that simply invokes the actual program. The only value add is that it writes the process identifier of the actual program to a file. You could possibly folder this functionality into the daemon itself, but I haven’t bothered to do that.
• The daemon script. It is what is located in /etc/init.d and ultimately is responsible for starting and stopping the service.

Actual Program – echoHW.py

For this example, it does nothing real. It just spins the process:

import time

while 1 < 2:
    time.sleep(10)

Wrapper Script – echoHW.sh

This script simply invokes the actual program. It does also write out the process identifier to the file.

# Execute the actual python script
# The "$@" passes in any parameters into the python exectuable
# The '&' puts the process into background (as a daemon)
# The 'echo $! > mydaemon.pid'  write the process id to a file

python echoHW.py "$@" &
echo $! > mydaemon.pid

Daemon Script – mydaemon

This file must be copied into /etc/init.d.

#!/bin/bash
#
# mydaemon          Start/Stop any shell script
#
# chkconfig: 345 95 65
# description: mydaemon 
# processname: mydaemond
#

# ENVIRONMENT
# Edit these for your configuration

# Name for the service, used in logging
NAME=mydaemon

# Name of the user to be used to execute the service
SCRIPT_USER=plaird

# Example of how to pass paramters into the command
PARAM2=whatever

# In which directory is the shell script that this service will execute
MYDAEMON_SCRIPTS_DIR=/home/plaird/dev/linux

# Construct the command the will cd into the right directory, and invoke the script
MYDAEMON_COMMAND="cd $MYDAEMON_SCRIPTS_DIR; ./echoHW.sh 'param1' $PARAM2"

# How can the script be identified if it appears in a 'ps' command via grep?
#  Examples to use are 'java', 'python' etc.
MYDAEMON_PROCESS_TYPE=python

# Where to write the log file?
MYDAEMON_SVC_LOG_FILE=$MYDAEMON_SCRIPTS_DIR/mydaemon.log

# Where to write the process identifier - this is used to track if the service is already running
# Note: the script noted in the COMMAND must actually write this file
PID_FILE=$MYDAEMON_SCRIPTS_DIR/mydaemon.pid

# Load system specific optional arguments
# Create and populate this file with machine specific settings
if [ -f /etc/sysconfig/mydaemond ]; then
    . /etc/sysconfig/mydaemond
fi

# Is the service already running? If so, capture the process id
if [ -f $PID_FILE ]; then
    PID=`cat $PID_FILE`
else
    PID=""
fi

# SERVICE ENTRY POINTS (START/STOP)

# Start Command
start() {
    if [ "${PID}" != "" ]; then
        # Check to see if the /proc dir for this process exists
        if [ -a /proc/${PID} ]; then
            # check to make sure this is likely the running service
            ps aux | grep ${PID} | grep $MYDAEMON_PROCESS_TYPE >> /dev/null
            # If it is a process of the right type assume that it is mydaemon and just exit
            # otherwise remove the subsys lock file and start mydaemon
            if [ "$?" = "0" ]; then
                exit 1
            else
                echo "mydaemon lock file still exists, removing..."
                rm /var/lock/mydaemond
            fi
        else
            # The process running as pid $PID is not a process of the right type, remove subsys
            # lock and start mydaemon
            echo "mydaemon lock file still exists, removing..."
            rm /var/lock/mydaemond
        fi
    fi
    echo -n "Starting mydaemon: "   
    su - $SCRIPT_USER -c "/bin/sh -c \"$MYDAEMON_COMMAND > $MYDAEMON_SVC_LOG_FILE 2>&1\"" & RETVAL=$?
    sleep 3
    touch /var/lock/mydaemond
    exit 0
}

stop() {

    echo -n $"Stopping mydaemon: "
    if [ "${PID}" != "" ]; then
        echo -n "killing " $PID
        kill ${PID}
        for i in {1..30}
        do
            if [ -n "`ps aux | grep $MYDAEMON_PROCESS_TYPE | grep mydaemon `" ]; then
                sleep 1 # Still running, wait a second.
                echo -n .
            else
                # stopped
                rm -f /var/lock/mydaemond
                rm -f $PID_FILE
                echo
                exit 0
            fi
        done
    else
        echo "$NAME is not running"
        exit 1
    fi
    echo "Failed to stop in 30 seconds."
    kill -QUIT ${PID} # Request a thread dump so we can diagnose a hung shutdown
    exit 1
}

case "$1" in
  start)
      start
    ;;
  stop)
      stop
    ;;
  *)
    echo $"Usage: $0 {start|stop}"
    exit 1
esac

Installation

1. Start by copying the actual program and wrapper script somewhere on your machine in a permanent location.

2. Next, edit the mydaemon script to reflect the location where the program was installed. You may want to rename the file, and update the name of mydaemon inside the file.

3. Copy the mydameon script to /etc/init.d. As in:

sudo cp mydaemon /etc/init.d

4. Install the service. Installation will vary depending on your flavor of Linux. If you have chkconfig installed, it will be:

sudo chkconfig –-add mydaemon

If you are on a distro like Ubuntu, use update-rc.d, as in:

sudo update-rc.d mydaemon defaults

Service Usage

The service should now start on startup the computer. But you can invoke it manually as root:

sudo service mydaemon start

or:

sudo service mydaemon stop

Technorati Tags: linux,bash,init.d