Monday, February 08, 2010

Passing custom objects with Apache Etch.

While using Apache Etch as the wire protocol of a distributed project, we had a requirement to send our custom Java objects across. This post demonstrates sending custom types and invoking different server methods without changing the idl.

Apache Etch has the concept of an extern, which can be used to define specific types. The IDL needs the definition of the type, and the class with which to serialize/deserialize it. This serializer helps Etch transfer the object across the network. Let us define our pojo - com.etchTrials.Base

package com.etchTrials;

import java.io.Serializable;
import java.util.Map;

public class Base implements Serializable{

private String name;
private int age = 5;

public Base(String values) {
this.name = values;
}

public Base() {

}


public String getName() {
return name;
}

public void setName(String name) {
this.name = name;
}

public int getAge() {
return age;
}

public void setAge(int age) {
this.age = age;
}

public String methodA() {
System.out.println("Inside method A");
return "A";
}

public String toString() {
return "Value is " + name + " and int is " + age;
}

}


Next we define the BaseSerializer. This implements Etch's ImportExportHelper to define how Base can be marchalled/unmarshalled.


package com.etchTrials;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;

import etch.bindings.java.msg.Field;
import etch.bindings.java.msg.ImportExportHelper;
import etch.bindings.java.msg.StructValue;
import etch.bindings.java.msg.Type;
import etch.bindings.java.msg.ValueFactory;
import etch.bindings.java.support.Class2TypeMap;
import etch.bindings.java.support.Validator_byte;
import etch.bindings.java.support.Validator_object;
import etch.bindings.java.util.StrStrHashMap;
import etch.bindings.java.util.StrStrHashMapSerializer;

public class BaseSerializer implements ImportExportHelper {

private final Type type;
private final Field field;

public final static String FIELD_NAME = "base";

public BaseSerializer(Type type, Field field) {
this.type = type;
this.field = field;
}

/**
* Defines custom fields in the value factory so that the importer can find
* them.
*
* @param type
* @param class2type
*/
public static void init(Type type, Class2TypeMap class2type) {
Field field = type.getField(FIELD_NAME);
class2type.put(Base.class, type);
type.setComponentType(Base.class);
type.setImportExportHelper(new BaseSerializer(type, field));
type.putValidator(field, Validator_byte.get(1));
type.lock();
}

@Override
public Object importValue(StructValue struct) {
Base base = new Base();
try {
byte[] bytes = (byte[]) struct.get(field);
ByteArrayInputStream stream = new ByteArrayInputStream(bytes);
ObjectInputStream ois = new ObjectInputStream(stream);
base = (Base) ois.readObject();
ois.close();
}
catch(Exception e) {
e.printStackTrace();
}
return base;
}

@Override
public StructValue exportValue(ValueFactory vf, Object arg1) {
StructValue struct = new StructValue(type, vf);
Base base = (Base) arg1;
try {
ByteArrayOutputStream stream = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(stream);
oos.writeObject(base);
oos.close();
byte[] bytes = stream.toByteArray();
struct.put(field, bytes);
}
catch(Exception e) {
e.printStackTrace();
}
return struct;
}

}


We define these two in the IDL:

module com.etchTrials

service Generic {
exception BaseException(string msg);

@Extern( java, "com.etchTrials.Base", "",
"com.etchTrials.BaseSerializer", "" )
extern Base;

object invokeServerMethod(string method, object[] params);

void connect( string host, int port );

void listen();

@Direction(Both)
void close();

}



The above idl has:
1. Definition of our custom type - Base as an extern. It tells Etch that we have a type called Base, which can be marshalled/demamarshalled using the BaseSerializer. Other pojos need to be defined similarly
2. Definition of a generic method invokeServerMethod. This is capable of accepting and returning Etch currently supported java types and Base types. It takes the name of the method to be executed, and its parameters.

We now define EtchClient, our client, The Etch client will be able to execute the following on the server:
a. Invoke method whoIsOnline which returns a String[].
b. Invoke method login which accepts a String and returns a boolean.
c. Invoke a void method with no params
d. Invoke a method which sends a custom pojo com.etchTrials.Base object.
Server alters this object and client receives the changed values

We also define EtchServer, which implements the methods defined in the IDL.


package com.etchTrials;

import java.lang.reflect.Method;

import etch.util.core.io.Transport;

public class EtchClient extends BaseGenericClient {

private RemoteGenericServer server;

public EtchClient() {
}

protected void setUp() {
try {
String uri = "tcp://0.0.0.0:4005?TcpTransport.reconnectDelay=4000";
System.out.println("URI " + uri);
final EtchClient client = this;
server = GenericHelper.newServer(uri, null,
new GenericHelper.GenericClientFactory() {
public GenericClient newGenericClient(
RemoteGenericServer server) throws Exception {
return client;
}
});

server._startAndWaitUp(4000);
System.out.println("Client server started");
} catch (Exception e) {
e.printStackTrace();
}

}

public void close() {
try {
System.out.println("Closing time");
server._transportControl(Transport.STOP, null);
} catch (Exception e) {
e.printStackTrace();
}
}

//test method which invokes some methods on Base object
public void testInvokeServer() {
String[] whoIsOnline = (String[]) server.invokeServerMethod("whoIsOnline", null);
for (String who: whoIsOnline) {
System.out.println("whoIsOnline returns " + who) ;
}
System.out.println("Login returns " + server.invokeServerMethod("login", new String[] {"Browser1"}));
System.out.println("Invoking voidMethod ");
server.invokeServerMethod("voidMethod", null);
System.out.println("Invoked voidMethod ");
//String user = "User";
Base base = new Base();
base.setAge(25);
base.setName("Any value");
String name = "Change from server";
Object[] params = new Object[] {base, name};
System.out.println("Invoking withParams");
Base returnBase = (Base) server.invokeServerMethod("withParams", params);
System.out.println("Base returned is " + returnBase);

}

public static void main(String[] args) {
EtchClient client = new EtchClient();
client.setUp();
client.testInvokeServer();
client.close();
}

}


The Etch client will be able to execute the following on the server:
a. Invoke method whoIsOnline which returns a String[].
b. Invoke method login which accepts a String and returns a boolean.
c. Invoke a void method with no params
d. Invoke a method which sends a custom pojo com.etchTrials.Base object.
Server alters this object and client receives the changed values

Finally, we have the EtchServer:

package com.etchTrials;

import etch.bindings.java.support.ServerFactory;
import etch.util.core.io.Transport;

public class EtchServer extends BaseGenericServer implements
GenericHelper.GenericServerFactory {

public EtchServer(RemoteGenericClient client) {
System.out.println("Client " + client);
this.client = client;
}

public EtchServer() {
System.out.println("Server constructor");
}

public Object invokeServerMethod(String method, Object[] params) {
//System.out.println("Inside invoking the method");
if (method.equals("whoIsOnline")) {
return whoIsOnline(params);
}
else if (method.equals("login")) {
return login(params);
}
else if (method.equals("voidMethod")) {
voidMethod();
return null;
}
else if (method.equals("withParams")) {
return withParams(params);
}
return null;
}

private Boolean login(Object[] params) {
String userName = (String) params[0];
System.out.println("login: User is logged in now " + userName);
return new Boolean(true);
}

private String[] whoIsOnline(Object[] params) {
String[] names = new String[] { "ChatterBoxA", "ChatterBoxB" };
return names;
}

private void voidMethod() {
System.out.println("voidMethod: Inside Void method");
}

private Base withParams(Object[] params) {
System.out.println("withParams: With params method");
//String clientName = (String) params[0];
//System.out.println(params[1].getClass().toString());
Base base = (Base) params[0];
//do some calculations
//return clientName +
System.out.println("withParams: Received base object from client" + base);
String name = (String) params[1];
base.setAge(base.getAge() * 2);
base.setName(name);
return base;
}

private RemoteGenericClient client;

public void connect(String host, String port) {
try {
String uri = "tcp://" + host + ":" + port;
System.out.println("URI " + uri);
Transport listener = GenericHelper.newListener(uri,
null, this);

listener.transportControl(Transport.START_AND_WAIT_UP, 4000);
System.out.println("Started");
} catch (Exception e) {
e.printStackTrace();
}
}

public void listen() {
System.out.println("listen");
}

public void close() {
System.out.println("close");
}

public GenericServer newGenericServer(RemoteGenericClient client)
throws Exception {
return new EtchServer(client);
}

public static void main(String[] args) {
EtchServer server = new EtchServer();
System.out.println("New Server");
server.connect("0.0.0.0", "4005");
System.out.println("Up and running");
}

}

Labels: , ,

Tuesday, December 15, 2009

JAQL

Wondering what's up at JAQL. Seems they are still at Hadoop 0.18, whereas Hadoop has moved to 0.20.1. Anybody besides IBM using it?

Labels: ,

Saturday, July 25, 2009

Unknown Entity with Hibernate 3

While using Hibernate 3 and JPA, remember to place persistence.xml in the classes/META-INF folder. Otherwise, you will get Exception in thread "main" org.hibernate.MappingException: Unknown entity

Labels: , ,

Friday, June 19, 2009

Hadoop On Ubuntu

I am ramping myself up to MapReduce and GFS as part of a very exciting new opportunity. I needed to install Hadoop-0.20 on my Ubuntu desktop. I followed Michael Noll's excellent tutorial as well as the quickstart for the release. Michael's tutorial is not updated yet for the split of config files in Hadoop 0.20, and he has mentioned that clearly. So, I followed his tutorial and when it comes to the configs, I used the quickstart information for core-site, hdfs-site and mapred-site.

I was however getting stuck with a strange issue. jps would show the jobtracker process for a while, and then it would get killed. The logs had a java.io.IOException: /tmp/hadoop....jobtracker-info file could only be replicated to 0 nodes instead of 1.
The Hadoop mailing lists did not have any reference to this issue. All I found was this bug. I read on Cloudera's page that start-all and stop-all are deprecated, but I could not find any reference to it on Hadoop's wiki.

After various hits and trials, what worked for me was to edit start-all.sh. I first started the dfs and then waited for 5 minutes. Only then did I brin up the mapred deamons. This solved it.

Labels: ,

Sunday, March 22, 2009

Infamous Error 150 while creating a table in MySQL

I am working on a data migration and defining the schema. The application will eventually be implemented in Ruby on Rails, but instead of migrations, the client needed the sql dump. My CREATE TABLE statement was failing with errno 150, and SHOW ENGINE INNODB STATUS was not helpful. The Foreign Key section kept telling me :

Cannot resolve table name close to:
(id)) engine=innodb

I was completely zapped, both the related tables were INNODB tables, the columns were same type and size, indices were defined correctly. It was after sometime that I realized that the syntax for creating the foreign key was wrong. I was saying:

FOREIGN KEY (parent_id) REFERENCES parent.id

But it should be:
FOREIGN KEY (parent_id) REFERENCES parent(id)

Labels: ,

Sunday, August 31, 2008

Buying from Dell.

I decided on buying a Dell laptop. Dell was offering 2 GHz+ processors, which was a very tempting proposition. As I needed the laptop as soon as possible, I bought it from the direct Dell stores instead of the site. There were two choices - Vista prebundled or no OS. Needless to say, I went with no OS. Here began the most harrowing customer experience of my life.

The FreeDOS CD which came with the laptop did not work. It failed to recognize the hard drive. RedHat could not be installed either, no hard disk again. Fedora Core got installed, but my wifi button would refuse to power on. Windows XP would not get installed either, the installation hung midway.

Dell refuses to provide drivers for any OS except Vista! A clear way to dissuade people from going the non proprietary, non-Microsoft way. An intelligent financial strategy to subvert legalities and yet force the customer to buy tightly integrated systems.

When I called Dell Support, a customer service representative with no know how whatsoever told me that :

1. Dell can not support non factory OS.
2. They have NO non Vista drivers.
3. They cant tell me why FreeDOS is not installing. Nor will they help me install it, even though they gave the CD with the laptop.

45 minutes of interaction with the customer support guy had my head spinning. I felt absolutely cheated. Luckily, threats of suing the company worked, and the rep digged through some more guides to tell me some BIOS changes needed to install Windows XP.
Eureka, dream machine and excellent customer service!!!!

Apparently, the bundling with Vista is so strong that the BIOS itself is pre-configured. No documentation on what to change obviously from Dell, lest some lowly mortal attempt a lesser OS than holier than thou Vista.

Anti-trust anyone ?

Labels: , ,

YouTube API does not work through Resin

I have run into a strange issue with the youtube api. I am able to upload videos through the browser and verify them in my youtube acocunt. However, when I try to retrieve the videos, I do not get them. I am able to retrieve correctly through a junit test case.

After a lot of digging on the forum, I found this post:
Different behaviour under Tomcat and Resin

The client api documentation does not suggest any supported environments or versions. GData Doc

Labels: , , ,

Tuesday, March 18, 2008

Pricing consulting services

I have been wondering a bit about the business model and economics of freelancing. Stumbled across some interesting links:

Freelance Consulting Rates
How to Price Consulting Services

Hotgigs has some good reference data based on skill set

Labels: , ,

Six resolutions for a new year of consulting | IT Consultant | TechRepublic.com

Came across some interesting new year resultions - Six resolutions for a new year of consulting | IT Consultant | TechRepublic.com