ETL Work - Talend

Extract Transform and Load (ETL) is a general concept of how to move data from one system to another.  There are a lot of tools, and a lot of different methodologies to successfully accomplish this.  But I want to touch upon a product and ECO system that I've never heard of before, until 4 months ago.

Talend is a open source eclipse project that allows you to design ETL work in a visual capacity.  It stores its' configuration within XML files, and when you are ready to execute, for debugging or production, it then generates java code and compiles it.

My group has bought into the entire Talend ECO system, which combines many different open source projects into an nicely integrated product.

But I'm a ColdFusion lover, and this blog is about ColdFusion, therefore, there is two integration points that Talend provides.

First, if you buy into the ECO System, then you'll get a Administration server that schedules you, and then you can use modules the consume web services and we all know how awesome ColdFusion is at building web services!

Second, Talend Studio - the eclipse development UI, is free.  Using this free product will provide you an ability to write ETL jobs in java, compile them and then execute them from ColdFusion.

Both scenarios allow you to be a novice JAVA developer and use the powerful more complicated parts of JAVA.  For example, it is a single click to on parallelism for the entire job, therefore, long running jobs would finish in a portion of the time.

Lastly, The use-case that we are using this tool is to transition from one system to another.  And because of the scope of the project it won't be one quick cut over.  Therefore, having the ability to run the ETL job over and over and over (you get it), while developing, testing, and then in production during the transition phase is very powerful.  Along with having the entire java core to be able to transform the data is a great resource to have under your belt.

Comments

Post a Comment

Popular posts from this blog

ColdFusion in an Enterprise Environment - Part 1 - Understanding how to use SubVersion (SVN)

Subversion can be used in many ways, so this isn't the holy grail of how you need to use subversion, but it does add application management to subversion. What does application management mean? Well, the definition for this article is how to control what gets released into production. This article is not going to go into how to setup version, or how to add an external to the repository tree. But how in concept to use SVN and an external to achieve application management. First of course you need a SVN repository, which for the purpose of this article our repository will be noted as "/". Second we need to know that will we still be using the concepts of trunk, tags, and branches, they just won't live in the root of the repository. The trunk is the current working copy of development, while the tag folder is to version releases (by tagging). Branches will be used when "new" development is going on and emergency coding is required on the existin...
Read more

coldfusion builder 2 extension not displaying browse button on type=projectdir

I'm in the process of creating an extension that fires from the RDSView (from a database table).  Basically I would like it to create 7 files for me by using the table's columns I selected.  *PERFECT*, I got the files being created, but I have to TYPE in the LOCATION to store the files <-- WHAT --  Yes, I said I have to type in the full path to where I want the files to be stored. So my question to anyone who might see this blog is -->  Is there a way via (call back, response, etc) in an cfbuilder extension for the user to pick the project and/or folder as an input? PS.  if you read this and don't know the answer, please "socialize" it (tweet, FaceBook, etc) so possibly the correct person with the answer can answer it.  *THANKS* **UPDATE 9/29/2011** I was able to email a contact at Adobe with this issue, and after working back and forth, it came up that the input of  type="projectdir" does NOT work when using a REMOTE server. How disappoin...
Read more

Being Thread Safe in Coldbox/Coldfusion

Thread safe never crossed my mind.  I never put CFC in the application scope to allow quicker execution (bypassing the createobject function).  But now that my office has implemented ColdBox, it cache's a lot for you, including its handlers - which are CFCs.  Well, we got bit by that evil words "THREAD SAFE".  Our code WASN'T... yes including my code. So what does that mean.  Well I'll try to do a quick explanation, thread safe is any variable that is dedicated to a SINGLE user will NOT be shared to other users.  For example, in a object a "private" variable is NOT thread safe.  So if many users use the same object (implementation of a class), then we must make sure the variables being passed around are LOCAL to that function.  Therefore, in a CFC (depending on language) you must force your function variables to be LOCAL to the function only. In CF8 use of the VAR keyword provides this <cfset var myLocalVar = "" /> In CF9 the use o...
Read more