EigenbaseCodeReorg2010

Introduction

This is the planning page for some code reorganizations taking place across the Eigenbase codebase during Q1 2010. We would like to carry these out with as few integration hiccups as possible across branches.

Reorg List

The following are the reorgs to be carried out:

• Run Jalopy on all LucidDB files eigenjira:LDB-205 and bring them into conformance with checkFile+checkPreambles eigenjira:LDB-36 and enable the trigger for all subsequent checkins on all branches
• Update year in copyright notices for any files modified during 2009
  • We decided to remove the end year from the copyright notices, so do that instead
• Globally replace LucidEra, Inc. with Dynamo BI Corporation in copyright notices
• Globally replace Disruptive Tech with SQLstream under Enki (something we left out of the last reorg)
• Move remaining LucidDB components under Farrago and Fennel from yellow zone to green zone
• Add SQLstream to copyright in all LucidDB files (leftover task from EigenbaseCodeReorg2008) as well as files moving from yellow zone to green zone (same question about start year?)
• Move code for applib from luciddb to extensions, and repackage it as org.eigenbase.applib in eigenbase-applib.jar
• Update this page

Yellow to Green Movement

Negotiations between Dynamo BI and LucidEra have resulted in an agreement to move all remaining LucidDB components from yellow-zone to green-zone:

• column store
• bitmap indexing
• optimizer rules
• miscellaneous integration classes

Details of code movement:

Origin	Destination	Notes
fennel/lucidera/colstore	fennel/lcs	we'll keep the "l" prefix in case other column-store implementations come in later
fennel/lucidera/bitmap	fennel/lbm	likewise, and also to discriminate from non-index bitmaps
fennel/lucidera/farrago	fennel/farrago
fennel/lucidera/libfennel_lu	fennel/libfennel
farrago/src/com/lucidera/lcs	org/luciddb/lcs
farrago/src/com/lucidera/lurql	deleted	this was just a dummy package left over from the last reorg; it is no longer needed since LucidEra red-zone dependencies no longer exist
farrago/src/com/lucidera/namespace/flatfile	deleted	this was just a dummy package left over from the last reorg; it is no longer needed since LucidEra red-zone dependencies no longer exist
farrago/src/com/lucidera/opt	farrago/src/org/luciddb/optimizer
farrago/src/com/lucidera/farrago,query,type,runtime	farrago/src/org/luciddb/session
farrago/src/com/lucidera/jdbc	farrago/src/org/luciddb/jdbc

Extensions Top Level Project

A new component for the Salesforce webservice connector will also be added (net.sf.farrago.namespace.sfdc). Build/test for this component depend on the proprietary Salesforce API, so sources for this component will be under a new extension project, //open/dev/extensions/.

Some Rules on Extensions Project:

- Farrago/Fennel can NOT have any required acceptance tests or requirements for extensions to be present.  There should be no build dependencies on anything in /extensions/.
- Extensions should run on vanilla Farrago, unless otherwise stated (applib-luciddb).
- They are still managed and covered by Eigenbase copyright assignments (contributions).
- Like thirdparty, they can include non-GPL open source libraries. --Ngoodman 20:56, 17 February 2010 (EST) I wonder, since they are separate can we use GPL libraries?
- External dependencie management systems, such as Ivy, may be used to manage dependencies for extension projects.
- LGPL Licensed

Overall Structure:

- /extensions/ (base folder, contains wrapper build scripts/build, blackhawk tests/ref)
- /extensions/<<extensionname>>/ (individual extensions, for instance applib-farrago, applib-luciddb, connector-sfdc, connector-mql, etc.

DynamoBI will make some proposals for overall extension project format (build.xml targets consistent, blackhawk testing, etc) in the future and will use splitting/moving Applib along with the LucidEra contributed SFDC connector a first trial to determine commonality needed. Additionally, per Julian suggestion we'll start a Wiki page similar to PDI to document extension projects (both @ Eigenbase and beyond).

--Ngoodman 14:57, 21 January 2010 (EST) I'd vote for adding these optional connectors as separate downloads/etc. They can be their own project, have their own packaging/jar/docs etc. Maybe another peer of applib? Or maybe the new toplevel project is extensions (that includes applib, connectors, etc). This work (separate project to build applib extension) will be valuable for Eigenbase regardless as we explore a new extensions area (perhaps "incubator-extensions") that is separate from the main project build /tests.

--Jvs 19:11, 23 January 2010 (EST): For further discussion, see this mailing list thread.