EigenbaseCodeReorg2008
Contents |
Introduction
This is the planning page for some code reorganizations taking place across the Eigenbase codebase during Q4 2008. We would like to carry these out with as few integration hiccups as possible across branches.
Reorg List
The following are the reorgs to be carried out:
- Bring all Farrago, Fennel, and Saffron files into conformance with checkFile and enable the trigger for all subsequent checkins on all branches
- Update year in copyright notices (this has not been done for 2008 yet)
- Globally rename Disruptive Tech to SQLstream anywhere it appears, including in copyright notices
- But the Perforce tree will still be named //open/dt for old time's sake
- Run Jalopy on all Farrago code
- Movement of a number of components from yellow-zone to green-zone (see next section); this will have impact on both LucidEra and SQLstream red-zone code
- If possible, JVS would also like to carry out the first two items from FarragoRelCleanup; this would have impact on SQLstream red-zone code
Yellow To Green Movement
Negotiations between SQLstream and LucidEra have resulted in an agreement to move the following components from yellow-zone to green-zone:
- hash join/agg
- external sort
- flatfile reader
- LURQL
- applib
- Fennel calc
- Volcano
In each case, either the corresponding source files need to be moved into an existing green zone location, or the entire directory in which they live needs to be redefined as green zone. In addition, the copyright notices need to be updated to include all three entities (Eigenbase, LucidEra, and SQLstream).
Details of code movement:
| Origin | Destination | Notes |
|---|---|---|
| fennel/disruptivetech/calc | fennel/calculator | |
| fennel/disruptivetech/xo | fennel/exec | (except Calc{ExecStream/Excn}, which will go to fennel/calculator) |
| fennel/disruptivetech/libfennel_dt | fennel/libfennel | |
| fennel/disruptivetech/test | fennel/calctest | (except {Collect/CorrelationJoin}ExecStreamTest{Suite}, which will go to fennel/test) |
| fennel/disruptivetech/farrago | fennel/farrago | |
| fennel/disruptivetech | deleted | |
| fennel/lucidera/hashexe | fennel/hashexe | |
| fennel/lucidera/flatfile | fennel/flatfile | |
| fennel/lucidera/sorter | fennel/sorter | |
| relevant portions of fennel/lucidera/test | fennel/test | |
| relevant portions of fennel/lucidera/farrago | fennel/farrago | |
| farrago/src/com/disruptivetech/farrago/calc | farrago/src/net/sf/farrago/fennel/calc | |
| farrago/src/com/disruptivetech/farrago/fennel | farrago/src/net/sf/farrago/defimpl | |
| farrago/src/com/disruptivetech/farrago/rel | farrago/src/net/sf/farrago/fennel/rel | this is also the destination package for the second item in FarragoRelCleanup |
| farrago/src/com/disruptivetech/farrago/sql/advise | farrago/src/org/eigenbase/sql/advise | |
| farrago/src/com/disruptivetech/farrago/test | farrago/src/org/eigenbase/sql/test, farrago/src/net/sf/farrago/test | other options are sql/advise, sql/advise/test, farrago/src/net/sf/farrago/fennel/calc(/test) |
| farrago/src/com/disruptivetech/farrago/volcano | farrago/src/org/eigenbase/relopt/volcano | |
| farrago/src/com/disruptivetech | deleted | |
| farrago/src/com/lucidera/lurql | farrago/src/org/eigenbase/lurql | perhaps eventually to enki; LucidEra has red-zone dependencies on this so we may need to let the old and new copies coexist for a short while |
| farrago/src/com/lucidera/opt/Lhx* | farrago/src/net/sf/farrago/fennel/rel | |
| farrago/src/com/lucidera/farrago/namespace/flatfile | farrago/src/net/sf/farrago/namespace/flatfile |
Rezoning
Everything under the //open/dev/luciddb project gets reclassified from yellow to green (with corresponding copyright change), but does not move anywhere. We can work on moving the relevant applib code to farrago/src/net/sf/farrago/applib, but it remains to be worked out how to do that with backwards compatibility for LucidDB's existing applib jar, and we may actually want applib as an independent project to make it easier to accept contributions for it. So TBD for now, and probably best done as an independent followup after the main reorg.
Everything under farrago/src/com/lucidera, fennel/lucidera, and saffron remain as yellow-zone.
Code movement should be carried out via p4 integrate where possible in order to preserve history.
Plan
- Test out all activities which are going to take place during the rest of the steps so that they can be executed as quickly as possible, minimizing global lockout time
- Arrange for a global quiescence. This means all branches have been fully integrated, so all trees are identical, and everyone stops checking in for the duration of the reorg, except for emergencies.
- Run Jalopy on //open/dev
- Run Julian's fixup script on //open/dev to make all files checkFile-conformant
- Enable checkFile trigger for farrago/fennel/saffron/luciddb across all branches
- --Jvs 19:05, 24 March 2009 (EDT): skipped saffron/luciddb for now
- Update copyright years on //open/dev, and global rename Disruptive Tech to SQLstream
- --Jvs 20:56, 11 October 2008 (PDT): haha, resgen still has Red Square too!
- Everyone syncs everything down to all branches ASAP, but quiescence remains in effect
- On //open/dt/dev, SQLstream carries out their yellow->green reorgs, as well as first two items from FarragoRelCleanup (dealing with their red-zone dependencies simultaneously)
- SQLstream integrates their reorgs up to //open/dev
- At this point, quiescence is ended for SQLstream, but not yet for LucidEra
- LucidEra syncs down from //open/dev to //open/lu/dev
- LucidEra carries out their yellow->green reorg on //open/lu/dev
- LucidEra syncs up to //open/dev (and sideways to lcs)
- Quiescence is over for LucidEra now too
- SQLstream syncs down to //open/dt/dev
- Done, yay...carry on with BoostStlportUpgrade2009!
--Jvs 17:16, 24 March 2009 (EDT): Actual execution for steps 8-16 was slightly different: JVS carried out all of the reorg himself on //open/dev, and SQLstream and LucidEra are taking care of impact as they integ down.
Integration Notes
--Jvs 17:16, 24 March 2009 (EDT): This section contains pointers for people absorbing the impact of the reorg above. This could be either external dependencies or Eigenbase in-flight changes during the quiescence (either sandbox or private branch). Eigenchange references are provided so that people carying out this work can reference the specific changes with p4 describe.
- eigenchange 12468 (move all of SQLstream portion of Fennel to green-zone): Extension projects which depend on the SQLstream yellow-zone libraries need to change their #includes to reference the new green-zone locations (e.g. including "fennel/calculator/Calculator.h" instead of "fennel/disruptivetech/calc/Calculator.h"; don't forget about TestSuite inheritance). This change also eliminates all libfennel_dt_*.so and libfarrago_dt.so libraries, so extension projects which depend on them need to change their linker directives accordingly (e.g. linking with libfennel_calculator instead of libfennel_dt_calc). Likewise, the JNI library libfarrago_dt.so no longer needs to be dynamically loaded from Farrago; see edit to FarragoDefaultSessionPersonality where the DisruptiveTechJni.registerStreamFactory call was removed.
- eigenchange 12470 (move some of LucidEra portion of Fennel to green-zone): I don't believe there are any external dependencies on these components yet, so nothing special needed here.
- eigenchange 12471 (move com.disruptivetech.farrago.calc to net.sf.farrago.fennel.calc): Extension projects which depend on these SQLstream yellow-zone Java packages need to update their imports accordingly (e.g. import net.sf.farrago.fennel.calc.* instead of com.disruptivetech.farrago.calc.*).
- eigenchange 12472 (move com.disruptivetech.farrago.rel to net.sf.farrago.fennel.rel): Similar to previous, just update imports.
- eigenchange 12474 (move com.disruptivetech.farrago.sql.advise to org.eigenbase.sql.advise, and com.disruptivetech.farrago.volcano to org.eigenbase.relopt.volcano): Similar to previous, just update imports (see for example Saffron dependency update in eigenchange 12481).
- eigenchange 12475 (delete com.disruptivetech, moving tests to org.eigenbase.sql.test and net.sf.farrago.test): all imports from com.disruptivetech.** should be gone now.
- eigenchange 12476 (move com.lucidera.lurql to org.eigenbase.lurql; move com.lucidera.opt.Lhx* to net.sf.farrago.rel; move com.lucidera.farrago.namespace.flatfile to net.sf.farrago.namespace.flatfile): To minimize impact from this change, I left behind deprecated wrappers in the old packages; e.g. com.lucidera.lurql.LurqlQueryProcessor stays behind as a wrapper for the new org.eigenbase.lurql.LurqlQueryProcessor. Once LucidEra has fixed up any red-zone dependencies, these deprecated wrappers can be deleted.
- eigenchange 12478 (move rules from org.eigenbase.rel to org.eigenbase.rel.rules; move Fennel-related subset of net.sf.farrago.query to net.sf.farrago.fennel.rel): update imports accordingly; this one takes a bit more work since it's a partial move, so blind search/replace can't be used.
- eigenchange 12480 (change AggregateRel EXPLAIN to use MAX($1) instead of MAX(1); make optimizer rule singleton pattern consistent per FRG-105): update EXPLAIN PLAN .ref files for the AggregateRel change; I took care of LucidDB checkin acceptance tests, but not full nightly regression, so LucidEra will need to handle that part. For the rule singletons, I left the existing constructors as public but deprecated; once all dependencies have been fixed (leaving references only to the public singletons), these constructors can be made private and undeprecated (this task belongs to SQLstream).
