MondrianDevelopment
This page is intended as a starting point and reference for anyone work on the development and testing of Mondrian.
Contents |
Basics
A good deal of information about Mondrian can be found on the Mondrian project page. For starters, you should read the following pages:
These will provide you with a high-level, introductory overview of Mondrian. The information is by no means comprehensive so you should consult other sources for a deeper understanding of OLAP and MDX.
Mondrian Community
The Mondrian community is led by Julian Hyde.
Mailing Lists
If you are going to be making changes in the Mondrian source code, it would be a good idea to subscribe to the Mondrian mailing list. You can also access that mailing list subscription page by clicking on the "Mailing List" link from the project page. That page also provides a link to an archive of past messages posted to the mailing list. However, that archive only contains more recent mailings. For the older archive, see the SourceForge archives.
Developer Guide
There are code guidelines and check-in policies that you should follow described here Developer Guide.
Automated Regression
Mondrian also has Cruise Control setup so that any check-in is compiled and validated through 1000+ unit tests within ~20 minutes. Mondrian CC
Setup Shop
Building Mondrian
In order to build Mondrian, you first need to add the following line to your p4 client spec:
//open/mondrian/... //calvin.treehouse.eigenbase/open/mondrian/...
replacing "calvin" with your username and "treehouse" with your machine name. Sync your client once you've added the line to bring the source files to your machine.
Go to the section entitled "Building the code" on the Developer's Guide section on the Mondrian project page. That section lists a number of 3rd party products that Mondrian depends on. Some of these you should already have as part of your Eigenbase build. Follow the links to download the ones that are missing, and make sure you have set the environment variables listed.
Be sure to use JDK 1.6 for Mondrian development and testing.
- --Zfong 19:49, 22 April 2008 (PDT) JDK 1.6 is required for javadoc to run without errors. However, when JDK 1.6 is used, I get errors in several of the standard regressions tests that expect specific SQL statements to be generated as part of the test. With JDK 1.6, the order of expressions in some of the WHERE clauses is different from what is generated using JDK 1.5. I used build 10934, although I suspect this behavior exists in earlier builds as well.
- --Jsichi 17:24, 20 September 2008 (PDT): I disabled one of these in eigenchange 11589 (it was failing for me on JDK 1.5 also). Any others?
Run "ant" in your mondrian directory. This will build the source jar files as well as setup the Derby database that you will most likely use as the Foodmart demo database, for testing purposes.
Running Mondrian
This section assumes you are using Derby as your backend database. Later sections describe how to use other databases, including LucidDB and MySql. mondrian.properties File In order to run Mondrian, you will first need to edit the property file. This is the file in mondrian/mondrian.properties. Search for "Derby" in the file and replace the commented out property values with the following values:
mondrian.foodmart.jdbcURL=jdbc:derby:demo/derby/foodmart mondrian.foodmart.jdbcUser=sa mondrian.foodmart.jdbcPassword=sa mondrian.jdbcDrivers=org.apache.derby.jdbc.EmbeddedDriver driver.classpath=testlib/derby.jar mondrian.test.jdbcUser=sa mondrian.test.jdbcPassword=sa mondrian.test.jdbcURL=jdbc:derby:demo/derby/foodmart mondrian.catalogURL=file:/home/calvin/open/mondrian/demo/FoodMart.xml
The last four lines are needed if you want to run within Eclipse, which will be described further below.
The property file also has a number of other configuration parameters. For example, if you want Mondrian to dump out the SQL statements it executes while processing an MDX statement, follow the example in log4j tracing on how to modify the property settings.
Other configuration values are described in the Configuration section on the Mondrian project page.
To avoid accidentally checking in your temporary modifications to mondrian.properties, create a separate Perforce changelist for the edit using "p4 change". Any additional temporary Mondrian edits can be added to this changelist, once you've created it, by either specifying the -c option when you "p4 edit" the file, or by moving the file into the existing changelist through the p4v GUI.
(JVS)If you want to avoid checking out the mondrian.properties file, you can put your local property settings in a file named local.properties and then use the command antLocal (from open/util/bin) instead of ant. However, this approach isn't Eclipse-compatible yet.
Foodmart Demo Database
Mondrian comes with a Foodmart demo database, which is very handy for testing. All unit tests are written using this database. A version of Foodmart stored in Derby is available in the mondrian/demo directory. Also in that directory is the schema file that specifies the multi-dimensional schema for the Foodmart database. That file is named FoodMart.xml. This is where you specify your cube definitions (both base and virtual) and the dimensions and measures that make up those cubes. The schema currently does contain one virtual cube -- [Warehouse and Sales].
For testing purposes, it generally is sufficient to use the existing schema defined in that file. However, on occasion, you may need to extend the schema to exercise new features. See the Writing a Schema section on the Mondrian project page for a description of the different constructs available.
CmdRunner
Mondrian comes with a standalone utility that allows you to run ad-hoc MDX queries, similar to sqllineEngine in Farrago.
Copy the JDBC jar of the database you're connecting to into the testlib directory. Build the mondrian.jar file:
ant jar ant cmdrunner
To invoke the utility, run the following command from the mondrian directory:
bin/run.sh -p mondrian.properties
For convenience, you might want to define an alias for the above command.
See the Command Runner section on the Mondrian project page for a full description of all the commands available within cmdRunner.
Eclipse
In addition to cmdRunner, you can also execute MDX queries within a java program running inside of Eclipse. This is useful when you need to debug Mondrian.
Create a new workspace in Eclipse and import your Mondrian project. Note that the import creates a "mondrian-2.0" project. This is correct.
To build the project with JDK1.5, right click on the "mondrian-2.0" project. Scroll down to "Properties". From the "Java Compiler" menu, choose "Enable project specific settings" and set "JDK Compiler compliance level" to 5.0.
If you take a look at the mondrian/.classpath file, you'll notice that the file references the variable, ANT_HOME. Therefore, make sure you have defined this variable in your Eclipse environment. To define the variable, within Eclipse, select
Project->Properties->Java Build Path->Libraries->Add Variable->Configure->New
From here, you can then run one of the existing test programs, or modify one of them to add the MDX query that you want to debug.
Note that buiild.xml under the mondrian project excludes certain files depending on the JDK version. Search for the exclude tags in the compile.java target. You will need to exclude the appropriate file(s) (depending on which JDK version you are using) manually in Eclipse by editing
Project->Properties->Java Build Path->Source->Excluded
If you need to exclude files, make sure you "p4 edit" the .classpath file so Eclipse can write to it.
Testing Mondrian
Regression Tests
Run "ant test" to run all Mondrian regression tests. The "Running the test suite" section on the Developer's Guide page shows a sample of what the output should look like. That output is slightly old, but gives you a general idea of what it should look like.
A failed test will be denoted by an "F" in place of one of the dots (.) in the output. The actual failure appears at the end of the output. Because the output can be somewhat lengthy when you do have failures, it might be a good idea to pipe the test output into a file so you don't lose it from your screen. To run the tests in the background, do the following:
set +m ant test >& test.out &
Note: when running ant test with LucidDB over FoodMart schema, some tests that succeed in derby can report errors, for example, "Transactions not supported". These errors can be ignored,
JVM Choice
All Mondrian tests run by "ant test" target pass with Sun HotSpot JVM; however, not all pass with BEA JRockit. For example, test mondrian.util.ObjectPoolTest.testString fails for both the R26.4 and R27.2 version of JRockit.
Running Individual Tests
runtest.sh
--Ngoodman 21:34, 4 May 2011 (EDT) Appears to no longer working (ivy broke it?) Running "ant test" automatically builds the tests before running them. If you want to explicitly build the test programs, run "ant compile.tests".
If you want to run a single unit test from the command line, use the bin/runtest.sh script file. You'll need to first edit the file to uncomment out the lines corresponding to Derby. Then, to run an individual test, do something like:
bin/runtest.sh -c mondrian.test.BasicQueryTest testSample0
The above example runs the testSample0 test in mondrian.test.BasicQueryTest.
Eclipse
Alternatively, you can run the test within Eclipse, which is what you have to do anyway if you want to debug a problem.
ant test
To run test method testSample0 in mondrian.test.BasicQueryTest, you can also use "ant test" after modifying these two mondrian properties:
mondrian.test.Class=mondrian.test.BasicQueryTest mondrian.test.Name=testSample0
Connecting to Derby Through sqlline
If, as part of debugging, you want to confirm the behavior of a SQL statement in Derby, you can directly connect to your Derby database using sqlline. Put the following lines into a shell script and execute the script file:
java
-cp /home/calvin/open/mondrian/testlib/derby.jar:/home/calvin/open/thirdparty/sqlline.jar
sqlline.SqlLine -d org.apache.derby.jdbc.EmbeddedDriver
-u jdbc:derby:/home/calvin/open/mondrian/demo/derby/foodmart -n sa
Note that the above command needs to be a single line in your script file, so place escape characters at the end of each line if you decide to split it across multiple lines.
Development Tricks and Tips
Regression tests should be run on changes before checkins. There are over a 1000 testcases in the suite so it does provide fairly comprehensive coverage. The tests take about 25 minutes to run against Derby, and a mere 5-10 minutes on MySql, so you can't use lack of time as an excuse for not running the tests. (Note that at one point, JVS was able to get all of the tests to run cleanly against LucidDB. However, they took over an hour to run, so you probably don't want to use LucidDB as your backend database for regression testing purposes.)
Ideally, the tests should be run on multiple backend databases because of subtle syntax and semantic differences between different database vendors. But that isn't always practical and necessary. There is a safety net though. There is a nightly cron job that runs through all tests on the commonly used databases (Oracle,MySql, Derby, etc.). So, if you do happen to break a test on a database platform you didn't test against, you'll be notified :-).
In general, database incompatibility issues aren't very common because Mondrian provides an encapsulation layer for SQL syntax generation in mondrian.rolap.sql.SqlQuery. This should shield you from these issues, provided you use that class appropriately.
Caching behavior can make writing a test tricky. For example, when run by itself, a test may work fine (because the cache is cold), but when run as part of a full regression, it may fail due to pollution from a previous test. Worse, it may only work as part of some full runs, but not others, based on timing, parallelism, memory available, and garbage collection. There are ways to insulate tests from these issues, e.g. by forcing usage of a fresh connection.
Likewise, there are pitfalls due to property settings. The nightly cron job mentioned above uses a randomized matrix of different property settings, so your test may work with the default settings, but fail with some other settings. When this happens, the usual procedure is to add checks to the test to disable it in case the property settings are incompatible. An alternative is to change the test to make it set its required properties explicitly. Tests which change properties must reset them to their previous state before teardown; see BasicQueryTest.testMembersOfLargeDimensionTheHardWay for an example.
Also, certain properties are cached at the schema level, requiring a test to force usage of a fresh connection if it needs to change them. Currently this is only true for the properties which control native evaluation (it's questionable whether they should even be cached at all, but for now, watch out).
Adding New Unit Tests
All Mondrian unit tests are located in the mondrian/testsrc/main directory. Most unit tests operate by executing an MDX query and then comparing the actual result against an expected result that's hard-coded into the test by using the method mondrian.test.TestContext.assertQueryReturns(). It's also possible to just execute and verify an axis expression (mondrian.test.TestContext.assertAxisReturns()) as well an arbitrary expression (mondrian.test.TestContext.assertExprReturns()).
Another test type that I found very useful are a set of tests that verify that elements of an MDX query are natively executed. See mondrian.rolap.NonEmptyTest. For example, there are tests in there that verify that non-empty cross joins are actually being pushed down into SQL.
mondrian.rolap.TestAggregationManager is another interesting test type. Tests in here verify that the SQL query generated by an MDX statement matches an expected SQL string.
There are also some diff-repository type tests. However, these are currently only being used to test XML/A functionality. See testsrc.main.mondrian.xmla.*.
As noted above, the existing Foodmart schema should suffice for most testing. If it's not sufficient, you have the option of extending the schema. However, that only makes sense if the extension is of general use. If it isn't, then you can directly add schema elements in your unit test. For example, mondrian.test.BasicQuery.testBadMeasure1() creates a cube with an invalid measure, while mondrian.test.TestCalculatedMembers.testCalculatedMemberInCubeViaApi() add a "Profit2" measure to the Sales cube.
Mondrian Tracing
There are two types of tracing in Mondrian.
SQL Tracing using java.io.PrintWriter
It prints out every sql statement issued by Mondrian. Turn on this tracing to observe the interaction between Mondrian and the underlying relational storage.
This trace is enabled by setting these mondrian properties:
mondrian.trace.level=1 (or anything > 0) mondrian.debug.out.file=<file location>
log4j Tracing for Debugging
Several Mondrian classes traced are documented here. However, it's easy to add new trace in other classes. To enable log4j tracing for a certain class, here are the steps:
- In log4j.properties file (under open/mondrian), add the class name and associated tracing level. e.g:
log4j.category.mondrian.rolap.RolapUtil=DEBUG
- In mondrian.properties file, set log4j.configuration to the full path of log4j.properties file. e.g:
log4j.configuration=file:///home/calvin/open/mondrian/log4j.xml
With the default log4j.xml, output will go to your console. If you want to redirect the output to a file, e.g., mondrian.log in the same directory, make the following modifications to your log4j.properties file:
Change the setting of log4j.appender.MONDRIAN as follows:
log4j.appender.MONDRIAN=org.apache.log4j.RollingFileAppender
and add the following lines to your log4j.properties file:
log4j.appender.MONDRIAN.File=mondrian.log log4j.appender.MONDRIAN.Append=false log4j.appender.MONDRIAN.layout=org.apache.log4j.PatternLayout
Both log4j.properties and log4j.xml are checked-in files, so if you need to change any of the settings, the best approach is to copy it, edit the copy, and change the log4j.configuration parameter to point to the location of the modified copy. That way, you don't have to worry about remembering to revert it before checkin.
Alternatively, you can also edit the file in a separate changelist, as described earlier when making modifications to mondrian.properties.
Note that the above settings make log4j tracing work with cmdRunner. Eclipse seems to require a bit more setup for log4j tracing to work.
Connecting Mondrian to Other DBs
Occasionally, Mondrian testing requires connecting to DBs other than the default Derby DB. Here are instructions for a few frequently used DBs.
Connecting Mondrian to LucidDB
Connecting to Foodmart
- You'll first need to create a Foodmart schema in LucidDB by running the following commands in sqllineEngine:
$cd ~/open/luciddb/ ~/open/luciddb$ ./sqllineEngine 0: jdbc:luciddb:> create schema foodmart; 0: jdbc:luciddb:> create user foodmart authorization 'Unused' default schema foodmart; 0: jdbc:luciddb:> grant execute on specific procedure applib.compute_statistics_for_schema to foodmart;
- Startup a LucidDB server instance and load the data by running the following commands or adding them into a file, and executing it as a shell script. Remember to copy luciddb/plugin/LucidDbClient.jar into mondrian/testlib, if you haven't already done so.
$cd ~/open/luciddb/ ~/open/luciddb$./lucidDbServer (in another shell) $cd ~/open/mondrian ~/open/mondrian$export CP=lib/eigenbase-properties.jar:lib/eigenbase-resgen.jar:lib/eigenbase-xom.jar ~/open/mondrian$export CP=$CP:lib/mondrian.jar:lib/log4j.jar:testlib/LucidDbClient.jar
(replace calvin with your home directory)
~/open/mondrian$java -Xms500m -Xmx500m -cp $CP \
mondrian.test.loader.MondrianFoodMartLoader \
-verbose -tables -data -indexes \
-jdbcDrivers=com.lucidera.jdbc.LucidDbRmiDriver \
-inputFile=/home/calvin/open/mondrian/demo/FoodMartCreateData.sql \
-outputJdbcURL=jdbc:luciddb:rmi://localhost \
-outputJdbcUser=FOODMART
Escape the lines above if you decide to split it across multiple lines in your shell script.
You might want to run analyze schema after all tables are loaded. Connect to the DB using sqllineClient and issue these two commands:
CALL APPLIB.GRANT_SELECT_FOR_SCHEMA('FOODMART', 'sa');
CALL APPLIB.ESTIMATE_STATISTICS_FOR_SCHEMA('FOODMART');
Note 'sa' is the user connected via sqllineClient.
- Edit your mondrian.properties file as noted in the second bullet item in the previous section, except set the following values instead:
mondrian.test.jdbcUser=FOODMART
mondrian.catalogURL=file:/home/calvin/open/mondrian/demo/FoodMart.xml
mondrian.test.connectString="Provider=mondrian; Jdbc=jdbc:luciddb:rmi://localhost;
JdbcUser=FOODMART; catalog=file:/home/calvin/open/mondrian/demo/FoodMart.xml;
JdbcDrivers=com.lucidera.jdbc.LucidDbRmiDriver"
Note that there's no need to set mondrian.test.jdbcPassword or switch mondrian.rolap.ignoreInvalidMembers to true. Also, note that mondrian.test.connectString needs to be on a single line.
Corresponding instructions for using a packaged build of LucidDB are also published in Eigenpedia, including how to run ANALYZE TABLE on the Foodmart schema.
Connecting Mondrian to MySql
Load Foodmart into MySql
First, use the red-zone thirdparty automatic MySQL setup:
$ cd /home/calvin/depot/platform/main/thirdparty $ make mysql
This will install mysql into depot/platform/main/dev/mysql, and will start it running as a daemon.
Next, you need to set an environment variable (do this in your .bash_profile for new shells too):
$ export MYSQL_HOME=/home/calvin/depot/platform/main/dev/mysql
From depot/platform/main, you can use these commands any time to shut down MySQL or restart it:
$ ant shutdown-mysql-server $ ant start-mysql-server
Before running Mondrian, you have to create a database to hold the FOODMART sample data:
$ cd /home/calvin/depot/platform/main/dev/mysql/bin $ mysqladmin -u root create foodmart $ mysql -u root mysql> grant all privileges on *.* to 'foodmart'@'localhost' identified by 'foodmart'; Query OK, 0 rows affected (0.00 sec)
mysql> quit Bye
To populate the sample data, here's a FoodMartLoader.sh script to run from modrian/ home dir(asuming all jar files are located under the respective directories lib and testlib). All in one line.
java
-cp "./lib/mondrian.jar:./lib/log4j.jar:./lib/commons-logging.jar:./lib/eigenbase-xom.jar:./lib/eigenbase-resgen.jar:
./lib/eigenbase-properties.jar:./testlib/mysql-connector-java-3.1.14-bin.jar"
mondrian.test.loader.MondrianFoodMartLoader
-verbose -tables -data -indexes -jdbcDrivers=com.mysql.jdbc.Driver -inputFile=./demo/FoodMartCreateData.sql
-outputJdbcURL="jdbc:mysql://localhost/foodmart?user=foodmart&password=foodmart"
--Rchen 15:31, 22 May 2009 (PDT)Some times you might get class loading error, even though the jar in path contains the "missing" class. Try changing the order of the .jar appearing in the -cp argument. Also, don't forget to unzip FoodMartCreateData.zip in ./demo.
Before running Mondrian unit tests, edit mondrian.properties accordingly:
mondrian.foodmart.jdbcURL=jdbc:mysql://localhost/foodmart?user=foodmart&password=foodmart mondrian.jdbcDrivers=com.mysql.jdbc.Driver driver.classpath=/home/calvin/depot/platform/main/thirdparty/mysql-connector/mysql-connector-java-3.1.14-bin.jar
Connect Mondrian to MySql
If you want to connect Mondrian to MySql through Eclipse and cmdRunner, do the following:
- Copy mysql-connector-java-3.1.14-bin.jar from your platform thirdparty directory into mondrian/testlib.
- Modify your mondrian.properties file to add the following definitions:
mondrian.foodmart.jdbcURL=jdbc:mysql://localhost/foodmart?user=foodmart&password=foodmart mondrian.jdbcDrivers=com.mysql.jdbc.Driver driver.classpath=/home/calvin/open/mondrian/testlib/mysql-connector-java-3.1.14-bin.jar mondrian.catalogURL=file:/home/rchen/open/mondrian/demo/FoodMart.xml mondrian.test.jdbcURL=jdbc:mysql://localhost/foodmart?user=foodmart&password=foodmart
- Add a reference to testlib/mysql-connector-java-3.1.14-bin.jar in your .classpath file.
See the Installation page on the Mondrian project page for a general description of how to load Foodmart into other databases.
Connecting Mondrian to Oracle
Steps:
- Download and install Oracle 10G XE.
- Copy ojdbc14.jar to the Mondrian testlib directory. (You'll find it in /usr/lib/oracle/xe/app/oracle/product/10.2.0/server/jdbc/lib.)
- Connect to Oracle using the system account, and create a user named foodmart with password foodmart.
- If you've already previously downloaded and installed Oracle, to restart it, execute the following command as root:
/etc/init.d/oracle-xe restart
- Run the loader script after exporting the necessary jars
export CP=lib/commons-dbcp.jar:lib/commons-collections.jar:lib/commons-pool.jar export CP=$CP:lib/eigenbase-properties.jar:lib/eigenbase-resgen.jar:lib/eigenbase-xom.jar export CP=$CP:lib/javacup.jar:lib/mondrian.jar export CP=$CP:lib/log4j.jar export CP=$CP:testsrc/main export CP=$CP:classes export CP=$CP:testclasses export CP=$CP:testlib/ojdbc14.jar
java \ -cp $CP mondrian.test.loader.MondrianFoodMartLoader -verbose -tables -data -indexes \ -jdbcDrivers="oracle.jdbc.driver.OracleDriver" -inputFile=../demo/FoodMartCreateData.sql \ -outputJdbcURL="jdbc:oracle:thin:foodmart/foodmart@localhost:1521"
