At the time of this writing, our unit tests are in bad shape because of a couple of reasons:
- We have to duplicate a lot of data between multiple module-specific DefaultTestData.sql files
- There is no common paradigm between all of the different test harnesses related to how test data is loaded
- Our KIM data and some other data is incomplete in our test harnesses
- There is a specific issue where some system parameters are required to exist for startup but our suite-level test harness setup does not allow for us to load those via SQL before the Rice jetty server is started.
During the @Before annotation on the unit tests, we have two different peices that run.
The first is the perSuiteLifecycles. These only run once per test run (a static boolean is used to determine if they have already executed or not).
The second is the perTestLifecycles. These execute prior to every single test.
What follows are proposals for how we can modify the existing test harness code in each of these two cases to hopefully help address the issues we are having.
- Start up spring context which contains database connections (TestHarnessSpringBeans.xml)
- Clear the database using ClearDatabaseLifecycle
- Execute a global suite-level SQL import, should include
- system parameters (KRNS_PARM_TYP_T, KRNS_PARM_DTL_TYP_T, KRNS_PARM_T)
- namespaces (KRNS_NMSPC_T)
- kim data (all tables begginning with KRIM)
- quartz lock table data (KRSB_QRTZ_LOCKS)
- Execute a module-specific suite-level SQL import (i.e. the KNS tests have some stuff in them about the TRAV tables). Or maybe just include the different suite level sql from all the modules into the global default file? Might be easier to do that.
- Module Specific - Start the Rice Jetty Server to initialize Rice inside of the test harness
Per Test Level
- If the module needs to clear the database on every test (i.e. KEW) then execute a ClearDatabaseLifecycle - for this we need to be sure to exclude the tables that are populated at the suite level!!! Perhaps a configuration file with file names of tables not to clear would best serve the purpose here? The trick here is to determinewhether or not it will differ across the different modules
- Execute default per test sql
- Ingest default per test XML
- Call into test's loadTestData to allow it to load any additional test data
- migrate all of our standard groups and principals/entities out of XML files (DefaultTestData.xml) into the suite-level SQL file
- export all KIM data from our master database as SQL statements and include in our suite-level SQL file (do this manually for now, in the future, would be nice to figure out an automated mechanism for doing this)
- export all namespaces and system parameter data from our master database into the master suite-level sql file (also, do this manually for now)
- Update the perSuiteLifecycles so that they load this SQL.
- The way these lifecycles are handled is currently vastly different in some cases for each of the test harnesses (KEW vs. KNS vs. KSB, etc.) It would be nice to bring some consistency across these if we could. My suggesstion would be that we start by trying to apply these changes to a single module first (i.e. the KEW module) and then work our way from there.
- ...? I'm sure there is something that I'm missing here, but this is a rough cut
- It will be very important that unit tests are careful if they modify any of the suite-level test data. Otherwise is could cause issues for downstream tests. We will need to document this in our unit testing guidelines because there are bound to be tests that will be required to modify suite-level test data. In those cases there are going to be a couple of options:
- They use a test that executes in a Tx and rolls back at the end (this will only work if the test is single-threaded! Some of the KSB tests are not)
- They manually clean up any changes they make to suite test data
- Without automating it, it will likely be difficult to keep the suite-level test data in sync with the KIM data that's in the database. There are a couple of solutions we could look at for this long term:
- Follow the KFS model where the CI database contains a knows dataset
- Come up with an automated process for updating the suite-level test sql file(s) in the project