Test Cleanup
Test Cleanup Project
See the mission statement mail and slides from a FOSDEM'10 talk on the subject.
User Interface
CppUnit
For tests in C++, plain CppUnit (at least version 1.12.1) is available. (OOo bundles CppUnit 1.12.1 as an external module; alternatively, you can configure --with-system-cppunit
.)
Implement the test code as a dynamic library, following CppUnit conventions (i.e., using CPPUNIT_TEST_SUITE_REGISTRATION
and CPPUNIT_PLUGIN_IMPLEMENT
macros). The following settings are needed in the makefile.mk
:
ENABLE_EXCEPTIONS = TRUE
CFLAGSCXX += $(CPPUNIT_CFLAGS)
SHLnRPATH = NONE
SHLnSTDLIBS += $(CPPUNITLIB)
SHLnVERSIONMAP = version.map
, where that file must contain the following (see details below for improving this):
UDK_3_0_0 { global: cppunitTestPlugIn; local: *; };
To run tests, sal/cppunittester
delivers a cppunittester
executable that has some advantages over the plain CppUnit DllPlugInTester
:
- It uses
SAL_IMPLEMENT_MAIN
to properly set up the per-process settings required by URE. Test code as well as tested code based on URE will expect this and would fail in a process that is not set up in this way. - Getting CppUnit to build as an external module within the Windows OOo build is a non-trivial task. The less parts of it that need to be built, the better.
Something like the following in makefile.mk
will execute the tests:
ALLTAR : test test .PHONY : $(SHL1TARGETN) $(CPPUNITTESTER) $(SHL1TARGETN)
$(CPPUNITTESTER)
internally uses $(AUGMENT_LIBRARY_PATH_LOCAL)
to favor library instances in the local output tree over ones in solver. Alternatively to the above, .INCLUDE: _cppunit.mk
calls $(CPPUNITTESTER)
on each of $(SHL1TARGETN)
, …, $(SHL10TARGETN)
that is defined.
JUnit
For tests in Java, plain JUnit (at least version 4) is available. (OOo requires JUnit as a prerequisite, see junit.org for downloads; you can configure --with-junit=…
if it is not automatically found; alternatively, you can configure --without-junit
, disabling JUnit-based tests.)
subsequenttests
Some tests have build and/or run time dependencies on modules further up in the hierarchy, or modules that the rest of the module in question does not have dependencies on (and which are thus not listed in the module's build.lst
). The extreme case are tests that need a complete OOo installation as their test environment.
For such tests, the whole makefile.mk
should be wrapped in:
.IF "$(OOO_SUBSEQUENT_TESTS)" == "" nothing .PHONY : .ELSE ... .END
(and the directory included in the module's build.lst
).
The idea is to do a complete OOo build first (cd smoketestoo_native && build --all
) and then build and execute those tests either selectively or wholesale.
Selectively building a specific such test directory can be done with cd … && OOO_SUBSEQUENT_TESTS=TRUE dmake
.
To build and execute all these test directories, the tool subsequenttests
(in solenv/bin
) exists. It scans the build.lst
files of all modules for makefile.mk
s that mention OOO_SUBSEQUENT_TESTS
and simply executes dmake
in all those directories, with OOO_SUBSEQUENT_TESTS
set. It supports a -Pn
switch to build n directories in parallel.
Helpers
A new module test
shall collect reusable code snippets for C++ and Java tests.
For now, it concentrates on letting a test connect to an OOo instance via remote UNO: A C++ test class can hold a test::OfficeConnection
as class member and, in its CppUnit setUp
and tearDown
functions, call OfficeConnection::setUp
and OfficeConnection::tearDown
, respectively. Tests obtain access to the OOo instance via OfficeConnection::getFactory
. Similarly, a Java test class can use an org.openoffice.test.OfficeConnection
instance, offering the same functionality as its C++ counterpart.
For this to work, three additional steps are necessary:
- First, in
instsetoo_native
, building an installation set for theopenoffice
product in the default language (en-US
) in any package format now takes care to always also build an archive package format (tar.gz
resp.zip
) of that product. - Second, building
smoketestoo_native
permanently installs that archive into the solver. (Except on Windows, where problems with resulting long pathnames prevent this; there, as a workaround, each test individually unzips the archive to some local temporary directory.) - Third, a test that requires an OOo instance includes
installationtest.mk
in itsmakefile.mk
. That file makes sure to call the CppUnit and JUnit test runners with the information necessary for the helper classes to start the correct OOo instance with a freshUserInstallation
and set up an URP pipe connection. The information is tunneled into the test runner executables as UNO bootstrap parameters.
unoapi
Tests
The unoapi
tests are redesigned to use the new test framework. They are plain JUnit tests and use the mechanisms described above to connect to running OOo instances. However, due to the large amount of logic encoded in qadevOOo
, those JUnit tests for now are only thin wrappers around org.openoffice.Runner.run
. This is intended to change over time.
The cwscheckapi
tool to run the unoapi
tests has been subsumed by subsequenttests
. The latter always runs all tests, while the former had mechanisms to run only tests for modules that had been changed by a CWS. The hope is that the parallel mode of subsequenttests
makes it fast enough so that running all tests all of the times becomes feasible.
Unfortunately, running the unoapi
tests is still not stable enough to give useful results. Therefore, running subsequenttests
has been disabled in the build bots for now. Making the tests sufficiently stable is ongoing work.
Details
Working on CWS sb118 , CWS sb120 .
CppUnit 1.12.1
- Replace existing
cppunit
andtestshl2
combo with latest CppUnit 1.12.1. - OOo
configure --with-system-cppunit
(for now only if also--with-system-stl
!); minimal required version for now arbitrarily 1.12.1. - TODO: Upstream
PATCH_FILES
(windows.patch
?!?). - TODO:
RPATH
in generated libraries and executables. - TODO: Improve
CPPUNIT_PLUGIN_EXPORT
and get rid of version maps (GCC exception problems?). - TODO:
cppunit/makefile.mk
MY_LIBS = -lm
Solaris hack. - TODO: In general OOo build environment, no longer set
CFLAGS
etc. as environment variables; have package-specificOOO_package_CFLAGS
etc. and combine those intoCFLAGS
etc. where necessary (seeOOO_STLPORT_CXXFLAGS
etc. incppunit/makefile.mk
). - TODO: Why use
cppunittester
instead ofDllPlugInTester
? (Remove building/deliveringDllPlugInTester
.) - TODO: Make
operator <<(std::basic_ostream &, rtl::OUString const &)
fromtest/oustringstreaminserter.hxx
generally available? - TODO: Make STLport
num_put
symbols available on platforms other than Linux (seesolenv/bin/addsym.awk
), too, so thatCPPUNIT_ASSERT_EQUAL
actually outputs numeric values.
testshl2
solenv/inc_cppunit.mk
/cppunit.mk
: “adaptedcppunit.mk
tocppunittester
and no longer require ‘dmake test
’ to run tests”.- TODO:
DEV300_m68
introducedTESTSHL2LIB
(which I removed again) and uses it in manyqa
makefiles (that I need to touch someday, anyway).
Unit Tests
OOO_SUBSEQUENT_TESTS
(build.lst
lists local dependencies of those directories, so that modifying a module and then calling “OOO_SUBSEQUENT_TESTS=x build
” rebuilds the module and executes the tests).- TODO: If Build Environment 2.0 would build directly to solver (instead of local output tree/deliver), that would simplify tests that currently need to take care to test local libraries instead of solver ones (see, for example,
OOO_TEST_PREFIX
in theservices.rdb
ofstoc/test/uriproc/makefile.mk
). - TODO: Constructs like “
-env:UNO_…=$(my_file)…
” would fail for problematic characters (spaces!) in paths.
Smoke Test
- Redesign
smoketestoo_native
to use CppUnit and pre-installed OOo. - TODO: For
smoketestoo_native/makefile.mk
“-env:arg-env=$(OOO_LIBRARY_PATH_VAR)"$${{$(OOO_LIBRARY_PATH_VAR)+=$$$(OOO_LIBRARY_PATH_VAR)}}"
” mechanism:osl_executeProcess
ustrEnvironments
members without “=
”: all platforms; documentation; documentation ofosl_executeProcess_WithRedicrectedIO
. - TODO: Dynamically create
smoketestdoc.sxw
? - TODO: “changed
tools/debug
to never fall back from writing to shell to writing to file, and readDBGSV_INIT
fromrtl::Bootstrap
; the latter causes problems iftools/debug
is called from static code executing beforeSAL_MAIN
has calledosl_setCommandArgs
→ removed statics from code and added assertions toosl/process
; various improvements tosmoketests.cxx
(incl. passing an appropriatedbgsv.ini
for non-pro builds)”: Instead of-env:DBGSV_INIT=
insmoketest.cxx
(and supportingDBGSV_INIT
as bootstrap variable), globally in the build environment letDBGSV_INIT
environment variable point to adbgsv.ini
witherror=coredump
, to catch all assertions thrown during build time (incl. ones from tests like smoke test; see Issue 109142 ). - TODO:
-headless
? (appears to still display windows on Windows and Mac OS X, however, see Issue 109052 )
Pre-Installed OOo
- For Sun Hamburg Release Engineering, take MWS archive installation sets from “S:” instead of solver: “take installation set from
SHIPDRIVE
if appropriate”. - TODO:
PKGFORMAT=installed
instead ofarchive
?
unoapi
Tests
- Known issues:
- Issue 109517 “sc/qa/unoapi fails for ScAnnotationObj etc.”
- Issue 109518 “sd/qa/unoapi crash in SfxViewShell::GetController”
- Issue 109643 “toolkit/qa/unoapi tests fail when losing focus”
- Issue 109728 “~ScAutoFormatObj uses cleared ScGlobal”
- Issue 109770 “ChartView::getExplicitValuesForAxis accessing destroyed VCoordinateSystem”
- Issue 109917 “fragile unoapi test using OnUnfocus event”
- Issue 109939 “toolkit: inconsistent UnoControl locking schemes”
- Issue 110253 “chart2: chart::WrappedPropertySet::setPropertyValue crash”
- Issue 110862 “sc/qa/unoapi fails for sc.ScDataPilotTableObj::com::sun::star::sheet::XDataPilotTable::refresh()”
- Issue 110988 “svtools/qa/unoapi fails for svtools.AccessibleBrowseBox etc.”
- Issue 111006 “toolkit: deadlock between UnoControls”
- Issue 111032 “sc/qa/unoapi sc.ScAccessibleCell::com::sun::star::accessibility::XAccessibleText getCharacterBounds() failure”
- Issue 111042 “sd/qa/unoapi sd::DrawController::dispose crash”
- Issue 111043 “sd/qa/unoapi sd::DrawController_OutlineView NullPointerException”
- Issue 111047 “toolkit/qa/unoapi VCLXAccessibleMenuItem::doAccessibleAction deadlock”
- Issue 111076 “vcl: [NSExtraMIData unregisterMouseEventListener:] crash”
- Issue 111102 “chart2: ChartModelHelper::findDiagram returns null”
- Issue 111111 “xmloff.Impress.XMLImporter::com::sun::star::document::XImporter”
- Issue 111113 “toolkit.AccessibleStatusBarItem unoapi failure”
- Issue 111114 “svx.AccessiblePresentationOLEShape unoapi failure”
- Issue 111148 “forms/qa/unoapi forms.OImageControlModel::com::sun::star::beans::XMultiPropertySet deadlock”
- Issue 111169 “svx.AccessiblePageShape unoapi failure”
- Issue 111170 “sun/i18n/qa/unoapi "creating a drawdoc" failure”
- Issue 111178 “sw/qa/unoapi SwView::SelectShell crash”
- Issue 111180 “framework/qa/unoapi fwk.Desktop::com::sun::star::frame::XComponentLoader::loadComponentFromURL() failure”
- Issue 111182 “Desktop::terminate -> ~DropTarget -> _objc_error crash”
- Issue 111184 “sal/qa/osl/pipe uses fixed pipe names”
- Issue 111185 “sw/qa/unoapi: various failures”
- Issue 111186 “sw/qa/unoapi sw.PageStyle::com::sun::star::beans::XPropertySet::setPropertyValue() failure”
- Issue 111190 “sw/qa/unoapi sw.SwXShape failure”
- Issue 111194 “sw/qa/unoapi sw.XMLExporter crash”
- Issue 111195 “toolkit/qa/unoapi toolkit.AccessibleScrollBar::com::sun::star::accessibility::XAccessibleValue::setCurrentValue() failure”
- Issue 111197 “sw/qa/unoapi sw.SwAccessibleDocumentPageView failure”
- Issue 111199 “sd/qa/unoapi sd.AccessibleOutlineView failure”
- Issue 111200 “xmloff/qa/unoapi xmloff.Draw.XMLMetaImporter failure”
- Issue 111216 “svx/qa/unoapi accessibility deadlock”
- Issue 111218 “sw/qa/unoapi sw.SwAccessibleParagraphView::com::sun::star::accessibility::XAccessibleEventBroadcaster::removeEventListener() failure”
- Issue 111220 “sw/qa/unoapi sw.XMLContentExporter::com::sun::star::document::XFilter::cancel() failure”
- Issue 111224 “xmloff/qa/unoapi xmloff.Impress.XMLContentExporter crash”
- Issue 111225 “toolkit/qa/unoapi toolkit.AccessibleStatusBar failure”
- Issue 111252 “svx/qa/unoapi svx.AccessibleOLEShape deadlock”
- Issue 111273 “sw/qa/unoapi sw.SwXTextEmbeddedObject::com::sun::star::document::XEmbeddedObjectSupplier::getEmbeddedObject() crash”
- Issue 111278 “svx/qa/unoapi svx.AccessibleEditableTextPara failure”
- Issue 111283 “sfx2/qa/unoapi sfx.StandaloneDocumentInfo failure”
- Issue 111285 “forms/qa/unoapi forms.OButtonModel failure”
- Missing tests:
- Issue 109564 “qadevOOo/tests/java/ifc/awt/_XSystemChildFactory.java”
- Sporadic failures:
- Issue 100159 “unexpected exception from oooimprovement during shut down”
- Issue 111166 “unexpected exception from InitUpdateCheckJobThread during shut down”
- Issue 111217 “BasicManager::SetGlobalUNOConstant crash during shutdown”
- Issue 111223 “cppu_threadpool::ThreadAdmin::remove during shutdown”
- Issue 111246 “svx::FontHeightToolBoxControl::statusChanged crash (memory corruption)”
- Issue 111256 “vcl: AquaSalFrame::initWindowAndView crash during loadComponentFromURL”
- Issue 111271 “Mac OS X: crash during shutdown, __tcf_2 on stack”