Difference between revisions of "QA/BVT"
Phoenixwang (talk | contribs) |
|||
(20 intermediate revisions by 4 users not shown) | |||
Line 1: | Line 1: | ||
= Introduction = | = Introduction = | ||
− | Build Verification Testing, BVT for short, is a set of tests that run on new build to verify that whether the build is testable or not. It is done prior to its release to test team for further testing. This testing is done for Build Validation and Build Acceptance.<br /> | + | Build Verification Testing, BVT for short, is a set of tests that run on new build to verify that whether the build is testable or not. It is done prior to its release to test team for further testing(So get latest build for test is more meaningful). This testing is done for Build Validation and Build Acceptance.<br /> |
− | {{ | + | {{Tip|'''Why do we need BVT?'''<br /> |
BVT is a type of regression test and only take short time, to verify that whether the build is stable and can be tested thoroughly. BVT can save the efforts of testers to setup and test a build when major functions are having defects. If any of the tests fails, BVT will report to developers immediately to remind relevant developers to review source codes. | BVT is a type of regression test and only take short time, to verify that whether the build is stable and can be tested thoroughly. BVT can save the efforts of testers to setup and test a build when major functions are having defects. If any of the tests fails, BVT will report to developers immediately to remind relevant developers to review source codes. | ||
}} | }} | ||
Line 9: | Line 9: | ||
Test machine with OS in below list: | Test machine with OS in below list: | ||
WindowsXP, Windows7 32bit, Windows7 64bit,Ubuntu, Suse, Redhat,Mac | WindowsXP, Windows7 32bit, Windows7 64bit,Ubuntu, Suse, Redhat,Mac | ||
+ | ==AOO Build== | ||
+ | AOO4.0 [http://ci.apache.org/projects/openoffice/index.html build snapshots]<br/> | ||
+ | Below are builds for AOO4.0 trunk | ||
+ | *Linux 32bit Nightly in buildbots table | ||
+ | *Linux 64bit Nightly in buildbots table | ||
+ | *Windows Nightly in buildbots table | ||
==Environment tools== | ==Environment tools== | ||
*SVN Client: | *SVN Client: | ||
On Windows: Download and Install TortoiseSVN as SVN Client to checkout sourcecode.<br/> | On Windows: Download and Install TortoiseSVN as SVN Client to checkout sourcecode.<br/> | ||
[http://tortoisesvn.net/downloads.html http://tortoisesvn.net/downloads.html]<br/> | [http://tortoisesvn.net/downloads.html http://tortoisesvn.net/downloads.html]<br/> | ||
− | *JDK: | + | *JDK: [http://java.sun.com/javase/6/ JDK 1.6] or above.<br /> |
*Ant: [http://ant.apache.org/bindownload.cgi Apache Ant 1.8.2] or above. | *Ant: [http://ant.apache.org/bindownload.cgi Apache Ant 1.8.2] or above. | ||
− | |||
==Soft Skills== | ==Soft Skills== | ||
*Basic skills about Windows Command line, Linux/Mac Terminal | *Basic skills about Windows Command line, Linux/Mac Terminal | ||
*Basic knowledge of Java | *Basic knowledge of Java | ||
− | =How to run BVT= | + | |
+ | =How to run BVT automatically= | ||
== Getting the source code == | == Getting the source code == | ||
Anyone can checkout source code from our Subversion repository. Run the following command. | Anyone can checkout source code from our Subversion repository. Run the following command. | ||
svn co https://svn.apache.org/repos/asf/openoffice/trunk/test/ test | svn co https://svn.apache.org/repos/asf/openoffice/trunk/test/ test | ||
+ | |||
+ | On Windows, you can use TortoiseSVN GUI Client to checkout, URL of repository is https://svn.apache.org/repos/asf/openoffice/trunk/test/ | ||
== Run testing == | == Run testing == | ||
'''As a normal user, how to run testing against an installed OpenOffice?'''<br /> | '''As a normal user, how to run testing against an installed OpenOffice?'''<br /> | ||
Line 42: | Line 50: | ||
result.html: Test result in HTML. Collect all case's result, to report as HTML. | result.html: Test result in HTML. Collect all case's result, to report as HTML. | ||
screenshot/: Screenshot pictures when test assert is failed. | screenshot/: Screenshot pictures when test assert is failed. | ||
+ | ==Sample on Windows== | ||
+ | Running Platform: Windows 7 64bit | ||
+ | *Install OO, use default install directory. "C:/Program Files (x86)/OpenOffice.org 3" | ||
+ | "OOo-dev 3.5 (en-US) Installation Files" is directory for install files. not default installation directory. | ||
+ | *Checkout sourcecode to "C:\test" | ||
+ | *Change CMD directory to "C:\test",then compile test use: ant -Dopenoffice.home="C:/Program Files (x86)/OpenOffice.org 3" compile | ||
+ | *Change CMD directory to "C:\test",then execute BVT: run -Dopenoffice.home="C:/Program Files (x86)/OpenOffice.org 3" -tp bvt | ||
==FAQ== | ==FAQ== | ||
− | {{ | + | {{Note| '''What is openoffice.home?'''<br /> |
The default installation directory of Apache OpenOffice:<br/> | The default installation directory of Apache OpenOffice:<br/> | ||
− | Mac OS X: " | + | Mac OS X: "/Applications/OpenOffice.org.app/Contents"<br /> |
− | Linux: " | + | Linux: "/opt/openoffice.org3"<br /> |
− | Windows: " | + | Windows: "C:/Program Files/OpenOffice.org 3"<br /> |
− | Windows 64b: " | + | Windows 64b: "C:\Program Files (x86)\OpenOffice 4" |
}} | }} | ||
− | {{ | + | {{Note| '''Can I do other operations when BVT is running?'''<br /> |
For BVT cases wrote via GUI(Graphical User Interface) not UNO interface, little display offset may impact test result. So not do other operations when BVT is running. | For BVT cases wrote via GUI(Graphical User Interface) not UNO interface, little display offset may impact test result. So not do other operations when BVT is running. | ||
− | |||
}} | }} | ||
− | + | {{Note| '''Do I need compile bvt source code anytime?'''<br /> | |
− | =Process= | + | If there is not any change of bvt source code, there is no need to compile it anytime. |
− | 1 | + | That is "Java is compile once, run everywhere" |
− | + | }} | |
− | * Platform: Platform which you can run BVT.<br/> | + | =Task Process= |
− | * Volunteer: Volunteer's name | + | 1)Track your execution task on [[http://wiki.openoffice.org/w/index.php?title=QA/BVT_Report BVT Report ]]<br/> |
+ | * Platform: Platform which you can run BVT on it.<br/> | ||
+ | * Volunteer: Volunteer's name who take this execution task.<br/> | ||
* Revision: build version of AOO4.0<br/> | * Revision: build version of AOO4.0<br/> | ||
* Status: Pass/Fail<br/> | * Status: Pass/Fail<br/> | ||
− | * | + | * Issue Number:Issue Number on BZ which failed BVT case .<br/> |
{| class="wikitable" border="1" | {| class="wikitable" border="1" | ||
|- | |- | ||
Line 71: | Line 87: | ||
! Volunteer | ! Volunteer | ||
! Status | ! Status | ||
− | ! | + | ! Issue Number |
|- | |- | ||
| | | | ||
Line 117: | Line 133: | ||
| | | | ||
|} | |} | ||
− | + | 2)Update your execution result on "Status" column, and add detail HTML result in new link(Record link on [[http://wiki.openoffice.org/w/index.php?title=QA/BVT_Report BVT Report ]] ).<br/> | |
− | + | 3)If have fail/error case, you need check it manually. To confirm it's exact a bug. (For some BVT cases write via GUI(Graphical User Interface), it have some offset).<br/> | |
In this step, you need study to read cases. [[http://wiki.openoffice.org/wiki/QA/dev_gui_testing Dev GUI Test]] [[http://wiki.openoffice.org/wiki/QA/vclauto/A_step_by_step_example A step by step example]]<br/> | In this step, you need study to read cases. [[http://wiki.openoffice.org/wiki/QA/dev_gui_testing Dev GUI Test]] [[http://wiki.openoffice.org/wiki/QA/vclauto/A_step_by_step_example A step by step example]]<br/> | ||
− | + | 4)If there is a defect, you can open it on [[https://issues.apache.org/ooo/ Bugzilla]],select specific editor as it's category, then record on [[http://wiki.openoffice.org/w/index.php?title=QA/BVT_Report BVT Report ]]<br/> | |
+ | 5)If you finish all steps above, send your result to QA mail list with your report link.<br/> | ||
+ | =Appendix= | ||
+ | ==BVT Case Detail== | ||
+ | Currently, BVT include 2 classes. These test cases are based on VCLAuto, and verify main functionalities. | ||
+ | #Integrating smoke test from AOO. | ||
+ | #Test cases cover 5 products: word processor, spreadsheet, presentation, drawing, and equation editor. | ||
+ | #Testing main operations about each products: new, save, load, reopen. | ||
+ | #Objects operations: picture, chart, table, function. | ||
+ | #Other test points: export as pdf, print, about dialog, slide show. | ||
+ | |||
+ | Below is the detailed introduction of each case. | ||
+ | |||
+ | ===FileTypeTest.java=== | ||
+ | {| class="wikitable" border="1" | ||
+ | |- | ||
+ | | | ||
+ | <center>'''Test Point'''</center> | ||
+ | |||
+ | | | ||
+ | <center>'''Function Description '''</center> | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testSaveNewODT | ||
+ | |||
+ | testSaveNewOTT | ||
+ | |||
+ | testSaveNewSXW | ||
+ | |||
+ | testSaveNewSTW | ||
+ | |||
+ | testSaveNewDOC | ||
+ | |||
+ | testSaveNewTXT | ||
+ | |||
+ | | | ||
+ | * Create a new text document. | ||
+ | * Input some texts by keyboard. | ||
+ | * Set the texts style. | ||
+ | * Save the document. One case is corresponding to one file type. | ||
+ | * Close the document. | ||
+ | * Reopen the saved file. | ||
+ | * Verify if the text still exists in the file. | ||
+ | |- | ||
+ | | | ||
+ | testSaveNewODS | ||
+ | |||
+ | testSaveNewOTS | ||
+ | |||
+ | testSaveNewSXC | ||
+ | |||
+ | testSaveNewSTC | ||
+ | |||
+ | testSaveNewXLS | ||
+ | |||
+ | | | ||
+ | * Create a new spreadsheet document. | ||
+ | * Input some texts into a cell by keyboard. | ||
+ | * Set style of the texts. | ||
+ | * Save the document. One case is corresponding to one file type. | ||
+ | * Close the document. | ||
+ | * Reopen the saved file. | ||
+ | * Verify if the text still exists in the file. | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testSaveNewODP | ||
+ | |||
+ | testSaveNewOTP | ||
+ | |||
+ | testSaveNewPPT | ||
+ | |||
+ | testSaveNewPOT | ||
+ | |||
+ | testSaveNewSXI | ||
+ | |||
+ | testSaveNewSTI | ||
+ | |||
+ | | | ||
+ | * Create a new presentation document. | ||
+ | * Input some texts by keyboard. | ||
+ | * Save the document. One case is corresponding to one file type. | ||
+ | * Close the document. | ||
+ | * Reopen the saved file. | ||
+ | * Verify if the text still exists in the file. | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testSaveNewODG | ||
+ | |||
+ | testSaveNewOTG | ||
+ | |||
+ | testSaveNewSXD | ||
+ | |||
+ | testSaveNewSTD | ||
+ | |||
+ | | | ||
+ | * Create a new draw document. | ||
+ | * Insert a picture from file. | ||
+ | * Save the document. One case is corresponding *to one file type. | ||
+ | * Close the document. | ||
+ | * Reopen the saved file. | ||
+ | * Verify if the picture still exists in the file. | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testSaveNewODF | ||
+ | |||
+ | testSaveNewSXM | ||
+ | |||
+ | testSaveNewMML | ||
+ | |||
+ | | | ||
+ | * Create a new formula document. | ||
+ | * Insert a formula. | ||
+ | * Save the document. One case is corresponding to one file type. | ||
+ | * Close the document. | ||
+ | * Reopen the saved file. | ||
+ | * Verify if the formula still exists in the file. | ||
+ | |||
+ | |} | ||
+ | |||
+ | |||
+ | ===BasicFunctionTest.java=== | ||
+ | {| class="wikitable" border="1" | ||
+ | |- | ||
+ | | | ||
+ | Test Point | ||
+ | |||
+ | | | ||
+ | Function Description | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | SmokeTest | ||
+ | |||
+ | | | ||
+ | *Create documents: word processor, spreadsheet, presentation,drawing, and equation editor. | ||
+ | *Save above documents to supported types. e.g. Save wordprocessor as .odt, .sxw files. | ||
+ | *Close above documents. | ||
+ | *Open a database, insert/delete/seek data in the database,close the database. | ||
+ | *Install and uninstall an extension. | ||
+ | *Write test results in log files. | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testExportAsPDF | ||
+ | |||
+ | | | ||
+ | *Open an exist .odt file and export it as pdf by menu operation. | ||
+ | |||
+ | *Use toolbar to export a file to pdf file. | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testPrinter | ||
+ | |||
+ | | | ||
+ | Check if print file dialog exist | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testRunMacro | ||
+ | |||
+ | | | ||
+ | Run basic Macro | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testHelp | ||
+ | |||
+ | | | ||
+ | Test the About Dialog show | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testInsertPictureInDocument | ||
+ | |||
+ | | | ||
+ | Test inserting a picture in text document | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testInsertPictureInSpreadsheet | ||
+ | |||
+ | | | ||
+ | Test inserting a picture in spreadsheet document | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testInsertPictureInPresentation | ||
+ | |||
+ | | | ||
+ | Test inserting a picture in presentation document | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testSlideShow | ||
+ | |||
+ | | | ||
+ | Check slide show correctly | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testFind | ||
+ | |||
+ | | | ||
+ | *Open an existing .odt file. | ||
+ | |||
+ | *Find a text in the file. | ||
+ | |||
+ | *Replace a text in the file. | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testFillInSpreadsheet | ||
+ | |||
+ | | | ||
+ | Test automatic filling function of spreadsheet. | ||
+ | |||
+ | Fill Down/Right/Up/Left/Series... | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testSort | ||
+ | |||
+ | | | ||
+ | Test sort function of spreadsheet. | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testInsertChartInDraw | ||
+ | |||
+ | | | ||
+ | Test inserting a chart in a draw document. | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testInsertChartInDocument | ||
+ | |||
+ | | | ||
+ | Test inserting a chart in a text document | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testInsertChartInSpreadsheet | ||
+ | |||
+ | | | ||
+ | Test inserting a chart in a spreadsheet document. | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testInsertChartInPresentation | ||
+ | |||
+ | | | ||
+ | Test inserting a chart in a presentation document. | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testInsertTableInDraw | ||
+ | |||
+ | | | ||
+ | Test inserting a table in a draw document. | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testInsertTableInDocument | ||
+ | |||
+ | | | ||
+ | Test inserting a table in a text document | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testInsertTableInPresentation | ||
+ | |||
+ | | | ||
+ | Test inserting a table in a presentation document | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testSumInFormulaBar | ||
+ | |||
+ | | | ||
+ | Test inserting a function in a spreadsheet document via Sum button. | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testInsertFunctionViaFormulaBar | ||
+ | |||
+ | | | ||
+ | Test inserting a function in a spreadsheet document via input line. | ||
+ | |||
+ | |- | ||
+ | | | ||
+ | testFunctionWizardInFormulaBar | ||
+ | |||
+ | | | ||
+ | Test inserting a function in a spreadsheet document via Function Wizard Dialog | ||
+ | |||
+ | |} | ||
=Reference= | =Reference= | ||
*[http://wiki.openoffice.org/wiki/QA/test_automation_guide Test Automation Guide] | *[http://wiki.openoffice.org/wiki/QA/test_automation_guide Test Automation Guide] | ||
*[http://wiki.openoffice.org/wiki/QA/BVT_Report BVT Report] | *[http://wiki.openoffice.org/wiki/QA/BVT_Report BVT Report] | ||
+ | *[http://ci.apache.org/projects/openoffice/install/ AOO Build] | ||
+ | |||
+ | [[Category:Quality Assurance]] |
Latest revision as of 10:04, 16 July 2018
Introduction
Build Verification Testing, BVT for short, is a set of tests that run on new build to verify that whether the build is testable or not. It is done prior to its release to test team for further testing(So get latest build for test is more meaningful). This testing is done for Build Validation and Build Acceptance.
Prerequisites
Platform
Test machine with OS in below list: WindowsXP, Windows7 32bit, Windows7 64bit,Ubuntu, Suse, Redhat,Mac
AOO Build
AOO4.0 build snapshots
Below are builds for AOO4.0 trunk
- Linux 32bit Nightly in buildbots table
- Linux 64bit Nightly in buildbots table
- Windows Nightly in buildbots table
Environment tools
- SVN Client:
On Windows: Download and Install TortoiseSVN as SVN Client to checkout sourcecode.
http://tortoisesvn.net/downloads.html
- JDK: JDK 1.6 or above.
- Ant: Apache Ant 1.8.2 or above.
Soft Skills
- Basic skills about Windows Command line, Linux/Mac Terminal
- Basic knowledge of Java
How to run BVT automatically
Getting the source code
Anyone can checkout source code from our Subversion repository. Run the following command.
svn co https://svn.apache.org/repos/asf/openoffice/trunk/test/ test
On Windows, you can use TortoiseSVN GUI Client to checkout, URL of repository is https://svn.apache.org/repos/asf/openoffice/trunk/test/
Run testing
As a normal user, how to run testing against an installed OpenOffice?
Firstly compile the project with the command under test module.
ant -Dopenoffice.home="OpenOffice installation directory" compile
Then start testing with the command "run". e.g. Run all test classes under package bvt
run -Dopenoffice.home="/Applications/OpenOffice.org.app/Contents" -tp bvt
e.g. Run the given test classes
run -Dopenoffice.home="/Applications/OpenOffice.org.app/Contents" -tc bvt.gui.BasicFunctionTest
Where to get the testing result?
By default, the testing output is stored in "test/testspace/output***". Open "test/testspace/output/result.html" in your browser to see the testing report.
Special stuff in the output directory.
result.xml: Test result in XML. Every case's detail result, may include case error stack. result.html: Test result in HTML. Collect all case's result, to report as HTML. screenshot/: Screenshot pictures when test assert is failed.
Sample on Windows
Running Platform: Windows 7 64bit
- Install OO, use default install directory. "C:/Program Files (x86)/OpenOffice.org 3"
"OOo-dev 3.5 (en-US) Installation Files" is directory for install files. not default installation directory.
- Checkout sourcecode to "C:\test"
- Change CMD directory to "C:\test",then compile test use: ant -Dopenoffice.home="C:/Program Files (x86)/OpenOffice.org 3" compile
- Change CMD directory to "C:\test",then execute BVT: run -Dopenoffice.home="C:/Program Files (x86)/OpenOffice.org 3" -tp bvt
FAQ
Do I need compile bvt source code anytime? If there is not any change of bvt source code, there is no need to compile it anytime. That is "Java is compile once, run everywhere" |
Task Process
1)Track your execution task on [BVT Report ]
- Platform: Platform which you can run BVT on it.
- Volunteer: Volunteer's name who take this execution task.
- Revision: build version of AOO4.0
- Status: Pass/Fail
- Issue Number:Issue Number on BZ which failed BVT case .
Revision | Platform | Volunteer | Status | Issue Number |
---|---|---|---|---|
WindowsXP | ||||
Windows7 32bit | ||||
Windows7 64bit | ||||
Ubuntu | ||||
Suse | ||||
Redhat | ||||
Mac |
2)Update your execution result on "Status" column, and add detail HTML result in new link(Record link on [BVT Report ] ).
3)If have fail/error case, you need check it manually. To confirm it's exact a bug. (For some BVT cases write via GUI(Graphical User Interface), it have some offset).
In this step, you need study to read cases. [Dev GUI Test] [A step by step example]
4)If there is a defect, you can open it on [Bugzilla],select specific editor as it's category, then record on [BVT Report ]
5)If you finish all steps above, send your result to QA mail list with your report link.
Appendix
BVT Case Detail
Currently, BVT include 2 classes. These test cases are based on VCLAuto, and verify main functionalities.
- Integrating smoke test from AOO.
- Test cases cover 5 products: word processor, spreadsheet, presentation, drawing, and equation editor.
- Testing main operations about each products: new, save, load, reopen.
- Objects operations: picture, chart, table, function.
- Other test points: export as pdf, print, about dialog, slide show.
Below is the detailed introduction of each case.
FileTypeTest.java
|
|
testSaveNewODT testSaveNewOTT testSaveNewSXW testSaveNewSTW testSaveNewDOC testSaveNewTXT |
|
testSaveNewODS testSaveNewOTS testSaveNewSXC testSaveNewSTC testSaveNewXLS |
|
testSaveNewODP testSaveNewOTP testSaveNewPPT testSaveNewPOT testSaveNewSXI testSaveNewSTI |
|
testSaveNewODG testSaveNewOTG testSaveNewSXD testSaveNewSTD |
|
testSaveNewODF testSaveNewSXM testSaveNewMML |
|
BasicFunctionTest.java
Test Point |
Function Description |
SmokeTest |
|
testExportAsPDF |
|
testPrinter |
Check if print file dialog exist |
testRunMacro |
Run basic Macro |
testHelp |
Test the About Dialog show |
testInsertPictureInDocument |
Test inserting a picture in text document |
testInsertPictureInSpreadsheet |
Test inserting a picture in spreadsheet document |
testInsertPictureInPresentation |
Test inserting a picture in presentation document |
testSlideShow |
Check slide show correctly |
testFind |
|
testFillInSpreadsheet |
Test automatic filling function of spreadsheet. Fill Down/Right/Up/Left/Series... |
testSort |
Test sort function of spreadsheet. |
testInsertChartInDraw |
Test inserting a chart in a draw document. |
testInsertChartInDocument |
Test inserting a chart in a text document |
testInsertChartInSpreadsheet |
Test inserting a chart in a spreadsheet document. |
testInsertChartInPresentation |
Test inserting a chart in a presentation document. |
testInsertTableInDraw |
Test inserting a table in a draw document. |
testInsertTableInDocument |
Test inserting a table in a text document |
testInsertTableInPresentation |
Test inserting a table in a presentation document |
testSumInFormulaBar |
Test inserting a function in a spreadsheet document via Sum button. |
testInsertFunctionViaFormulaBar |
Test inserting a function in a spreadsheet document via input line. |
testFunctionWizardInFormulaBar |
Test inserting a function in a spreadsheet document via Function Wizard Dialog |