some unit tests do not work ...

Jan 25, 2011 at 9:29 AM

When building the DNI from sources, some unit test fail on my machine.

The unit tests that make problems are:
  DVLib::UnitTests::ConfigFilesUnitTests::testNoMatchingConfiguration
  ConfigFilesImpl::OnLoad(D:\Source\DotNetInstallerTrunk\Samples\Invalid\NoMatchingOSFilter.xml) ...
Expected exception : [Expected error : no match in 1 setup configuration(s).] : OK
=> D:\Source\DotNetInstallerTrunk\dni.proj(217,5): error MSB3073: The command "UnitTests\dotNetInstallerLibUnitTests\Debug\dotNetInstallerLibUnitTests.exe" exited with code -1.

when building UITests I get the follwing restults:

  Tests run: 80, Errors: 0, Failures: 5, Inconclusive: 0, Time: 240,564 seconds
    Not run: 0, Invalid: 0, Ignored: 0, Skipped: 0

  Errors and Failures:
  1) Test Failure : dotNetInstallerUnitTests.CompleteCommandTests.TestDontWaitForCompleteCommand
       Expected: True
    But was:  False

  bei dotNetInstallerUnitTests.CompleteCommandTests.TestDontWaitForCompleteCommand() in d:\Source\DotNetInstallerTrunk\
  UnitTests\dotNetInstallerUnitTests\CompleteCommandUnitTests.cs:Zeile 162.

  2) Test Failure : dotNetInstallerUnitTests.LanguageUnitTests.TestComponentLcid
       Expected: True
    But was:  False

  bei dotNetInstallerUnitTests.LanguageUnitTests.TestComponentLcid() in d:\Source\DotNetInstallerTrunk\UnitTests\dotNet
  InstallerUnitTests\LanguageUnitTests.cs:Zeile 52.

  3) Test Failure : dotNetInstallerUnitTests.LanguageUnitTests.TestConfigNoLangID
       Expected: 1031
    But was:  1033

  bei dotNetInstallerUnitTests.LanguageUnitTests.TestConfigNoLangID() in d:\Source\DotNetInstallerTrunk\UnitTests\dotNe
  tInstallerUnitTests\LanguageUnitTests.cs:Zeile 137.

  4) Test Failure : dotNetInstallerUnitTests.LanguageUnitTests.TestConfigurationLcid
       Expected: True
    But was:  False

  bei dotNetInstallerUnitTests.LanguageUnitTests.TestConfigurationLcid() in d:\Source\DotNetInstallerTrunk\UnitTests\do
  tNetInstallerUnitTests\LanguageUnitTests.cs:Zeile 99.

  5) Test Failure : dotNetInstallerUnitTests.LanguageUnitTests.TestOsLangID
       Expected: 1031
    But was:  1033

  bei dotNetInstallerUnitTests.LanguageUnitTests.TestOsLangID() in d:\Source\DotNetInstallerTrunk\UnitTests\dotNetInsta
  llerUnitTests\LanguageUnitTests.cs:Zeile 120.


D:\Source\DotNetInstallerTrunk\dni.proj(202,5): error MSB6006: "nunit-console-x86.exe" exited with code 5.
Done Building Project "D:\Source\DotNetInstallerTrunk\dni.proj" (uitests target(s)) -- FAILED.


Build FAILED.

"D:\Source\DotNetInstallerTrunk\dni.proj" (uitests target) (1) ->
(uitests target) ->
  D:\Source\DotNetInstallerTrunk\dni.proj(202,5): error MSB6006: "nunit-console-x86.exe" exited with code 5.

    0 Warning(s)
    1 Error(s)

Result for "Editortests":

  Tests run: 34, Errors: 27, Failures: 0, Inconclusive: 0, Time: 79,375 seconds
    Not run: 0, Invalid: 0, Ignored: 0, Skipped: 0

[here comes the detailed errors]

"D:\Source\DotNetInstallerTrunk\dni.proj" (editortests target) (1) ->
(editortests target) ->
  NUNIT : 1) Test error : InstallerEditorUnitTests.UIConfigurationUnitTests.TestAddDownloadDialog
  NUNIT : 2) Test error : InstallerEditorUnitTests.UIConfigurationUnitTests.TestAddSetupConfigurationComponents
  NUNIT : 3) Test error : InstallerEditorUnitTests.UIConfigurationUnitTests.TestAddSetupConfigurationControls
  NUNIT : 4) Test error : InstallerEditorUnitTests.UIConfigurationUnitTests.TestAddSetupConfigurationLicenseAgreement
  NUNIT : 5) Test error : InstallerEditorUnitTests.UIConfigurationUnitTests.TestEditWithNotepad
  NUNIT : 6) Test error : InstallerEditorUnitTests.UIConfigurationUnitTests.TestNewConfiguration
  NUNIT : 7) Test error : InstallerEditorUnitTests.UIConfigurationUnitTests.TestOpenCloseConfiguration
  NUNIT : 8) Test error : InstallerEditorUnitTests.UIConfigurationUnitTests.TestOpenRecent
  NUNIT : 9) Test error : InstallerEditorUnitTests.UIConfigurationUnitTests.TestSaveAsConfiguration
  NUNIT : 10) Test error : InstallerEditorUnitTests.UIConfigurationUnitTests.TestSaveConfiguration
  NUNIT : 11) Test error : InstallerEditorUnitTests.UIMakeExeUnitTests.TestMakeExe
  NUNIT : 12) Test error : InstallerEditorUnitTests.UIMoveUpDownUnitTests.TestMoveUpMoveDownChecks
  NUNIT : 13) Test error : InstallerEditorUnitTests.UIMoveUpDownUnitTests.TestMoveUpMoveDownComponent
  NUNIT : 14) Test error : InstallerEditorUnitTests.UIMoveUpDownUnitTests.TestMoveUpMoveDownConfiguration
  NUNIT : 15) Test error : InstallerEditorUnitTests.UIMoveUpDownUnitTests.TestMoveUpMoveDownEmbedFileFolder
  NUNIT : 16) Test error : InstallerEditorUnitTests.UITemplateUnitTests.TestAddSetupConfigurationLang
  NUNIT : 17) Test error : InstallerEditorUnitTests.UITemplateUnitTests.TestCustomizeTemplates
  NUNIT : 18) Test error : InstallerEditorUnitTests.UIUnitTests.TestExit
  NUNIT : 19) Test error : InstallerEditorUnitTests.UIUnitTests.TestExpandCollapseRoot
  NUNIT : 20) Test error : InstallerEditorUnitTests.UIUnitTests.TestHelpAbout
  NUNIT : 21) Test error : InstallerEditorUnitTests.UIUnitTests.TestMainMenu
  NUNIT : 22) Test error : InstallerEditorUnitTests.UIUnitTests.TestMainMenuEdit
  NUNIT : 23) Test error : InstallerEditorUnitTests.UIUnitTests.TestMainMenuFile
  NUNIT : 24) Test error : InstallerEditorUnitTests.UIUnitTests.TestMainMenuHelp
  NUNIT : 25) Test error : InstallerEditorUnitTests.UIUnitTests.TestMainMenuTools
  NUNIT : 26) Test error : InstallerEditorUnitTests.UIUnitTests.TestMainMenuView
  NUNIT : 27) Test error : InstallerEditorUnitTests.UIUnitTests.TestRefresh

    0 Warning(s)
    27 Error(s)

I am not using VS2005, but 2008 to build DNI, this might be a reason why the test fail? Are there any known problems? If that is not the case I will investigate further.

I want to develop and submit some patches for DNI and thought I should run unit test before submitting them. So I want to make sure there initially pass.

Developer
Jan 25, 2011 at 11:35 AM

When I started work on DNI I noticed and fixed some issues because my locale was not US-English. What locale are you using?

How are you build it with VS2008? Have you upgraded all the project files?

Jan 25, 2011 at 12:51 PM

my display language and keyboard layout is german, the "language for unicode incompatible programs" (free translation) is US-English

I build with VS2008 after upgrading project files, Visual studio does this for you. And I changed the path to msbuild.exe from .Net Version 2.0.50727 to 3.5 in build.cmd

It is working fine, except for the unit tests.

Developer
Jan 25, 2011 at 1:16 PM

I haven't tried VS2008 but my guess it that it has something to do with the fact that your display language is German, you probably need to step through the test to see whether this is correct.

Coordinator
Jan 25, 2011 at 1:32 PM

Indeed, the failed tests are probably actual bugs in the tests that manifest themselves on non-English machines, please feel free to fix and submit patches.

Jan 25, 2011 at 3:47 PM

ok, will have a closer look into the test, will come back if I find any fixes or more detail on the reasons why the test fails.

Jan 28, 2011 at 4:55 PM

so, I had a closer look at the unit test:

failing test from dotNetInstallerLibUnitTests.exe comes when running unit tests with msbuild from .Netframework higher than V2.XXX (e.g V3.5 or V4.XXX), so the fix here is to use right msbuild for unit test execution

when running dotNetInstallerToolsLibUnitTests.exe (not in first post)
I get 12 failures:
     these all seem to result from non english(us) operating system(what nsleightholm allready guessed) , since english strings are compared with german ones....  a solution would be to get the "strings to compare" with dvlib method results from some windows API and to not set them statically in the test.
     any ideas how to do that? may be someone had the same problem?

for the uitests: 4 "only" get 4 failures now the 5th seem to have been a problem of wrong msbuild also(or may be nunit integration helped...) => not investigated yet

for editortests: same, not investigated yet

 

Is it anytime planed to move to a higher Visual Studio/MSBuild version? since there are 2 more versions available it might be the time?

 

Coordinator
Jan 31, 2011 at 12:17 AM

No plans to move to a higher MSBuild / VS version because of OS backward compatibility (this still runs on Win95 believe it or not, although I think maybe it's time to drop it ;)).

Submit patches for any (even partial) fixes. You'll have to find a creative way to fix those string things - basically you need to be in control of the string content.

Jan 31, 2011 at 7:11 PM

so, I have no creative idea at the moment. The question is what the use is to verify if localised strings are formatted...

anyway, the editortest have also a localisation issue where I found no solution:

27 test fail, because the menuitem can not be found....

problem is that e.g. in

UIAutomation.Find<MenuBar>(mainWindow, "Application").MenuItem("File", "New").Click();

"Application" is somehow translated to "Anwendung". So the unit test will never find the application menubar and all test that use menubar fail.

I so far have not found out how to set language or get the localised name from API. May be you have any ideas on that?

A way of workaround/ease would be to use a global variable that could be changed at one point to get the test to pass again....

Coordinator
Feb 2, 2011 at 5:18 PM

InstallerEditor is English only. So I would think that nothing gets translated. That looks like a bug, no?

Feb 3, 2011 at 7:50 AM
Edited Feb 3, 2011 at 10:49 AM

I have the feeling that this is not really something dotNetInstaller does, but that comes from the Automation API from windows. Unfortunately I have not found any hint where this name is set, or how to change this localisation problem ...

Coordinator
Feb 3, 2011 at 1:06 PM

You can definite change the language of the UI thread, maybe that will fix it?

Thread.CurrentThread.CurrentUICulture = ...

Feb 3, 2011 at 4:16 PM
Edited Feb 3, 2011 at 4:57 PM

dblock, that did a lot, aftewards I "only" get 5 errors, so a big step forward. Will send the changed unittest files as patch.
again, the rest has to do a lot with localisation again.

Mainly it searches the title of the "SaveFileDialog" and "openFileDialog", but the default titles are localised by OS. May be it is possible to set them in installereditor? I think this will not be overridden.
Anyway I feel the gui tests are really os language dependent, which is bad. Unfortunately I also do not know a better way.

Coordinator
Feb 6, 2011 at 9:29 PM

Thanks. Can you please make an SVN patch and not a zip of the project :) Sorry to be a pest - I got a bunch of changes that need merging.

Feb 7, 2011 at 8:06 AM

how to do a svn patch? Is there a docu somewhere? Does it mean a "normal" commit?

Developer
Feb 7, 2011 at 10:16 AM

If you use TortoiseSVN you can just right click on the project folder and create a patch file, make sure you only select the files that a related to your change. For example you would probably want to exclude the csproj files as you have upgraded them to VS2008.

Feb 7, 2011 at 10:24 AM

thanks, have done it how described and submitted a new patch with the .patch file. hope is right format now.

Coordinator
Feb 7, 2011 at 12:45 PM

I refactored this into a base class and committed the fix. Thanks!

So the remaining problems are in system dialogs. Google tells me there's a Title property on those dialogs. Didn't check ;)

Feb 7, 2011 at 1:54 PM

that's right. If you set no title, then you get localised ones that make problems on testing. I did not try to set them manually, but I think this would be a solution for installer editor if it only has english gui.

Coordinator
Feb 7, 2011 at 5:24 PM

Installer editor today only has English GUI. It would be nice to localize it I guess, but frankly who has time for that? ;) I think that broken unit tests should be fixed first. Go for it.

Feb 8, 2011 at 9:10 AM

dblock, english is full sufficient for the editor in my eyes. if so the fix would be to give the open/save dialogs a dedicated title using the property and so it can be found in unit tests. unfortunately I do not have the time to do it.

Coordinator
Feb 8, 2011 at 10:48 AM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.
Coordinator
Feb 8, 2011 at 10:48 AM

Thanks for all your help so far. I created a workitem to track the remaining issues.

Feb 8, 2011 at 10:58 AM

good point, but I think there are more issues remaining ... this one is for the editorunittests, but for some formatting tests I think there was the problem that error messages are also localized by windows...