VirtualStore and other frustrating problems

[NigelMills]

Hi all,

We've had several instances where we have had the compiler use an out of date source file, to the extent that it was reporting #Warnings from the source file that are no longer present. Eventually we worked out that there was a path (win 7) C:\Users\<user>\AppData\Local\VirtualStore\Swordfish, where it was keeping out of date copies of some source files, and bringing those in instead of the ones in the project directory. I'll refrain from commenting about my lost days trying to figure that one, but deleting the Swordfish directory under this path seems to improve the situation. As I have more faith in the Swordfish compiler than I do in Microsoft, I'll assume this is some problem with Microsofts cacheing that is invisible to the compiler.

Since then we've seen continuing issues with the compiler picking up similarly named files from other projects it has been used on that day instead of the current project. We've been through the process of using absolute references in the include statements (ig INCLUDE "M:\COMMON\USART.bas", mapping M: to the relevant directory using DOS Subst), and more recently relative references to avoid the Subst command eg (..\COMMON\USART.bas). We've followed the guidance in this topic.. http://sfcompiler.co.uk/phpBB3/viewtopi ... hilit=path, and now use the forward slash instead of the back slash in paths for includes and now have a single main file in the project directory and below it subdirectories with code separated relating to the various functional blocks (USART, I2C etc).

Sadly, that still has not totally solved the problem.

We are still left with another completely unexplainable issue, and I throw myself upon the mercy of this forum for some help in resolving it !

We have a project we compile with and without USB functionality, and use #define Include_USB_Code as a switch set in the main file to build in USB functions (or a USART if the USB is not included). The following bit of code is quite literally nearly driving me to a breakdown....

Code: Select all

Include "COMMON/BITDEFS.bas"

Include "NM_SENSOR/S_GLOB_VAR.bas"

<snip>

  ' CC1101 GD02 on RP24 is mapped to INT3
	Public Dim Radio_GDo2_Int_Pri  As INT3IP     
	Public Dim Radio_GDo2_Int_Edge As INTEDG3
	Public Dim Radio_GDo2_Int_Flag As INT3IF
	Public Dim Radio_GDo2_Int_En   As INT3IE

The definition for INT3IP (and all the other peripheral control registers) is in COMMON/BITDEFS.bas included at the top of the code snippet.

When built WITHOUT USB, it works.

When built WITH USB, it fails on the line (and every subsequent line) referencing any peripheral control bit (the first being INT3IP). Inserting #Warning statements into the BITDEFS.bas Include file shows the file is NOT being included, BUT if you intentionally corrupt the file name, it complains that it can't find it.

Comment out the #define Include_USB_Code in the main code (which doesn't directly affect any of these parts of the code), and suddenly the #warnings from BitDefs.bas show up in compilation and it compiles and runs correctly !

We've looked at this older thread which is not specified as resolved.... http://sfcompiler.co.uk/phpBB3/viewtopi ... lude#p7982, and of course other sections of the code do have Include statements for the USB functions within #if statements (or even nested #if statements). But nothing in this file selectively includes depending on the USB functions. It simply works without the USB and doesn't with it.

WHAT is making the compiler complete ignore the content of the BITDEFS file when I include the #define for the USB ?

For my 2 cents worth, I'm not blaming the USB code. it might be any section of code that's selectively brought in. There is some unexplained process causing the compiler to fail here, and I really am tearing my hair out.

I will happily send a project zip file to anyone who can show me how to solve it please (but just moving stuff around till it `goes away without explanation` isn't a solution, as this sort of problem keeps recurring).

Thanks for reading this far, and even more thanks if you can try and help.

Nigel Mills
BSc CEng FIET and generally decent chap with too little hair to keep tearing out.

PC: Vanilla Core i7, SSD C: Win 7. - Identical problems on two similar workstations (and my behemoth PC at home)

[David Barker]

> AppData\Local\VirtualStore

If the folder you are trying to compile in does not have the correct privileges, windows 7 uses "virtual store". It's the OS doing this, not Swordfish. Check out UAC and virtual storage on the MS site. If using win7 and above, you should keep all of your project files in a folder that has the correct privileges (like my documents)

[NigelMills]

Hi David,

Thanks for that virtually instantaneous answer.

We both use administrator accounts on the PC's and have the projects on c:\users\<user>\work\MyProject, and so we are not using any areas of the disk to which we do not have full access rights. The possibility here, now you've put the `access rights` thought in my head is that we do use on-line backup software (Mozy), and it's possible that Mozy has a file open to back it up, and the compiler can't open it for write access and so lovely old Microsoft makes a copy. That is possible, and having apparently eliminated VirtualStore by deleting the Swordfish subdirectory, I'm will to take that at face value.

Sadly that still leaves the issue for which I'm tearing my hair out.... any suggestions, or could I interest you in a zip file ?

Best regards,

Nigel Mills

[David Barker]

You can email me the project files and I can take a look, but please reduce the error to the smallest possible piece of code and include files...

[David Barker]

I've taken a look at your code, but to be honest the project is huge. It could take me days just to try and get to grips what the program is doing and how it all links in. You need to reduce the size of the code base for me to investigate. However, it's very possible you have a mismatched #if #endif or #elseif somewhere. If these are mismatched they can carry from module to module, causing problems.

[NigelMills]

David,

Yes it's big, but we only sent you one third of it !

I have a feeling that sometime through attempting to reduce the code, the problem will simply disappear. We have hit similar problems before, and after spending a day shifting code around between modules we somehow then are able to carry on without ever actually finding the source of the problem.

It's very frustrating as I'm desperate to move the project forward, but feel I have spent days fighting the compiler. We love your product and have 20 months and £100k invested in our project completely dependent on Swordfish after converting from C when we discovered your product a few months in.

I'm sure you are busy, but rather than me reducing the project until the problem disappears without explanation (possibly only to surface on other peoples big projects), could I pay for a day of your time, for you to find the exact indisputable cause for me ? Sure it'll help me (not go crazy), but if it turns out to be a compiler issue it'll help you too (and I'll still pay).

Please give me a price, I'm desperate for the solution to what I see as a recurring problem.

Best regards,

Nigel Mills

[David Barker]

I had a look at your code last night. May I suggest the following:

(1) delete any device files you have in you include file (for example, 18F67J94.bas)
(2) Remove every single backslash reference for include files (every one!) for example "COMMON/CC1101.bas" becomes "CC1101.bas". I cannot stress enough, please remove them all from very single module.
(3) Clear out or backup your UserLibrary folder and place all of your modules in there. For example,

Code: Select all

UserLibrary\NigelMills\Common
UserLibrary\NigelMills\I2C

and so on. The next release of Swordfish will auto search you root source folder, so make these changes now and put your modules in the UserLibrary.

[NigelMills]

David,

I did finally solve the problem just before midnight last night. I do not understand why what I had done caused an issue, but I will highlight it here in the hope that you might be able to explain why it causes the problem and why this solved it.

The problem was that my module "Sensor_Hardware" (a code snippet shown previously) was completely ignoring the content of the Include file "BitDefs", although changing the include to a non existent file caused a "file not found" error.

The cause appears to have been where "BitDefs" was included also in an earlier file in the compilation process (shown below), but it was included in both the #if and #else parts of the same #if statement (whereas it should have been outside the #if statement as it was required in both cases). It was the inclusion of this "include BitDefs.bas" statement twice in the one #if that caused the file to be unavailable to be included a little later in the compilation (of the code snippet shown earlier). Moving it outside the #if, makes it now be included correctly in the later module !

So this doesn't work (it causes an error making the include file not available in a subsequent file when compiled):

Code: Select all

#if defined (BUILD_AI_HUB)
  Include "COMMON/BITDEFS.bas"
  Include "NM_HUB/H_TIMERS.bas"
#elseif defined (BUILD_AI_SENSOR)
  Include "COMMON/BITDEFS.bas"
  Include "NM_SENSOR/S_TIMERS.bas"
#else
  #Error "No BUILD_ defined for High level interrupt definition"
#Endif

But this does:

Code: Select all

Include "COMMON/BITDEFS.bas"

#if defined (BUILD_AI_HUB)
  Include "NM_HUB/H_TIMERS.bas"
#elseif defined (BUILD_AI_SENSOR)
  Include "NM_SENSOR/S_TIMERS.bas"
#else
  #Error "No BUILD_ defined for High level interrupt definition"
#Endif

OK, so we're all shouting, "Yes well I wouldn't have done it that way, I'd have put the include outside the #if to start with". Which is all well and good, but there's two of us working on a joint project here, and we do our best not to tread on each others toes by making unneccessary changes (or to use the technical term `to dick with`) each other sections of code. Hence the reason I put his Includes inside a #if without change (he's working on the hub), and added mine for the sensors in the #else (even although one was common).

In fact the statements in the #if part are never executed in this build, (as they are for the hub device and this is the sensor build), it is the #else that is always executed, and so why listing the Include line inside the #if as well as the #else causes a problem is a mystery to me. An EVEN BIGGER MYSTERY is that if you leave the two statements inside the #if and #else, but then corrupt the file name of the Include within the #if (the part that's never executed on this build), then the problem also goes away !

So this also works....

Code: Select all

#if defined (BUILD_AI_HUB)
  Include "COMMON/XXX-BITDEFS.bas"
  Include "NM_HUB/H_TIMERS.bas"
#elseif defined (BUILD_AI_SENSOR)
  Include "COMMON/BITDEFS.bas"
  Include "NM_SENSOR/S_TIMERS.bas"
#else
  #Error "No BUILD_ defined for High level interrupt definition"
#Endif

I cannot understand this at all (possibly something to do with release times and file locking ?), but it does make me wonder with the compiler whether it detects if the included files are available on an early pass, but then fails to notify the user and skips over the error if the file is unavailable for a subsequent pass?

I know you're busy guys, and "my problem is solved", but it's only solved until someone else finds themselves tearing their hair out with some other unexplained problem that "should work". I personally would have my confidence in the compiler massively improved if you can give me a solid reason why this fails and the changes make it work ?

As a side note, should you try and compile the code sample I have e-mailed. Once the fault above has been resolved, it highlights that the following two include files needs adding just below the code snippet in usbcdc.bas in order to solve a remaining error.

Code: Select all

Include "NM_SENSOR/SENSOR_HARDWARE.bas"
Include "NM_SENSOR/S_GLOB_VAR.bas"

Please try and explain why this fails !

Best regards,

Nigel Mills.

[David Barker]

> I did finally solve the problem just before midnight last night.

I think your solution is just masking another problem. However, if you are happy with it, then that's great. If not, you need to perform the steps I outlined earlier in order to move towards some understanding of the problem.

> Please try and explain why this fails !

Because I do not think the problem is resolved

[NigelMills]

David,

I have followed your guidelines explicitly opening all .bas source files at once in an editor and making global changes.

1) There were not any Include <device> lines
2) There are now no paths specified on any include files
3) The directory structure under the project directory has now moved to the User Library
(only mplabicd.bas, util.bas, and reatme.txt appear in the user library root).

The result is unchanged (and the old fix still works). Shall I send you a new zip file ?

Best regards,

Nigel Mills

Update coming....

[David Barker]

> 1) There were not any Include <device> lines

There is a device file in your module directories

> 2) There are now no paths specified on any include files

OK

3) The directory structure under the project directory has now moved to the User Library

OK

> Shall I send you a new zip file ?

Yes please

[NigelMills]

David,

To add to my confusion, even although I cut/pasted the subdirectories containing my modules into:

C:\ProgramData\Mecanique\Swordfish\UserLibrary

When I open one in Swordfish (clicking on the listed includes in code explorer and hover over the button for that file), I see it lists the path as coming from:

D:\Users\Nige\Documents\Swordfish\UserLibrary

and although I have NEVER placed anything in this folder, I see it is a mirror of the original one. Worse, it is a COPY not a redirection, as editing a file name in one does not show up in another !

Did I put them in the right user library first time ? How do we manage this from a version control point of view ?

Re latest post: Sorry misunderstood point (1), thought you meant Included files. Removed and still no change.

I will send a new zip file, shall I include the user library now. Which one ?

Best regards,

Nigel Mills

[David Barker]

> and although I have NEVER placed anything in this folder, I see it is a mirror of the original one.
> Worse, it is a COPY not a redirection, as editing a file name in one does not show up in another !

Why is it worse? You placed the user libraries in the windows "global" all users folder. When the application starts, it copies user libraries and code to the users (my documents) folder. This is correct for a win7 installation.

> include the user library now. Which one

as you pointed out, they are copies so either will do.

[NigelMills]

Sorry, we have no understanding of there being anything other than a global user library !

So is the contents of C:\ProgramData\Mecanique\Swordfish\UserLibrary copied to D:\Users\Nige\Documents\Swordfish\UserLibrary when the IDE starts or the compiler ?

If I copy things into C:\ProgramData\Mecanique\Swordfish\UserLibrary\nige, is that then specific to my user name on the PC ? Does it end up also under `nige` in the D: copy, or one level further up matching the global stuff ?

I will prepare a zip file now.

Best regards,

Nigel Mills

[David Barker]

I have examined the files you sent me yesterday. This is what I did

(1) removed all backslash references for include file paths
(2) placed all of the module code in the user library
(3) added

Code: Select all

Include "sensor_hardware.bas" 
Include "s_glob_var.bas"    

to the top of module cdc as you are referencing two variables which are contained in these files. You project (as sent to me) now compiles correctly without having to make any other code alterations