Where does the compiler search for libraries?

General discussion relating to the library modules supplied with the compiler

Moderators: David Barker, Jerry Messina

SHughes_Fusion
Posts: 219
Joined: Wed Sep 11, 2013 1:27 pm
Location: Chesterfield

Where does the compiler search for libraries?

Post by SHughes_Fusion » Tue Jul 15, 2014 3:05 pm

Does the compiler search to the sub-folder level for user libraries?

The reason I ask is that I'm seeing some odd behaviour. I'm sharing a project I'm working on via Mercurial. I don't store any libraries within my project folder directly, but I've created a subfolder called 'Copy to Swordfish UserLibrary' so my colleague can get a copy of my libraries. The plan is if I update a library I will copy it to this folder and it will automatically be shared.

However, Swordfish appears to be opening libraries from here rather than the default UserLibrary folder and as I've got several projects using the same libraries I'm finding that sometimes I'm working on the wrong copy of a library.

What I've read suggests Swordfish should only be looking in the folder the project is in, but I don't know if this has been increased in scope to include subfolders at some point. Has it? And if so, can this be disabled in some way?

If not then I guess I'll have to find another way of sharing libraries as this is too dangerous....

User avatar
David Barker
Swordfish Developer
Posts: 1194
Joined: Tue Oct 03, 2006 7:01 pm
Location: Saltburn by the Sea, UK
Contact:

Re: Where does the compiler search for libraries?

Post by David Barker » Tue Jul 15, 2014 3:11 pm

http://www.sfcompiler.co.uk/wiki/pmwiki ... ionHistory
http://www.sfcompiler.co.uk/wiki/pmwiki ... leLocation

There is no way to disable this feature. It was added due to numerous requests to make it easier to share projects between users.

SHughes_Fusion
Posts: 219
Joined: Wed Sep 11, 2013 1:27 pm
Location: Chesterfield

Re: Where does the compiler search for libraries?

Post by SHughes_Fusion » Tue Jul 15, 2014 3:16 pm

Drat, didn't notice that update....

I've been checking and I now appear to have three copies of my modules all at differing stages with some having updates that others don't.... Going to take me a while to unpick all the changes!

I'd suggest that rather than searching any sub-folder it would be better to have a particular folder name for this. Either a 'default' one, or if not too hard maybe allow a 'path' to be specified in the code? I can see why the change was made but where libraries are shared across multiple projects it actually harms more than it helps.

User avatar
David Barker
Swordfish Developer
Posts: 1194
Joined: Tue Oct 03, 2006 7:01 pm
Location: Saltburn by the Sea, UK
Contact:

Re: Where does the compiler search for libraries?

Post by David Barker » Tue Jul 15, 2014 3:27 pm

I think once the mechanism is understood, it works very well.

SHughes_Fusion
Posts: 219
Joined: Wed Sep 11, 2013 1:27 pm
Location: Chesterfield

Re: Where does the compiler search for libraries?

Post by SHughes_Fusion » Tue Jul 15, 2014 3:40 pm

I'm not so sure, I'll let you know when I'm more used to it but I can see more problems than solutions to be honest.

I use the same libraries across several projects. Keeping them in the UserLibrary folder I can just have a simple update script which copies the latest version of the libraries to my working folder and these are shared with a colleague who is working on one particular project with me. He then copies them back to his User Library folder and all is good.

Now if I change a library I have to remember to copy it back to the user library folder before using it anywhere else and if I forget I can easily end up with multiple libraries that are subtly different. Also, if I modify a library in a non-shared project then it modifies the 'main' copy which means more hassle making sure I've not also modified the local version on a shared project...

Personally, I think the previous system was better where you had one set of libraries you were able to supercede in specific circumstances.

I guess a simple solution to my problem is just to zip up my user library folder and share that instead. It is a bit frustrating when a fundamental change to the way the compiler works is introduced without a way to disable it though for backwards compatibility...

User avatar
David Barker
Swordfish Developer
Posts: 1194
Joined: Tue Oct 03, 2006 7:01 pm
Location: Saltburn by the Sea, UK
Contact:

Re: Where does the compiler search for libraries?

Post by David Barker » Tue Jul 15, 2014 4:07 pm

The main library and user library work in the same way. Traversing sub-folders is an addition and many people have found this useful for making copies and backups. I see more benefits than problems to be honest.

SHughes_Fusion
Posts: 219
Joined: Wed Sep 11, 2013 1:27 pm
Location: Chesterfield

Re: Where does the compiler search for libraries?

Post by SHughes_Fusion » Wed Jul 16, 2014 7:34 am

It just seems a bit odd to me that there were two specific folders for libraries and you've now just said 'and anything else inside the project directory'.

To me, adding something like a 'ProjectLibrary' folder within the directory would have been a good move but I've got a lot of projects where I previously stored a copy of the libraries I used at build time in the project folder which I'll now have to change for fear of getting libraries mixed up.

I can see the benefit of allowing the user to specify a different folder to get libraries from but to me the default really should be from the 'central' folders. This has actually made it harder for me to share the project not easier...

All that said, maybe I'm in the minority here, it would be good for other users to leave their views.

Jerry Messina
Swordfish Developer
Posts: 1245
Joined: Fri Jan 30, 2009 6:27 pm
Location: US

Re: Where does the compiler search for libraries?

Post by Jerry Messina » Wed Jul 16, 2014 9:29 am

but I've got a lot of projects where I previously stored a copy of the libraries I used at build time in the project folder which I'll now have to change for fear of getting libraries mixed up
Perhaps I'm missing something, but doesn't the new method actually help here? Now, you'd automatically be using the local copies in the project directory vs having to remember to copy them to some central place before building and managing those.

Personally, I prefer option #3 where you'd actually have to specify the path to the include files, but that has its issues too. I'm in the minority in that I always use local copies of all the files. That way I know I can rebuild the project later and not have to rely on getting the correct version of any "libraries". Sometimes I find that global changes to the libraries aren't necessarily a good thing when trying to go backwards in time.

If you're ever in doubt as to which copies you're using, open the project .IDF file in a text editor and check the 'files' section at the beginning. There you'll find a list of all the files and paths used by the project.

SHughes_Fusion
Posts: 219
Joined: Wed Sep 11, 2013 1:27 pm
Location: Chesterfield

Re: Where does the compiler search for libraries?

Post by SHughes_Fusion » Wed Jul 16, 2014 9:45 am

The problem really is that when I open a library from within a project to edit it, I don't know where that library has come from.

I've been updating libraries for the 18F46K22 I've started using and spent ages trying to work out why things worked in one project but not another. This is what I liked about the original system, you could store a project-specific library in the project root if necessary, but otherwise it would always use the main folder.

Now when I make a change for some projects the change only happens to the local copy where as for others it happens in the central folder. I have one case where I'd modified both and had to try and merge the changes to a single library. Luckily the rest I'd only changed one so I just used WinMerge to find the latest copy and manually copied that back to the UserLibrary folder.

I have several projects on the go which work in conjunction with each other so I need the libraries to be consistent across the projects.

What I'd been trying to do was a variation on how you say you work, I was keeping a copy of the libraries in the project folder so I could rebuild with those libraries at a later date if necessary. However, during normal development any changes I made were copied to the central UserLibrary folder so the improvements would be available to all projects.

My colleague is trying to work out ways around this, his suggestion was to use NTFS junction points which sort of 'clone' between several directories. I'm still not sure in my mind if this will work as we still don't know where Swordfish is opening the library from, especially as the IDE doesn't show a full path where the library doesn't come from the default location. He says any changes will be cloned across all copies but as one location is networked I wonder what will happen if the network is down at any point...

The current plan is to just set up a new Mercurial project for the UserLibrary folder and share things that way. More of a hassle to update but I guess it does have the benefit of adding version control to the library files as well.

Jerry Messina
Swordfish Developer
Posts: 1245
Joined: Fri Jan 30, 2009 6:27 pm
Location: US

Re: Where does the compiler search for libraries?

Post by Jerry Messina » Wed Jul 16, 2014 10:13 am

The problem really is that when I open a library from within a project to edit it, I don't know where that library has come from
In the IDE there's an option to display the full path to the file (View | Editor Options... | General | Display full filename path in the application title bar).

Does that help?

SHughes_Fusion
Posts: 219
Joined: Wed Sep 11, 2013 1:27 pm
Location: Chesterfield

Re: Where does the compiler search for libraries?

Post by SHughes_Fusion » Wed Jul 16, 2014 1:34 pm

In some ways, but it still isn't immediately obvious without studying the path carefully. I'd prefer to know for sure if I wasn't working on the 'normal' library.

User avatar
kanderson
Posts: 17
Joined: Tue Oct 15, 2013 4:28 pm
Location: Canada
Contact:

How does the <processor>.inc file get included

Post by kanderson » Thu Jul 31, 2014 3:28 pm

I recently got a code repository going on a local company server. While setting up the repo, I used the opportunity to reorganize some folders. I ended up putting my libraries in project subfolders... which brought me to this thread because my stuff no longer compiles.

My project is now throwing ASM errors:
[ASM Error]: Error[113] c:\dev\<my path>\mainFile.asm 6257 : Symbol not previously defined (POSTINC1)
[ASM Error]: Error[113] c:\dev\<my path>\mainFile.asm 6363 : Symbol not previously defined (PR4)
[ASM Error]: Error[113] c:\dev\<my path>\mainFile.asm 6295 : Symbol not previously defined (OSCCON)
... and so on

When I look at the generated .LST file, I can see that I'm missing the contents of P18F87J11.inc which used to get included. The .LST file is also missing the comments
00001; SWORDFISH STRUCTURED BASIC COMPILER FOR PICMICROS.
00002; VERSION 1.1.6.4 COPYRIGHT MECANIQUE 2014
00003;
00004 ; C:\DEV\PROJECTS\<myPath>\<myMainFile>.BAS <DATE>

I added the "Includes" folder to my project subdirectory but that didn't help.

So my questions are:
What actually happens when I write Device = 18F87J11?
How do I get my .bas file to compile such that the contents of P18F87J11.inc are added to my .ASM?

Note:
I kept a copy of my old folder structure and it still compiles OK, but I'd very much prefer to use the new subdirectories way. FWIW I like the new subdir search function. I used to have this funny script that would go to My documents and make symbolic links named "Library", "User Library", "Includes", "Samples" to fool SF to finding the different versions of libraries that I wanted.

User avatar
kanderson
Posts: 17
Joined: Tue Oct 15, 2013 4:28 pm
Location: Canada
Contact:

Re: How does the <processor>.inc file get included

Post by kanderson » Thu Jul 31, 2014 5:23 pm

I think I got my problem fixed.

I looked back through my old .IDF file and found that the 18F87J11.BAS file was actually coming from C:\ProgramData\Mecanique\Swordfish\Includes, which at some point in the past I made into a symbolic link to a place in my old file structure. Thus changing to the new file structure invalidated the link and the compiler couldn't find the processor file.

What is the difference in files located in 'ProgramData' vs in 'My Documents'? Why didn't it find the 18F87J11.bas in my project subdirectory?

Also, if I have
/c/<myproject>/libraries/Library/mylib.bas and
/c/<myproject>/libraries/UserLibrary/mylib.bas and I still have
/c/.../My Documents/Swordfish/Library/mylib.bas and
/c/.../My Documents/Swordfish/UserLibrary/mylib.bas and
/c/ProgramData/Mecanique/Swordfish/Library/mylib.bas and
/c/ProgramData/Mecanique/Swordfish/UserLibrary/mylib.bas
Which one can I expect to get compiled? I noticed that I was getting 'this was already declared' errors until I deleted mylib.bas from /c/<myproject>/libraries/Library. This was only happening when there was also a file /c/<myproject>/libraries/UserLibrary/mylib.bas

Is there an order to searched directories & subdirectories? Why was it trying to include some files twice?

SHughes_Fusion
Posts: 219
Joined: Wed Sep 11, 2013 1:27 pm
Location: Chesterfield

Re: Where does the compiler search for libraries?

Post by SHughes_Fusion » Fri Aug 01, 2014 8:34 am

[quote]Is there an order to searched directories & subdirectories? Why was it trying to include some files twice?[/quote]

Previous to the latest change the search order was:

Project folder root
UserLibrary folder in Swordfish install directory (usually My Documents/Swordfish)
Default libraries - not sure where these are actualy stored!

It may be that the file you are struggling with is a different type. The device files are stored in the 'Includes' folder in the Swordfish directory. There is a .bas and a matching .inc (albeit with a P infront of the file name).

I don't know if Swordfish needs these files to be in the includes folder or if it will search in the same folders as for libraries though.

I also don't know the search order wrt sub-folders in the project folder, hopefully David will advise. However, that is one problem I can see with this 'searching any subdirectory' way of looking for libraries - if you have two copies in different directories in the project folder who knows which order they will be found in? I'd guess alphabetic but who knows in what order directory names are handed to Swordfish by Windows?

User avatar
David Barker
Swordfish Developer
Posts: 1194
Joined: Tue Oct 03, 2006 7:01 pm
Location: Saltburn by the Sea, UK
Contact:

Re: Where does the compiler search for libraries?

Post by David Barker » Fri Aug 01, 2014 9:01 am

Device files have to work differently from standard files. The compiler will only ever look for a device file in one place. For Win7 and above, this is

x:\ProgramData\Mecanique\Swordfish\Includes

This is fixed. You should never place device files in any other folder.

Post Reply