Bluetooth (serial) Bootloader

Coding and general discussion relating to user created compiler modules

Moderators: David Barker, Jerry Messina

Post Reply
SHughes_Fusion
Posts: 219
Joined: Wed Sep 11, 2013 1:27 pm
Location: Chesterfield

Bluetooth (serial) Bootloader

Post by SHughes_Fusion » Mon Aug 04, 2014 2:53 pm

I'm looking at the options for writing a bootloader for my project and was hoping I could get some tips and pointers from the board before going too deeply in to researching this.

I'm using the PIC18LF46K22. I have a RN42 on the UART and was hoping to update via this Bluetooth link.

My understanding is that I need to find some way to trigger the bootloader code - I'm planning to look for a particular button press on power-up. Easy enough.

I have a 64K EEPROM in my circuit so I'm thinking that it would be safer to write my new code to this EEPROM, do some checking (CRC) to see if the data has transmitted correctly then write it to the PIC. This allows me to recover the device to an operating unit if the upload fails to work. The other option would be the opposite way, I copy the current program memory to the EEPROM, write directly from the input data stream and recover from EEPROM if necessary.

So, the PIC side of things doesn't seem overly complex, the only hassle is fitting it in to 4k or less of memory.

What I don't know is how to get my compiled program out to the bootloader.

I've tried loading in a compiled .hex file and comparing the contents to what I see in the PICKit 3 programmer window. I can see some of the data but not all and I can't see a pattern as to how the data is stored. Can anyone point me in the direction of how to interpret this data?

Is there anything else I need to be aware of or to cover?

Any help much appreciated.

Jerry Messina
Swordfish Developer
Posts: 1469
Joined: Fri Jan 30, 2009 6:27 pm
Location: US

Re: Bluetooth (serial) Bootloader

Post by Jerry Messina » Sat Aug 09, 2014 12:48 pm

I've tried loading in a compiled .hex file and comparing the contents to what I see in the PICKit 3 programmer window. I can see some of the data but not all and I can't see a pattern as to how the data is stored. Can anyone point me in the direction of how to interpret this data?
The hex file produced by SF is a standard Intel hex file, so there's more than just the programming data in the file itself. Also, the Pickit3 Program Memory and EEPROM Data windows will show the entire device memory space and not just the contents of what's in the hex file.

There's a handy little free utility I ran across a few years back that will parse an Intel hex file and show the contents based on the addresses in the file, so it's easy to see program data, config fuses, EEPROM init data, etc. It seems the original site isn't available anymore, so I've attached a copy of it here.


There are two basic approaches when it comes to bootloaders:

- Write/use a PC application that reads the Intel .hex file and transfers the binary data to the PIC using a light-weight protocol. This is the most common approach and lets you use a small bootloader on the PIC side since most of the "heavy lifting" is on the PC side. I've seen bootloaders as small as a few hundred bytes using this method. The downside is you need to use a special PC app to do the transfers, but it does give you control of both ends that way.

- Use a bootloader that can directly interpret an Intel hex file. That's the approach I normally use in my bootloaders, and I usually add the ability to transfer the file via a serial connection using XMODEM-CRC. That way you can directly transfer the hex file from the compiler with no massaging and all you need on the PC side is a terminal app that supports XMODEM transfers, so no special app required.
XMODEM-CRC uses small 128-byte packets so it works with pretty much any PIC, and the protocol is simple to implement. Another nice thing about this is that you get the benefits of two checksums, since the Intel HEX file has a checksum per line and XMODEM checksums the 128-byte packet. That helps overcome one of the complaints about XMODEM's simple xsum which can be fooled by multiple bit errors.

The downside to this is the size... the ones I've written in SF tend to be MUCH larger than the first approach. With a simple command-line interface, UART driver, hex file parser and XMODEM transfer the bootloader ends up being just under 4K bytes in size. I've never tried using the RN42. so that would add to the size/complexity.


As far as where to write the data (internal vs ext EEPROM) that's up to you. I usually just write it directly to the device. If the download/programming fails for some reason you can always use the bootloader to retry the operation, or load in the old version.

Keep in mind that if you want to support programming of all three memory spaces on a 46K22 (program, internal EEPROM, config), you'll need a bit more than 64K of storage, although allowing the bootloader to change CONFIG data can be dangerous.
Attachments
HexLister.zip
(266.75 KiB) Downloaded 276 times

SHughes_Fusion
Posts: 219
Joined: Wed Sep 11, 2013 1:27 pm
Location: Chesterfield

Re: Bluetooth (serial) Bootloader

Post by SHughes_Fusion » Mon Aug 11, 2014 8:41 am

Hi Jerry,

Thanks. I'd found conflicting information when I googled the file type for PIC hex files. Good to have it confirmed.

I almost certainly won't need to write the EEPROM - I store configuration data in there which would carry over to the updated software. I might have to be clever with the update process though in case I need to add extra configuration information if, for example, I add a new function to the unit.

Luckily I won't need to do any error checking - that is all handled by the Bluetooth protocol so I can assume if data arrives it has transmitted correctly. The RN42 should be fairly easy to use actually as it acts just like a serial device to the PIC.

Would you say there is any real benefit in keeping the bootloader size down to under 2k if possible so I can take advantage of the boot block code protection? All updating will be done either from a PC or a smartphone and luckily we have PC developers in another part of the company I can ask to do the software if necessary.

SHughes_Fusion
Posts: 219
Joined: Wed Sep 11, 2013 1:27 pm
Location: Chesterfield

Re: Bluetooth (serial) Bootloader

Post by SHughes_Fusion » Mon Aug 11, 2014 10:10 am

Daft question, but how do you decode the data in to a 16 bit word for the PIC?

For example, the first data line of a hex file is:
:0400000032EF0EF0DD

The PICKit software shows the first two words as being EF32 F00E.

From the datasheet you use a table write, one byte at a time, to write the program memory.

So, do I just write bytes as they are presented in the hex file, or do I need to swap each pair?

Also, it appears the hex file doesn't always contain enough data to fill a block. The example above is only 4 bytes but the next data is for location 8. Does it matter if I fill the 'missing' bytes with 00 or FF?

Lastly, is it 'safe' to just add each incoming byte using a byte variable then invert and add 1 to get the checksum?

Jerry Messina
Swordfish Developer
Posts: 1469
Joined: Fri Jan 30, 2009 6:27 pm
Location: US

Re: Bluetooth (serial) Bootloader

Post by Jerry Messina » Mon Aug 11, 2014 11:35 am

Would you say there is any real benefit in keeping the bootloader size down to under 2k if possible so I can take advantage of the boot block code protection?
If you're using a bootloader that resides in low-memory the extra protection is a nice feature, if you can use it. Either way I always add logic to the programming/erasing part to protect writes to the bootloader area just in case.
So, do I just write bytes as they are presented in the hex file, or do I need to swap each pair?
Write them as they appear. The PICkit app presents the prog memory as a word, but you write them in the order they are in the file.
Also, it appears the hex file doesn't always contain enough data to fill a block. The example above is only 4 bytes but the next data is for location 8. Does it matter if I fill the 'missing' bytes with 00 or FF?
The erased state of the flash is FF, so I always start with an "empty" buffer the size of a flash page that's filled with all FF's. Both instruction codes (00 or FF) are a NOP, so there's no real point in programming an unused byte with 0.
Lastly, is it 'safe' to just add each incoming byte using a byte variable then invert and add 1 to get the checksum?
I just add all the bytes (including the xsum) in a hex line together, skipping the beginning ':' character. The sum should be 0. The math is inherently done in 2's compliment.

SHughes_Fusion
Posts: 219
Joined: Wed Sep 11, 2013 1:27 pm
Location: Chesterfield

Re: Bluetooth (serial) Bootloader

Post by SHughes_Fusion » Mon Aug 11, 2014 3:34 pm

Thanks again, Jerry.

A few more questions....

Are there any reasons for or against using an interrupt in the bootloader? And if I do use one, what options do I need to use in the bootloaded program to ensure its interrupt works correctly? I was planning to use an interrupt to handle serial reception but there is no reason why I can't just do it polled if using an interrupt causes problems.

And on that subject, what options do I need to set to allow the main program to run still and are there any options I need to set in the bootloader code to ensure it runs correctly? Or do I just write the bootloader like a normal program?

Is there any way to tell Swordfish to only include the code in the hex file? If I don't have to watch out for the config settings and skip them then that will save a few words of code space. I know I can edit them out but would be easier to not have them generated in the first place as I'm sure to forget to remove them manually at some point.

And lastly, I'm planning to test a byte in EEPROM to see if there is a working program in the chip. The main program will write a value to this byte early on when it starts. What is the best way to jump from the bootloader to the main program?

All new to me but potentially a quite exciting thing to implement.

Jerry Messina
Swordfish Developer
Posts: 1469
Joined: Fri Jan 30, 2009 6:27 pm
Location: US

Re: Bluetooth (serial) Bootloader

Post by Jerry Messina » Mon Aug 11, 2014 6:07 pm

Are there any reasons for or against using an interrupt in the bootloader?
All against, at least in my book.

Take a look at what's in the first page of the PIC18 program memory. There you'll find three important sections:

addr 0x0000 - reset vector. this is the always the first instruction executed, and is normally a GOTO instruction.
addr 0x0008 - high priority intr vector
addr 0x0018 - low priority intr vector

You need to think about where you're going to locate the bootloader (bottom of memory/top of memory), and what effect that has on those first few locations. When you go to program the flash, you must first erase it, and erase operations are done in blocks (aka pages). The size of the erase block varies with device, but for an 18F45K22 it's 64-bytes. If you want to change any of those three sections you have to erase them all. This can leave you in a bad position since if power goes down after the first page is erased you now have a bricked device, unable to bootload.
do I just write the bootloader like a normal program?
When you're building your code/bootloader SF has a few built-in options that can help relocate things. Check out the Code Relocation section in the Language Reference portion of the online help for #option org_reset, vector_isr_hi, vector_isr_lo, and org_program

There's a couple of approaches here...

1). Application in low memory/bootloader at top

This is probably the simplest, since the reset/intr vector/app program can be built normally without having to massage anything.
It can be tricky reprogramming though since the only way to get into the bootloader is for the application to jump to it somehow,
or if you have a completely erased device (except for the bootloader). Remember all FF's are NOP's, so if you try to run an erased device eventually you'll get around to executing the bootloader at the top of memory. #option org_reset may help here, but there's always a hazard if you erase that first page.

2). Bootloader in low memory/application after it

With this arrangement the bootloader always runs first, and it's up to it to decide if it's ok to branch to the application or not.
This is nice in that you can do whatever you like initially... wait for a byte, read an input pin, check EEPROM, run a CRC on the flash, etc. Here, the reset vector and intr vectors are part of the bootloader space unless you move the beginning of the bootloader past the first page. But, since you NEVER want to erase the bootloader, you can't reprogram the intr vectors. You need to build the bootloader using #option vector_isr_hi/vector_isr_lo and build the application using #option org_program to get it located after the bootloader. While you have to build things correctly, this is the method I prefer.


Here's a snippet from the low-memory bootloader I use, and a sub showing how to jump to the application from the bootloader

Code: Select all

//
//-----------------------------------------------------------------------------
// rom addresses and interrupt vector relocation
//-----------------------------------------------------------------------------
//
// boot block size, in bytes (2K)
#option BOOT_BLOCK_SIZE = 2048

// application program start address
// this must match the application '#option org_reset = xxxx' setting
#define PROGRAM_START = BOOT_BLOCK_SIZE

const PROGRAM_START_ADDR as longword = PROGRAM_START

// redirect interrupts to application code
#define VECTOR_HI_OFFSET = $0008
#define VECTOR_LO_OFFSET = $0018

#option vector_isr_hi = PROGRAM_START + VECTOR_HI_OFFSET
#option vector_isr_lo = PROGRAM_START + VECTOR_LO_OFFSET

// constrain ROM so that it doesn't exceed the bootblock size
#variable _maxrom = BOOT_BLOCK_SIZE
#if _maxrom <> BOOT_BLOCK_SIZE
  #warning "_maxrom unconstrained"
#endif


sub run_application()
    // set default registers so the app thinks we just started...
    SetRegisterDefaults()

    // clear STKPTR, leaving the stack error bits intact
    CLEAR_STKPTR()

    // reset wdt
    clrWDT()

    // here we go...
    asm
        goto PROGRAM_START_ADDR
    end asm
end sub
There's a number of variations of the above and different methods you can use. Just take the time to think through the ramifications of each one. There's nothing worse than an unreliable bootloader.
Is there any way to tell Swordfish to only include the code in the hex file?
Not that I know of. I usually just have the bootloader ignore those sections just in case I screwup and build the bootloader/app with different CONFIG settings. While you can reprogram them, that's asking for trouble since you could brick the device this way too.

Search the forums here for more info on the #options and how to use them. Mchip also has some app notes on bootloaders that might be worth a read.

SHughes_Fusion
Posts: 219
Joined: Wed Sep 11, 2013 1:27 pm
Location: Chesterfield

Re: Bluetooth (serial) Bootloader

Post by SHughes_Fusion » Tue Aug 12, 2014 8:33 am

Hi Jerry,

Excellent points. I must admit I'd had brainfade when it came to the interrupt vectors, for some reason I'd thought the were RAM so could be changed when the main program ran...

I did read the code relocation section but couldn't seem to make much of it. I don't know if it is just me but I find a lot of the help files are written more as an aide memoire rather than a clear description of what the commands or options actually do and how to use them.

Am I right in thinking the following:
* org_reset in effect sets an 'offset' when building a program so everything is shifted up in ROM by that much.

* If you use vector_isr_xxx then your interrupt will take slightly longer to execute as the PIC will jump an additional time, once to the original vector, from there to the 'new' vector and from there to the interrupt itself? And org_reset will build the program so that the jump to the interrupt code is in the same relative location so you need to set these as in your sample code?

* org_program actually controls the Goto instruction written to address zero so is best left alone if I'm writing a low bootloader?

I don't use both interrupts in my program so I wonder if I can use high priority in the bootloader and low priority in the main program or vice-versa. I'll have to read up on the interrupt setting options on this PIC to see if there will be any clash.

Jerry Messina
Swordfish Developer
Posts: 1469
Joined: Fri Jan 30, 2009 6:27 pm
Location: US

Re: Bluetooth (serial) Bootloader

Post by Jerry Messina » Tue Aug 12, 2014 11:44 am

You've got it right.
I don't use both interrupts in my program so I wonder if I can use high priority in the bootloader and low priority in the main program or vice-versa
I've never tried that but off the top of my head it sounds like it might have issues. When using interrupt priorities the high priority has an effect on the low. While there is an individual low priority intr enable flag (GIEL), disabling the high intr (GIEH) actually disables both.

It might work if you're really careful but you'd have to make sure that all of the high priority sources were masked when you wanted to use the LP. FWIW I've never had a situation where the bootloader required interrupts, and I prefer it that way. Interrupts just complicate things, especially when you're erasing/programming memory. You want the bootloader to be as bulletproof as possible. If it were me, I wouldn't try it.

SHughes_Fusion
Posts: 219
Joined: Wed Sep 11, 2013 1:27 pm
Location: Chesterfield

Re: Bluetooth (serial) Bootloader

Post by SHughes_Fusion » Tue Aug 12, 2014 12:46 pm

I'm trying re-writing it to see how it runs. The main thing I need to be aware of is that the RN42 transmits at 115200 baud so even with my PIC running at 64MHz I need to respond fairly quickly to cope. I'm hoping I can pause transmission after each record by controlling the RTS line - I've done this on the main app but that is more tolerant of data over-runs and the like. The new code will accept one record but I don't know what happens if I send more than one and I've not got it actually writing code memory yet.

One question I can't find a clear answer to anywhere.

Is there a 'standard' for the Intel Hex files. I'm writing the parser and am wondering if I can make certain assumptions, for example that a record will be no more than 16 data bytes, that data will always be sequential, even that each record will always cover 16 bytes of program memory. I of course want to make it robust but the more I can assume the smaller the parser will be.

Jerry Messina
Swordfish Developer
Posts: 1469
Joined: Fri Jan 30, 2009 6:27 pm
Location: US

Re: Bluetooth (serial) Bootloader

Post by Jerry Messina » Tue Aug 12, 2014 1:49 pm

I'm hoping I can pause transmission after each record by controlling the RTS line
That's where using some sort of download protocol on the PC side helps. I use 115K baud @ 64MHz (no interrupts, no handshaking) w/out any real issues, but that's with XMODEM's 128-byte blocks.
Is there a 'standard' for the Intel Hex files
Sure... just google Intel Hex file format

About the only thing Mchip has to say is from the MPASM users guide:

Code: Select all

1.7.5.3 INTEL HEX 32 FORMAT
The extended 32-bit address hex format is similar to the hex 8 format, except that the 
extended linear address record is output also to establish the upper 16 bits of the data 
address. This is mainly used for 16-bit core devices since their addressable program 
memory exceeds 64 kbytes.

Each data record begins with a 9-character prefix and ends with a 2-character 
checksum. Each record has the following format:

	:BBAAAATTHHHH....HHHCC

where:

BB 		A two digit hexadecimal byte count representing the number of data bytes that will appear on the line.
AAAA 	A four digit hexadecimal address representing the starting address of the data record.
TT 		A two digit record type:
			00 - Data record 
			01 - End of File record
			02 - Segment address record
			04 - Linear address record
HH 		A two digit hexadecimal data byte, presented in low byte/high byte combinations.
CC 		A two digit hexadecimal checksum that is the two's complement of the sum of all preceding bytes in the record.
The record length COULD be up to 255, but realistically normally you'll only see up to 16 bytes of data per record. There's nothing that guarantees the records are in order, but I can't say I've ever seen a hex file that isn't, at least with SF and the Mchip tools. Dealing with out of order records would be a major pain.

The one thing you do have to deal with is data records of varying length... they aren't always 16 bytes. For example you'll get shorter records for the reset and intr vectors, and also the very last data record unless the program is exactly a multiple of 16 bytes in length. That HexLister program is handly for looking at some of this.

SHughes_Fusion
Posts: 219
Joined: Wed Sep 11, 2013 1:27 pm
Location: Chesterfield

Re: Bluetooth (serial) Bootloader

Post by SHughes_Fusion » Tue Aug 12, 2014 2:57 pm

At the moment I'm hoping I can just dump the hex file in to a terminal and have it program automatically. I think my terminal allows a delay after each line so I could use that if necessary. Longer-term a proper app would be the way to go though.

I've looked at plenty of sites for the Intel standard, including the original Intel document that describes it, but none seem to lay down any guidelines. I guess the idea is that they don't want you to assume anything to make it as future-proof as possible.

As you say, the biggest issue would be if data wasn't in order, especially if it stepped backwards!

With the processor I have there is no great hassle in allowing for bigger records so I'll increase the buffer size accordingly. I will assume they are in order though and will see how big it makes the program to allow for packets of a size where they will go across device page boundaries. I'd hope that would never happen in a hex file so unless I can allow for it without taking up a lot of space I'll just work under that assumption.

SHughes_Fusion
Posts: 219
Joined: Wed Sep 11, 2013 1:27 pm
Location: Chesterfield

Re: Bluetooth (serial) Bootloader

Post by SHughes_Fusion » Thu Aug 14, 2014 8:58 am

I think I've got the bootloader cracked - it accepts a hex file from a serial terminal and writes it to the PIC. The code executes as expected although I've not done a full test.

At the moment it all fits in under 1600 bytes so there is even room for improvements! At the moment there is no way to enter the bootloader other than via a command to the main program to do so, I need to add a button sequence to do this.

Once I've got it all sorted I'll see if I can post a cut-down version on here. My version is based on hardware which uses the DOGM132 display from Electronic Assembly - it does display a message on the display so I had to adapt and cut down my code for that.

Thanks again for your help, Jerry!

Post Reply