External program not synchronous with writing files

[jpsc]

Cumulus 3032 on Raspberry Pi

I am processing a webcam image by the simple expedient of writing a shell script file with the tags processed then running it with /bin/bash as the external program. This is the template for the script, for what it's worth:

Code: Select all

#!/bin/bash
daylight=<#isdaylight>
if [ $daylight -eq 1 ]
   then
     raspistill -w 1280 -h 960 -q 75 -dt -o /mnt/thumb/CMX/webcam/%d.jpg -l /mnt/thumb/www/webcam/latest.jpg -a "Mount Pleasant\n<#update>  <#temp><#tempunitnodeg> <#temptrendenglish>  <#currentwdir> <#beaudesc>  <#press> <#pressunit> <#presstrendenglish>"  -a 1025 -ae 32,0x808000,0x8080FF
fi

Several times a day I get errors like "syntax error near unexpected token 'fi'", ie really not that often, say 1 in 100 as I am processing every two minutes.

I think this is because the file is being written as bash is trying to read it.

Is there any way to ensure the file is closed when the external program runs? At the moment it is ExtraLocal14, would moving it up the list help?

[steve]

Thanks for reminding me about that, it's something I noticed myself recently and intend to change. It's a difference between Cumulus 1 and MX - C1 doesn't run the the external program until all of the files have been processed, and MX should do the same. I'll fix it in the next build.

[jpsc]

I still get script errors with 3033 and 3034

[steve]

I moved the execution of the external command to after the processing of the extra files, so I don't understand why you still have an issue, unless it occurs for some other reason. I'll do some experimenting when I get a minute. What settings do you have for this file?

[jank]

Hello jpsc
Change your script like this. Maybe it fixes your Problem
(Improvments are marked in red)

#!/bin/bash
daylight=<#isdaylight>
if [ "$daylight" -eq 1 ] ; then
raspistill -w 1280 -h 960 -q 75 -dt -o /mnt/thumb/CMX/webcam/%d.jpg -l /mnt/thumb/www/webcam/latest.jpg -a "Mount Pleasant\n<#update> <#temp><#tempunitnodeg> <#temptrendenglish> <#currentwdir> <#beaudesc> <#press> <#pressunit> <#presstrendenglish>" -a 1025 -ae 32,0x808000,0x8080FF
fi

[jpsc]

What settings do you have for this file?

The file is marked as executable but I explicitly run it with bash -
Program:

Code: Select all

/bin/bash

Program parameters:

Code: Select all

/mnt/thumb/CMX/webcam/webcam.sh

Change your script like this.

What difference would that make? I've made the change and it hasn't happened since, but the problem is very intermittent.

[steve]

The file is marked as executable but I explicitly run it with bash -

Sorry, I meant the settings in 'extra files' for processing the file.

I've created a test file similar to yours but simpler:

#!/bin/bash
daylight=<#isdaylight>

if [ $daylight -eq 1 ]
then
ls -al > lsal.txt
fi

and I've got Cumulus processing it like this:

2015-11-25 17_24_43-Cumulus MX.png

I'm executing the output file like this:

2015-11-25 17_26_27-Cumulus MX.png

I've had it running for about four hours now without any errors (but I upload at 5 minute intervals). I'll leave it running to see what happens.

I added some more debug logging to be sure that the file was being processed and copied before the attempt to execute it:

2015-11-25 17:25:00.730 Processing extra file /mnt/samsung/cumulus/testcommand.sh
2015-11-25 17:25:00.732 Finished processing extra file /mnt/samsung/cumulus/testcommand.shtmp
2015-11-25 17:25:00.732 Copying extra file /mnt/samsung/cumulus/testcommand.shtmp
2015-11-25 17:25:00.733 Finished copying extra file /mnt/samsung/cumulus/testcommand.shtmp
2015-11-25 17:25:00.735 Executing program /bin/bash /mnt/samsung/cumulus/testcommandop.sh
2015-11-25 17:25:00.742 External program started

MX copies the file using File.Copy(), which is supposed to be synchronous, but I'm wondering if there is actually some delayed writing going on somewhere which means it hasn't necessarily finished writing it when it's executed (but the system should handle that anyway, even if that is the case). What type of drive are you using - SD card? I'm on a USB hard drive. From your file path it looks like you may be using a USB flash drive?

I'm assuming that you are copying the file (i.e. you don't have 'ftp' set), and if so then you could try executing the tmp file which MX creates as the result of the processing, before it does the copy. That would eliminate the copy as the cause of the problem.

Something else you could try (if the problem is something to do with delayed file writes) would be to have a static script launched from MX which does a small delay before it then executes the processed script.

[jpsc]

All of those things.

but, dammit, I also had realtime checked. I guess that may be on a separate thread.

I am writing to a USB stick (old Sandisk U3 1GB), and maybe that does not flush the files. I will be getting a shiny SSD for christmas to go inside my Plusberry Pi case so that will change things in that quarter.

If you do not think that the realtime processing is the problem I will implement your suggestion of a delay via a static script.

[steve]

Yes, selecting realtime processing is quite likely to be the problem, as you have no synchronisation between the processing of the file and executing it. My fix made sure that the external program wasn't run until the 'normal' interval file processing was complete, but that fix has no effect for you as your file isn't being processed at that time. It's inevitable that at some point a realtime update will be taking place and processing the file at the same time that the 'normal' update is trying to execute the file.

[jpsc]

OK, sorry to have wasted your time

[steve]

That's OK - you earned that for reminding me about the genuine issue in the code that needed fixing - which you probably would have hit anyway before the fix if you hadn't had realtime set.