Page 1 of 1

import Cumulus file.php doing bad, worse and worse

Posted: Fri 16 May 2014 6:43 am
by mm23
So, in the beginning everything works well and importCumulus php file worked well. By sometime, I have trouble opening realtime.txt file from the server. Sometimes it opens the entire file, sometimes just part of the file and sometimes just the last line.
Almost everything is working well :D , including a cron job every minute, however, missing data in the realtime table but cron job is executed well. People from web hosting say that everything works well, but I'm missing data in realtime table in the database and on the graph on the website.

Any help, please........

http://www.vrijemeusibeniku.com/weather/graf1.php

Re: import Cumulus file.php doing bad, worse and worse

Posted: Fri 16 May 2014 7:51 am
by mcrossley
Do you have the ftp 'rename' option enabled in Cumulus?

Re: import Cumulus file.php doing bad, worse and worse

Posted: Fri 16 May 2014 5:23 pm
by rogerthn
mm23 wrote:... sometime, I have trouble opening realtime.txt file from the server. Sometimes it opens the entire file, sometimes just part of the file and sometimes just the last line....
I've seen something similar (not Cumulus) and the reason was that sometimes transfer of the file wasn't complete when processing started and sometimes a new file was created before processing was complete.

Re: import Cumulus file.php doing bad, worse and worse

Posted: Fri 16 May 2014 6:19 pm
by mm23
mcrossley wrote:Do you have the ftp 'rename' option enabled in Cumulus?
No, I have not

Re: import Cumulus file.php doing bad, worse and worse

Posted: Fri 16 May 2014 6:25 pm
by mm23
rogerthn wrote:
mm23 wrote:... sometime, I have trouble opening realtime.txt file from the server. Sometimes it opens the entire file, sometimes just part of the file and sometimes just the last line....
I've seen something similar (not Cumulus) and the reason was that sometimes transfer of the file wasn't complete when processing started and sometimes a new file was created before processing was complete.
Hm, I am not sure.. In the beginning everything was ok .... now, there is no rule ..... sometimes works great, sometimes not and since last night does not work at all.....I mean web pages...

Re: import Cumulus file.php doing bad, worse and worse

Posted: Fri 16 May 2014 10:41 pm
by mcrossley
mm23 wrote:
mcrossley wrote:Do you have the ftp 'rename' option enabled in Cumulus?
No, I have not
If you enable it and your web site supports it OK, then that may fix the problem of the truncated files for you.

Re: import Cumulus file.php doing bad, worse and worse

Posted: Fri 16 May 2014 11:48 pm
by mm23
OK Mark, I'll try it. ....
Today afternoon, I turned off the cron job on the server and turned on again http comand from Toolbox and everything is start to "working" again. The only problem is that the http command activates on every hour and cron job every minute..... The question is why the "system" does not work when it activates by a cron job, and it works when it is activated by http command from the Toolbox ....?

Re: import Cumulus file.php doing bad, worse and worse

Posted: Sat 17 May 2014 6:47 am
by rogerthn
mm23 wrote:... The question is why the "system" does not work when it activates by a cron job, and it works when it is activated by http command from the Toolbox ....?
What does The crontab line look like?
Maybe you can try adding something like >> /tmp/my.log 2>&1
Please note the >> i.e. append i.e. the file will get bigger and bigger ;-)

Re: import Cumulus file.php doing bad, worse and worse

Posted: Sat 17 May 2014 10:21 am
by mm23
After the http command from Toolbox worked well all night, I decided to turn back to old settings (re-activate the cron job on the server - one minute interval) and now everything is ok again. I did not activate the ftp rename option (for now) as Mark suggested - thanks Mark. What happened still do not know.
Php command is ......php -q /path to/importcumulusfile.php type=realtime key=***** table=***** file=yourrealtimefile retainVal=2 retainUnit=day......and as you see the table size is limited on two days

Anybody have any suggestion what might be causing my problem because I doubt that my problem still has not been solved by long-term?

Thanks

Re: import Cumulus file.php doing bad, worse and worse

Posted: Sat 17 May 2014 1:11 pm
by mcrossley
It just a clash between the FTP upload and the cron job reading the file. Using the rename should eliminate the small time period when there is a partial file available to read.

Edit: phone tryping

Re: import Cumulus file.php doing bad, worse and worse

Posted: Sat 17 May 2014 3:21 pm
by mm23
OK, Mark thanks. I will do so and see what will happen......