Monthly detail log file

[serowe]

Not for this - I ran into this previously and I get a different error (can't recall which one right now) when they time out. But on this - the host is 'me' (as it is being run on a server I physically run at home) and I had already increased the script time out from the default to (again from memory) 20 or 30 minutes and the error comes up after about 2-3 minutes. (Just checked and it is currently set to 15 minutes:

Code: Select all

max_execution_time =900

[serowe]

I wonder if doing mysql_unbuffered_query() instead of mysql_query() would help. But I know very little about MySQL so I'm just guessing.

I just tried this and it comes up with the same result.

I also looked into the COMMIT possibility - MySQL has commit turned on by default for every transaction - so, according to the manuals, MySQL will COMMIT the data after EVERY transaction and then proceed to the next one - so don;t believe it is this.

At this stage I am going have to implement one of two things - both a pain really.

Either:

1. split the monthly data file manually every day at a point I know it can process the maximum (in this case around the 18th) and then only process, on a daily basis, the data from that point onwards. This means editing the file every day - not the best scenario.

2. copy then delete the file in the cumulus/data directory at the point at which the file can no longer be processed properly and let Cumulus start from scratch - I need to get Steve to confirm what will happen though if I do this ie will Cumulus, if it can't find the log file, simply start up another one?

Steve - I know you said that it would need a lot of programming to split this, but from what I can see you are already starting a new file at the start of each month and taking the file name from the month and year. Is ther any reason this can't also include the day number? This would allow a daily file to be generated and, IMHO, also allow for a more manageable file size for anyone wanting to analyse or use the data. Because your files don't include headers (this is the one major difference, btw, between Cumulus and all other weather software packages - no, not a criticism, just an observation) if anyone wanted to create a lengthy monthly (or even annual) file, concatenation of the files can be very easily performed on our part.

[steve]

2. copy then delete the file in the cumulus/data directory at the point at which the file can no longer be processed properly and let Cumulus start from scratch - I need to get Steve to confirm what will happen though if I do this ie will Cumulus, if it can't find the log file, simply start up another one?

Yes, it will start a new one. But note that Cumulus won't operate fully without access to complete data files.

Steve - I know you said that it would need a lot of programming to split this, but from what I can see you are already starting a new file at the start of each month and taking the file name from the month and year. Is ther any reason this can't also include the day number?

See my previous reply, I wasn't making it up. I can't believe there isn't a straightforward solution to your problem that doesn't involve a Cumulus redesign.

[serowe]

OK think what was missing in my understanding of the file before - Cumulus actually uses this file then? It isn't just an external log of what has happened?

[steve]

OK think what was missing in my understanding of the file before - Cumulus actually uses this file then? It isn't just an external log of what has happened?

Yes, it uses the data logs.

Here's a suggestion for a solution. Well, not a solution - but I'm sure there is one - more a work around. The script opens a database connection, does N REPLACEs and then closes the connection. This works fine until N reaches a certain value, then something crashes. So put an inner loop in the script which closes the connection before the crash value is reached, and then opens it again. I'm not saying this is a guaranteed fix, because we don't know what the problem actually is, but if it doesn't work it might at least tell you more about the problem.

[serowe]

I've been working on something similar to that today to try and see why it is failing. The only common denominator I have so farcome up with is file size - between 1.8 and 2 Mb the script fails. I broke the log into 3 parts each 10 days in length and they process all OK - they come in at 800k, 1M and 300k.

I then went back to see if the same thing happened in the original script (ie without any of my modifications to add in ther FreeWx log coding) and it does - so it is either in daj original script or connected with size.

I'll get it licked - but having a break tomorrow to try and learn some video editing I have to have done for 7 weeks time - just have to find some appropriate musi to put to videos of mountain bike racing (and have been threatened with castration if I even think of Queen's Fat Bottomed Girls! )

[mcrossley]

By default doesn't PHP have a file upload limit of 2MB, though it can open open local files of 4GB? Is it treating the log file as a file upload for some reason?
(more straw clutching!)

[serowe]

That's what I initially thought but when I re-checked the FreeWx files I was processing, none of them have been over 1.8 M so it wasn'ty possible to check (they didn'thave the same problem).

I've had to implement a maual 'chopping' of the file in order to process it - a PITA but I can't see any other way around it at present.

[steve]

I've just generated a 3MB monthly log file, and successfully created a database table from it on my sandaysoft account at Dreamhost, using David's script.