Daily charts not updating

[packman2008]

I've looked at this for a few hours and can't figure it out. My hourly based charts (default and my own extras) are all up to date, but my daily based charts only have data until 20th November, so I'm missing a day. Also I fixed an incorrect data value in the dayfile for 13th Nov and the daily charts that display that value aren't being re-generated so still show atmospheric pressure hitting zero!

Some background:

1) I upgraded CMX from b4027 to b4043 late on 18th Nov. This doesn't seem to have affected anything because I have charts with daily data showing for 19th/20th Nov.
2) I was running CU 7.0.0b1 until this morning. I upgraded to v7.2.0 to see if that would fix the problem but it's still happening with the later version. v7.2.0 seems to be running OK because that's the version being reported by the website and the generated date/time are both updating after uploads.
3) I can't see any errors in the CU logs that seem to relate to this. In particular the data error from 13th Nov shows on a DailyPressure chart and there's no error logged when that chart is being created.
4) I have two sets of extra charts being compiled (charts1.txt and charts2.txt). These two chart files are being created and uploaded and have current timestamps on the web server filesystem.
5) I've run all of my scheduled CU tasks (10min/30min/daily) multiple times both from task scheduler (so pretty much invisible) and manually (so I can see any displayed errors) but the daily charts still don't update.

What have I not checked?

[PaulMy]

Hi,
I presume this is yours https://blackpool.weather-watch.com/cumulus
and presume you mean under the Graphs tab not Charts tab.
The Daily Rain shows to Nov 21st which I presume is correct as the last full day https://blackpool.weather-watch.com/cum ... =DailyRain
Am I looking in the right place?

Enjoy,
Paul

[freddie]

Zip up the contents of the MXdiags folder and attach to a reply to this topic.

How did you change the data on the 13th? Did you use the dayfile editor in the MX web interface, or did you open dayfile.txt in a text editor?

[packman2008]

https://blackpool.weather-watch.com/cum ... lyPressure shows the original dodgy data.

I edited the dayfile using Notepad++ and deleted the whole of the 13th Nov then ran CreateMissing to rebuild it. There was more wrong than just the min pressure for the day so that was easier than trying to figure out which values were wrong and fix them individually. Not sure what went wrong on the 13/14th to cause that.

MXDiags has about 250MB of data in it (I've kept loads of logs from my import of data back to 1979). Is there a specific file or files that are needed? Also are the CMX logs needed or the CU logs from utils\utilslog?

[freddie]

Daily graphs stopping at a date are generally a sign of something wrong in the dayfile. This can show up in logs at start-up, so attach a log file that covers a start-up. The MX log file names just contain numbers that include the date information and have extension .txt

[packman2008]

I restarted to get a new log file (attached). I don't see any errors reported when reading the dayfile but then I'm not an expert at looking at the logs!

[freddie]

Sorry, I forgot to say you likely need debug logging switched on.

[packman2008]

Debug logging was turned on for that restart.

[freddie]

Yes, it was switched on. I couldn't find any errors relating to the dayfile.

However, looking at the graphs on your site, they all appear up to date to me, with the last displayed date of the 21st on the daily graphs. I could only find one graph with pressure on and that shows normal ranges - definitely no zero pressure reading.

So I'm confused as all looks well to me.

[HansR]

When back on monday I'll take a look, now for a few days away. Sorry.

[packman2008]

The charts have updated since I last looked at them and the zero pressure problem has gone away. I'm not sure what caused the charts to update though. I ran all the scheduled CU tasks manually and through task scheduler yesterday without any charts updates.

It shouldn't be caching...I cleared the cache of the browser I normally use and tried another browser that I hadn't accessed the site with recently. I also turned off Cloudflare caching for a while but that didn't help either.

The charts still seem to be a day behind though. It's 23rd Nov now but the daily charts only include up to 21st Nov. Are there some daily files updated by CMX that CU uses which aren't updated by the time I run the daily CU task? I run the daily task at 01:15 so that should give enough time.

[packman2008]

I just fixed this. The daily charts only had data up to 21st Nov which seemed odd as I'd only seen them a day behind before. I found that the alldaily* data files weren't being uploaded at the end of each day. Looking at the diagnostic logs for the nightly upload showed an error, but not where I was expecting. I'd forgotten to update the server directory for the NOAA files when I changed my server directory structure. The update of those files was generating an error which killed the SFTP process. With no SFTP process all the nightly files queued up for upload after the NOAA files failed. I've fixed the directory error and forced an upload. The daily charts are all now showing data up to midnight last night.

My mistake was to look for errors with the CU uploads rather than elsewhere in the logs

[HansR]

Great reading while underway in a delayed train ( )
Another issue solved not a cause being CU

[mcrossley]

Do you have a log file for that failure please Chris, I've tried to make all the different "bits" of the upload independent so the failure of one should not affect the others, obviously I failed, or the FTP component failed in a way that it could not continue.