Files not transfrerred to website

[newburyweather]

Hi,

I'm trying CumulusMX and have part of of it working but some parts don't. I have read through and executed the instruction in the wiki but obviously I may have made a mistake.

I have manage to create easyweather.dat file and using http://192.168.x.x:8998/ I can see that CMX is running and I'm getting data.

On this(these) page(s) I can see real time data and gauges but no graphs. Is this normal or is something wrong?

I have set it up to ftp files to my web site. The raspberry pi I'm using is the web server and the ftp transferrs them from one directory to another. It manages to ftp realtime.txt and there are no failures in the ftp log but there are no htm files ftp'd. They are not listed as transfer failures in the log, there is no attempt to transfer them. There are htm files in the web directory under CMX. No index.htm is ftp'd

[edit] the files in the wiki "("graphconfig.json", "tempdata.json", "pressdata.json", "winddata.json", "wdirdata.json", "humdata.json", "raindata.json", "solardata.json", "dailyrain.json", "sunhours.json", "dailytemp.json")." are not generated in the CumulusMX top level directory.

Also I have a crash report from a previous run if that it useful for general debug. I don't think it's related to this but from error in saving the ini file.

Also in the MXDiag files I get sequences of "*** Data input appears to have stopped" but every time I look at the dat file under "data" it contains a date of a few moments ago.

Something is wrong and it's probably finger trouble on my part but can someone help?

I'll keep trying to find a solution but if there's a simple solution it'd be great to know it.

If needed I can upload the log files.

Many thanks

Andy

I'm running this on raspbian wheezy on a raspberry pi 2+

[water01]

Have you ticked "Include graph data files" in the Internet Settings.

Also if you are using the Pi as a web server as well as running Cumulus why are you FTP the files. Why not just copy them from one location to another within the Pi file system. This is achieved by have the local file system in the remote file area and not ticking FTP and doing the same for the standard files i.e. putting the local file system directory structure in.

See here in the wiki http://wiki.sandaysoft.com/a/FAQ#Runnin ... as_Cumulus Note the file limitation bit in the interface does NOT apply to MX you can access all 100 files via the web interface.

[newburyweather]

Have you ticked "Include graph data files" in the Internet Settings.

Yes I have.

I think I understand what you mean about

This is achieved by have the local file system in the remote file area

I think you mean map the web server root directory to the MX directory so it can server the files directly. OR just using the extra web files copy table instead of FTP.

BUT I'm not getting any htm or json files generated as suggested by the wiki so I can't transfer any thing. Something is stopping the generation of the files or I'm misunderstanding the wiki.

On thing that comes to mind is if the Esayweatherplus.dat file was poorly formed. Could Cumulus pull out a number of data points and then fail. I have no way of validating the data other than by running CumulusMX

Many thanks
Andy

[steve]

On thing that comes to mind is if the Esayweatherplus.dat file was poorly formed. Could Cumulus pull out a number of data points and then fail.

This is the most likely explanation for what you're seeing - or it could simply be a bug in MX with the operation of the easyweather.dat code, which is relatively untested. But more information would be useful, it's hard to keep guessing (and I hate having to guess) without a clear description of what you are doing and what you see in MX (do you see all of the data displayed on the user interface, for example? Do the graphs in the UI work OK?). Attaching an example of your easyweather file would be helpful, as would zipping up the MX diags folder and attaching it (with debug logging turned on in Cumulus.ini).

[newburyweather]

Many thanks for your reply.

I'm busy this afternoon and early evening but I'll try and give all info I can asap.

Cheers

Andy

[newburyweather]

Managed to get some time.
On the UI I Get

Dashboard: Sensible readings for Temperature, Wind Chill Wind Dir(fixed at N) and update times,
Now: Sensible readings for Temperature/Humidity table, indoor humidity Rain rate(Fixed at 0.5mm)
Gauges: Sensible readings for Inside and outside temps, inside humidity, wind dir (fixed at N), UV index (fixed at 1)
Charts: 1 blank chart with Temperature up lhs vertical.
Today/Yesterday and record have values were I've provided some value.

The EasyWeatherPlus.dat file is based on the wiki table. The tail end of the data string is taken from the wiki example.

Cheers

Andy

[steve]

You're not providing a valid pressure reading, so this FAQ applies - http://wiki.sandaysoft.com/a/FAQ#Cumulu ... eb_uploads

There may or may not be other issues, I'll check further when I get the time.

[newburyweather]

Hi,

I have some success. I have corrected some of the data file issues, found a few more and corrected them.

I now get the creation of the htm, json files but I get

Code: Select all

2015-12-27 23:00:02.071 Connecting to newburyweather.org.uk
220 FTP Server Ready
USER newburyweather
331 Password required for newburyweather
PASS <omitted>
230 User newburyweather logged in
FEAT
211-Features:
 MDTM
 MFMT
 TVFS
 UTF8
 MFF modify;UNIX.group;UNIX.mode;
 MLST modify*;perm*;size*;type*;unique*;UNIX.group*;UNIX.mode*;UNIX.owner*;
 SITE MKDIR
 SITE RMDIR
 SITE UTIME
 SITE SYMLINK
 REST STREAM
 LANG en-GB.UTF-8*
 SITE COPY
 SIZE
211 End
Text encoding: System.Text.UTF8Encoding
OPTS UTF8 ON
200 UTF8 set to on
2015-12-27 23:00:02.480 Uploading web/index.htm to newburyweather.org.uk/public_html/index.htm
DELE newburyweather.org.uk/public_html/index.htm
550 newburyweather.org.uk/public_html/index.htm: No such file or directory
2015-12-27 23:00:02.497 Uploading web/today.htm to newburyweather.org.uk/public_html/today.htm
DELE newburyweather.org.uk/public_html/today.htm
550 newburyweather.org.uk/public_html/today.htm: No such file or directory
2015-12-27 23:00:02.510 Uploading web/yesterday.htm to newburyweather.org.uk/public_html/yesterday.htm
DELE newburyweather.org.uk/public_html/yesterday.htm
550 newburyweather.org.uk/public_html/yesterday.htm: No such file or directory
2015-12-27 23:00:02.524 Uploading web/record.htm to newburyweather.org.uk/public_html/record.htm
DELE newburyweather.org.uk/public_html/record.htm
550 newburyweather.org.uk/public_html/record.htm: No such file or directory
2015-12-27 23:00:02.537 Uploading web/trends.htm to newburyweather.org.uk/public_html/trends.htm
DELE newburyweather.org.uk/public_html/trends.htm
550 newburyweather.org.uk/public_html/trends.htm: No such file or directory
2015-12-27 23:00:02.550 Uploading web/gauges.htm to newburyweather.org.uk/public_html/gauges.htm
DELE newburyweather.org.uk/public_html/gauges.htm
550 newburyweather.org.uk/public_html/gauges.htm: No such file or directory
2015-12-27 23:00:02.565 Uploading web/thismonth.htm to newburyweather.org.uk/public_html/thismonth.htm
DELE newburyweather.org.uk/public_html/thismonth.htm
550 newburyweather.org.uk/public_html/thismonth.htm: No such file or directory
2015-12-27 23:00:02.579 Uploading web/thisyear.htm to newburyweather.org.uk/public_html/thisyear.htm
DELE newburyweather.org.uk/public_html/thisyear.htm
550 newburyweather.org.uk/public_html/thisyear.htm: No such file or directory
2015-12-27 23:00:02.593 Uploading web/monthlyrecord.htm to newburyweather.org.uk/public_html/monthlyrecord.htm
DELE newburyweather.org.uk/public_html/monthlyrecord.htm
550 newburyweather.org.uk/public_html/monthlyrecord.htm: No such file or directory
2015-12-27 23:00:02.608 Uploading graphconfig.json to newburyweather.org.uk/public_html/graphconfig.json
DELE newburyweather.org.uk/public_html/graphconfig.json
550 newburyweather.org.uk/public_html/graphconfig.json: No such file or directory
2015-12-27 23:00:02.622 Uploading tempdata.json to newburyweather.org.uk/public_html/tempdata.json
DELE newburyweather.org.uk/public_html/tempdata.json
550 newburyweather.org.uk/public_html/tempdata.json: No such file or directory
2015-12-27 23:00:02.637 Uploading pressdata.json to newburyweather.org.uk/public_html/pressdata.json
DELE newburyweather.org.uk/public_html/pressdata.json
550 newburyweather.org.uk/public_html/pressdata.json: No such file or directory
2015-12-27 23:00:02.651 Uploading winddata.json to newburyweather.org.uk/public_html/winddata.json
DELE newburyweather.org.uk/public_html/winddata.json
550 newburyweather.org.uk/public_html/winddata.json: No such file or directory
2015-12-27 23:00:02.664 Uploading wdirdata.json to newburyweather.org.uk/public_html/wdirdata.json
DELE newburyweather.org.uk/public_html/wdirdata.json
550 newburyweather.org.uk/public_html/wdirdata.json: No such file or directory
2015-12-27 23:00:02.678 Uploading humdata.json to newburyweather.org.uk/public_html/humdata.json
DELE newburyweather.org.uk/public_html/humdata.json
550 newburyweather.org.uk/public_html/humdata.json: No such file or directory
2015-12-27 23:00:02.692 Uploading raindata.json to newburyweather.org.uk/public_html/raindata.json
DELE newburyweather.org.uk/public_html/raindata.json
550 newburyweather.org.uk/public_html/raindata.json: No such file or directory
2015-12-27 23:00:02.706 Uploading solardata.json to newburyweather.org.uk/public_html/solardata.json
DELE newburyweather.org.uk/public_html/solardata.json
550 newburyweather.org.uk/public_html/solardata.json: No such file or directory
2015-12-27 23:00:02.719 Uploading dailyrain.json to newburyweather.org.uk/public_html/dailyrain.json
DELE newburyweather.org.uk/public_html/dailyrain.json
550 newburyweather.org.uk/public_html/dailyrain.json: No such file or directory
2015-12-27 23:00:02.735 Uploading sunhours.json to newburyweather.org.uk/public_html/sunhours.json
DELE newburyweather.org.uk/public_html/sunhours.json
550 newburyweather.org.uk/public_html/sunhours.json: No such file or directory
2015-12-27 23:00:02.748 Uploading dailytemp.json to newburyweather.org.uk/public_html/dailytemp.json
DELE newburyweather.org.uk/public_html/dailytemp.json
550 newburyweather.org.uk/public_html/dailytemp.json: No such file or directory
Disposing FtpClient object...
QUIT
221 Goodbye.
Disposing FtpSocketStream...

and they don't get ftpd to the site.

So I manually copied the json and the htm files that are in the web directory using filezilla and I have a sort of working site.

I get NOW and TREND values on the web site but the gauge screen shows all zero's and there is a box above the gauges that has "error:Not Found" in it. What is it that it has not found?

You can view this at newburyweather.org.uk if it helps.

I'll get back to this in the morning.

Many thanks

Andy

[steve]

and they don't get ftpd to the site.

Are there any error messages in the MX diags regarding the upload of the file? The ftp component doesn't appear to be attempting the upload after the delete fails because the file doesn't exist (there's no particular reason that it should care). And if this is your server, you could turn off 'delete before upload' and configure your server to not require that - it's not a good thing to require. But as has been said, copying the files makes much more sense than using FTP on the same machine.

there is a box above the gauges that has "error:Not Found" in it. What is it that it has not found?

Your browser will tell you, if you look at the F12 console. It's the realtimegauges.txt file:

GET http://newburyweather.org.uk/realtimega ... 1296532196 404 (Not Found)

[newburyweather]

Hi Steve,

Much progress made.
I disabled the ftp and used the extra files to transfer them.
All pages seem to work except the trend pages. These seems to work for 1/2 hour and are now stopped at 16:30, the time now is 17:50.

I'm getting "*** Data input appears to have stopped" in the MXdiag files but as the current data is OK I can't see if that's the problem.
The json and htm files are only a minute out of date so they must be getting transferred.

Any thoughts as to the issue?

Cheers

Andy

[steve]

The graphs use the json files, so if they're being updated and copied successfully, the graphs should be up to date. Perhaps the files are being cached by your browser? You should be able to diagnose the problem using the browser console.

Regarding the 'data stopped' messages - at what interval have you configured Cumulus to read the file? How are you generating the file? It does a File.Exists() before attempting to read the file, perhaps it doesn't exist when it needs to read it. MX doesn't actually check the timestamps in the file, it just uses the last line as new data every time it reads it.

[water01]

"*** Data input appears to have stopped"

means Cumulus is telling you it is not receiving data from your Weather Station console for over a minute.

[newburyweather]

Hi,

The data file were being updated every minute on the minute. I only write one line. I have change it so that they are produced 5 secs before the minute in case of clashes. It didn't make any difference to the " Data input appears to have stopped" message.

The trends are still ending at 16:30. The files are update every 10 secs or so. I've changes that to 45 secs.
The first part of the temdata.json file trasnferred to the webserver area reads
{"intemp":[[1451319000000,22.5],[1451319540000,22.5],[1451319600000,22.5],[1451319660000,22.5],[1451319720000,22.5],

I deduce the numbers 1451319000000 represent epoch time in millisecs. The first one converts to GMT: Mon, 28 Dec 2015 16:10:00 which is the beginning of the graph.
In the root under CumulusMX there are two files tempdata.json and tempdata.jsontmp. The .json file is timestamped 16:30 and the .jsontmp is timestamped 23:01 but they are both identical. I seems that they are not being updated.

And thoughts?

I've attached MXdiags and ini file if it helps.

Cheers

Andy

[steve]

The data file were being updated every minute on the minute. I only write one line. I have change it so that they are produced 5 secs before the minute in case of clashes. It didn't make any difference to the " Data input appears to have stopped" message.

But at what interval have you configured Cumulus to read the file?. Cumulus checks for data stopped once a minute, so if you are only reading them once a minute, there's a chance that the data stopped check will trigger before the new data is read. You can turn off the message if you wish by adding ReportDataStoppedErrors=0 to the [Station] section of Cumulus.ini.

In the root under CumulusMX there are two files tempdata.json and tempdata.jsontmp

Cumulus only creates files locally with tmp on the end for processed files - it sounds like you have ticked 'process' for these files? That's likely to cause problems, given the structure of these files. 'Process' is only for files which need to be processed to convert web tags to real data.

[steve]

I see from your Cumulus.ini file that you do have 'process' set for the graph data files, so you need to unset that. I also see that you are using the 'realtime' update, with an interval of 65 seconds, to do your copying. This is somewhat unexpected, as the graph data files are generated at the 'normal' update interval (which you also have set to 65 seconds). It is possible that the realtime copying of the files is interfering with the generation of the files. It would be better not to have realtime set for the json files, so that their copying is guaranteed to take place just after their update.

You are also using the realtime update to process the web pages. These are also processed by the 'normal' update thread. It creates them even though you don't have 'Include standard files' set, so that they are available if required even though they are not selected for FTP. It is possible that the standard processing and your realtime processing will interfere. It would be better to unset 'realtime' for those, and you could also unset 'process' but copy the already processed file instead of processing the template. There probably should be a separate setting to turn of automatic processing of the standard files completely.