Cumulus Support

Over the last few days I've had the opportunity to start testing the Cumulus MX beta, and certainly the logging seems to work OK, and I was beginning to sort out the format changes. I've yet to understand how the new charts work, but that was my next task. I gave it a 6 hour run yesterday to see how it went, but then got an email from my hosting company to say that my ftp rate had increased 300 fold, and as a result I had now exceeded my allotted monthly bandwidth (5Gb) in 1 day. Fortunately they have now doubled my bandwidth for this month to allow me to continue using my web pages, but they obviously expect me sort out the problem, and I've stopped using CumulusMX

I checked my ISP usage and it is quite normal, so the CumulusMX program running on the Pi is not directly responsible for the FTP problem. The hosting company say some scripts or programs in my weather directory are running on the server and are continuously making FTP connections to an external location and loading files from it almost constantly. They suggest that this is the source of the massive bandwidth increase. The last setting was for the pages to update every five minutes, and the 'upload standard pages' was ticked. The index page uploaded successfully at that rate.

I'm running the latest MX version on a Raspberry Pi B v1 with 8Gb SD card with a direct Lan connection. I took my Cumulus 1 directory and loaded it on to the SD card, and then unzipped the MX software into the same directories. My pages don't use gauges or 'realtime', just the index and a page for the graphs. Being well away from any BT exchange I use a satellite internet connection.

Were you using the standard pages supplied with MX, or your own pages with the same names? None of the scripts supplied with the standard web site make ftp connections - well, my scripts certainly don't, and I'm reasonably sure that none of the third-party scripts supplied do either.

The only differences between the supplied pages with MX and Cumulus 1 are that the gauges page uses Mark's SteelSeries gauges, which are in use by a large number of people without any problems, and the graphs use Highcharts and are drawn in the browser using data files loaded (using Ajax, not ftp) from the server (upload by MX at the same interval as the html pages).

It seems rather odd that the best the hosting company can say is that "some scripts or programs" on the server are "making FTP connections to an external location". Can they not be more specific? Which scripts, and which external location? Note that the scripts supplied with the standard pages are all javascript and hence run in the browser, not on the server. The scripts only run when someone is viewing the associated page. There is no server-side scripting supplied with the standard pages. To run on the server they would have to be PHP or similar.

I asked my web host to give me further details about the scripts they thought were generating the FTP connections, and after checking further they are saying that they made a mistake, and it is NOT your scripts that are generating the FTP connections. I pass that apology on to you. They say that an external source is responsible for making FTP connections resulting in 5Gb of uploads to the server in the last few days. I've checked my ISP usage chart and in the last 6 days I have only uploaded 212Mb of data, which is typical for me.

Before I tackle my webhost over their figures, is there any way that CumulusMX can ftp 5Gb of data using standard pages over about the 12 hours or so of testing that I have done over the last few days?

Is there anything fundamentally different with the the way CumulusMX uploads its data, (or the data itself) that might cause a server's logger to misread, or be confused about the amount of data it is being sent. I'm now back to running Cumulus 1, and the server is recording the usual amount of data being sent to it.

The ftp code in MX is by necessity completely different from the code in Cumulus 1. In both cases they are handled by a third party component and Cumulus itself has no involvement in the actual transfers.

If they are now saying that it is your uploads (the 'external source' must be you, if it's your ftp account) that account for 5GB in 12 hours, it's difficult to see how that could be the case when your usage is only 212Mbytes in 6 days. So presumably their meters are wrong, as you suggest. I don't know what it could be about the way the ftp component in MX works that could confuse their servers. The component is well respected and widely used - https://netftp.codeplex.com/

The standard pages are less than 100Kbytes in total, and the graph data is about 500Kbytes. To generate traffic of 5GB in 12 hours, it would need to be uploading those files every 5 seconds. But there is still the matter of your ISP meters.

If you want to try a test for a few minutes with MX, you could turn on ftp logging and see what the ftplog.txt file says. You could also monitor your network usage locally at the same time.

Actually I did have the ftp log ticked for my extended test yesterday - in the spirit of a beta test!

There was an ftp session every 5 minutes between 09:50 and 17:35
This is the last session at 17:35, but it is typical of all of them. On this occasion it uploaded 577kb which is in keeping with your estimate. They range between 300 and 580kb each time.
There may, or may not be a clue amongst all these 4 letter abbreviations and 3 number codes!

220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
220-You are user number 11 of 50 allowed.
220-Local time is now 17:35. Server port: 21.
220-This is a private system - No anonymous login
220-IPv6 connections are also welcome on this server.
220 You will be disconnected after 15 minutes of inactivity.
USER greetham
331 User greetham OK. Password required
PASS <omitted>
230 OK. Current restricted directory is /
FEAT
211-Extensions supported:
EPRT
IDLE
MDTM
SIZE
MFMT
REST STREAM
MLST type*;size*;sizd*;modify*;UNIX.mode*;UNIX.uid*;UNIX.gid*;unique*;
MLSD
AUTH TLS
PBSZ
PROT
UTF8
TVFS
ESTA
PASV
EPSV
SPSV
ESTP
211 End.
Text encoding: System.Text.UTF8Encoding
OPTS UTF8 ON
200 OK, UTF-8 enabled
2015-07-16 17:35:16.342 Uploading web/index.htm to public_html/weather/index.htm
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/index.htmtmp
550 Can't check for file existence
EPSV
229 Extended Passive mode OK (|||36661|)
STOR public_html/weather/index.htmtmp
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
226 2.191 seconds (measured here), 2.79 Kbytes per second
RNFR public_html/weather/index.htmtmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/index.htm
250 File successfully renamed or moved
2015-07-16 17:35:28.306 Uploading web/today.htm to public_html/weather/today.htm
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/today.htmtmp
550 Can't check for file existence
EPSV
229 Extended Passive mode OK (|||20907|)
STOR public_html/weather/today.htmtmp
150 Accepted data connection
Disposing FtpSocketStream...
Disposing FtpSocketStream...
226-File successfully transferred
226 2.210 seconds (measured here), 2.30 Kbytes per second
RNFR public_html/weather/today.htmtmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/today.htm
250 File successfully renamed or moved
2015-07-16 17:35:40.792 Uploading web/yesterday.htm to public_html/weather/yesterday.htm
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/yesterday.htmtmp
550 Can't check for file existence
EPSV
229 Extended Passive mode OK (|||15979|)
STOR public_html/weather/yesterday.htmtmp
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
226 1.865 seconds (measured here), 1.93 Kbytes per second
RNFR public_html/weather/yesterday.htmtmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/yesterday.htm
250 File successfully renamed or moved
2015-07-16 17:35:51.412 Uploading web/record.htm to public_html/weather/record.htm
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/record.htmtmp
550 Can't check for file existence
EPSV
229 Extended Passive mode OK (|||33427|)
STOR public_html/weather/record.htmtmp
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
226 2.175 seconds (measured here), 3.15 Kbytes per second
RNFR public_html/weather/record.htmtmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/record.htm
250 File successfully renamed or moved
2015-07-16 17:36:02.454 Uploading web/trends.htm to public_html/weather/trends.htm
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/trends.htmtmp
550 Can't check for file existence
EPSV
229 Extended Passive mode OK (|||13022|)
STOR public_html/weather/trends.htmtmp
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
226 1.174 seconds (measured here), 1.23 Kbytes per second
RNFR public_html/weather/trends.htmtmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/trends.htm
250 File successfully renamed or moved
2015-07-16 17:36:10.591 Uploading web/gauges.htm to public_html/weather/gauges.htm
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/gauges.htmtmp
550 Can't check for file existence
EPSV
229 Extended Passive mode OK (|||45355|)
STOR public_html/weather/gauges.htmtmp
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
226 2.322 seconds (measured here), 12.99 Kbytes per second
RNFR public_html/weather/gauges.htmtmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/gauges.htm
Disposing FtpSocketStream...
Disposing FtpSocketStream...
Disposing FtpSocketStream...
Disposing FtpSocketStream...
250 File successfully renamed or moved
2015-07-16 17:36:21.420 Uploading web/thismonth.htm to public_html/weather/thismonth.htm
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/thismonth.htmtmp
550 Can't check for file existence
EPSV
229 Extended Passive mode OK (|||8178|)
STOR public_html/weather/thismonth.htmtmp
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
226 2.309 seconds (measured here), 3.22 Kbytes per second
RNFR public_html/weather/thismonth.htmtmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/thismonth.htm
250 File successfully renamed or moved
2015-07-16 17:36:33.698 Uploading web/thisyear.htm to public_html/weather/thisyear.htm
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/thisyear.htmtmp
550 Can't check for file existence
EPSV
229 Extended Passive mode OK (|||46652|)
STOR public_html/weather/thisyear.htmtmp
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
226 2.225 seconds (measured here), 3.46 Kbytes per second
RNFR public_html/weather/thisyear.htmtmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/thisyear.htm
250 File successfully renamed or moved
2015-07-16 17:36:46.666 Uploading web/monthlyrecord.htm to public_html/weather/monthlyrecord.htm
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/monthlyrecord.htmtmp
550 Can't check for file existence
EPSV
229 Extended Passive mode OK (|||12226|)
STOR public_html/weather/monthlyrecord.htmtmp
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
226 2.346 seconds (measured here), 10.62 Kbytes per second
RNFR public_html/weather/monthlyrecord.htmtmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/monthlyrecord.htm
250 File successfully renamed or moved
2015-07-16 17:36:58.888 Uploading graphconfig.json to public_html/weather/graphconfig.json
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/graphconfig.jsontmp
550 Can't check for file existence
EPSV
229 Extended Passive mode OK (|||28671|)
STOR public_html/weather/graphconfig.jsontmp
Disposing FtpSocketStream...
Disposing FtpSocketStream...
Disposing FtpSocketStream...
Disposing FtpSocketStream...
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
226 1.376 seconds (measured here), 133.76 bytes per second
RNFR public_html/weather/graphconfig.jsontmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/graphconfig.json
250 File successfully renamed or moved
2015-07-16 17:37:10.825 Uploading tempdata.json to public_html/weather/tempdata.json
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/tempdata.jsontmp
550 Can't check for file existence
EPSV
229 Extended Passive mode OK (|||17767|)
STOR public_html/weather/tempdata.jsontmp
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
226 3.686 seconds (measured here), 39.58 Kbytes per second
RNFR public_html/weather/tempdata.jsontmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/tempdata.json
250 File successfully renamed or moved
2015-07-16 17:37:23.846 Uploading pressdata.json to public_html/weather/pressdata.json
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/pressdata.jsontmp
550 Can't check for file existence
EPSV
229 Extended Passive mode OK (|||32464|)
STOR public_html/weather/pressdata.jsontmp
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
226 2.358 seconds (measured here), 13.69 Kbytes per second
RNFR public_html/weather/pressdata.jsontmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/pressdata.json
250 File successfully renamed or moved
2015-07-16 17:37:35.560 Uploading winddata.json to public_html/weather/winddata.json
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/winddata.jsontmp
550 Can't check for file existence
EPSV
229 Extended Passive mode OK (|||18105|)
STOR public_html/weather/winddata.jsontmp
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
226 2.699 seconds (measured here), 20.86 Kbytes per second
RNFR public_html/weather/winddata.jsontmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/winddata.json
250 File successfully renamed or moved
2015-07-16 17:37:47.922 Uploading wdirdata.json to public_html/weather/wdirdata.json
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/wdirdata.jsontmp
550 Can't check for file existence
EPSV
Disposing FtpSocketStream...
Disposing FtpSocketStream...
Disposing FtpSocketStream...
Disposing FtpSocketStream...
229 Extended Passive mode OK (|||33601|)
STOR public_html/weather/wdirdata.jsontmp
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
226 2.973 seconds (measured here), 18.58 Kbytes per second
RNFR public_html/weather/wdirdata.jsontmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/wdirdata.json
250 File successfully renamed or moved
2015-07-16 17:38:00.931 Uploading humdata.json to public_html/weather/humdata.json
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/humdata.jsontmp
550 Can't check for file existence
EPSV
229 Extended Passive mode OK (|||17874|)
STOR public_html/weather/humdata.jsontmp
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
226 2.722 seconds (measured here), 19.60 Kbytes per second
RNFR public_html/weather/humdata.jsontmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/humdata.json
250 File successfully renamed or moved
2015-07-16 17:38:13.530 Uploading raindata.json to public_html/weather/raindata.json
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/raindata.jsontmp
550 Can't check for file existence
EPSV
229 Extended Passive mode OK (|||18145|)
STOR public_html/weather/raindata.jsontmp
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
226 2.800 seconds (measured here), 20.05 Kbytes per second
RNFR public_html/weather/raindata.jsontmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/raindata.json
250 File successfully renamed or moved
2015-07-16 17:38:26.129 Uploading solardata.json to public_html/weather/solardata.json
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/solardata.jsontmp
550 Can't check for file existence
EPSV
229 Extended Passive mode OK (|||9194|)
STOR public_html/weather/solardata.jsontmp
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
226 3.013 seconds (measured here), 26.64 Kbytes per second
RNFR public_html/weather/solardata.jsontmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/solardata.json
250 File successfully renamed or moved
2015-07-16 17:38:39.720 Uploading dailyrain.json to public_html/weather/dailyrain.json
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/dailyrain.jsontmp
550 Can't check for file existence
EPSV
229 Extended Passive mode OK (|||30521|)
STOR public_html/weather/dailyrain.jsontmp
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
226 1.843 seconds (measured here), 346.12 bytes per second
RNFR public_html/weather/dailyrain.jsontmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/dailyrain.json
250 File successfully renamed or moved
2015-07-16 17:38:50.846 Uploading sunhours.json to public_html/weather/sunhours.json
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/sunhours.jsontmp
550 Can't check for file existence
EPSV
229 Extended Passive mode OK (|||28999|)
STOR public_html/weather/sunhours.jsontmp
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
226 1.351 seconds (measured here), 470.13 bytes per second
RNFR public_html/weather/sunhours.jsontmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/sunhours.json
250 File successfully renamed or moved
2015-07-16 17:39:00.008 Uploading dailytemp.json to public_html/weather/dailytemp.json
TYPE I
200 TYPE is now 8-bit binary
SIZE public_html/weather/dailytemp.jsontmp
550 Can't check for file existence
EPSV
229 Extended Passive mode OK (|||14040|)
STOR public_html/weather/dailytemp.jsontmp
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
226 1.931 seconds (measured here), 1.00 Kbytes per second
RNFR public_html/weather/dailytemp.jsontmp
350 RNFR accepted - file exists, ready for destination
RNTO public_html/weather/dailytemp.json
250 File successfully renamed or moved
Disposing FtpClient object...
QUIT
221-Goodbye. You uploaded 577 and downloaded 0 kbytes.
221 Logout.
Disposing FtpSocketStream...
Disposing FtpClient object...
Disposing FtpSocketStream...
Disposing FtpSocketStream...
Disposing FtpSocketStream...
Disposing FtpSocketStream...
Disposing FtpSocketStream...
Disposing FtpSocketStream...
Disposing FtpSocketStream...

That all looks fine to me. If it's running every 5 minutes, it's hard to see how it could be uploading 5GB in 12 hours.

In some ways that's good to know. I'll have to challenge their bandwidth figures, and get them to look at it in detail. (if they have the inclination) If they do, and if they find anything, I'll let you know. It might be useful.

Its a shame that I'll have to put any further testing on hold though. Like a lot of people here, I was impressed with the way you got the little plastic box to do everything the big PC was doing. I was looking forward to eventually freeing the laptop from its burden!

I'd like to turn off the standard files and graph files and just select files one by one from the 'user files' list. That way I'm hoping to see if any particular file is the trigger for this server problem.

I can see how to turn those off but I'm having difficulty working out the format for the 'local file' box.
In Cumulus 1, I would use something like C:\Cumulus\web\indexT.htm but I'm not familiar with the format used in the Raspbian OS. The Pi is simply 'pi', and the folder is CumulusMX and the sub-folder 'web'. If someone could provide me with the correct format - I would be very grateful.

Linux uses forward slashes for directory separators (like ftp servers), so if you have installed MX in the home directory of the 'pi' user, then you want

/home/pi/CumulusMX/web/indexT.htm

Ah - that is where I was going wrong. Many thanks

Steve

I've had a opportunity to upload each of the standard files individually over a period of time. None of those have triggered a problem with my web server. So from my aspect they are 'safe' for me to use. I can at least get the index page etc running from the Pi which is good news.

I now want to try out these 'json' files in a similar way. They seem to be located in the main CumulusMX, but it looks as if they have not been generated since I switched off the uploading.
Firstly I just wanted to check that although the standard files are still generated by CumulusMX when the upload is switched off, the 'json' files are not generated when their upload is switched off. I just want to make sure it is working for me as you intended.

If that is the case, I can still experiment with the last generated files, by uploading them singly and checking the effect (if any) on the bandwidth.

logjam wrote:Firstly I just wanted to check that although the standard files are still generated by CumulusMX when the upload is switched off, the 'json' files are not generated when their upload is switched off.

Yes, that's correct. Inconsistent, really.

Well, I've finished testing uploading those json files, and I can confirm that it is uploading them by ticking the box that triggers my hosts bandwidth logging problem. I accept that no one else seems to have this problem, and that I'll have to find my own solutions. I must say that it is still well worth having Cumulus on the Raspberry Pi, even with the absence of graphs.

If it is possible, at some time, to work it so that the json files are generated without having to upload them all, I may be able selectively upload a few of them at a reduced frequency, which may help. I accept that it would be a very low priority considering the work that still has to be done, - - but I hope there is no harm in asking.

Well I do not understand this as I have been uploading using MX since Steve first released it i.e since January.

I am not uploading the processed web pages as I use PHP but I am uploading the JSON files, various PHP webtag files (including one of 88K) every 15 minutes, and 3 realtime files every 30 seconds, plus the dayfile.txt(400K) and another 25k PHP file daily, and at no time in the last 7 months has my FTP bandwidth exceeded 4.3GB which would suggest you are measuring something incorrectly.

I would also suggest that 5GB is an extremely small bandwidth per month, my hosting service (Vidahost) gives me 25GB per month for the princely sum of £23.20 per year!!

It sounds like a bug in the ftp server software, where it calculates the size of the upload incorrectly for json files, for some reason. Clearly you aren't actually using the bandwidth that it says you are, as it would show up in your ISP upload stats as well, and also on your PC - presumably you have looked at the transfer stats on your PC and confirmed that it's actually only using bandwidth appropriate to the size of the files.