For New Users

[HansR]

@all:
I understand you are experimenting to get things running, but I notice on the map that you don't have name, description nor latitude or longitude filled in the basic Cumulus Setup. So I will list in this message some preconditions needed for CumulusUtils (and Cumulus itself for that matter) to run properly. If you follow these rules you can avoid fails and frustration. If you have any remarks or run into problems, let me know and I will adapt this message so to smooth the process of starting up. It is not all that difficult, but there are some rules to follow.

NOTE: a map cleanup exists. If you do not run cumulusutils website, you will be removed from the map. For sending the signature file and receiving the assembled map, on Linux you need to have OpenSSL 1.1.0. See release notes for version 6.6.0 on how to do that.

You cannot use the website generator if there is less than 32 days of data and/or elementary information does not exist in the system. The generator will then exit. In such case, see the log for information. Ask questions in this thread if you don't understand.

This message contains some information on how to use CumulusUtils as a starter.

1. Read the Wiki. It is far from perfect, but it does contain some important information. If you find errors, please notify me.
2. The working directory MUST be the CumulusMX directory. Because you are to install CMX wherever you want, there is no assumption of where it is and the computer is not being searched. You just have to start there. All utils directories are relative to the CMX directory.
3. To use CumulusUtils properly, you need at least 32 days of data. With less days in the system only some of the data driven modules will work. SysInfo, MapsOn and Forecast will always work because they do not need weather data. So set up your weather station, fine tune it and be patient for a month. Check the running of Cumulus[MX].
4. If you already have a long series of data you may want to use Create Missing utility.
5. Make sure your Cumulus version is running smoothly and that your station has it's name, description and location defined in Cumulus.ini . These are important identification parameters. CumulusUtils' Website Generator won't run without these at all.
6. Follow the installation instruction in the Wiki and run the command once with the WEBSITE or ALL commandline parameter (depending on whether you wish to use the website or just the modules):

   Code: Select all

   utils/bin/cumulusutils.exe website

   This will create the cumulusutils.ini file with most of the required parameters.
7. Fill in the parameter 'Website' in the section [Maps]
8. If you run into problems, check your data first (or start with this):
   • utils/bin/cumulusutils.exe CheckOnly
   • Correct any errors
   • utils/bin/cumulusutils.exe CheckOnly
   • Repeat while necessary
9. Try running with the default values first. Then start changing the inifile bit by bit. Focus on content first, then on appearance.
10. If you wish to create your website with CumulusUtils, try running the modules first without uploading. Then upload. If all works well, run the website generator:

    Code: Select all

    utils/bin/cumulusutils.exe website

11. Decide how you wish to use cumulusutils.ini: modulewise - using the modules in your existing website - or as a website generator, being the basis of your website. Make yourself acquainted with the important related parameters:

    Code: Select all

    [General]
    IgnoreDataErrors=true/false
    DoLibraryIncludes=true (for modular use)/ false (for Website Generator use)
    GeneratejQueryInclude=true (for modular use) /false (for Website Generator use)
    Language=nl-NL
    

    And:

    Code: Select all

    [FTP site]
    DoUploadFTP=true/false
    UploadDir=
    

    And:

    Code: Select all

    CumulusRealTimeLocation=
    

12. When running as a website, make sure you have configured CMX properly for receiving the realtime data.
13. Having run several times with different commandline parameters, recheck the inifile: some parameters may have been added.
14. Don't ignore the logfiles!
15. Best is to run in a 24 hr cycle for website or individual modules. Run a sysinfo command at any frequency you want.

Most of what follows after these first steps should be self-evident but if you have questions, don't hesitate to ask.
If I don't reply, the other (first) users have a lot of experience by now and no doubt can answer.

When you are sufficiently confident about CumulusUtils result, use your own locale (and translation), start changing it's appearance and create your usermenu for the personal touch. Step by Step is the way to go.

Despite the original ambition to create something simple, it turns out the variety of user demands and possibilities is so great the complexity increases automatically.
We have to live with it.

[rogerthn]

I've been using Cumulus since 2008 and I'm now on my third WS an Ecowitt GW1003.
The first (National Geographic 265 NE) did die on 2017-12-19.
The second (Oregon Scientific WMR88) has been running since 2013-04-07 but I did started to see some ???issues??? with the wind speed and/or direction hence the Ultrasonic Anemometer Sensor
The GW1003 has been operational with CumulusMX since yesterday at 10AM
https://rogerthn.se/weather3/index.htm

Now the question, do I need to wait 31 more days before I can use cumulusutils?
From the log file

Code: Select all

2020-07-04 17:33:36.008 Timing of Dayfile load = 37 ms
2020-07-04 17:33:36.012  Main CmulusUtils: Not enough data. Only 1 lines in dayfile.txt
2020-07-04 17:33:36.012  Main CmulusUtils: Need at least 32 days for useful output.
2020-07-04 17:33:36.012 Main CmulusUtils: Exiting!

and

Code: Select all

2020-07-04 17:31:43.045 CumulusUtils : Entering Main
2020-07-04 17:31:43.102 CommandLineArgs : starting
2020-07-04 17:31:43.102  CommandLineArgs : handling arg: SysInfo
2020-07-04 17:31:43.103 CommandLineArgs : End
2020-07-04 17:31:43.111 SystemStatus : starting
2020-07-04 17:31:43.115 Exception Unknown : Index was outside the bounds of the array.
2020-07-04 17:31:43.121 Data (cont): cumulusutils
2020-07-04 17:31:43.134 Data:   at CumulusUtils.Support.StationInUse (System.Int32 i) [0x00062] in <55e5f348dd014d098d5e31987fd9a8ff>:0
  at CumulusUtils.SysInfo.GenerateSystemStatus () [0x00056] in <55e5f348dd014d098d5e31987fd9a8ff>:0
  at CumulusUtils.CMXutils.RealMain (System.String[] args) [0x000e1] in <55e5f348dd014d098d5e31987fd9a8ff>:0
  at CumulusUtils.CMXutils.Main (System.String[] args) [0x000ee] in <55e5f348dd014d098d5e31987fd9a8ff>:0
2020-07-04 17:31:43.134 Exiting - check log file

[HansR]

I handle the second issue first: your run for sysinfo.
It is clearly a crash when it starts to try to find the information for the station you have entered in CumlusMX. Clearly that is a station unknown to CumulusUtils meaning that recently CMX must have added one or more devices and I did not enhance the list. This is a bug, both for the missing device as well for the fact that it crashes. I will fix that and let you know.

The second issue is that of 32 days.
Apparently you started all over again and did not keep your old data files. And yes, the dashboard and Home-graphs are the 'real-time' weatherdata.
But because the main reporting of CumulusUtils is on statistical reporting on the weather time series I thought it better to have at least one month and a day in the database. I haven't given that much thought, you are the first to meet that limit. If you wish you could really try from scratch, I can put that version out with the fix I'll make.

[rogerthn]

Thanks!
Please let me know if you need any information from me!!

Since the WS is new there isn't any old date
Please don't spend any time on fixing < 32 days but "maybe" there should be a note for new users

[HansR]

@rogerthn:

1. Attached is a patch release which accepts only 2 days of data. I cannot guarantee all statistical reports/graphs will run without error, you will have to try. If it does not work, I'll bring it back to 32 days.
2. The patch also has added the following devices: "Davis WLL", "GW1000". If CMX receives a new device for which I delay introduction in CumulusUtils now "Unknown Station" will be displayed.

[EDIT:]the 32 day fix was so easy I included it. Just worth a try.

[rogerthn]

Better, actions and logs
FileZilla cumulusutils v396patch.zip to /home/pi

Code: Select all

cd ~/CumulusMX
unzip ../cumulusutils\ v396patch.zip
chmod +x cumulusutils.exe
./cumulusutils.exe all

cumulusutils.ini and utils/HighchartsLanguage.js created as below

Code: Select all

cat cumulusutils.ini
[General]
Language=en-GB
IgnoreDataErrors=true

[Forecasts]
ForecastSystem=Yourweather
NorwegianURL=
SevenDayPredictionURL=
WXsimURL=

Code: Select all

cat utils/HighchartsLanguage.js
Highcharts.lang = {lang:{
months:['January','February','March','April','May','June','July','August','September','October','November','December'],
shortMonths:['Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct','Nov','Dec'],
weekdays:['Sunday','Monday','Tuesday','Wednesday','Thursday','Friday','Saturday'],
}
};

highchartsOptions = Highcharts.setOptions(Highcharts.lang);

Log file

Code: Select all

cat utils/utilslog/2007042150cumulusutils.log
2020-07-04 21:50:11.286  Support constructor
2020-07-04 21:50:11.353  Support constructor : Unit Wind (m/s, mph, km/h, kts): 0
2020-07-04 21:50:11.354  Support constructor : Unit Pressure (mb,hPa,inHg): 1
2020-07-04 21:50:11.354  Support constructor : Unit Rain (mm,in): 0
2020-07-04 21:50:11.354  Support constructor : Unit T (C,F): 0
2020-07-04 21:50:11.358  CUstrings[xx].ini : CUstringsEN.ini looked for.
2020-07-04 21:50:11.358  No CUstringsEN.ini found.
2020-07-04 21:50:11.360 CumulusUtils : ----------------------------
2020-07-04 21:50:11.360 CumulusUtils : Entering Main
2020-07-04 21:50:11.417 CommandLineArgs : starting
2020-07-04 21:50:11.418  CommandLineArgs : handling arg: all
2020-07-04 21:50:11.418 CommandLineArgs : End
2020-07-04 21:50:11.429 Prediction : No URL in Prediction ini section.
2020-07-04 21:50:11.430 Prediction : Impossible to continue, exiting procedure.
2020-07-04 21:50:11.433 Timing of WeatherForecast generation = 12 ms
2020-07-04 21:50:11.433 CumulusUtils : Creating class dayfile -> Opening dayfile.txt
2020-07-04 21:50:11.438 Dayfile constructor: Using path: |data/|; file: |dayfile.txt
2020-07-04 21:50:11.440 Dayfile constructor: Working on: data/copy_dayfile.txt
2020-07-04 21:50:11.441 Dayfile constructor: DayfileType is SlashCommaPoint
2020-07-04 21:50:11.471 Dayfile : EOF detected
2020-07-04 21:50:11.472 Timing of Dayfile load = 38 ms
2020-07-04 21:50:11.475  Main CmulusUtils: Not enough data. Only 1 lines in dayfile.txt
2020-07-04 21:50:11.475  Main CmulusUtils: Need at least 2 days for useful output.
2020-07-04 21:50:11.476 Main CmulusUtils: Exiting!
2020-07-04 21:50:11.476 CumulusUtils : Exiting Main
2020-07-04 21:50:11.484  Ini Flush cache modif:True, cumulusutils.ini
2020-07-04 21:50:11.486  Ini Flush writing:General, Language/en-GB
2020-07-04 21:50:11.487  Ini Flush writing:General, IgnoreDataErrors/true
2020-07-04 21:50:11.487  Ini Flush writing:Forecasts, ForecastSystem/Yourweather
2020-07-04 21:50:11.487  Ini Flush writing:Forecasts, NorwegianURL/
2020-07-04 21:50:11.487  Ini Flush writing:Forecasts, SevenDayPredictionURL/
2020-07-04 21:50:11.487  Ini Flush writing:Forecasts, WXsimURL/
2020-07-04 21:50:11.489  Ini Flush cache modif:False, CUstringsEN.ini

./cumulusutils.exe sysinfo

Code: Select all

cat cumulusutils.ini
[General]
Language=en-GB
IgnoreDataErrors=true

[Forecasts]
ForecastSystem=Yourweather
NorwegianURL=
SevenDayPredictionURL=
WXsimURL=

[SysInfo]
ReportWidth=700

Code: Select all

cat utils/systeminfoTable.txt
<div style='margin:auto; text-align:left; width:700px'><pre>
Cumulus version: <#version>, uptime: <#ProgramUpTime>
Weather station: GW1000

Linux/Unix

System uptime: 6 days, 3 hours, 17 minutes
Cumulus version: <#version>, uptime: <#ProgramUpTime>

System/Processor: Unknown (exception) - ApplicationName='lshw', CommandLine='-quiet -class system', CurrentDirectory='', Native error= Cannot find the specified file
Please install lshw (apt-get install lshw)

sudo apt install lshw
./cumulusutils.exe sysinfo

Code: Select all

cat utils/systeminfoTable.txt
<div style='margin:auto; text-align:left; width:700px'><pre>
Cumulus version: <#version>, uptime: <#ProgramUpTime>
Weather station: GW1000

Linux/Unix

System uptime: 6 days, 3 hours, 20 minutes
Cumulus version: <#version>, uptime: <#ProgramUpTime>

System: Raspberry Pi 2 Model B Rev 1.1
Processor: ARMv7 Processor rev 5 (v7l)
Nr of processors: <#CpuCount>
OS: Linux 4.19.118-v7+ armv7l
Description: Raspbian GNU/Linux 10 (buster)
Mono JIT compiler version 5.18.0.240 (Debian 5.18.0.240+dfsg-3 Sat Apr 20 05:16:08 UTC 2019)

Memory info:
MemTotal:         948284 kB
MemFree:          303680 kB
MemAvailable:     739864 kB

Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/root      ext4  916G  3.5G  866G   1% /
/dev/mmcblk0p1 vfat  253M   52M  202M  21% /boot
</pre></div>
<div style ="width:100%; margin-left: auto; margin-right: auto; text-align: center; font-size: 11px;"><a href="https://cumulus.hosiene.co.uk/viewtopic.php?f=44&t=17998" target="_blank">CumulusUtils</a> Version 3.9.6 - generated at 04/07/2020 21:59 - &copy; Hans Rottier </div>

Log file enclosed, more testing tomorrow

[HansR]

Looks good to start with.
Now configure

[sutne]

I've been using Cumulus since 2008 and I'm now on my third WS an Ecowitt GW1003.

Why do you only have data for 2 days when you have been running Cumulus for 12 years?

If you have a new setup for the Ecowitt why don't you just include old /data?

[rogerthn]

Why do you only have data for 2 days when you have been running Cumulus for 12 years?

If you have a new setup for the Ecowitt why don't you just include old /data?

Because:
The first (National Geographic 265 NE) with data from 2008 to 2017 was using Cumulus v3.0.0 (3038)
The second (Oregon Scientific WMR88) with data from 2013 and still running is also using Cumulus v3.0.0 (3038)
The work to convert from Cumulus v3.0.0 (3038) to Cumulus v3.6.10 (3086) is not "insignificant" and with different sensors and different locations ...

[rogerthn]

Looks good to start with.
Now configure

Since there are "Index was out of range" in log files it looks like I need to wait, I CAN DO THAT!
BUT
Trying ./cumulusutils.exe MapsOn I do get

Code: Select all

cat utils/utilslog/2007051150cumulusutils.log
2020-07-05 11:50:02.757  Support constructor
2020-07-05 11:50:02.835  Support constructor : Unit Wind (m/s, mph, km/h, kts): 0
2020-07-05 11:50:02.836  Support constructor : Unit Pressure (mb,hPa,inHg): 1
2020-07-05 11:50:02.836  Support constructor : Unit Rain (mm,in): 0
2020-07-05 11:50:02.837  Support constructor : Unit T (C,F): 0
2020-07-05 11:50:02.841  CUstrings[xx].ini : CUstringsEN.ini looked for.
2020-07-05 11:50:02.842 CumulusUtils : ----------------------------
2020-07-05 11:50:02.843 CumulusUtils : Entering Main
2020-07-05 11:50:02.911 CommandLineArgs : starting
2020-07-05 11:50:02.912  CommandLineArgs : handling arg: MapsOn
2020-07-05 11:50:02.912 CommandLineArgs : End
2020-07-05 11:50:03.176 Maps GetFTPLock : Checking for a lockfile
2020-07-05 11:50:20.791 Map generation : Not a successful lock - abandoning Map generation
2020-07-05 11:50:20.799 Timing of Map generation = 17883 ms
2020-07-05 11:50:20.800 Thrifty: DoTop10 && (!Thrifty || Thrifty && ThriftyTop10RecordsDirty) - False | Uploading Top10 None
2020-07-05 11:50:20.800 Thrifty Thrifty && RunStarted.DayOfYear % 7 == 0 || !Thrifty - True | Uploading = maps.txt
2020-07-05 11:50:20.810  UploadFile: Start
2020-07-05 11:50:20.812  UploadFile Upload=false -> No file(s) uploaded.
2020-07-05 11:50:20.813 Thrifty: DoRecords && (Thrifty && ThriftyRecordsDirty || !Thrifty)- False | Uploading Records None
2020-07-05 11:50:20.813 Thrifty: DoNOAA && (Thrifty && RunStarted.Day == 2 || !Thrifty) - False | Uploading NOAA None
2020-07-05 11:50:20.813 Thrifty: DoDayRecords && (!Thrifty || Thrifty && ThriftyDayRecordsDirty) - False | Uploading DayRecords None
2020-07-05 11:50:20.814 Thrifty: DoWindRose && (Thrifty && RunStarted.Day == 2 || !Thrifty) - False | Uploading WindRose None
2020-07-05 11:50:20.814 Overall Timing all Modules = 17906 ms
2020-07-05 11:50:20.815 CumulusUtils : Exiting Main
2020-07-05 11:50:20.828  Ini Flush cache modif:False, cumulusutils.ini
2020-07-05 11:50:20.830  Ini Flush cache modif:False, CUstringsEN.ini

[sfws]

I can't comment on Cumulusutils or Ecowitt, but since you say the following, I'm hoping Hans won't mind me responding to that in his sub-forum...

The second (Oregon Scientific WMR88) with data from 2013 and still running is also using Cumulus v3.0.0 (3038)
The work to convert from Cumulus v3.0.0 (3038) to Cumulus v3.6.10 (3086) is not "insignificant" and with different sensors and different locations ...

Somebody else on this forum a little while ago updated from v3.0.0 directly to Cumulus v3.5.4 with no issues.

I believe you could do that for your Oregon Scientific WMR88 similarly with no issues. Although you might not be aware of the bugs in your 3.0.0, almost certainly some are present, and your data quality would improve by updating to 3.5.4 with minimal effort (download MX 3.5.4 from https://github.com/cumulusmx/CumulusMX/releases over your existing set-up, best if you take a backup first).

Notice I am NOT suggesting that you join series with different sensors/locations, nor suggesting updating to any MX version 3.6.x.

[rogerthn]

Somebody else on this forum a little while ago updated from v3.0.0 directly to Cumulus v3.5.4 with no issues.

I believe you could do that for your Oregon Scientific WMR88 similarly with no issues. Although you might not be aware of the bugs in your 3.0.0, almost certainly some are present, and your data quality would improve by updating to 3.5.4 with minimal effort (download MX 3.5.4 from https://github.com/cumulusmx/CumulusMX/releases over your existing set-up, best if you take a backup first).

Notice I am NOT suggesting that you join series with different sensors/locations, nor suggesting updating to any MX version 3.6.x.

Thanks!
On top of different sensors/locations the old setups was/is using -lang sv-SE and since I've had e.g. file naming issues with mono updates (still running 3.2.8) I've decided to skip -lang sv-SE.

[HansR]

On top of different sensors/locations the old setups was/is using -lang sv-SE and since I've had e.g. file naming issues with mono updates (still running 3.2.8) I've decided to skip -lang sv-SE.

Yes, unfortunately that is still an issue. The implementation of the locales in mono has become much stricter and as CMX uses the locale for the naming of files, there are countries which can't use CMX if they have the machine locale set to such a value. That is a known issue. CMX should change the filenaming technique such that those datafiles have the same name everywhere.

Trying ./cumulusutils.exe MapsOn I do get etc...

Try again. Apparently there was an issue with the lockfile.
Do you sometimes have internet problems? Are you on a line which can be interrupted?

[rogerthn]

Trying ./cumulusutils.exe MapsOn I do get etc...

Try again. Apparently there was an issue with the lockfile.
Do you sometimes have internet problems? Are you on a line which can be interrupted?

My internet connections is VERY stable but response time could be better

Annotation 2020-07-05 151251.png

Finally

Code: Select all

grep rogerthn /home/pi/CumulusMX/stationswithutils.xml
    <Website>https://rogerthn.se/weather3</Website>

[HansR]

@rogerthn:
So, MAP is OK then. Weird though why it did not work first time. I'll keep an eye on it.

And could you send me the log for a normal run? You say there is still 'index out of range messages.
Not sure I 'll change it but I might, depending on the complexity and consequences.
Just curious.