v3.24.1-b3234 EOD PHPupload dayfile.txt not uploading

[PaulMy]

Hi,
All the extra web files are being updated except dayfile.txt
I have many to upload at realtime and at the 5-minute interval, and 3 at End of Day. Two of the EoD upload ok at midnight but dayfile.txt has failed the last 3 days:

Code: Select all

2023-03-14 00:00:01.770 PHP[Int]: Uploading dayfile.txt

Code: Select all

2023-03-14 00:00:12.565 PHP[Int]: dayfile.txt: Response code = 422: 422
2023-03-14 00:00:12.565 PHP[Int]: dayfile.txt: Response text follows:
Error: TimeStamp is out of date
Data TS   = 1678766401
Server TS = 1678766413

2023-03-14 00:00:12.565 PHP[Int]: Upload process complete

The dayfile.txt is quite large at 1,517,800
The next largest file is alldailytempdata.json at 1,311,550 and this is uploading.
Could the size affect the TimeStamp discrepancy?

Enjoy,
Paul

[mcrossley]

Possibly - the error message is saying there is a 12 second difference between the timestamp on CMX and your web server. But because of the way PHP works with web servers, it cannot check the timestamp on the server until all the file content is uploaded.

I do have a ToDo already about this, but that doesn't help you [now] with your massive dayfile

Your only option would be to increase the allowed time window from 10 seconds to say 15 - line 79 of upload.php.


[Ideally I would use a two-step upload known as 100 Continue. This allows you to send the request headers, the server checks these are Ok and sends back a 100 continue message, then CMX would upload the data. It avoids this type of issue where you waste time and bandwidth sending all the data, only for it to be rejected when a check fails. Unfortunately, you cannot use that protocol with PHP ]

[PaulMy]

Thanks Mark.
I hadn't detected/understood that 12 seconds difference. I've changed upload.php and will see test result after midnight.

Enjoy,
Paul

[freddie]

For something like the dayfile, could just the delta be uploaded?

[mcrossley]

For something like the dayfile, could just the delta be uploaded?

see...

I do have a ToDo already about this, but that doesn't help you [now] with your massive dayfile

[freddie]

Fair enough

[PaulMy]

I can confirm that dayfile.txt was uploaded at midnight rollover after changing line 79 in upload.php from 10 to 15

Code: Select all

if (abs($receivedTime - $requestTime) > 15) {

MXdiags:

Code: Select all

2023-03-15 00:00:01.757 PHP[Int]: Upload process starting
2023-03-15 00:00:01.760 PHP[Int]: Uploading Extra file[0]: C:\KomokaWeather\CumulusMXwll\data\dayfile.txt

2023-03-15 00:00:02.167 PHP[Int]: Uploading dayfile.txt

2023-03-15 00:00:10.731 PHP[Int]: dayfile.txt: Response code = 200: OK
2023-03-15 00:00:10.731 PHP[Int]: dayfile.txt: Response text follows:
Unzipping data
Opening text file dayfile.txt for replacement
Writing text file dayfile.txt
Write complete

Enjoy,
Paul