Page 1 of 1
v3.24.1-b3234 EOD PHPupload dayfile.txt not uploading
Posted: Tue 14 Mar 2023 3:19 pm
by PaulMy
Hi,
All the extra web files are being updated except dayfile.txt
I have many to upload at realtime and at the 5-minute interval, and 3 at End of Day. Two of the EoD upload ok at midnight but dayfile.txt has failed the last 3 days:
Code: Select all
2023-03-14 00:00:01.770 PHP[Int]: Uploading dayfile.txt
Code: Select all
2023-03-14 00:00:12.565 PHP[Int]: dayfile.txt: Response code = 422: 422
2023-03-14 00:00:12.565 PHP[Int]: dayfile.txt: Response text follows:
Error: TimeStamp is out of date
Data TS = 1678766401
Server TS = 1678766413
2023-03-14 00:00:12.565 PHP[Int]: Upload process complete
The dayfile.txt is quite large at 1,517,800
The next largest file is alldailytempdata.json at 1,311,550 and this is uploading.
Could the size affect the TimeStamp discrepancy?
Enjoy,
Paul
Re: v3.24.1-b3234 EOD PHPupload dayfile.txt not uploading
Posted: Tue 14 Mar 2023 3:45 pm
by mcrossley
Possibly - the error message is saying there is a 12 second difference between the timestamp on CMX and your web server. But because of the way PHP works with web servers, it cannot check the timestamp on the server until all the file content is uploaded.
I do have a ToDo already about this, but that doesn't help you [now] with your
massive dayfile
Your only option would be to increase the allowed time window from 10 seconds to say 15 - line 79 of upload.php.
[Ideally I would use a two-step upload known as 100 Continue. This allows you to send the request headers, the server checks these are Ok and sends back a 100 continue message, then CMX would upload the data. It avoids this type of issue where you waste time and bandwidth sending all the data, only for it to be rejected when a check fails. Unfortunately, you cannot use that protocol with PHP
]
Re: v3.24.1-b3234 EOD PHPupload dayfile.txt not uploading
Posted: Tue 14 Mar 2023 4:16 pm
by PaulMy
Thanks Mark.
I hadn't detected/understood that 12 seconds difference. I've changed upload.php and will see test result after midnight.
Enjoy,
Paul
Re: v3.24.1-b3234 EOD PHPupload dayfile.txt not uploading
Posted: Tue 14 Mar 2023 6:13 pm
by freddie
For something like the dayfile, could just the delta be uploaded?
Re: v3.24.1-b3234 EOD PHPupload dayfile.txt not uploading
Posted: Tue 14 Mar 2023 7:24 pm
by mcrossley
freddie wrote: ↑Tue 14 Mar 2023 6:13 pm
For something like the dayfile, could just the delta be uploaded?
see...
mcrossley wrote: ↑Tue 14 Mar 2023 3:45 pm
I do have a ToDo already about this, but that doesn't help you [now] with your
massive dayfile
Re: v3.24.1-b3234 EOD PHPupload dayfile.txt not uploading
Posted: Tue 14 Mar 2023 7:25 pm
by freddie
Fair enough
Re: v3.24.1-b3234 EOD PHP upload dayfile.txt not uploading - Resolved
Posted: Wed 15 Mar 2023 3:49 pm
by PaulMy
I can confirm that dayfile.txt was uploaded at midnight rollover after changing line 79 in upload.php from 10 to 15
Code: Select all
if (abs($receivedTime - $requestTime) > 15) {
MXdiags:
Code: Select all
2023-03-15 00:00:01.757 PHP[Int]: Upload process starting
2023-03-15 00:00:01.760 PHP[Int]: Uploading Extra file[0]: C:\KomokaWeather\CumulusMXwll\data\dayfile.txt
2023-03-15 00:00:02.167 PHP[Int]: Uploading dayfile.txt
2023-03-15 00:00:10.731 PHP[Int]: dayfile.txt: Response code = 200: OK
2023-03-15 00:00:10.731 PHP[Int]: dayfile.txt: Response text follows:
Unzipping data
Opening text file dayfile.txt for replacement
Writing text file dayfile.txt
Write complete
Enjoy,
Paul