Page 1 of 1

how to best handle a lot of data

Posted: Sun 24 May 2015 5:37 pm
by n9mfk
Hi All,
i have monthly data back to 2009
what the best way to load it in a database?
thanks Beau

Re: how to best handle a lot of data

Posted: Sun 24 May 2015 7:44 pm
by mcrossley
Use the ImportCumulus script?

You either have to run it manually for each month, or write a little wrapper script to loop through each monthly file in a folder.

Re: how to best handle a lot of data

Posted: Mon 25 May 2015 6:43 am
by steve
Or use ExportMySQL from MX, which lets you upload a month at a time, but will also do all months in one go (which may take some time). The ImportCumulusFile script may well be quicker as it will typically be running on the same machine as the the MySQL server (or be very close to it with a good link, as is the case with some providers).

Re: how to best handle a lot of data

Posted: Mon 25 May 2015 11:20 pm
by n9mfk
Mark i i did the script how long would you pause it to give the php time to run?
Beau

Re: how to best handle a lot of data

Posted: Tue 26 May 2015 7:24 am
by mcrossley
If you run it synchronously then there would be no need to pause. Remember though that PHP normally has a maximum default run time of 30 seconds (iirc), so if your DB is a bit slow the script may time out before all the entries are added, so keep a log and check for error messages.

I'd forgotten about Steve's new utility, that may be a better option, as Steve said possibly a bit slower, but utilmately simpler to use.

Re: how to best handle a lot of data

Posted: Tue 26 May 2015 12:29 pm
by n9mfk
I tried ExportMySQL with mx stoped it seemed slow

Re: how to best handle a lot of data

Posted: Wed 27 May 2015 11:49 pm
by n9mfk
Hi all here a update I have tested both options i have 77 fines in both seem to overwhelm godady msql sever
is there a easy way i can setup a mysql database on my pc fill it then export it to my host?
beau