Register Members List Search Today's Posts Mark Forums Read

Reply
 
Mod Options
Import External Images Details »
Import External Images
Mod Version: 1.0.5, by y2ksw (Coder) y2ksw is offline
Developer Last Online: Jan 2021 I like it Show Printable Version Email this Page

vB Version: 4.x.x Rating: (38 votes - 4.68 average) Installs: 301
Released: 07 Nov 2010 Last Update: 14 Feb 2011 Downloads: 1905
Supported DB Changes Uses Plugins Additional Files Translations  

This vBulletin 4 product imports external images in your posts from external servers or paths into a specific path, normally your forum's image folder.

Differently than the previous product Image Cache for vBulletin 3.x., it parses the posts at regular intervals for new images and replaces the external references in the background. Also, instead of using only a single folder, it creates folders for years and months, making it much easier to handle large quantities of images. The source and destination image URL's are stored into database, too, avoiding unnecessary duplicates.

It moves and optionally converts images from one place to another. It is a handy tool in order to keep your images local to your forums.

Version 1.0.4 is temporarily in BETA stage until the users have confirmed the newly requested features.
Version 1.0.5 is still in BETA stage. It adds a new bunch of additional features, one of which allows you to test your mathematical understanding ... and a few other to fine-tune CURL and socket timeouts, and resizing on the fly of too large images. If you don't know what these options do, please leave them at their default values.

Download Now

Only licensed members can download files, Click Here for more information.

Supporters / CoAuthors

Screenshots

Click image for larger version

Name:	08-11-2010-11.32.53.jpg
Views:	4025
Size:	112.4 KB
ID:	123480   Click image for larger version

Name:	02-02-2011-22.06.39.jpg
Views:	2793
Size:	126.0 KB
ID:	126356  

Show Your Support

  • To receive notifications regarding updates -> Click to Mark as Installed.
  • If you like this modification support the author by donating.
  • This modification may not be copied, reproduced or published elsewhere without author's permission.
  #796  
Old 25 Oct 2017, 16:24
Emad ELsayed's Avatar
Emad ELsayed Emad ELsayed is offline
 
Join Date: Feb 2013
my dear
Does not work on 4.2.5
php 5.6.31
mysql 5.6.37
any help ..
Best Regards
Reply With Quote
  #797  
Old 26 Oct 2017, 09:20
y2ksw's Avatar
y2ksw y2ksw is offline
 
Join Date: Aug 2003
Real name: Giovanni
Originally Posted by Emad ELsayed View Post
my dear
Does not work on 4.2.5
php 5.6.31
mysql 5.6.37
any help ..
Best Regards
for me it works
__________________
Giovanni
Italian support: http://www.vbulletin.it/
Reply With Quote
  #798  
Old 14 Nov 2017, 10:55
Afrika Afrika is offline
 
Join Date: Aug 2004
Is it possible to import images from a specific path from the own forum? The own forum is not imported by default
Reply With Quote
  #799  
Old 15 Nov 2017, 17:45
y2ksw's Avatar
y2ksw y2ksw is offline
 
Join Date: Aug 2003
Real name: Giovanni
Originally Posted by Afrika View Post
Is it possible to import images from a specific path from the own forum? The own forum is not imported by default
Not without changing the code base.

But in reality, you may move the files directly and then change their path with a simple query in all of your posts. This is much more effective than importing and storing the images elsewhere.


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

where old_url is the path as found in one in you posts now, and new_url the path as you wish to have as the new destination.

Keep in mind that the db connection may time out, but you can repeat as often as needed, since UPDATE is smart enough not to change the records which already have been changed. Best of course is to call mysql from the shell.
__________________
Giovanni
Italian support: http://www.vbulletin.it/
Reply With Quote
  #800  
Old 17 Nov 2017, 16:33
gerryvz gerryvz is offline
 
Join Date: Aug 2011
Hi Giovanni,

Great mod, and thank you so much for creating it. I am running 4.2.4 and installed the mod yesterday. I have around 7,300 users and 150,000 total posts, with a relatively small number of overall posts that would require images to be downloaded (I would say the majority of images on my forum are directly attached and residing already in the file system).

Server - Linux + Apache
vbulletin - 4.2.4
php - 5.6.31
php max post size - 64MB
php max upload size - 64MB
php memory limit - 256MB
mySQL - 5.5.7
mySQL packet size - 10MB

The mod started downloading and it got through about 4 of the 9 years of posts on my forum. Now it has frozen since last night. I turned on the logging this morning and only came across this single entry in the log since I turned it on, which is attached. The offending post has some images attached to it, using the external host home.btconnect.com

The images seem to be small in size and pixels, and I downloaded them manually and attached them to the post to see if I could have the mod continue downloading other posts. So far, I haven't been able to get it started again.

Attached is the error, as well as my settings screens.

Also, I have now received two times the mySQL error as attached, which I read appears to be the database server "timing out."

The mod is definitely enabled and running, download folder and sub-folders are 777 permissions, etc.

Does it look to you that there is an image that is too large, or do you have other ideas as to what is causing the mod to stop indexing and downloading?

EDIT: A new post that was made by one of my users today was indexed and the external images were replaced, so I know the code is working. However, it seems stalled and won't continue indexing beyond a certain point in the past. It must be hanging up on something that is keeping it from continuing past indexing. Please advise

SECOND EDIT: I emptied the table in the database (it has a few more than 3,000 records in it) and the code executed again, but then stopped, and it had seven new records in the database. I emptied the table again and it sent through one more cycle of downloading a few more images, and putting seven new records in the database again.

So, something related to the database and/or php memory seems to be preventing the code from executing correctly.

Any thoughts?

Thanks
Gerry





Attached Images
File Type: jpg Screenshot 2017-11-17 11.09.31.jpg (31.5 KB, 14 views)
File Type: jpg Screenshot 2017-11-17 11.29.35.jpg (34.1 KB, 16 views)
File Type: jpg Screenshot 2017-11-17 11.29.49.jpg (32.8 KB, 15 views)
File Type: jpg Screenshot 2017-11-17 11.32.11.jpg (119.9 KB, 15 views)
File Type: jpg Screenshot 2017-11-17 11.32.26.jpg (125.9 KB, 17 views)

Last edited by gerryvz; 18 Nov 2017 at 13:04.
Reply With Quote
  #801  
Old 17 Nov 2017, 16:49
gerryvz gerryvz is offline
 
Join Date: Aug 2011
In addition, here is the error I received a few times relating to the database timing out. What changes can I make to the settings to avoid this?

Database error in vBulletin 4.2.4:

Invalid SQL:
INSERT IGNORE INTO vb_iei_img (
oldurl,
newurl
) VALUES (
'http://217.118.128.96/charlesironside/FMPro?-db=ecar2dta.fp5&key=41457&-img',
'https://www.500eboard.com/forums/clear.gif'
);

MySQL Error : MySQL server has gone away
Error Number : 2006
Request Date : Friday, November 17th 2017 @ 11:43:51 AM
Error Date : Friday, November 17th 2017 @ 11:44:52 AM
Script : http://www.500eboard.com/forums/admi...ron&cronid=171
Referrer : https://www.500eboard.com/forums/adm....php?do=modify
IP Address : 100.16.101.179
Username : gerryvz
Classname : vB_Database_MySQLi
MySQL Version :
Reply With Quote
  #802  
Old 18 Nov 2017, 20:13
y2ksw's Avatar
y2ksw y2ksw is offline
 
Join Date: Aug 2003
Real name: Giovanni
Originally Posted by gerryvz View Post
Hi Giovanni,

Great mod, and thank you so much for creating it. I am running 4.2.4 and installed the mod yesterday. I have around 7,300 users and 150,000 total posts, with a relatively small number of overall posts that would require images to be downloaded (I would say the majority of images on my forum are directly attached and residing already in the file system).

Server - Linux + Apache
vbulletin - 4.2.4
php - 5.6.31
php max post size - 64MB
php max upload size - 64MB
php memory limit - 256MB
mySQL - 5.5.7
mySQL packet size - 10MB

The mod started downloading and it got through about 4 of the 9 years of posts on my forum. Now it has frozen since last night. I turned on the logging this morning and only came across this single entry in the log since I turned it on, which is attached. The offending post has some images attached to it, using the external host home.btconnect.com

The images seem to be small in size and pixels, and I downloaded them manually and attached them to the post to see if I could have the mod continue downloading other posts. So far, I haven't been able to get it started again.

Attached is the error, as well as my settings screens.

Also, I have now received two times the mySQL error as attached, which I read appears to be the database server "timing out."

The mod is definitely enabled and running, download folder and sub-folders are 777 permissions, etc.

Does it look to you that there is an image that is too large, or do you have other ideas as to what is causing the mod to stop indexing and downloading?

EDIT: A new post that was made by one of my users today was indexed and the external images were replaced, so I know the code is working. However, it seems stalled and won't continue indexing beyond a certain point in the past. It must be hanging up on something that is keeping it from continuing past indexing. Please advise

SECOND EDIT: I emptied the table in the database (it has a few more than 3,000 records in it) and the code executed again, but then stopped, and it had seven new records in the database. I emptied the table again and it sent through one more cycle of downloading a few more images, and putting seven new records in the database again.

So, something related to the database and/or php memory seems to be preventing the code from executing correctly.

Any thoughts?

Thanks
Gerry





Please try to disable the URL Cache. Then, all research moves from memory to database, a bit slower but less memory hungry.
__________________
Giovanni
Italian support: http://www.vbulletin.it/
Reply With Quote
  #803  
Old 18 Nov 2017, 20:19
y2ksw's Avatar
y2ksw y2ksw is offline
 
Join Date: Aug 2003
Real name: Giovanni
Originally Posted by gerryvz View Post
In addition, here is the error I received a few times relating to the database timing out. What changes can I make to the settings to avoid this?

Database error in vBulletin 4.2.4:

Invalid SQL:
INSERT IGNORE INTO vb_iei_img (
oldurl,
newurl
) VALUES (
'http://217.118.128.96/charlesironside/FMPro?-db=ecar2dta.fp5&key=41457&-img',
'https://www.500eboard.com/forums/clear.gif'
);

MySQL Error : MySQL server has gone away
Error Number : 2006
Request Date : Friday, November 17th 2017 @ 11:43:51 AM
Error Date : Friday, November 17th 2017 @ 11:44:52 AM
Script : http://www.500eboard.com/forums/admi...ron&cronid=171
Referrer : https://www.500eboard.com/forums/adm....php?do=modify
IP Address : 100.16.101.179
Username : gerryvz
Classname : vB_Database_MySQLi
MySQL Version :
The database server connection times out before the page or script finishes. This should never occur, but quite some hosting services "enforce" this policy in order to keep up with thousands of sites in a box. In a normal server, the connections time out once a day or so.

If this is your box, please make sure that the mysql server never drops connections, but also disallow permanent connections. The speed loss is won back by making less connections due to timed out pages and user refreshes.
__________________
Giovanni
Italian support: http://www.vbulletin.it/
Reply With Quote
  #804  
Old 19 Nov 2017, 01:11
gerryvz gerryvz is offline
 
Join Date: Aug 2011
Hi Giovanni,

Thank you for your responses. Rarely is a developer so responsive and engaged, particularly so many years after creating the code. For that, I thank you.

I solved the problem of my image downloads stopping, and I hope this solution is helpful for other people.

What I found is that the database is timing out, because the image downloader is waiting for hosts that once had images that were posted in forum posts, that are not responding or no longer have the images on them.

What I do is to run the image downloader code manually from the vBulletin Administrative control panel, in the "Scheduled Task Manager" panel, hitting the "Run Now" button.

This forces your image downloader code to run manually (in between the 10-minute cron job period). The downloader code will run until it encounters an unresponsive host, and times out. There will be a database error (like I showed in the text I posted previously...you see the URL with the "charlesironside" text in it) that shows the host URL where the timeout occurred.

I note this in the database error, and then I add this host (not the entire URL, just the "www.xxxxx.com" portion or equivalent) to the "Ignore" list.

After adding the unresponsive host to the Ignore list, then I restart the image downloader manually (as above)...or wait for the cron job to start it again. It will automatically begin the downloading of the images, until it hits another unresponsive host.

I check on the downloading every couple of hours. If via my FTP program, I see that there have been no images downloaded for at least 30 minutes, then I run the code manually until I receive another database error. Then I add the offending host to the Ignore list, and start the process again.

So, what it comes down to, is hosts that are now off-line or otherwise unresponsive in posts made years ago.

This won't be an issue going forward (after all past posts' images are downloaded) because of the ongoing cron job, which will download external images from all future posts regularly -- before the hosts go off-line or otherwise become unresponsive.

I hope this makes sense, and I think it is probably the cause of many other peoples' problems with the image downloader abruptly stopping. The solution has worked for me and allowed me to "catch up" on the downloading for all of my past posts.

Regards,
Gerry
Reply With Quote
  #805  
Old 20 Nov 2017, 09:25
y2ksw's Avatar
y2ksw y2ksw is offline
 
Join Date: Aug 2003
Real name: Giovanni
Originally Posted by gerryvz View Post
Hi Giovanni,

Thank you for your responses. Rarely is a developer so responsive and engaged, particularly so many years after creating the code. For that, I thank you.
(...)
Regards,
Gerry
Hello Gerry,

I am an old style worker and it is my duty and pleasure to reply when I can

Regards,
Giovanni
__________________
Giovanni
Italian support: http://www.vbulletin.it/
Reply With Quote
  #806  
Old 25 Nov 2017, 18:28
Afrika Afrika is offline
 
Join Date: Aug 2004
Originally Posted by y2ksw View Post
Not without changing the code base.

But in reality, you may move the files directly and then change their path with a simple query in all of your posts. This is much more effective than importing and storing the images elsewhere.


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

where old_url is the path as found in one in you posts now, and new_url the path as you wish to have as the new destination.

Keep in mind that the db connection may time out, but you can repeat as often as needed, since UPDATE is smart enough not to change the records which already have been changed. Best of course is to call mysql from the shell.
The problem is not to change the path, but there are too much pictures in the folder. Many of them are not used. The importer were the best possibility for that problem.
Reply With Quote
  #807  
Old 26 Nov 2017, 15:52
y2ksw's Avatar
y2ksw y2ksw is offline
 
Join Date: Aug 2003
Real name: Giovanni
Originally Posted by Afrika View Post
The problem is not to change the path, but there are too much pictures in the folder. Many of them are not used. The importer were the best possibility for that problem.
ok then:
__________________
Giovanni
Italian support: http://www.vbulletin.it/
Reply With Quote
  #808  
Old 26 Nov 2017, 22:44
Afrika Afrika is offline
 
Join Date: Aug 2004
Is there really no way to solve this with the importer? That would be an elegant way and the function to re-sort is already built.
Reply With Quote
  #809  
Old 27 Nov 2017, 10:45
y2ksw's Avatar
y2ksw y2ksw is offline
 
Join Date: Aug 2003
Real name: Giovanni
Originally Posted by Afrika View Post
Is there really no way to solve this with the importer? That would be an elegant way and the function to re-sort is already built.
You may change the code of the importer because it does not exclude the forum itself.

However, the most elegant function is to move the files and update the database with queries. It has a much smaller impact on a system than importing, namely the additional space it would eat up.
__________________
Giovanni
Italian support: http://www.vbulletin.it/
Reply With Quote
  #810  
Old 29 Nov 2017, 10:48
Afrika Afrika is offline
 
Join Date: Aug 2004
OK thanks. What I have to change if the folder named picupload?
Reply With Quote
Reply

Tags
lisansdevir


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Mod Options

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


New To Site? Need Help?

All times are GMT. The time now is 13:20.

Layout Options | Width: Wide Color: