Seiko 7A38 - by the numbers

At the beginning of this week - Monday 31st August 2020, I received this initially innocuous sounding email from our Webs host:

It doesn't appear to hold that much significance, as this site has previously survived a couple of Webs earlier transitions (for example, see this older thread). They have merely succeeded in screwing up the (full width) page formatting and caused some annoyingly frustrating loss of functionality - primarily through lack of maintenance and Webs abysmal implementation of Captcha updates, starting in January 2017.

Note that under the paragraph headed: If you are paying for a website It states:

Don’t worry – we will take care of everything for you! We will ensure that your website design, domain, email, and all other data associated with your subscription will be available on the Vistaprint platform.

That is until you check out their FAQ's and read 'the small print'. Scroll down to:

In case you can't read the paragraph I've outlined in read, it nonchalantly states:

Unfortunately we are not able to migrate the Members or Forums apps.

These pages will be omitted from any sites being migrated to Vistaprint.

Unfortunately 99% of the useful data on this website is written in the Forum app !

I haven't bothered updating the text on the website's home page in a few years ....

There is actually a good reason for that. Were I to do so, I then need to re-publish the site (using Webs ShiteBuilder3) and I'm always wary of further screwing up the page formatting and headers. Currently the forum contains 922 topics / threads including nearly 18700 posts.

There's some irony to my original verbiage, which is not lost on me:

Whether it was a correct decision or not, a little over a week after the site was created, I added a forum module. Since then, 99% of all activity has taken place there.

Using the forum app (the way I have) has allowed me to continously add updates to a large number of topics, instead of re-editing a similar number of blog articles, ad nauseam, which frankly would have been competely unmanageable. In the earlier years of the forum there was a lot interaction / input from certain other members, to whom I remain eternally grateful for their contributions. Sadly this has declined over the last couple of years, to the level where the site has effectively almost reverted to being the blog it was originally intended to be. 

  All may not yet be completely lost though. Given Webs 'retirement' date of 31st March 2021, potentially we have over 6 months to save the forum data in some useable form or other. My current Webs pre-paid subscription does not expire until January 2022, but given that they are not migrating the all important forum data, it's extremely unlikely that I will be renewing with VistaPrint when it becomes due ! 

Some readers may not have been aware, but we've been in a similar situation (but without the proverbial 'Sword of Damocles' hanging over the forum) once before. Prompted by yet another of Webs various Captcha-related glitches, almost exactly two years ago, myself and a couple of other rather more IT-savvy members were investigating the possibility of migrating the forum to another software platform - specifically phpbb. This since abandoned thread in the Member's Lounge refers:

Proposed move of existing forum to phpBB

Using phpBB (open source) forum software was my personal choice, based on a now sadly defunct phpBB hosted 'specialist' forum: www.watchfetishforum.com. That particular forum had been customised from the standard phpBB format using CSS, and I found its aesthetics particularly pleasing. Almost a year earlier, I had made contact with the web designer 'Arty' on his own forum. See this thread:

https://www.artodia.com/threads/some-really-dumb-newbie-questions.3116

Unfortunately, he personally no longer supported nor provided enhancements to phpBB and tried to steer me in the direction of XenForo, which I wasn't keen on. 

I'm not sure why the 'proposed move' thread quietly died, the way it did, but Webs announcement has obviously spurred the need to revisit it - whichever replacement forum platform we ultimately decide on. I will endeavour to re-enlist the assistance of previous contributors Bill, Mark and Steve. Anyone else with the necessary IT skills, please feel free to give input.

I suspect the reason that earlier effort stalled was partly my fault, because of my foolhardy reluctance to leave Webs. I believe Mark may also have experienced some dificultly scraping the large volume of data. In view of Webs proposed move to VistaPrint, without migrating the forum data, it may place them under some legal obligation to provide it as a download - something they'd refused to do previously.

There is another option (he says tongue planted firmly in cheek). I could always try selling the forum to VerticalScope - as SCWF and WatchUseek have both gone down that route - and let them have the worry of converting it. Funnily enough, they now use enhanced XenForo forum software !!

200 days and counting until the imminent demise of this forum. 

As very few people seem to have read this thread (and I've still got new members registering) I've just added an update to the home page, re: Web's announcement.

To do this I had to re-publish the site. I did risk editing my original paragraph (just updating the forum statistics), which didn't adversely affect it. However, as you can see from the extra text box I've added, I was unable to format it to look exactly the same - even though it's supposedly using the same Comic Sans MS font. 

Hi Paul, any plans or ideas how to move/save all the data in the forum? It would be such a great loss to just see it all wiped out...

Yes  - for the most part this is simply an amazing resource for all of us here and whilst Paul is doing all the legwork we are reaping the benefits.

I'm truly grateful for the work he's put in here to build this and it would be very sad to see it go . 

Thanks for the words of encouragement, Guys - I just wish more members would read the thread so at least they'd be aware of the forum's impending demise. 

I have made a little progress over the last month. Two weeks ago, I managed to make almost direct contact with 'John' from Webs (the person who penned the announcement). He apologized, but implied that the decision was out of his hands. In response to my list of requirements (I'd sent in an attached Word.doc) he stated: 

I can certainly provide you with CSV files for each level of your forums at any time between now and March 31st, 2021:

Forums

Categories

Topics

Posts

Member Details 

These files would allow you to read the information offline, but I'm not sure if would be as usable as you may desire since they would be separate CSVs. I'm also not certain what format is required for importing into a new forum solution. Would you be able to let me know when you decide on a new forum provider that has the capability to import data? I can then look into their requirements for import and try to manipulate the data into a format that is compatible.

Prior to that, he had actually sent me a CSV file of the Member data, which is both useful and useable. CSV (Comma Separated Values) is a fairly archaic format. I was using it myself 30 years ago, to download parts information from mainframe databases. Simplistically, it converts the format and imports it into Excel. I doubt the large number of small files generated by the forum would be manageable. If and it's a big IF, I do decide to try and import the existing forum data onto another provider's forum, something like JSON Lines (as utilised by Mark H. two years ago, in the abortive attempt to migrate the forum to phpBB) would probably make more sense.

Selfish as it may sound, my primary aim at the moment is to create a searchable offline record of the forum - initially for myself, with a view to uploading it as static online publically available archive.

Those of you who've been around for a while and witnessed some of the earlier tribulations with Webs transitions may remember this from almost four years ago:

Originally posted by myself on: October 12, 2016 at 9:26 AM

Here's another extract from an email from my work colleague:

I did kick off another download overnight last night and it ran through quite a bit of today (I wasn’t sure if it would finish before they fiddled with the site), but it looks to be right on this version. I had to write a convertor to get a lot of the images to display and interpret properly after Teleport Pro did it’s magic. Let me see if I can make today’s download visible…

This is a screen print of the bottom half of the forum home page from his 'mirror image' copy. It's all there. 

I remembered that my work colleague - an IT guru named Kev, had created that download of the forum (as it existed @ 12th October 2016) and asked if he'd still kept it on his server. Fortunately he had. A couple of weeks ago, he emailed me a Zip file. I downloaded it and now have a copy on my home PC. It's complete (apart from the page headers and footers and colour themes) and still works - though obviously it's not possible to update it.

So I asked him if he'd be prepared to do it again. The tool he'd previously used to scrape the site - Teleport Pro was quite old. After Webs eventually converted the site from the original SiteBuilder2 (in which it was created) to their SiteBuilder3, nearly a year later (when the full width page formatting got screwed), it wouldn't work any more. This time Kev's using a different crawler tool: www.httrack.com

Kev started two weeks ago, in mid-September. Obviously the forum is a lot bigger than it was 4 years ago (certainly in terms of the number of posts, increased from 11790 then to 18840 today. It's taking much longer to crawl the site - I'm talking days !! The problem is that the forum has been very flakey of late, randomly returning the Hmm... and Uh Oh error screens, instead of turning the page.

In case (by the remotest chance) you're not familiar with them ....

These temporary outages can last for a few seconds or minutes; sometimes for an hour or more. Obviously this is adversely affecting Kev's ability to obtain a complete capture of the forum. Not just that either. To enable Kev to run these crawls, I've had to restore the forum back to the public domain (from its recent 'lockdown' condition). So naturally Google is trying to re-crawl it. They're also reporting multiple errors, as shown in this screen print of just one of them:

Here's extracts from a couple of Kev's recent emails:

Well, this is interesting… the grab finished this morning, and had a number of the “hmmm” pages. I tried a refresh and it picked up a few more, but still looked like problems. I’ve tried this 3 times during the day and it’s been the same. Until the update I kicked off about 4pm, which is still running! I’ll probably leave it running overnight and see what it shows I just hope the “doing more” means it is succeeding and we’ll get a lot of the pages.

It finished in the early hours of yesterday morning – and seems to have picked up a lot more – the WRUW thread has most pages, but there’s a “hmm” on something like page 62 to 75. I kicked it off to do another update and see what happened, and this is also still comparing and running, so I’m hoping it’s picking up more and more on each run.

I need to catch up with Kev to see how he's getting on. I sincerely hope he hasn't given up out of sheer frustration. 

 180 days and counting until the imminent demise of this forum.

Sad news. The forum has been incredibly helpful for me. I wish there could be a transfer to a new forum software. Maybe I would be more active myself if the forum software would be more convenient to use.

I can’t help technically but would do with a small financial donation in case that helps. 

Just a quick interim update on the progress so far of saving the forum data. Kev's persistent valiant efforts at running a scrape are beginning to bear fruit. Yesterday, he sent me a link (to his server) for his latest saved version. My initial impressions on seeing it were extremely favourable - read WOW !! 

Looking at the statistics, at the bottom of the page, it's nearly all there.  Webs Hmm... Let's try that again error message, which has been popping up all too frequently of late, causes the crawler to skip pages. Comparing Kev's scrape with the current forum numbers, it's only missed one complete thread (from the eBay section) and approximately 100 posts in total (from a number of different threads). Oh and 5 members have disappeared. 

However a quick check of a few threads (starting with the WRUW thread) revealed that all was not well. Virtually all legacy PhotoBucket remotely hosted images are missing from every thread. Not just mine - everybody's: Simon's, Don's, Arpad's, Paul K's, to name but a few major past contributors. They've been replaced by a small spaceball (1000's of them). Here's how page 87 of the WRUW thread looks:

The images that I've hosted in the website's photo gallery since the first week of June are all there, as (best I can tell) are all Flickr, Imgur and Postimage hosted images. It appears that even though correctly linked and currently displaying properly on the live forum, Photobucket is denying access to Kev's scraper. 

What a bummer - I hope you can get the data extracted OK ... 

Gosh, I'd read in about these problems ages ago but hadn't realised the current issues.

And I'd only just posted on PPRuNe dot org / Jet blast /

"Without that site I'd have been lost." 

As an octogenarian I'm pleased to be able to still 'go deep' into servicing my 40-odd 7axx collection.  It would be impossible without referring to this site.

Fingers crossed.

With the added problem of tens of thousands of legacy PhotoBucket hosted images missing from Kev's valiant efforts at scraping, I'm still struggling to figure out how (if at all possible) to save a full offline copy of the forum (for myself) at the moment. 

Over the last couple of months, which have simply flown by since Webs dropped their bombshell, I have thought about creating another replacement website / forum.

I'd also been wondering how many other Webs site owners have found themselves in this predicament. I don't think I've mentioned them before, in any of the previous threads, but going back four years, to October 2016, I remember trading emails with the administrator of another Webs hosted site, which incidentally included a large forum module, about the problems we were both experiencing back then. Absolutely nothing to do with watches, but another online community of enthusiasts using the same teetering platform. This afternoon, I remembered the name of their website.

http://www.dollshousespastandpresent.com/

This is, or should I say was their Webs-hosted forum:

http://www.dollshousespastandpresent.com/apps/forums/

Note the statistics at the bottom: 1163 members | 2519 topics | 32972 posts - Not dissimilar to this forum's numbers. The last posts were made by their administrator 'Brooksey' 7 months ago. Apart from those, it appears that they abandoned using it (along with Webs hosting) over a year ago and started a completely new website:

http://www.dollshousespastandpresent.co.uk/

Their new forum follows a similar format, with a left-hand side-bar. Although there are no statistics / totals visible at the bottom, the number of topics / comments (read threads / posts) are noticeably lower. This points to them having started again from scratch, rather than making any attempt at conversion.

http://www.dollshousespastandpresent.co.uk/Forum

However, what is mightily impressive is that they managed to raise over £10,000 in individual contributions from nearly 400 members to fund this new website in only 42 weeks !! 

http://fundrazr.com/c1NSU4?ref=ab_3iHWCWcoSTC3iHWCWcoSTC

It's worth reading some of the other text which follows underneath:

120 days and counting until the imminent demise of this forum. 

Though in fact we may potentially have even less time left than that.  Overnight, I received another rather disconcerting email - not from Webs, but from a 'Mike' at Vistaprint. You may remember that 'John' from Webs had cited a deadline of 31st March 2020. which theoretically still gives us 4 months to save the forum in one form or another. This latest email implies that Vistaprint intend to start the conversion process in 2 months time - if not earlier !! 

Needless to say, I shall be replying to Mike - in the strongest possible terms.

Something else that has been nagging me concerns Flash. I know there are some elements of Flash used in a few places on the site - for example the flakey Avatar Upload button on the Member Profile screen, which comes and goes. I have no idea how much of the forum coding uses Flash, but when Adobe discontinue it and it's no longer supported in Chrome from 31st December 2020, potentially the website / forum could be flagged as Not Secure - and worst case, no longer accessible. 

See: http://www.blog.google/products/chrome/saying-goodbye-flash-chrome/

and: http://support.google.com/chrome/thread/65589573?hl=en

Time for a long overdue update and finally some good news regarding the future of the forum. It has found a 'saviour'. At the end of November, I was contacted through Instagram by forum member Johnny_E, who kindly (and boldly) offered to attempt a full scrape of the existing forum, with a view to hosting it as an online archive. 

As my work colleague, Kev, had tried many many times before, John first attempted a scrape using HTTrack which seems to be the most popular free website copier. I'm sure he won't mind my disclosing, but his first effort was almost an unmitigated disaster. Like all these tools, there are a number of selectable settings, where you can pick or exclude certain links. Unfortunately John didn't check all the right boxes.

The scraper started picking up the 'Flag for Abuse' in every single post - and flagged every one of them !! The following morning, I logged on to find that nearly 2000 posts had been flagged !! Luckily John was keeping a watching eye on the scraper, realized what it was doing and aborted it. Somebody in Webs Support must also have seen what was going on and banned John for his troubles !!

I spent a couple of hours the following afternoon removing all the flags. I reinstated John's membership and upgraded him to 'Administrator' (in case anybody noticed and wondered why). Over the next few days, having suitably tweaked his settings, John attempted a number of scrapes using HTTrack, with varying levels of success. However, like Kev's previous efforts, John's scrapes all failed to capture all and any Photobucket hosted images. 

John then presented me with 'Option 2' - a scrape using Cyotech WebCopy. It's a little different in the way it works and as stated on their website is really intended to: Copy websites locally for offline browsing. Becoming desperate by now - and selfish as it may sound, that's all I was really interested in at this rather late stage - saving an offlne copy for myself. John gave me blow by blow updates as it downloaded individual files - some 75,000+ of them !! As I said, it works differently to most scrapers, in simplistic layman's terms, that it compiles and re-makes all the links after it's finished downloading all the files. I must admit that I had visions of it being a horrible jumbled mess. 

When John sent me a link to what he'd downloaded, I was initially very pleasantly surprised to be met with this:

It's the forum home menu, as it would have looked, 3 weeks ago on 4th December - the summary totals at the bottom: 1381 Members | 923 Topics | 19086 Posts are, best I can recall, just about spot on. 

Unfortunately there was a gotcher.  Clicking on any forum section and thread returned just coding. It would only work properly in Internet Explorer or MS Edge. John did explain the reasons to me, but simplistically put, it's because Webs forum software is Javascript based. Like the majority of websurfers, I use Google Chrome as my default browser (worldwide users Chrome = 63.5% versus MS Edge = 3.4%). Then, even if John did host it and make it available (as a searchable archive) in the public domain, who would actually see it ? In terms of search engines, Google has an even bigger market share: 87.9% versus 5.3% MS Bing.

I should qualify my statement about 'using Chrome as my default browser'. I still currently work for a UK company in a very secure business environment. They do permit employees to browse the Internet from their network, but it is extremely restrictive and usage is monitored. Their default browser is a modified form of Internet Explorer, which blocks all social media sites. I'm not just taking about Facebook, Instagram, etc., but all image hosting sites, including PhotoBucket. 

So, in the past, on the rare occasions when I'd viewed the forum from a work PC, all the tens of thousands of PhotoBucket hosted images were blocked - replaced by a small black X placeholder. As I was leaving the company at end of the year, I took a calculated risk and forwarded the link John had sent me to my work email address and opened it in their 'cut down' version of Internet Explorer. It worked perfectly ! Because all the (ex PhotoBucket) images had been downloaded by the scraper and were now hosted on John's server, they all displayed correctly. I spent a couple of evenings after work checking the WRUW thread, some of the 7A38-xxxx model specific threads and quite a few others besides. Everything, including mine and everybody else's remotely hosted photos appears to be there.

Whereas I was (and still am) extremely grateful to John for saving the forum as a static archive, I didn't relish the prospect of having to view it using Internet Explorer (either at work, or home) and was obviously a little despondent about not being able to share it easily. Then last weekend, to my surprise, John sent me an encouraging sequence of messages via Instagram DM:

I've a VERY talented developer and ex staff member who I'm going to show this to. Using the F12 developer tools in Chrome, you can see what the browser is trying to do and where it hits a dead end. If he can suggest a code fix, I can script that fix into the pages. All theory... But worth a shot.

Followed on Monday by:

We have a fix. ... It'll take a few days.... But fingers crossed. Tested it on one page and it renders in Chrome and Firefox ok. Just need to repeat 19k times! Plan is to tell the server to do this automatically when it finds a page like this... So hopefully.... 1 file on the server will hit it in 1 shot.... Trying it tonight.

Followed on Tuesday by:

Well... 2am? Lol... Took longer than I thought but I'm happy to say that your archive is 100% in all browsers now! Had to rename every post and forum file to add .html to the end of its name. Then, add server level URL rewriting so that when a URL is clicked (which has no .html) on the end... Append .html to said link. Bottom line, it works!

I've since checked a number of threads using Chrome and indeed it works !! 

 90 days and counting until the imminent demise of this forum.