Seiko 7A38 - by the numbers

A couple of weeks ago, Steve ('a1topdog') asked me to start a new topic for him. (He's since resolved the problem himself). I hadn't realized before, but the inability of normal forum members to start new topics, due to the non-functional Captcha issue (and Webs general negligence) is yet another glitch afflicting the forum.

Forum members, Bill ('goodbot'), Mark ('H') and Steve ('a1topdog') have be urging me to move the forum to a more stable platform. Bill and Mark have both made considerable progress, over the last couple of weeks, in both setting up a new phpBB proposed replacement forum and crawling the existing Webs forum to capture the data. For which I am extremely grateful.

They've both emailed me loads of info, which I won't pretend to completely understand. As I wrote to Bill in reply to one of his lengthy emails earlier today, I'm already feeling a little overwhelmed. Rather than have all this crosstalk / potential duplication of effort, can I suggest we use this thread to continue the discussions ?

Thanks for creating this topic Paul. Hopefully as a community we can pull together and save this valuable reference resource from the clutches of webs.com

I'll chip in where I can, having already been through the the pain of converting one forum to phpbb, and I still manage another phpbb forum, so although I don't claim to be a techie guru I may be of some help.

Hi all, it should be possible, I guess, to import the forum into phpbb although I've no experience with software.

Bill - if there's an import file format for phpbb do send me a link to the spec or any info you have.

I'll help out when I have time, but I'm currently in the middle of relocating my family to the UK/changing job/finding a school etc. My time's somewhat limited for the next few months.

Hi All:   There doesn't appear to be any facilty within the phpBB standard system to effect "imports" from other BBS systems... but if you look on the phpBB support board, you'll find a category (or forum) on the topic of available converter utility apps. These all appear to be out-of-date, but offered the ability to import data from various other popular BBS systems.  The phpBB3 app has 50 separate php tables, all readily accessible via the common popular open source utility "PhpMyAdmin". Using a little bit of SQL, we can import the Webs content into the phpBB.  There are a number of steps to this process. But essentially, first, we must get some data out of the webs BBS. Hmmm... Paul... in your Webs admin interface, do you have access to this same PhpMyAdmin tool?  If so... this is our ticket! Another thing to think about... its likely that Webs didn't code/create their own BBS, but are using one or the other of the several available free (php?) BBS's... if we can find out which BBS they're using... this will definitely help us.  I'll start perusing google to see if I can up with any hits on this info. Stay tuned.    Bill.

This is the link to the old phpbb converter info.

https://www.phpbb.com/community/viewtopic.php?f=65&t=1761395

A couple of points to mention. 1. None of these were written for webs.com forums, 2. They were converting to what is now a very old version of phpbb. 

It may be we have to install an old version of phpbb, convert to that, then use the other phpbb upgrade scripts to upgrade to the latest version of phpbb.   

The first thing that has to exist is an abiity within the webs.com admin environment to either save/backup or export the forum database contents. Without any of these, there can't exist a "conversion app" for php (or any other platform) because there's nothing to convert. The only other option we'll have is to do an html scrape... this will be better than nothing, but it's going to involve alot of manual work.  I'm not comfortable proceeding with this big manual work option before I'm able to positivey ascertain that there are no tools in the webs.com admin gui to either save/backup or access/dump the raw content of the forum database (like phpBB, probably mysql).  I've tried unsuccessfully to create a new account at webs.com myself so that I can access this admin gui... but it's not possible to create any new accounts at webs.com at this time (not a good sign, right?).  I've asked Paul for screen prints of his admin gui so that I can get a feel for whats available in there.  I'm not for proceeding in any other "conversion" direction at this time without having a certain understanding of what's possible in the webs.com gui.  I'm standing still at this time until I can see that admin gui and I'm personally satisfied that there's no save/backup/dump capabilities.  To do anything else at this time is not a good idea.

Bill.

Well here you go, Bill - here's 3 screen prints of Webs / Vistaprint S(h)iteBuilder 3 Control Panel. If you can find anything remotely of use in there.

It really is quite naff - compared to the previous Webs SiteBuilder2 interation.

Paul:  Nothing here... yet... Can we dig a little deeper?  On the Dashboard, what's under the left hand second tab "Webs Premium"? Is this the drag & drop gui website builder?  What's in the "Settings" tab?  and finally, how about the last "Matching Products" tab? Is there anything in there like a backup utility? Is there a support forum? If so, did you peruse through there to see if anyone has previously raised these same questions of Webs?  If they really have over 100k websites, I'd imagine quite a few would be facing the same situation as you and would be interested in ths same topic.  Is there a support forum you can access and research this issue?  Maybe if you write a request to tech support they'll do a DB backup for you... You gotta ask... at least.   Bill.

OK Bill.

Under the Webs Premium tab there's a table of different levels of site ownership: Starter / Enhanced / Pro, with features that are are available to different levels. My site is Enhanced (i.e. Premium, but not Pro). There is the option to upgrade to another level (Pro) but all that would give me is more bandwidth that I don't need and phone support (as opposed to IM Chat).

Under Settings are: Site Name; Site Description, Tags / Keywords, etc. Under the old Webs Sitebuilder2, the Settings tab is where I would have set up the forum and other pages. I can't fathom what anyone is supposed to be able to do using this SB3 / Vistaprint crap.

Under Matching Products there's:

Custom Printed Marketing Materials

Get great deals on business cards, postcards, calendars, and much more.

Brought to you by Vistaprint, a Webs partner

Premium Business Cards, etc. and a whole load of other useless shite.

Trust me Bill. I looked a long time ago (under the old Webs SB2) Control Panel / Dashboard and there was nothing vaguely resembling any kind of download tool. So I don't expect there to be one under S(h)iteBuilder 3.

Yes, there is allegedly a Webs Community Support forum. I have referred to it MANY times in the various threads in the General Discussion area and posted numerous links to threads on it. I regularly refer to it deliberately sarcastically as the TOTAL LACK OF* .... 

http://support.webs.com/hc/en-us/community/topics/200811888-Known-Issues

*Try running a search on this forum and see how many results it returns. 

Looks hopeless... So instead... Let's start our html scrapping process experiement with a small sub-set of your data - membership info.

First off, it looks to me that you have one unified membership DB shared by both your general website & the forum... correct? (I need to clarify this because our proposed follow-on solutions such as WordPress+phpBB each have their own separate member DBs... that are effectively unified when you use one of these ($) encapsulator apps)...

Do you have a membership listing screen where you can either scroll through a summary of your entire membership DB a screen at a time (with a given number of lines, each line one member, per screen)? Or by increasing the number of lines displayed per screen, can you generate one screen with summary lines of all members (~1000)? If you don't have this, can you display each member one screen at a time? If you have either of these capabilities, can you share examples of these with us via screen prints? You can use my member profile page as our example.

These are the screens that you (only you with your admin screen access) run on your PC while running the scrapping program and scrolling through this data. The scrapping program will generate all of your screen captures (each of your different screens) into separate tables and records.  We require the scrapper to generate these tables and records into a format that can be readily imported into an Excel spreadsheet. 

In Excel, we create separate sheets that mirror the format and data of each phpBB table we'll eventually import our data into. It's good to start with the member info (before any content-type info is imported) because the member info is atomic in the sense that it doesn't reference other content on the site (whereas all content-type info relies on membership info for ownership/creator links).  

It seems likely that to capture the most complete set of member info, you're going to have to go through the laborious task of capturing every single members (admin view) profile pages.  To help us get a better idea of the magnitutude of this chore, can you give us sample screen captures of all admin screens you can call up for admin viewing my goodbot member profile info pages?  

Bill.

Can I just stop you there, please Bill.

The idea of having this discussion on the forum was to prevent crosstalk and duplication of effort. I thought I'd forwarded you all what I considered were the most important parts of each other's emails. Mark had already been working on taking a crawl of the forum at the end of August. Here's a copy and paste of his email:

So far my program's downloaded a total of 33,178 posts from 4,523 threads which should be all forums/topics EXCEPT the members-only threads. I've not had time to make the script work with logins and it looks like webs.com are doing something unusual behind the scenes with logins. As I'm not able to do logins at this time with the script, I also can't download the member data yet. I'll try again when I get more time. I can send you the data if you wish, but it's not very user-friendly in it's current format (in a file format called "json lines".

My last reply to him was that his numbers (posts / threads) were out of whack. I assume he hasn't had time to revisit it since, with his relocation back to UK, etc.

Regarding your questions about the Members app ....

On the Control Panel / Dashboard, I can see the Member data two ways:

Like this: http://www.seiko7a38.com/apps/members/ with approx. 35 thumbnails per page, which you (or any other signed in member) can openly view on the live site.

Also from a Management standpoint, with approx. 60 lines of member data per page, like this:

That's the admin page which I previously would have used to approve / invite new members / ask them to complete profiles. Unfortunately, as I wrote in this thread, Webs have recently managed to break that too !!

Paul:

The output of Mark’s scrapping results (in the usable json format) are the potential eventual input to the "excel-to-phpBB-tables" process I described... but not up front here/now... it's closer to the last step in the migration.

This migration work has to be done in a certain fixed order: First the top (front page) Categories have to be imported, then the Forums under each Category were manually recreated by me and completed (these steps were manually recreated by me and are evident on the top page of seiko7a38.xyz). The reason I was able to so readily manually reproduce these first steps is because the owner/creator of all these Categories & Forums is the sole "seiko7a38" admin user... "seiko7a38" is IN the new database, so it was possible to assign ownership of all these Categories and Forums to that one admin user. Also, since there are just a handful of Categories and Forums in total, it wasn't hard for me to manually rekey these in.

As I earlier read that Mark was not able to get his scrapper past a login (so he was only able to scrape what’s viewable by a public guest/non-member), the immediate ramification of this is that he was not able to scape any membership info (which is only displayable to registered/signed-in members). So the reason I’m talking about doing new/additional scrapping at this time is because we need to scrape the membership info and Mark’s tool was not able to do that. Hence… we need to continue investigating alternative scrapping tools that CAN get past a login and can then scrape our membership info.

It’s possible that a latter version of Mark’s tool may be able to do this, or maybe an enhanced/premium ($) version of Mark’s tool can do this. It’s also possible that a different tool may be able to do this where Mark’s cannot. If so, we’ll need to acquire that alternative tool to do this required work. If this happens, we’ll also use this new tool to re-do Mark’s earlier site-wide scrapping. Afterwards, we’ll compare the two tools output to determine which is easier to work with for our “excel-to-phpBB” needs. If Mark’s tool’s output is superior, that’s the one we should use… if the new tool’s is better… that’s the one we should use.

In either case… unless you’re willing to share your admin credentials, you’re going to have to load this scrapper software on your own PC and do (re-do) this work yourself. Looking over your above screen prints, it appears that critical member info (essentially the user name and corresponding email address) are only available together on your admin view… not the general registered members view. Ideally it would be best if there was one screen with all the members info together… but so far I’m not seeing that here. Other essential info we need together on each member (besides the user name and email address) is their board privilege level, their registration date, and maybe their sequential member #. Only the board admin (you) can see this required member info… so you have to run the scrapper program yourself…

I'll start looking for the latest scrapping programs tonight. Mark - can you provide me info oh what program (and version) you used previously? Can I get a sample of this tool's json output? Do you have the option to choose the output format? Are there alternative format options, like CSV?

Bill

Hi Bill,

The output could be CSV but it's not easy to work with programatically when some fields will be long HTML strings. I can convert it if you're ok to work with it though. There seems to be a bug somewhere that's causing the post count to be wrong and I'll fix that in due course.

To one of your previous comments, the dump does preserve the site structure (forums, topics, posts) as I wasn't intending to recreate it manually - probably easier to use phpbb's API features, I guess.

The crawler is something I wrote specifically for working on this forum, not off-the-shelf software. I've not really tried implementing login in the crawler yet - the whole thing is just a quick hack to test the feasibility of crawling webs. There's no reason why login can't be added, just needs time to do it.

Some example output (I've removed the https link prefix as webs turns the colon-slash-slash into an emoji) - top-level forum list entry:

{"section": "Welcome", "link": "http://www.seiko7a38.com/apps/forums/show/6578536-general-discussion", "title": "General Discussion", "subtitle": "Please read the forum rules before posting in any section.", "num_topics": 48, "num_posts": 710}

Topic link:

{"forum": "http://www.seiko7a38.com/apps/forums/show/6635367-7a38-xxxx-model-specific-area-", "link": "http://www.seiko7a38.com/apps/forums/topics/show/7203419-7a38-6030-sjs028-black-chrome-small-diver-jdm-only-", "title": "7A38-6030 SJS028 - Black Chrome small 'Diver' (JDM only) ", "sticky": false}

Post (content removed here for brevity):

{"topic_ref": "http://www.seiko7a38.com/apps/forums/topics/show/7280029-the-all-too-common-displaced-finger-tension-spring-chrono-sweep-hand-problem", "post_id": "54396270", "poster": "http://www.seiko7a38.com/apps/profile/58260590/", "content": "...HTML POST CONTENT HERE...", "when": "August 24, 2013 at 8:38 AM"}

Mark, Steve:

I'm trying to get you two as "friends" here so that I can PM instructions to you with admin login credentials to the dev platform running phpBB (seiko7a38.xyz/forum). Look for my friend request and OK it... thank you.

I found this website listing several data extraction programs... all of these except one have (apparently) very large price tags (you have to schedule an appointment to talk to a salesperson to learn what the price is).  One of them has a free "community edition" (the last one in that list "UIPath Robotic Process Automatiion"). It's the right price... but data extraction is not its prime design objective, and there's a bit of a learning curve to get it to do what we want.

Mark's approach is way more targeted-specific than any of these general purpose data extractors could be... so I'm hopeful that his tailor made ("bespoke"?) solution is useful...

Mark - Is it possible to run your script on just one small isolated forum or topic only (instead of the entire forum) so that I can see a defined subset of your output? Pick any small forum or small topic... your choice... and (for now) json is fine... If you can do this, email me your output (goodbot@gmail.com)... whenever, of course. Thank you Mark!

Bill.

Hi Bill , Friend request accepted.