Server Move - Discussion and Status

[namida]

I think having them on the boards is fine, especially that I have now successfully been able to move them to the bottom; ie: have posts in the correct order.

My attempts to create some SQL code to do this failed, but the keyword here is "SQL". I used some PHP instead, and this was much more successful.

As there's now no issue of new topics getting lost below old ones, I have re-enabled posting on the new forum.

At this point, it's really safe to post on whichever forum you like - if we do get the database from this one, I'll be able to come up with some workarounds to merge the two. If we don't, then nothing is lost and no work is created by posting on the new one.

(Also: All LemmingsWelt topics are locked, and the same will be true for the IsaMedia ones. But the ones from this forum won't be.)

[Simon]

Good, topic lock for ancient posts is as good as an extra locked board.

-- Simon

[namida]

Yep. And if a need arises, they can be unlocked on a case-by-case basis.

[namida]

IsaMedia forum posts are now up.

So now, the question is... import this forum now (and then ideally get everyone moved over to avoid having to import it twice, which could be difficult), or wait a bit longer on the DB?

Advantages of waiting for the DB:

• Private messages can be transferred - otherwise, people will need to make their own backup of important ones (which they COULD pm to themself on the new board if they want it in their PMs specifically)
• Existing user accounts can be transferred rather than re-registration being needed
• Posts from this board won't need to be manually re-linked to accounts
• Even if the database isn't directly re-usable for some reason, I can most likely re-use most of the code used to import the IsaMedia forum
• The super-secret Staff Discussion board can be transferred

Disadvantages of waiting for the DB:

• We may be waiting for something we'll never end up getting or not for an unreasonably long time
• The new site is still costing me whether we're using it or not - though on the other hand, in the meantime it does still function as a searchable archive of the two old boards in one place. The same is most likely true for this board costing Adam.
• There's no guarantee it'll be fully usable
• This forum may go down without warning, and currently my backup is a few days old - but I will update it very soon and keep it updated on a fairly regular basis
• Some users may prefer to get this over and done with ASAP

Stuff that it won't make a difference about:

• Attachments from this board. They'll be safe either way.
• Links to posts on this board. Due to the ordering fix needed which modifies the post IDs, they are likely to break either way.
• Whether or not we get the lemmingsforum.com URL. Although we need to contact the same person, there's no guarantee he'll give us both.

EDIT: A compromised measure that has been discussed is importing what we can from the January backup, and then capturing posts for the rest. I'm not too keen on this idea myself as it'll be very messy and difficult, especially in the case of posts that were captured in it but have been edited since, and posts that have been deleted since then. Importing attachments from it could also be more difficult compared to from a recent backup or capturing the posts in HTML and parsing them. While I could look into this if there's very high support for it, it's a method I'm very not keen on. In short; while it is plausible, it'll also be by far the most difficult method without having much advantages (basically just people registered before this won't have to re-register, and it'll preserve private messages from before this time), which is why I'll need to see a lot of support for it if I'm going to consider this.

I doubt the staff here are the kind to have concerns about this, but just in case; I'm more than happy to let existing staff (provided they're reasonably active) retain their positions. Just make sure you PM me on this forum (or talk to me on #lix) to confirm your new account is really you.

[LJLPM]

Very good job, namida! 

[namida]

Given that it's now at least a useful archive of the older forums, I'm going to post the link.

http://lemforums.neolemmix.com/old/

Obviously, once it's fully established, I'll get a proper domain name for it.

For now - register there, know the link just in case this goes down. However, while you can post there (and it won't be lost), unless it relates specifically to that site itself, it might make more sense to keep your posts here in the meantime. And of course, feel free to use it to look through the old posts. Both LemmingsWelt and IsaMedia are added (no attachments, sorry, just the posts).

[Nepster]

Is it possible to use the database to transfer only certain parts? Then I would be in favor of transferring the accounts and PMs as of January 2014 using the database and get the rest (i.e. the posts) by hand.
This way most of us could just continue as usual and only a few newly joined members have to register anew (which the most active ones, i.e. exit, NaOH and myself, already did).

[namida]

Is it possible to use the database to transfer only certain parts? Then I would be in favor of transferring the accounts and PMs as of January 2014 using the database and get the rest (i.e. the posts) by hand.
This way most of us could just continue as usual and only a few newly joined members have to register anew (which the most active ones, i.e. exit, NaOH and myself, already did).

Yes, this is definitely possible. How easy it will be is another matter, though without a doubt easier than importing the entire database then adding the missing bits since. Especially importing the user accounts could be messy if it's done without wiping the ones already there; as far as PMs go, there is always the option of manually importing what's available of a user's PM's on request (from the January database). It's possible it may not correctly link who the PM is from, but I could possibly get around this by having them listed in the database as coming from no account (or if that doesn't work, my own account or a dummy account set up for that purpose) and add a note at the start of them saying "PM originaly from (blahblah)". Just for peace of mind; this can be done completely automated without me having to specifically look at and modify the contents of them.

Either way; there is nothing I can do about PMs more recently than the January backup. You'll have to back those up in some form yourself - one thing I could suggest is PM copies of them to yourself on the new board, if you don't want to save them offline. (Yes, you can send PMs to yourself, on both this board and the new one.)


As far as parsing the HTML goes - I haven't started on any code yet (though a small portion of the code from the other two will be re-usable; the same portion that was also reusable between LemmingsWelt and IsaMedia), but I have taken a look at the HTML layout and the layout the website dumper I used puts the files, to determine what approach to use.

So far, I have identified how to crawl the topic lists and get the links to the topics' content themself, but haven't looked at how to crawl this last part yet (which will no doubt be the hardest part). I'll also have to take into account that HTML will need to be reverted to BBCode.

[geoo]

I think that would be my personal preference: Import the users from this forum from January (while wiping the few that have registered on the new forum already), and link the accounts to the posts while at the same time getting most of the PMs (I have 211 PMs myself, about 80% from before January, so that'd make things a lot easier for me). Maybe we could even get the PMs from isamedia?

I really wanted to have a look at all these things one of these days, but somewhat unexpectedly I have more things on my mind right now than I hoped to have...

[ccexplore]

First off, thanks for all the work getting this done!

I see that some people have already re-registered themselves on the new forum, but I thought at some point you'd use the Jan 2014 database backup so that only those users who registered on the current forum after the Jan 2014 date needs to re-register?  Is that the case and if so, any chance you can transfer the users from the database backup now?

=================

Advantages of waiting for the DB:

• Private messages can be transferred - otherwise, people will need to make their own backup of important ones (which they COULD pm to themself on the new board if they want it in their PMs specifically)

Is access the only issue with transferring PMs?  That is, you can't use the same technique you plan to use for regular posts purely because PMs require the user's password/login to access?

I'm getting the (possibly wrong) impression that only a few people seem to need their PMs transferred.  If that's the case, perhaps it's a workable solution if you provide us the same tools you are using to do the transfer and have us do it ourselves (or at least the extraction portion, since obviously we won't have database access to import into the new forum)?

Links to posts on this board. Due to the ordering fix needed which modifies the post IDs, they are likely to break either way.

Hmm, there's no way for you to keep track of how the old IDs mapped to the new IDs, and then apply a second pass to fix up the IDs?

[mobius]

I just copied all my PMs to a document. Honestly I thought I was alone in having way too many PMs 
I keep meaning to take care of them and delete them all but I haven't gotten around to it. In the end I'll most likely delete them all including ones which I've meant to adress but haven't.

The only ones I currently really need are the ones for the Lix contest. [which make up a good portion]

[namida]

ccexplore: I could always just load users that havent already registered.

The problem with getting PMs (and for that matter the staff board) is an inability to do it with a logged in account. Though it is possible to log in, one of the first things the app does is grab a copy of the log out page, and you can guess the results....

I might be able to do that with fixing post links, now that you mention it. I can at least look into it.

[geoo]

What kind of app are you using? I'd think it should be possible to write a little python script to grab the PMs...I've written some sort of grabber in python before without login though, but you can send e-mails with the right python lib so...

[namida]

Im using WinHTTrack to grab the content as HTML files and download attachments, and custom-written QB64 code to parse it. I havent written a parser for these forums yet, but thats the plan, and its what I did for the old sites.

The actual importing to the database is pure SQL (geneeated by the QB64 code), and the order fix is a PHP/SQL hybrid as I wasnt able to get it working in pure SQL.

[ccexplore]

The problem with getting PMs (and for that matter the staff board) is an inability to do it with a logged in account. Though it is possible to log in, one of the first things the app does is grab a copy of the log out page, and you can guess the results....

Have you looked at the "filters" feature and see if that can be put to use to block WinHTTrack from navigating to the log out page?  Seems like an -*action=logout* filter should work, assuming the filtering works the way I think it does (it isn't clear from the page whether the filtering actually looks at the query portion of the URL [ie. the stuff after the ?]).

Failing that, another possibility is to install a web debugging proxy such as Fiddler, which can be set up on the machine you're running WinHTTrack on to observe and modify inbound and outbound http requests/responses on the fly, for example to effectively replace the outgoing request for logout to something else so that the logout doesn't actually happen while Fiddler is active.  It's been a while since I've used Fiddler but if it comes to that, I can try to help you with the setup needed.