Post by: geoo on December 18, 2014, 11:52:34 AM
We haven't managed to contact the server admin, Adam, in a very long time. He seems to be paying the server bill, but there's uncertainty for how long that would last, and the forum could go down at any point in time, and data might be lost. Without server access we also can't update the forum software, which right now is outdated, buggy and probably prone to exploits for spambots like the recent invasion. On a new server with someone active being server admin we have complete control over everything.
This is what has happened so far:
- With Mr. K's help, we now have access to the admin control panel, but nobody of us has server access
- Temporarily, to fight the spam bots, user registration was turned off for a day, which I changed to new users requiring approval by an administrator. It seems like there haven't been new members trying to register within the past two days, so I'm setting in back to normal now and see what happens.
- We can't get hold of an up-to-date version of the database (that'd contain all user information, posts, everything we'd need to just move the forum over without issues)
- Adam (and maybe Liam, whoever he is) is the only one able to get us the full database and the domain www.lemmingsforums.com, we have tried to contact him, but without success so far
This is what we have access to right now:
- Threads and posts from the Lemmingswelt forum (2004-2007 or so)
- Complete database from the isamedia forum (2007-2009 or so)
- Database of this forum from January 09th, 2014. All newer data is missing
- namida can get the complete set of theads/posts/attachments of this forum at any point in time (I believe)
My suggestion is to allow some more time to contact Adam, discuss and not rush things, while still keeping posting in this forum and leaving namida's forum as the emergency option should the forum suddenly go down. (I don't think it's very likely that the forum will go down within the next month.) We should prepare everything for a swift move, and once preparations are finished set a date (and announce it a week in advance) for when the move shall take place. That day namida would grab all the data off the forum and import it over on the new forum, and then this forum would be closed (or at least become read-only to ensure new content will be posted on the new server).
The reason I suggest to keep posting here for now is that moving to the new forum would encourage reposting topics which would have to be merged later, which becomes messier the more time passes and thus puts us under time pressure.
If we can somehow get hold of Adam, this would make things a lot easier as we could just import the whole database and get a full copy of the forum, and in addition the domain www.lemmingsforums.com. So I encourage everyone to help us in our effort to contact Adam!
Should this not work, I'd suggest purging the few posts from the new forum and importing the members from the January 2014 database (this means we'd also have most PMs, and almost all posts would automatically already be linked to the users), while taking all the posts that namida will grab from here over. Furthermore, we could add the posts from the isamedia and lemmingswelt forums and try to link them to the user accounts (something that I feel very strongly about). If you forgot your password from January 2014, or if you registered later than January 2014 you could identify yourself and get access to your account/get it linked to the posts that you made. Maybe we could also get the PMs from the isamedia forum working, I personally find having an archive of my PMs important, but I don't know what other people think about that.
The plan I outlined is just my opinion.
Feel free to discuss!
Post by: ccexplore on December 18, 2014, 08:47:24 PM
I'm actually surprised that the most recent database backup we had access to is from within this year, I would've thought it was a bit older. Remind me how were we able to get the Jan 2014 database without involving Adam or otherwise actually getting direct access to the database?
Ideally we should carry over as many things as we are able to. The priority would be posts after 2014, attachments, PMs, and finally the old stuff from isamedia/lemmingswelt.
I'm sure it's all implied but, when the move completes and the old forum closed or read-only, make sure to update home page of old forum with links to the new forum, if we aren't able to get access to the domain to simply redirect the current URL to the new forum. Actually, I suppose that info could be put up even as soon as the move starts.
If we aren't able to migrate everything over, the old forum should be made read-only but not closed (at least for a short time), so access to the stuff that wasn't migrated is still possible. In this scenario, the stuff that was successfully migrated could potentially be deleted by admin, so it stops showing up in search engine results. If everything got migrated then obviously closing the forum altogether would be the cleanest option, as long as it's still possible to display something to direct the user to the new forum (or otherwise be able to redirect the current URL).
=============
While not the highest priority: if we go down the route where we are porting recent threads and posts outside of a full database import, see if you can, in the process of porting, fix up URL bbcode tags and similar in posts that are linking to content in the existing forum, like this link here as an example. I know I do that from time to time. Failing that, being able to compile a list of all such posts could also work and then you could leave it up to individual users to edit and fix the posts themselves after the migration, as they see fit.
Post by: Prob Lem on December 18, 2014, 09:11:49 PM
Remind me how were we able to get the Jan 2014 database without involving Adam or otherwise actually getting direct access to the database?Something at the back of my mind is saying that that's when the previous server move (the one that broke notifications of PMs/subscriptions, and lost some attachments) was, so I'm guessing it might be something to do with that.
Post by: namida on December 18, 2014, 10:12:34 PM
If anyone does manage to get contact with Adam, what we need is just the database plus the contents of the "attachments" folder on the server. (Yes, I do at this point have a backup of all current attachments, but if we can get the folder as-is that'll be easier to import, whereas the form I have them in would require parsing the pages to get the correct filenames and other metadata for them.)
Meanwhile, I'm going to experiment with the older forum backups to see if I can get them going well on the new server.
As far as contacting Adam or Liam goes, I dont' have any ideas for Liam so we'll ignore that for now. As for Adam, I notice his youtube channel has uploads as recent as one week ago, so I'd say it's still active; I left a message on his channel (I noticed Akseli had already done this too). I also identified that the phone number in the domain WHOIS lookup appeared to be a cellphone so I've sent an SMS message to it.
Post by: mobius on December 18, 2014, 10:23:10 PM
Btw, do I read that you are able/willing to upload posts from previous forums like Lemmingswelt onto this new forum? If so, why wasn't this done before? [geoo or you(s) weren't in charge is the answer I'm guessing]
Post by: namida on December 18, 2014, 10:30:36 PM
I'm pretty sure someone said he's on facebook-- might be a good idea to try contacting him there. I do not use face book regularly so I don't know how to do that.
Btw, do I read that you are able/willing to upload posts from previous forums like Lemmingswelt onto this new forum? If so, why wasn't this done before? [geoo or you(s) weren't in charge is the answer I'm guessing]
I have Facebook, so I'll have a go at contacting him on there.
EDIT: Unless someone has a more up-to-date email address for him, no luck (maybe geoo can find one from the user database here?). However, I have managed to locate Liam's Facebook - no idea how useful that'll be; he hasn't posted anything publically since mid-late 2013, but it's possible more recent posts just weren't set to public; I won't be able to see that unless/until he accepts a friend request.
As for the old forums, one issue is a difference in format. But yes, the main issue is that doing so will require server access, which on this forum, only Adam (and maybe Liam) have, and they were either unable or unwilling to do so (or maybe just never thought of it). But yes, geoo has a backup of the old forums, which he's passed on to me, and I should be able to do something to get them up on the new board - I'm going to start looking into it very soon.
In case this one disappears without warning, I'm going to update my HTML backup of it on a daily basis where possible. Worst case scenario; we can at least get the posts and attachments back, which I believe is the most important thing.
Post by: namida on December 20, 2014, 07:28:46 AM
Anyway, in terms of importing the old posts, I've done a bit of looking into exactly how to do it, and I should be able to write SQL files (or more accurately, write an app to generate SQL files) to batch-add old posts. Shortly I'm going to try just adding a small hand-written snippet to make sure it works* (this'll be added to a mod/admin-only board on the new forum so most of you won't see it); once that works, I'll try to parse the LemmingsWelt database, which is ironically the one that's in a more convenient format for me to write the app for. If all goes well, I'll add IsaMedia next, and once both of those are in, I can look at doing something with *this* board, since it doesn't look like we're going to be getting a copy of the database...
One issue geoo raised (and this may be very important to some people) is private messages. Unless we get the database, there's simply no way I can import the private messages from this site, for obvious reasons. So, if we don't get the database then when we do move, make sure you save a backup copy of any important private messages somewhere.
EDIT: * This was 100% successful on the first attempt. There is one minor issue - even if the posts (going by timestamp) is older, the newly-added post is pushed to the top of the topics list. Therefore, until further notice, I've disabled posting on the new board. I'll re-enable it when either:
a) I find a way around this
b) These forums go down for more than 24 hours, unless it's an intentional action as part of the move
c) All posts have been imported
And just to confirm 100%: Yes, we will have the LemmingsWelt and IsaMedia posts imported too, not just the ones from *this* forum.
However, it looks like we'll only have the attachments from *this* forum; the other two will be posts only.
Post by: Mr. K on December 20, 2014, 05:24:03 PM
I'm actually surprised that the most recent database backup we had access to is from within this year, I would've thought it was a bit older. Remind me how were we able to get the Jan 2014 database without involving Adam or otherwise actually getting direct access to the database?
Admin panel allows a database export. I have been taking a backup whenever I log in, but it seems this broke around the Jan 9th backup. Every export I have after that is broken-- it just cuts off halfway through an early admin panel table and everything of use to us is completely missing. I'm thinking there may have been a host-side change that is restricting how big a file it can generate so it gets truncated before we receive it.
Also, I did a search on my backups and it doesn't look like I have a backup of any of the old attachments. I'm trying a few different queries to see if I might have named it weirdly. It's hard to find anything at all by hand when you have about 10TB of files. :/
Post by: Luis on December 20, 2014, 07:26:46 PM
Post by: Prob Lem on December 20, 2014, 07:38:29 PM
What about all the pictures that are in the posts? Will those be carried over too? I had posted a lot of pictures in this forum and it would be a shame for all of that to be gone and have to post them again.How did you post them? As attachments here, or using an outside image-hosting service?
Post by: Luis on December 20, 2014, 08:01:17 PM
Post by: Prob Lem on December 20, 2014, 09:23:49 PM
Post by: namida on December 20, 2014, 11:34:36 PM
EDIT: I mentioned yesterday that I successfully added a post directly via the database, using hand-written SQL queries. I've now successfully done the same with automatically-generated ones based on simply supplying a board ID, a subject, a message, a username and a post time to a self-written piece of code. This means that now, all that needs to be done is writing the side that parses the existing forum files in some format and supplies the data to this code snippet. LemmingsWelt will be first, both due to it being the oldest and due to it being in an easier format to work with.
Post by: namida on December 21, 2014, 03:18:11 AM
EDIT: After posting this, I noticed a small error; namely that the timestamps on the messages are wrong. Time to look into that...
Next up; IsaMedia! Due to the format this one is in, which isn't particularly well-suited to anything I've made so far, the easiest way might actually be to first write an app to convert it to a format similar to that which LemmingsWelt is in. However, I'm taking a short break for now.
(As in a few hours or so, not a few days. )EDIT: Got the timestamps fixed. I'm now trying to come up with a piece of SQL code that'll fix the message ordering with minimal side effects. That way, I can allow posting on the new forum without having to worry about disappearing messages (not as in disappearing completely; more accurate would be to say "dropping to the bottom of the list even if the added messages are older").
Post by: Simon on December 21, 2014, 06:11:59 AM
Do we want the ancient posts from Lemmingswelt and Isamedia in the same postable boards as the 2009-2014 posts? The alternative is to make read-only boards for the ancient posts at the bottom of the list. I.e., do we want to bloat the boards or the board list? :->
-- Simon
Post by: namida on December 21, 2014, 06:38:41 AM
My attempts to create some SQL code to do this failed, but the keyword here is "SQL". I used some PHP instead, and this was much more successful.
As there's now no issue of new topics getting lost below old ones, I have re-enabled posting on the new forum.
At this point, it's really safe to post on whichever forum you like - if we do get the database from this one, I'll be able to come up with some workarounds to merge the two. If we don't, then nothing is lost and no work is created by posting on the new one.
(Also: All LemmingsWelt topics are locked, and the same will be true for the IsaMedia ones. But the ones from this forum won't be.)
Post by: Simon on December 21, 2014, 07:03:11 AM
-- Simon
Post by: namida on December 21, 2014, 07:21:37 AM
Post by: namida on December 21, 2014, 08:56:28 AM
So now, the question is... import this forum now (and then ideally get everyone moved over to avoid having to import it twice, which could be difficult), or wait a bit longer on the DB?
Advantages of waiting for the DB:
- Private messages can be transferred - otherwise, people will need to make their own backup of important ones (which they COULD pm to themself on the new board if they want it in their PMs specifically)
- Existing user accounts can be transferred rather than re-registration being needed
- Posts from this board won't need to be manually re-linked to accounts
- Even if the database isn't directly re-usable for some reason, I can most likely re-use most of the code used to import the IsaMedia forum
- The super-secret Staff Discussion board can be transferred
Disadvantages of waiting for the DB:
- We may be waiting for something we'll never end up getting or not for an unreasonably long time
- The new site is still costing me whether we're using it or not - though on the other hand, in the meantime it does still function as a searchable archive of the two old boards in one place. The same is most likely true for this board costing Adam.
- There's no guarantee it'll be fully usable
- This forum may go down without warning, and currently my backup is a few days old - but I will update it very soon and keep it updated on a fairly regular basis
- Some users may prefer to get this over and done with ASAP
Stuff that it won't make a difference about:
- Attachments from this board. They'll be safe either way.
- Links to posts on this board. Due to the ordering fix needed which modifies the post IDs, they are likely to break either way.
- Whether or not we get the lemmingsforum.com URL. Although we need to contact the same person, there's no guarantee he'll give us both.
EDIT: A compromised measure that has been discussed is importing what we can from the January backup, and then capturing posts for the rest. I'm not too keen on this idea myself as it'll be very messy and difficult, especially in the case of posts that were captured in it but have been edited since, and posts that have been deleted since then. Importing attachments from it could also be more difficult compared to from a recent backup or capturing the posts in HTML and parsing them. While I could look into this if there's very high support for it, it's a method I'm very not keen on. In short; while it is plausible, it'll also be by far the most difficult method without having much advantages (basically just people registered before this won't have to re-register, and it'll preserve private messages from before this time), which is why I'll need to see a lot of support for it if I'm going to consider this.
I doubt the staff here are the kind to have concerns about this, but just in case; I'm more than happy to let existing staff (provided they're reasonably active) retain their positions. Just make sure you PM me on this forum (or talk to me on #lix) to confirm your new account is really you.
Post by: LJLPM on December 21, 2014, 10:20:45 AM
Post by: namida on December 21, 2014, 11:03:30 AM
http://lemforums.neolemmix.com/old/
Obviously, once it's fully established, I'll get a proper domain name for it.
For now - register there, know the link just in case this goes down. However, while you can post there (and it won't be lost), unless it relates specifically to that site itself, it might make more sense to keep your posts here in the meantime. And of course, feel free to use it to look through the old posts. Both LemmingsWelt and IsaMedia are added (no attachments, sorry, just the posts).
Post by: Nepster on December 21, 2014, 11:05:59 AM
This way most of us could just continue as usual and only a few newly joined members have to register anew (which the most active ones, i.e. exit, NaOH and myself, already did).
Post by: namida on December 21, 2014, 11:11:10 AM
Is it possible to use the database to transfer only certain parts? Then I would be in favor of transferring the accounts and PMs as of January 2014 using the database and get the rest (i.e. the posts) by hand.
This way most of us could just continue as usual and only a few newly joined members have to register anew (which the most active ones, i.e. exit, NaOH and myself, already did).
Yes, this is definitely possible. How easy it will be is another matter, though without a doubt easier than importing the entire database then adding the missing bits since. Especially importing the user accounts could be messy if it's done without wiping the ones already there; as far as PMs go, there is always the option of manually importing what's available of a user's PM's on request (from the January database). It's possible it may not correctly link who the PM is from, but I could possibly get around this by having them listed in the database as coming from no account (or if that doesn't work, my own account or a dummy account set up for that purpose) and add a note at the start of them saying "PM originaly from (blahblah)". Just for peace of mind; this can be done completely automated without me having to specifically look at and modify the contents of them.
Either way; there is nothing I can do about PMs more recently than the January backup. You'll have to back those up in some form yourself - one thing I could suggest is PM copies of them to yourself on the new board, if you don't want to save them offline. (Yes, you can send PMs to yourself, on both this board and the new one.)
As far as parsing the HTML goes - I haven't started on any code yet (though a small portion of the code from the other two will be re-usable; the same portion that was also reusable between LemmingsWelt and IsaMedia), but I have taken a look at the HTML layout and the layout the website dumper I used puts the files, to determine what approach to use.
So far, I have identified how to crawl the topic lists and get the links to the topics' content themself, but haven't looked at how to crawl this last part yet (which will no doubt be the hardest part). I'll also have to take into account that HTML will need to be reverted to BBCode.
Post by: geoo on December 22, 2014, 09:49:17 PM
I really wanted to have a look at all these things one of these days, but somewhat unexpectedly I have more things on my mind right now than I hoped to have...
Post by: ccexplore on December 22, 2014, 10:48:42 PM
I see that some people have already re-registered themselves on the new forum, but I thought at some point you'd use the Jan 2014 database backup so that only those users who registered on the current forum after the Jan 2014 date needs to re-register? Is that the case and if so, any chance you can transfer the users from the database backup now?
=================
Advantages of waiting for the DB:
- Private messages can be transferred - otherwise, people will need to make their own backup of important ones (which they COULD pm to themself on the new board if they want it in their PMs specifically)
Is access the only issue with transferring PMs? That is, you can't use the same technique you plan to use for regular posts purely because PMs require the user's password/login to access?
I'm getting the (possibly wrong) impression that only a few people seem to need their PMs transferred. If that's the case, perhaps it's a workable solution if you provide us the same tools you are using to do the transfer and have us do it ourselves (or at least the extraction portion, since obviously we won't have database access to import into the new forum)?
Links to posts on this board. Due to the ordering fix needed which modifies the post IDs, they are likely to break either way.
Hmm, there's no way for you to keep track of how the old IDs mapped to the new IDs, and then apply a second pass to fix up the IDs?
Post by: mobius on December 22, 2014, 10:57:33 PM
I keep meaning to take care of them and delete them all but I haven't gotten around to it. In the end I'll most likely delete them all including ones which I've meant to adress but haven't.
The only ones I currently really need are the ones for the Lix contest. [which make up a good portion]
Post by: namida on December 23, 2014, 12:28:16 AM
The problem with getting PMs (and for that matter the staff board) is an inability to do it with a logged in account. Though it is possible to log in, one of the first things the app does is grab a copy of the log out page, and you can guess the results....
I might be able to do that with fixing post links, now that you mention it. I can at least look into it.
Post by: geoo on December 23, 2014, 12:31:41 AM
Post by: namida on December 23, 2014, 12:36:26 AM
The actual importing to the database is pure SQL (geneeated by the QB64 code), and the order fix is a PHP/SQL hybrid as I wasnt able to get it working in pure SQL.
Post by: ccexplore on December 23, 2014, 01:31:51 AM
The problem with getting PMs (and for that matter the staff board) is an inability to do it with a logged in account. Though it is possible to log in, one of the first things the app does is grab a copy of the log out page, and you can guess the results....
Have you looked at the "filters" feature and see if that can be put to use to block WinHTTrack from navigating to the log out page? Seems like an -*action=logout* filter should work, assuming the filtering works the way I think it does (it isn't clear from the page whether the filtering actually looks at the query portion of the URL [ie. the stuff after the ?]).
Failing that, another possibility is to install a web debugging proxy such as Fiddler, which can be set up on the machine you're running WinHTTrack on to observe and modify inbound and outbound http requests/responses on the fly, for example to effectively replace the outgoing request for logout to something else so that the logout doesn't actually happen while Fiddler is active. It's been a while since I've used Fiddler but if it comes to that, I can try to help you with the setup needed.
Post by: namida on December 23, 2014, 03:19:19 AM
Post by: geoo on December 23, 2014, 03:22:08 AM
Post by: namida on December 23, 2014, 03:33:56 AM
Anyway, I'm going to attempt to import all existing user registrations (EDIT: not including those who had zero posts at this time) from the old backup. I'll let you know soon whether or not it was successful.
Post by: namida on December 23, 2014, 04:26:36 AM
Check this list to see if your username has been imported. If it hasn't, it's because you either registered after the January database backup, or you had zero posts at this time; if this is the case, you'll have to register a new account. If you had already registered an account on the new one, your old account would've been skipped during the import if the username and/or the email address were the same. This only affects a small handful of users, but if you don't see your name on the list it may also be because you already registered an account on the new forum. I didn't compare IP addresses or other profile information to check for duplicates, so if you do have a duplicate account, please let me know so I can merge them or delete one.
Most profile information was also imported along with the accounts themself. However, some things (such as mod/admin status, post count) weren't.
I'll look into importing the old private messages soon.
This way, you'll only need to back up (if we can't find a way around this) ones received since the backup.EDIT: Working on it now!
Post by: namida on December 23, 2014, 06:45:10 AM
Please do check that none are missing just in case. And there likely won't be any way to transfer newer ones, so make sure you back those up somehow.
EDIT: Does anyone think the IsaMedia PMs should be restored too? I could probably restore these too, although more will probably be missing due to username/email changes (as I'm using these to match accounts from the backups to those on the new forum - only one has to match, not both).
At this stage, pretty much all that's left that's possible (aside from the IsaMedia PMs if anyone thinks it's worth bothering - it shouldn't be too much effort, as long as they're in the backup I have - haven't checked if they are) is the posts (which unlike LW and IsaMedia, will include attachments
) from here. I don't want to do them while people are still posting here though, for the reasons I've mentioned before. So... what to do from here?(By the way, I probably won't be doing much on Christmas Day itself. But the days before and after should be perfectly fine.)
Post by: Simon on December 23, 2014, 07:28:56 AM
[08:22] * PhilM has quit (Read error: Connection reset by peer)
[08:24] <namida42> i didnt attempt to do anything on the sent side, didnt really cross my mind
[08:24] <namida42> i can probably patch it up though
[08:25] <SimonN> hnn :)
[08:27] <namida42> remind me a bit later on, kinda had enough of it for now xD
[08:27] <SimonN> okay, will make a post to the forum
-- Simon
Post by: namida on December 23, 2014, 09:58:36 AM
[08:22] <SimonN> namida42: incoming messages up to early 2014 are all there, thanks. Outgoing messages are empty still
[08:22] * PhilM has quit (Read error: Connection reset by peer)
[08:24] <namida42> i didnt attempt to do anything on the sent side, didnt really cross my mind
[08:24] <namida42> i can probably patch it up though
[08:25] <SimonN> hnn
[08:27] <namida42> remind me a bit later on, kinda had enough of it for now xD
[08:27] <SimonN> okay, will make a post to the forum
-- Simon
Simon already knows this due to being a guinea pig for checking if it works
but for everyone who isn't currently on #lix; this is now done.
Post by: namida on December 23, 2014, 11:25:25 AM
Note that they'll only have been imported if:
1. You have an account on the new forum. In other words, either a) you registered manually, or b) you had an account on here, with at least 1 post, at the time of the January backup.
2. Your username and/or email address is the same on the new forum (it will have been imported from what it is on *this* forum (or was at the time of the January backup, to be more precise) if your acccount was imported as described above) as it was on the IsaMedia forum.
3. In the case of sent messages, most likely they'll only have been imported if the recipient also met the above criteria.
Additionally, an order fix has been applied to the PMs similarly to how it was with the regular posts.
Post by: namida on December 23, 2014, 12:23:36 PM
That's all from me for today, will continue tomorrow.At this point, I'm suggesting we set a move-over date, and once this has passed we disable posting (but not viewing - I need to be able to grab a content dump as up-to-date as possible of course xD) on this board. Between the closing date here and the posts being transferred (it may take a few hours - ideally I'll have everything ready and working by then so the only thing left to do is actually run the import), you can always reply to existing topics here on the new board by creating a new topic there, and I can merge them once the posts from here are transferred over. My suggestion would be new year's day - that way most of the users will be too busy to need to post much here anyway - but of course this is something we all need to decide on rather than one of us saying "this is the date, and that's final", so please give your input.
Once the posts are transferred, we can start on linking them to actual accounts (instead of as guest posts).
Also - when this happens, we'll need to update the News thing at the top of the board to reflect this. I believe geoo (or Mr K) should be able to do this from the admin panel, unless that's broken at some point.
Post by: Prob Lem on December 23, 2014, 01:06:21 PM
If you had already registered an account on the new one, your old account would've been skipped during the import if the username and/or the email address were the same. This only affects a small handful of users, but if you don't see your name on the list it may also be because you already registered an account on the new forum.I'm one of the ones affected by this, as I'd already registered on the new forum quite early on in case this one fell over. I assume this won't present a problem for linking my posts from here to my account there, once the posts are moved over?
Latest update; we now have the Lemmings-y emoticons on the new board.Yay! I love the emoticons from here, and I was missing them over on the new forum.
Quote
My suggestion would be new year's day - that way most of the users will be too busy to need to post much here anyway - but of course this is something we all need to decide on rather than one of us saying "this is the date, and that's final", so please give your input.New Year's Day is fine by me - new year, new start, right?
I guess we'll have to make our forumy New Year wishes either a little early, or a little late, eh?
Post by: namida on December 23, 2014, 01:10:26 PM
If you had already registered an account on the new one, your old account would've been skipped during the import if the username and/or the email address were the same. This only affects a small handful of users, but if you don't see your name on the list it may also be because you already registered an account on the new forum.I'm one of the ones affected by this, as I'd already registered on the new forum quite early on in case this one fell over. I assume this won't present a problem for linking my posts from here to my account there, once the posts are moved over?
Nope, it wont' present any problem at all. The only disadvantage you may have is that you'll have to manually re-add any profile info you want to carry over (such as signatures, contact info in your profile, etc).
So, my task over the next few days is to get together an app that can parse this boards. I'm already mostly done with the planning stage, but still need to do the actual coding. Of course, I won't actually add the posts until the deadline, but needless to say I'll be doing a few test runs (basically, generating the data for importing, just without actually doing the final import to bring it live) beforehand - one reason for choosing new years' is this should give me plenty of time to do it.
Post by: Prob Lem on December 23, 2014, 01:23:38 PM
Nope, it wont' present any problem at all. The only disadvantage you may have is that you'll have to manually re-add any profile info you want to carry over (such as signatures, contact info in your profile, etc).That's awesome, thanks.
The profile info thing isn't a problem for me at all - I was going to rejig my signature a little bit once the move was done, anyway!
EDIT: Just had a browse on over to the new forum, and noticed that I'm unable to post or reply (I just wanted to try out the emotes, haha). I assume posting is disabled over there for now, even though there's a news note about merging duplicated posts?
Post by: namida on December 23, 2014, 10:53:03 PM
Fixed now.The reason for this is that before importing any old forum data (or even applying minor fixes usually) I back up the database, so that if it goes wrong (which it often does on a first attempt), I can simply jump back to the previous state rather than having to manually undo the damage. So, to avoid loss of posts, I also disable posting before the backup is taken. In this case, I must've forgotten to re-enable it.
Post by: ccexplore on December 24, 2014, 12:23:21 AM
It'd be the PM recipients who'd need to run that, not me. :p
I did some testing and have confirmed that the filters feature in WinHTTrack is sufficient to control its navigation in a logged in session, making the tool viable for grabbing PMs. Which is a very good thing as it is more than just logout--there are other kinds of links like "delete" that could potentially do other unwanted stuff.
I'll admit that the steps are not easy for computer novices unfortunately, partly due to having to deal with cookies in order for WinHTTrack to make use of your active login session. If you don't have too many PMs that need transfer (keeping in mind that everything before Jan 2014 are already in the database backup), you may find it easier to just use your web browser to save the few pages of PMs that need transferring, instead of getting the tool to do it for you.
For reference, here's a rough outline of how I've tested (only on a test account so far, since I want to be absolutely sure it doesn't accidently navigate a link that would perform a destructive action, one of the potentialdangers of crawling through a logged-in forum session):
1. create a directory (folder) to serve as working directory for WinHTTrack for crawling your PMs. Then create another directory under it with a simple name like "myPMs".
2. on web browser, logged in to desired user account with "stay logged in" checked
3. grab/export the cookie file associated with lemmingsforum, which contains the data necessary to allow WinHTTrack to use your active logged-in session as it crawls (obviously needed in order to browse your PMs). You may need a tool besides your own browser to properly get the cookie file, as WinHTTrack requires the cookie file to be in a specific format, which may not be the one used by your browser for storing the cookies. You may also need to slightly modify the contents of the exported cookie file to make WinHTTrack happy. For example, the tool I used seems to leave an extra "." before "www.lemmingsforum.com" inside the cookie file, and I had to remove that extra period to get things to work.
4. Put the exported cookie file into your directories from step 1 (one copy in the working directory, and another copy in the "myPMs" sub-directory or whatever you named that), make sure the filename is exactly cookies.txt
5. now start WinHTTrack and start a new project. For "Project name", set it to exactly the name of your sub-directory (eg. myPMs) from step 1. Set the "Base path" to the working directory from step 1. Hit Next.
6. Set "Action" to either "Download web sites" (if first time) or "Update existing download". For the "URL", use "http://www.lemmingsforums.com/index.php?action=pm" (no quotation marks of course). Then click "Set options". An "WinHTTrack" options window pops up.
7. Here's the important part: go to "Scan Rules" tab, and add the following to the end of the line you see in the text box:
-*?topic=* -*?board=* -*?action=* +*?action=pm* -*sa=* -*togglebar*
A quick explanation of what this does: the "topic" exclusion blocks crawling of forum posts, "board" exclusion blocks crawling of forum boards, and "action" exclusion blocks lots of actions you don't want the crawler to crawl and trigger unintentionally, like the link in the "logout" button for example (or worse, the "delete" buttons for your PMs!). The "action=pm" inclusion then allows the crawler to crawl just the URLs that are specific to your PMs (both the inbox and sent). The "sa" exclusion blocks site actions, which are associated with other unwanted actions like "prune messages" or changing your settings. The "togglebar" exclusion blocks the "<<" thing you see near the top-right of the PM pages, which apparently changes your user preference of the way the PM page is laid out.
8. Now go to "Limits" tab and set "Maximum mirroring depth" to a reasonable number. My suggestion is to look at how many pages of PMs you have in your inbox and in your sent box, take the maximum of the two, and then add like 3 to it and use that number. I actually don't know if you really need to set a max here, but it does ensure that the crawler won't get into a runaway situation where it is not able to accurate detect things it already crawled and therefore never stops crawling. Higher number simply means you are more likely to get pages saved by the crawler that are effectively duplicates.
9. Now click OK to commit the settings. Then "Next" through the rest and let the tool do its thing. It may take some time before it's done so be patient.
10. Once it's done, click the " view mirrored website" button and see what you get. If all goes well, the page that you first see would be a mirrored copy of your PM inbox page. If instead you get the page that asks for username and password, then the cookies you set up from steps 1-4 are apparently not getting used. Manually browse through a few links to verify that your entire inbox and sent box get mirrored (by seeing that the URL displayed in the browser is referring to a local file on your computer, rather than the usual HTTP web address). (Note: you may notice that the user avatar pictures are missing in the mirrored pages. This is a side effect of the "action" exclusion rule, since the URLs to the avatar pictures do contain that "action" keyword in their URLs. While it can fixed fairly easily by refining the scan rules, it really doesn't matter since the point is just to capture the contents of your PMs.)
Anyway, with that being a success, I'll take a look at capturing the PMs for my own account and for the QuizMaster account, and see if namida can do anything with the resulting files.
Post by: namida on December 24, 2014, 12:31:58 AM
Keep in mind that, while of course I'm not going to intentionally read through them, if any part is causing errors at any stage of the import process, I will need to take a quick glance at the part that's causing the error
By the way, ccexplore: I've been meaning to ask if you'd be interested in an admin position, with server access, given your technical skill level? I wouldn't be expecting you to do much, it's just so that in the event server access (or just a regular board admin) is needed, we have multiple people rather than ending up in a situation like we have here where two admins can't be contacted and the third doesn't have server access.
Post by: ccexplore on December 24, 2014, 03:59:05 AM
-*?topic=* -*?board=* -*?action=* +*?action=pm* -*sa=* -*togglebar* -*?action=pm;view;*
This last rule corresponds to blocking the invocation of the leftmost column header that you can click to apparently toggle between the "show all at once"/"show as a conversation" views of your PMs (at least I think that's what it does). While there's no harm in not including this in the scan rules, it does help speed things up as well as avoid duplications, as triggering the alternate view (ie. show as conversation) tend to show only one PM on each page, depending on the ratio of conversation threads to posts for your PMs. It also seems to affect the user setting (which of course you can always change back afterwards manually, so no harm but still a nuisance) that determines whether the default view of your PMs is show all vs show as conversation. Blocking it with the additional rule above avoids all that.
I PM'd the resulting files from the QuizMaster PMs to namida just now over at the new forum, so there is at least one set of files he can use to test out whether he can import those successfully into the new forum.
By the way, ccexplore: I've been meaning to ask if you'd be interested in an admin position, with server access, given your technical skill level? I wouldn't be expecting you to do much, it's just so that in the event server access (or just a regular board admin) is needed, we have multiple people rather than ending up in a situation like we have here where two admins can't be contacted and the third doesn't have server access.
Yeah sure, I can do that, just consider me as a backup.
Post by: namida on December 24, 2014, 04:54:46 AM
Post by: namida on December 24, 2014, 05:43:49 AM
NOTE: It's likely the more complex tags, I won't "translate" back but will just directly put the HTML into the new database. This will only matter if someone attempts to later edit the post, in which case they'll have to manually fix these tags - if the post remains unedited, the direct HTML will work fine.
ABBR
Test 2
ACRONYM
Test 2
ANCHOR
[anchor="Test 1"]Test 2[/anchor]
B
Test
BDL
[bdl=ltr]LTR[/bdl]
[bdl=rtl]RTL[/bdl]
BLACK
Test
BLUE
Test
BR
Test
Test
CENTER
Test
CODE
Code: [Select]
TestCOLOR
Red Test
Blue Test
#dee7ef Test
Test 2
Test 3
FLASH
Sized :: http://Test
Unsized :: [flash]http://Test[/flash]
FONT
Test 2
FTP
Test 1
Test 3
GLOW
[glow=orange, 2]Orange 2[/glow]
[glow=red, 3]Red 3[/glow]
[glow=#dee7ef, 4]#dee7ef 4[/glow]
GREEN
Test
HTML
Test 1 <b>Test 2</b>
HR
I
Test
IMG
Sized ::
Unsized ::
IURL
Test
LEFT
Test
LIST / LI
- Test 1
- Test 2
- Test 3
- Test 4
LTR
Test
ME
[me]Test 1[/me]
* Test 2 Test 3
MOVE
NOBBC
Test
PHP
Test
PRE
Test
QUOTE
Quote
Test 1
Quote from: Test 2
Test 3
RED
Test
RIGHT
Test
RTL
Test
S
SHADOW
Red Left
Blue Right
#dee7ef Top
SIZE
X-Large
Small
16pt
SUB
Test
SUP
Test
TABLE / TR / TD
| Test 1 | Test 2 |
| Test 3 | Test 4 |
TIME
2005-11-24 00:10:40 (1132812640)
TT
Test
U
Test
URL
Test 1
Test 3
WHITE
Test
Post by: namida on December 24, 2014, 05:46:16 AM
MAKESHIFT SPOILER TAG
Quote
Test 1
Quote from: Test 2
Test 3
Post by: namida on December 24, 2014, 08:14:36 AM
EDIT: Change that "pretty much" to "completely". The last thing remaining was how to add attachments to the database; it is a bit complex but I think it's doable, and if not, I have a plan B that definitely will work at the cost of not exactly replicating the normal attachment configuration. (Specifically; uploading the files themself to the server and adding links to them to the ends of the messages they're attached to.)
Post by: namida on December 24, 2014, 08:59:12 AM
www.lemmingsforums.net
This should be pretty easy for people to remember, since it's the same as the current one except .net instead of .com.
Any links using/containing the old URLs will just redirect to the correct new ones, though if you chose the "keep me log in" option you will have to log in again. If you can't access the site at the moment, just try again after a few hours - it should already be working for the majority of people though.
Post by: namida on December 26, 2014, 02:06:55 AM
It doesn't appear there's an option to set "no limit" to the number of posts where this option will still be available, but a workaround is to just set it to a much higher number than any topic on the board has.
This doesn't need to be done right now - I can *test* the import code fine without it (as there are topics that DO display this option), I'll just need the setting changed for the actual import.
Had one other minor issue so far, but that was related to the settings when I saved an offline mirror of the site, so doesn't need anything done on the admin side to fix; I just have to adjust the setting and redownload it.
That aside, it looks like there'll be little trouble with importing this current forum's posts.
Post by: geoo on December 26, 2014, 02:12:23 AM
It doesn't appear there's an option to set "no limit" to the number of posts where this option will still be available, but a workaround is to just set it to a much higher number than any topic on the board has.Changed this setting. That limit has in fact been annoying me in the past already when I wanted to save topics offline, and the only way to do it was using the print option.
Post by: namida on December 26, 2014, 04:04:30 AM
After that, PMs is my next target.
A couple of heads ups (both in reference to the new forum):
- At some point I'll have to disable posting for about 20 minutes or so to do a test run of importing posts from the old forums.
- When doing the real import, I'll have to at least disable attachments for a short time.
Post by: namida on December 27, 2014, 10:43:32 PM
Unless anyone thinks we should do it "as soon as we're able to", I think we'll keep the target date for the move as New Years' - as mentioned, I will be doing at least one or two test runs before this of transferring the posts - but in the meantime, I've used my newly-added admin powers on this side to add a note about this to the news flasher on the main page about the move.
(Speaking of which, who actually added me as such?)
Post by: namida on December 28, 2014, 02:15:38 AM
It's in testing at the moment. Due to the need to parse HTML it's a lot slower than the LemmingsWelt and IsaMedia codes were; whereas those generally took about 3 minutes each to run, the first "dry" (ie: not actually uploading, just generating output files) code of this is currently about 10 minutes into running and only up to the Level Designing board's second page. Still, it seems to work now, based on smaller test runs (only importing a small selection of posts instead of the entire forum). If the output from this looks decent, I'll do a small import test run of actually importing a few topics to the new board (which will of course be cleared after being confirmed as working, to avoid duplicating them when the full import is done).
One thing I haven't implemented yet is the timestamp. In order to keep this simple, I'll need to get a new download of the site with a couple of timestamp-related settings changed. So, at some point, the timestamps on this board are going to have to be changed to "YYYY-MM-DD HH:MM:SS" format, without the special display of "Today" and "Yesterday". This will have to be done on two occasions, both of which a full download of the site (which takes a couple of hours) will have to be run:
a) At some point to run a test run *with* timestamps.
b) When we do the actual final import.
My apologies if this is inconvenient to anyone, though it shouldn't be a major issue.
Note that about the timestamp thing; if you have a custom timestamp format enabled in your account settings, you don't have to change this. It's just how *guests* view it that must be changed (which also gets applied as the default for anyone who hasn't set a custom one).
I haven't done anything on the importing PMs yet. If you make backups as ccexplore described, you've at least got them in some form; I can sort out doing something with this later. The way I see it, although it is something I want to deal with eventually, it's a less critical issue than the posts themself and thus I'll deal with that once posts are fully handled.
EDIT: It appears the import posts code is successful on a "dry" test run. I'm going to now do a proper test with just 2 or 3 topics (including some attachments).
EDIT: The results are somewhat successful. A few tweaks and it should work better... Specifically, no problems at all with the post's content - it imported fine - but attachments are somewhat faulty.
EDIT: Complete success has now been acheived.
The only remaining todo is to add code for handling the timestamp. EDIT: The date format has been changed as mentioned above, in order to do the first test run. I will change it back when finished; but if you want to display dates normally in the meantime, go to Profile > Account Settings > Look and Layout, and enter this in the Time Format input box:
%B %d, %Y, %I:%M:%S %p
Post by: Prob Lem on December 28, 2014, 12:37:23 PM
I can only see one issue right now, and that's that my post-count seems to be out-of-whack with the actual number of posts I have (I see that they all seem to be correctly linked to my account :D).
I absolutely *LOVE* the new icons on the main index page, by the way - they look really smart.
Post by: namida on December 28, 2014, 12:54:24 PM
I can only see one issue right now, and that's that my post-count seems to be out-of-whack with the actual number of posts I have (I see that they all seem to be correctly linked to my account :D).
This is a known issue that'll be fixed eventually. I just haven't done it yet partly because I want to wait and attach more posts to the users, and partly because I'm kind of sick of doing move-related coding for one day - I'll do more stuff tomorrow, but for the rest of tonight, I'm only fixing things if they're very major; which post count isn't. :P As you may have noticed, I haven't even moved the NeoLemmix and Lemmings Plus topics to the correct boards yet (since they'd be more suited to the Other Projects board, or in the case of NeoLemmix, it has its own dedicated board now).
Post by: Prob Lem on December 28, 2014, 01:06:06 PM
Best of luck with the rest of what needs to be done. :D
Post by: namida on December 28, 2014, 01:13:26 PM
Aye, I figured it wasn't a big enough deal to be sorting out right now. :thumbsup: I'm just making it a note of it so it's noted, really.
Best of luck with the rest of what needs to be done. :D
Basically, the remaining todos are:
1) Sort posts into their correct boards, in cases where there's now a more suitable board that didn't exist before (eg: NeoLemmix and SuperLemmini now have dedicated boards, there's a "Other Projects" board which things like large level packs could go in).
2) Fix up missing attachments that didn't transfer over properly - the majority of them did, but there are some that didn't.
3) Link remaining posts to the users.
4) Fix the post counts.
1 and 2 can be done by any of the staff members, including board-specific moderators for posts on their boards, or (if the posts have been linked to the user already) by the original poster of the messages themself. 3 and 4 are tasks that require admins - most likely myself, though it's not entirely impossible for ccexplore or geoo to do it (but 3 in particular would be difficult as server access for them hasn't been set up yet; only Admin panel access and database access has been).
Post by: Prob Lem on December 28, 2014, 02:09:00 PM
If you need someone to help manually retrieve them, I've got a bit of free time right now, if you can point me to what needs grabbing.
Post by: namida on December 28, 2014, 02:13:00 PM
Any posts with this issue have a note added at the end of them saying "X attachments could not be loaded" (or something along those lines); if you find any such posts, make a note of the URL. I'm going to put up a dedicated topic for this, once the ones that *did* make it finish uploading - currently there's about 400 remaining, but this isn't requiring any effort; it's just a LOT of data to upload so it's taking quite some time.
Post by: Prob Lem on December 28, 2014, 02:14:46 PM