Server Move - Discussion and Status

[ccexplore]

Quick update:  I have successfully got the contents of all the PMs (inbox + sentbox) for both ccexplore and QuizMaster, using the method outlined above, with one minor modification:  for the WinHTTrack scan rules, add the following additional entry:

-*?topic=* -*?board=* -*?action=* +*?action=pm* -*sa=* -*togglebar* -*?action=pm;view;*

This last rule corresponds to blocking the invocation of the leftmost column header that you can click to apparently toggle between the "show all at once"/"show as a conversation" views of your PMs (at least I think that's what it does).  While there's no harm in not including this in the scan rules, it does help speed things up as well as avoid duplications, as triggering the alternate view (ie. show as conversation) tend to show only one PM on each page, depending on the ratio of conversation threads to posts for your PMs.  It also seems to affect the user setting (which of course you can always change back afterwards manually, so no harm but still a nuisance) that determines whether the default view of your PMs is show all vs show as conversation.  Blocking it with the additional rule above avoids all that.

I PM'd the resulting files from the QuizMaster PMs to namida just now over at the new forum, so there is at least one set of files he can use to test out whether he can import those successfully into the new forum.

By the way, ccexplore: I've been meaning to ask if you'd be interested in an admin position, with server access, given your technical skill level? I wouldn't be expecting you to do much, it's just so that in the event server access (or just a regular board admin) is needed, we have multiple people rather than ending up in a situation like we have here where two admins can't be contacted and the third doesn't have server access.

Yeah sure, I can do that, just consider me as a backup.

[namida]

If you make a PM backup as ccexplore described, you'll have a backup of your PMs for your own purposes. My first priority is on importing the posts and attachments; after I've got this working I will definitely look at importing PMs from these dumps next.

[namida]

Ignore this post, I'm just doing it to test how various BBCode tags here are rendered in HTML. The reason I'm posting it rather than just previewing is firstly, in case there's any rendering difference between various ways of viewing (eg: as a guest vs logged in vs preview), and secondly so I don't have to remake it later if need be.

NOTE: It's likely the more complex tags, I won't "translate" back but will just directly put the HTML into the new database. This will only matter if someone attempts to later edit the post, in which case they'll have to manually fix these tags - if the post remains unedited, the direct HTML will work fine.

ABBR
Test 2

ACRONYM
Test 2

ANCHOR
[anchor="Test 1"]Test 2[/anchor]

B
Test

BDL
[bdl=ltr]LTR[/bdl]
[bdl=rtl]RTL[/bdl]

BLACK
Test

BLUE
Test

BR
Test
Test

CENTER

Test

CODE

Code: [Select]

Test
COLOR
Red Test
Blue Test
#dee7ef Test

EMAIL
Test 2
Test 3

FLASH
Sized :: http://Test
Unsized :: [flash]http://Test[/flash]

FONT
Test 2

FTP
Test 1
Test 3

GLOW
[glow=orange, 2]Orange 2[/glow]
[glow=red, 3]Red 3[/glow]
[glow=#dee7ef, 4]#dee7ef 4[/glow]

GREEN
Test

HTML
Test 1 <b>Test 2</b>

HR

I
Test

IMG
Sized ::
Unsized ::

IURL
Test

LEFT

Test

LIST / LI

• Test 1
• Test 2

• Test 3
• Test 4

LTR

Test

ME
[me]Test 1[/me]

* Test 2 Test 3

MOVE
Test
NOBBC
Test

PHP
Test

PRE

Test

QUOTE

Test 1

Test 3

RED
Test

RIGHT

Test

RTL

Test

S
Test

SHADOW
Red Left
Blue Right
#dee7ef Top

SIZE
X-Large
Small
16pt

SUB
_Test

SUP
^Test

TABLE / TR / TD

Test 1	Test 2
Test 3	Test 4

TIME
2005-11-24 00:10:40 (1132812640)

TT
Test

U
Test

URL
Test 1
Test 3

WHITE
Test

[namida]

One more I forgot to include...

MAKESHIFT SPOILER TAG

Test 1

Test 3

[namida]

I haven't done the actual coding for it yet, but I've pretty much worked out everything needed to import the posts from here except exactly how to add attachments properly to the database - once I work that out, that's all the pieces and I can start actually putting together some code to import them.

EDIT: Change that "pretty much" to "completely". The last thing remaining was how to add attachments to the database; it is a bit complex but I think it's doable, and if not, I have a plan B that definitely will work at the cost of not exactly replicating the normal attachment configuration. (Specifically; uploading the files themself to the server and adding links to them to the ends of the messages they're attached to.)

[namida]

I've now given the new site a proper domain name instead of just a subdomain on NeoLemmix.com. From now on you can use the following link to access the new forum:

www.lemmingsforums.net

This should be pretty easy for people to remember, since it's the same as the current one except .net instead of .com.

Any links using/containing the old URLs will just redirect to the correct new ones, though if you chose the "keep me log in" option you will have to log in again. If you can't access the site at the moment, just try again after a few hours - it should already be working for the majority of people though.

[namida]

I've been working on the import code and noticed one problem with the current settings which will make it far more complicated to parse. Specifically, to keep things simple, I'm using the "All" option on topics that are more than one page, but this doesn't display on very large topics. So, I'd need an admin to adjust the settings so that all topics display this option.

It doesn't appear there's an option to set "no limit" to the number of posts where this option will still be available, but a workaround is to just set it to a much higher number than any topic on the board has.

This doesn't need to be done right now - I can *test* the import code fine without it (as there are topics that DO display this option), I'll just need the setting changed for the actual import.

Had one other minor issue so far, but that was related to the settings when I saved an offline mirror of the site, so doesn't need anything done on the admin side to fix; I just have to adjust the setting and redownload it.

That aside, it looks like there'll be little trouble with importing this current forum's posts.

[geoo]

It doesn't appear there's an option to set "no limit" to the number of posts where this option will still be available, but a workaround is to just set it to a much higher number than any topic on the board has.

Changed this setting. That limit has in fact been annoying me in the past already when I wanted to save topics offline, and the only way to do it was using the print option.

[namida]

Alright, may not be able to do too much over the next couple of days (though I'll be able to do more later today), but at the moment, I've got it able to parse all the board links, page links within a board, and links to each topic (though the last one is untested as of yet). All that's not even attempted yet is finally parsing the posts themself after grabbing all the links.

After that, PMs is my next target.

A couple of heads ups (both in reference to the new forum):
- At some point I'll have to disable posting for about 20 minutes or so to do a test run of importing posts from the old forums.
- When doing the real import, I'll have to at least disable attachments for a short time.

[namida]

I'm back - and now I can put full focus onto getting this done!

Unless anyone thinks we should do it "as soon as we're able to", I think we'll keep the target date for the move as New Years' - as mentioned, I will be doing at least one or two test runs before this of transferring the posts - but in the meantime, I've used my newly-added admin powers on this side to add a note about this to the news flasher on the main page about the move.

(Speaking of which, who actually added me as such?)

[namida]

I've finished coding (but not yet fully testing) the code to import posts from this site - attachments included of course!

It's in testing at the moment. Due to the need to parse HTML it's a lot slower than the LemmingsWelt and IsaMedia codes were; whereas those generally took about 3 minutes each to run, the first "dry" (ie: not actually uploading, just generating output files) code of this is currently about 10 minutes into running and only up to the Level Designing board's second page. Still, it seems to work now, based on smaller test runs (only importing a small selection of posts instead of the entire forum). If the output from this looks decent, I'll do a small import test run of actually importing a few topics to the new board (which will of course be cleared after being confirmed as working, to avoid duplicating them when the full import is done).

One thing I haven't implemented yet is the timestamp. In order to keep this simple, I'll need to get a new download of the site with a couple of timestamp-related settings changed. So, at some point, the timestamps on this board are going to have to be changed to "YYYY-MM-DD HH:MM:SS" format, without the special display of "Today" and "Yesterday". This will have to be done on two occasions, both of which a full download of the site (which takes a couple of hours) will have to be run:

a) At some point to run a test run *with* timestamps.
b) When we do the actual final import.

My apologies if this is inconvenient to anyone, though it shouldn't be a major issue.

Note that about the timestamp thing; if you have a custom timestamp format enabled in your account settings, you don't have to change this. It's just how *guests* view it that must be changed (which also gets applied as the default for anyone who hasn't set a custom one).



I haven't done anything on the importing PMs yet. If you make backups as ccexplore described, you've at least got them in some form; I can sort out doing something with this later. The way I see it, although it is something I want to deal with eventually, it's a less critical issue than the posts themself and thus I'll deal with that once posts are fully handled.



EDIT: It appears the import posts code is successful on a "dry" test run. I'm going to now do a proper test with just 2 or 3 topics (including some attachments).

EDIT: The results are somewhat successful. A few tweaks and it should work better... Specifically, no problems at all with the post's content - it imported fine - but attachments are somewhat faulty.

EDIT: Complete success has now been acheived. The only remaining todo is to add code for handling the timestamp.



EDIT: The date format has been changed as mentioned above, in order to do the first test run. I will change it back when finished; but if you want to display dates normally in the meantime, go to Profile > Account Settings > Look and Layout, and enter this in the Time Format input box:

%B %d, %Y, %I:%M:%S %p

[Prob Lem]

Congratulations on a successful server move!

I can only see one issue right now, and that's that my post-count seems to be out-of-whack with the actual number of posts I have (I see that they all seem to be correctly linked to my account ).

I absolutely *LOVE* the new icons on the main index page, by the way - they look really smart.

[namida]

I can only see one issue right now, and that's that my post-count seems to be out-of-whack with the actual number of posts I have (I see that they all seem to be correctly linked to my account ).

This is a known issue that'll be fixed eventually. I just haven't done it yet partly because I want to wait and attach more posts to the users, and partly because I'm kind of sick of doing move-related coding for one day - I'll do more stuff tomorrow, but for the rest of tonight, I'm only fixing things if they're very major; which post count isn't. As you may have noticed, I haven't even moved the NeoLemmix and Lemmings Plus topics to the correct boards yet (since they'd be more suited to the Other Projects board, or in the case of NeoLemmix, it has its own dedicated board now).

[Prob Lem]

Aye, I figured it wasn't a big enough deal to be sorting out right now. I'm just making it a note of it so it's noted, really.

Best of luck with the rest of what needs to be done.

[namida]

Aye, I figured it wasn't a big enough deal to be sorting out right now. I'm just making it a note of it so it's noted, really.

Best of luck with the rest of what needs to be done.

Basically, the remaining todos are:
1) Sort posts into their correct boards, in cases where there's now a more suitable board that didn't exist before (eg: NeoLemmix and SuperLemmini now have dedicated boards, there's a "Other Projects" board which things like large level packs could go in).
2) Fix up missing attachments that didn't transfer over properly - the majority of them did, but there are some that didn't.
3) Link remaining posts to the users.
4) Fix the post counts.

1 and 2 can be done by any of the staff members, including board-specific moderators for posts on their boards, or (if the posts have been linked to the user already) by the original poster of the messages themself. 3 and 4 are tasks that require admins - most likely myself, though it's not entirely impossible for ccexplore or geoo to do it (but 3 in particular would be difficult as server access for them hasn't been set up yet; only Admin panel access and database access has been).