Forum Topic Pruning: SOS needs your help

[Forum Admin]

Hi everyone,

Our site is being negatively affected by the massive size of our database, and a vast amount of server processing and resources/caching are having to be devoted to the (phpBB) SOS Forum topics/posts. This is impacting on page-load times all across the rest of the site and our search engine's ability to perform optimally without timing out.

The SOS Forum currently has 616,472 Posts and 57,317 Topics. Regrettably, we need to reduce its Post Count by approximately one-third, circa 200,000. Post-purge, we will also need to implement phpBB's regular Post-pruning routines, which can be configured individually per sub-forum, so the post quantity does not grow massively and the oldest topics/posts will get constantly erased.

We need to action this initial pruning as soon as possible and propose to cull the oldest Topics/Posts on one or two forums at a time to reduce the massive server overhead pruning causes. We anticipate the pruning and our search engine re-indexing the remaining content taking 2 weeks to complete.

From our checks, we will need to erase every Topic (and its Posts) that has not been posted to since 01 January 2012. Achieving this will reduce the overall Post Count by the approximated 200,000 figure.

Here's where SOS needs your help...

Topics from pre-2012 that you value and see as a benefit to this community can be retained.

However, we need your help to identify them. So please look through your saved Bookmarked Topics or jump to the oldest pages of your favourite sub-forum(s) and skim through the Topic list.

When on a Topic you wish us to keep, please click on the REPORT THIS POST icon (an exclamation mark inside a circle) and type something like 'Do Not Prune' in the message box and submit it.

One of the Moderators or myself will then move that Topic and all its Posts into the existing USEFUL INFORMATION ARCHIVE (UIA) sub-forum and the Topic will become a read-only item. It will be searchable using the Forum/Main Search functions. If you then wished to 'reply' to the read-only Topic, you'd need to start a new Topic, link to the archived one, and continue discussions in the new Topic. Not ideal, but workable.

When can we start and how much time do we have?
You can start reporting valued pre-2012 Topics from today and we'll need to stop taking requests by 7pm UK time on Friday 15 May 2020.

Thanks for your understanding.

cheers,
Ian G

[muzines]

Is there any way to export a pre-pruned text dump of the forums, which can then be hosted online?

It's a shame in the age of big data that online history might be deleted, but I understand the performance and business needs to do this. But if some kind of text archive could be exported and hosted, then at least the data would be retained and be searchable just in case, rather than have a bunch of text gone forever.

Admittedly, how much value is in those posts is probably questionable, but even so...

Or is it all just a big conspiracy to stop me reaching my 10K post count..? (Again!)

[Forum Admin]

Hi Desmond,

We plan to take a full database backup prior to pruning and store it offline.

As to a text-dump, I've been told it's not a simple task since phpBB pulls in who posted, etc from our CMS user base, and it'll require our web team to spend expensive time to achieve this. That's time we don't have at present with all our other pressing projects and we urgently need to lighten the server load or we may find ourselves with patches of 'dark time', temporary Forum suspension, or any other undesirable scenario -- and commercially, that simply can't happen.

So the proposed solution is the optimal one.

[muzines]

Understandable - thanks Ian.

How are important/saved threads immune to auto-pruning in the future?

[Hugh Robjohns]

The UIA forum will be the only forum excluded from the pruning regimen.

[Forum Admin]

I think desmond is referring to the ongoing auto-pruning of sub-forums after the initial prune has occurred.

Valid question. Each forum will have settings for its Topics and only Topics that have not been posted to in NNNN days would be candidates. And we still have to decide on each setting for the post-purge era. Unfortunately, we can't use a parameter like Days since last viewed, because there are so many search engine bots coming to the SOS Forum and crawling old (and new) posts on a daily basis.

[muzines]

Sorry - yes, Ian is correct - to be more clear, let's say there was a thread from 2013 that contained content that some felt was important to be preserved. At the moment, we'd alert a mod and say "don't delete this" as part of the general prune.

But later on, auto-prune would be deleting old threads without mod intervention. So even though people had informed mods that the thread was important and shouldn't be deleted, once it fitted the auto-prune criteria in regards to date range / last posted etc (whatever that ends up being), then it will presumably be deleted automatically.

I wondered if there was going to be some mechanic or flag on threads that could be used so auto-prune wouldn't automatically delete some old but gold content...

[Hugh Robjohns]

I think the plan is to ~move~ any threads or individual posts that have been deemed valuable across into the Useful Information Archive.

As I understand it, the alternative way to prevent auto-deletion is to mark posts as 'Stickies', but as they then inherently appear at the top of the first page in each forum it wouldn't take that many saved threads before there'd be nothing new on the first page, which is clearly not a good thing!

[Arpangel]

No problems here, anything to keep the forum going.

[muzines]

I think the plan is to ~move~ any threads or individual posts that have been deemed valuable across into the Useful Information Archive.

Ok, so they'd come out of their old sub-forum home to the new archive place. Would external links pointing directly to those threads continue to work, or would they break?

As I understand it, the alternative way to prevent auto-deletion is to mark posts as 'Stickies', but as they then inherently appear at the top of the first page in each forum it wouldn't take that many saved threads before there'd be nothing new on the first page, which is clearly not a good thing!

Yes, I don't think that's a workable option, really...

[Forum Admin]

Ok, so they'd come out of their old sub-forum home to the new archive place. Would external links pointing directly to those threads continue to work, or would they break?

I'm pretty certain phpBB forum software creates an automatic redirect to its new home in the UIA forum, and then search engines would re-index it.

I'll do some tests over the weekend to confirm or deny this.

[Forum Admin]

OK, my test is 100% conclusive...

Original URL in SOS Support Forum:

https://www.soundonsound.com/forum/view ... 30&t=69575

Topic moved to UIA forum (using Quick Mod Tools at the foot of the Topic) where the URL becomes:

https://www.soundonsound.com/forum/view ... 30&t=69575

As you can see, they are identical URLs. To convince myself I closed my browser (Safari), opened Chrome and pasted in the original URL. Naturally, it took me directly to the Topic and its three posts, all of which now reside in the Useful Information Archive forum.

Result.

[muzines]

Great!

[ConcertinaChap]

Does that mean our post counts will all be reduced and we'll all lose our Jedi status and be busted back to the ranks?



CC

[MOF]

Everything I’ve written on this forum is pure gold and should be preserved for the nation.
Does this technical change affect my chance of ever becoming a Jedi poster?

[Hugh Robjohns]

Probably... but possibly not...

[MOF]

Probably... but possibly not...

[Forum Admin]

Does that mean our post counts will all be reduced and we'll all lose our Jedi status and be busted back to the ranks?

We're working on that... but it's easy to change the threshold for each status level in the back office to compensate... don't worry too much at this stage, please.

[ConcertinaChap]

CC

[Forum Admin]

Does this technical change affect my chance of ever becoming a Jedi poster?

It would be possible to create a special MOF-Jedi status for you alone, for an attained count between 807 and 807 posts, then set up user-pruning only on your account so that it never goes up but you never lose MOF-Jedi status?

[I am of course jesting.]