Original Bottlehead Forum Archive

[Wardsweb]

Rod if you wouldn't mind answering a couple questions on the database. What size is it? Can you tar or gzip it and place that file where someone could pull it off your server? Maybe a better question would be, what is the least amount of work for you?

[Len]

I don't know in what database format the archive exists or can be translated into, nor do I know the size. Additionally, I work 12 to 13 hours a day in my own business and don't have a lot of time.

All that said, I am an Oracle master and have some experience with systems integration. If I knew what I was looking at and who else would want to pitch in, I could give some indication of how much work I'd be willing to donate. I would need to have a working copy of the database engine.

There are a couple of ways I see to do this. One would be to convert the whole thing into text or binary files of complete threads and use an individual's own (OS) computer's search function for simple searches. Another would be to provide a run time version of a database engine on DVD set that includes the complete archive and use its search functions. Doc and Rod would need to work that out, since there would probably need to be some type of fee to the user for such a set. Again, I don't know the size, so I don't know what's feasible.

Another way to do it would be to do a translation (or not, depending on the software available) and set up a totally different Bottlehead server with just the archive. If user identification is a problem, we might be able to just keep the old names without linking them to present users.

Of course if Rod and Doc agree to keep it up at a very reasonable fee on AA, Doc would need to decide what he feels is fair to Rod, who offered it for free. Once we know what that is, we can determine how many of us are willing to chip in, and thus determine if we'd like to go ahead with it that way. This last way, of course, would be cleanest.

The most thorough, elegant and pain in the ass way to do it would be to get it put up on a separate Bottlehead server and have users tag posts as they find them to indicate usefulness to a topic. I am not up to that task.

The threads are redundant, but often show a learning curve IMO. Cherry picking, to me, would be the same as just maintaining the new forum alone. Percentages concerning search topics is not a valid criterion in my view. What seems more important is that it is all available for search. A case in point is that I recently decided to attempt going all DHT in my Excite, and I found Doc's post regarding better op points than was originally designed. Who else is interested in that? It wouldn't even show up on a histogram.

Just my 2 cents, though you know I am shy when it comes to stating my opinion...

[Len]

Rod if you wouldn't mind answering a couple questions on the database. What size is it? Can you tar or gzip it and place that file where someone could pull it off your server? Maybe a better question would be, what is the least amount of work for you?

Hey Wardsweb,

Rod may or may not be a regular here, and he may or may not wish to get involved with every non-customer just to do us a favor.

Maybe we should just start from the beginning. We already have a baseline of people who are willing to pitch in financially, and maybe more will join in if it looks doable. I think we need to ask Doc how much he feels would be fair to pay Rod annually just to leave it up, and then see if we can raise it. That might not be wise to do publicly in its first stages. Do you think you could deal with that, being the middleman on this? My personality is kinda rough.

I think it's really nice of Rod to offer. Just to clarify (repeat) something I or someone else may have mentioned before: There's something in it for Rod when it comes to leaving the archive running. It's certain to pull more traffic, and the long search times keep the traffic there. This in turn should help pull in more advertising dollars, maybe more sales for those left hosting in the asylum, and therefore more value for his product. So though there is certainly a large favor component present in leaving it up, there is value to Rod as well.

Which brings up another long shot. I don't have anything to advertise. Maybe others here do, and maybe advertising dollars could go towards the cost of keeping it running.

Does anybody else have any ideas?

[Rod M]

No problem, Dan. I should give you a call and explain the options. I'm assuming you still want your logo on the site ;)

BTW: The Perl script that I just wrote (actually updated from an old one that was updated from an older one that was originally written to populate a db), is surely available to you or one of the guys that would know what to do with it. They'd have to figure out the db schema on your end and message formats changed over time, so there could be a little tweaking required for older posts.

You could also just add some sticky posts that point to particular posts there. And add a search box to search old archives on your site. That's simple. But really, keeping the old archive is not any real work as there is no maintanance. It's easy than taking it down.

[Doc B.]

Thanks Rod,

We're in the midst of the flu bug here and I just threw by $##@!$! back out (which might account for my seeming grouchy this week - apologies to all), so it's probably best if we talk when I'm recovered. Brain is more of a sieve than usual...

[Rod M]

Rod if you wouldn't mind answering a couple questions on the database. What size is it? Can you tar or gzip it and place that file where someone could pull it off your server? Maybe a better question would be, what is the least amount of work for you?

The whole db? 25GB, give or take.

The size on disk is 874MB and 163,814 files with all the thread and forum indexes.

I could gzip it if I knew the command to gzip a directory structure.

The mysql files for Bottlehead are 154MD, an 87MB data file plus a search index.

But that file is just for searching and doesn't have all the thread data. We use a transaction file for all the forum data and then make a smaller forum file for searches. Extracting just the Bottlehead info is possible but would take some effort.

To me, it would seem like using the parsing script that pulls the data from messages might be easier, at least for me. That script then puts the data elements into a variety of tables which is the bigger problem which is dealing with the new schema and tranlating the fields to fool the system into adding the posts.

[Wardsweb]

Thanks for the info. I will wait for you and Dan to talk and see what you come up with. If needed, we can readdress at that time.

[slomatt]

I could gzip it if I knew the command to gzip a directory structure.

Here's the command to gzip a directory structure in case it's useful.

    tar -cvzf <output file> <directory>

So, for example if your directory is named "forum" you could do:

    tar -cvzf forum.tar.gz forum

- Matt