Silk Road forums
Discussion => Silk Road discussion => Topic started by: dmtnexus on October 04, 2013, 10:23 pm
-
Let's make a list.
1. dmtnexus
2. datalore
3. StExo
4. snailgod
5. peaceloveharmony
Who else?
StExo's backup: https://anonfiles.com/file/2cb4eefb1492f43b9510a6186bc66259
.onion mirror: http://2bewt43pu6ymqutq.onion/
-
anyone who has a complete backup let us know and we will put it on our own onion site as well for future traders to learn all these great things !
please let us know!
ponderingthoughts@Safe-mail.net
-
How do you do that?
-
How do you do that?
The two methods I'm using are:
torify httrack http://dkn255hz262ypmii.onion/index.php -W -O "/home/sapient/websites/sr_sec" -%v -r4 -c5 -C -i
---and/or---
torify wget -mk http://dkn255hz262ypmii.onion/index.php
I'm not really sure if one is better than the other, although the httrack command I'm using running 5 parallel connections while wget only fetches one page at a time. I'm pretty sure you could also use curl (lol).
-
I hope someone is pulling the vendor only forum. There is probably some valuable information on there that we should preserve.I wasn't a vendor or I would do it myself.
-
We shouldn't really encourage more people to put the strain on the server. Remember yesterday, errors popping up all the time? Those error pages will make their way into our backups should the same thing happen again. I will publish mine once it's done, which by my calculations will be in about 9 days.
Edit: I'm making my backup whilst logged in, so the vendor roundtable will be included.
-
I'm using 16 simultaneous connections and still the ETA is 9 days. I must be doing something wrong, LOL.
-
Does anyone have a full backup yet? Mine is at 25%.
-
If someone could let me know too of any backups once completed..
Also why are backups being created, are the forums going down ?
-
http://dkn255hz262ypmii.onion/index.php?action=stats
Total topics: 84488
Actually, I never realised that the reason every topic ID doesn't correspond to an actual topic is because of spammers. The forum software is at fault because you only have to enter the captcha once and you can spam to your heart's content.
-
I'm downloading it but it's taking its time. Apparently wget doesn't support multiple concurrent connections. I'm at 2.3 gigs right now or over 51,000 files.
-
I'm at 2.4GB. 160 simultaneous connections.
Are you running wget -m? I approximated it would take me at least 25 days, so that plan was scrapped.
-
can anyone point me to a backup of the silkroad site for the texts of the listings?
I remember a site that was sorting by price before sr could, that was basically a archive of sr lisitngs in text
-
I'm at 2.4GB. 160 simultaneous connections.
Are you running wget -m? I approximated it would take me at least 25 days, so that plan was scrapped.
I originally went with -m, but after a couple of days, I decided to restart it with just -rk as I figured five levels were enough. Now I'm thinking it won't make much of a difference. I don't see an option for multiple connections though. And how do you know how far you have to go?
-
But whenever somebody manages to get a decent recursive crawl of the forums together, posting a copy on Freenet is the way to go.
Before somebody posts the archive somewhere redistributable, they should make sure:
1. That they weren't logged in when they grabbed it (so their username isn't at the head of every HTML file). Shouldn't be an issue with wget.
2. It might not be a bad idea to do a 'touch' on each file to get rid of the specific timestamps of each HTML download in the archive before compressing and archiving it. I think something like 'find ./my_archive -exec touch {} \;' should do it.
3. And a 'chown -R root ./my_archive' wouldn't be a bad idea either, just to strip UIDs of file owners if you're tar'ing it.
I manually grabbed the threads from Security that I didn't want to lose, but would love the whole archive.
-
Phase I is complete: I've downloaded 218879 topic IDs (5.8 GB). Phase II will be to download the remaining pages for topics that contain more than 50 posts. Working on it now!
-
Please keep this forum updated and let us know how we can download a backup copy of the forums!
Many thanks for your hard work.
-
Hello - I am on the scrounge.
Do you StExo or you DMTNexus have any desire to make those archives available for download? I have downloaded several threads, mostly from the security forum that I found interesting but would love to have a more inclusive copy.
regards
This is not SOCA
-
After considerable effort (I'm not a coder), I've whipped up a bash script that's at this very moment generating a list of the remaining URLs to be downloaded. Phase II will begin soon.
Yeah, of course our backups will be available for download; that's the whole point.
-
Again if somebody can provide me with an easier way of providing this file for people to download, I will happily distribute it! But uploading 300mb over Tor is an extremely slow process.
I've been thinking about it, and I don't have any good ideas other than uploading it over Tor or to Freenet. Anything else is going to provide a trail back to the last guy on SRF who said he was gonna upload it. i.e. you.
heh, actually, at this point, the *first* person to upload a tarball of SRF to a clearnet site (unless it was available elsewhere first) is gonna look like StExo to anybody watching these forums. :)
I say u/l via Tor from a dedicated server somewhere that has a Tor instance running and good bandwidth. I think that's your best bet. It's a huge pain in the ass, I know. But I really hope you find a way to do it, because I have this feeling that in a few months, I'll wish I had a thread about X that I remember, and it won't be one I thought to grab a copy of.
-
Phase II has begun. 13111 URLs left to download.
Personally, I was thinking of creating an .onion mirror. Who needs the entire forum on their hard drive anyway?
-
well.. i'd like to read "drug safety" on my tablet offline, 'cause i don't have adsl at home, just mobile 3g, and can't use tor on tablet.
so i could download anywhere then reading on my bed without wasting my few MB/day
(tnx for the machine) :P
-
THANK YOU ! i'm going to my mom just to download it :)
-
Here we are guys, FINALLY got it uploaded:
https://anonfiles.com/file/2cb4eefb1492f43b9510a6186bc66259
Take a copy and pass it along peeps :)
Thanks! Will keep it safe for future generations.
-
Received. Cheers.
-
I also started started downloading the complete forum excluding the user profile pages yesterday.
As i rented a server for downloading the forum, i will probably set up an hidden service on this server with the cleaned up forum backup running on it. Unfortunately the download is quite slow so this will take some time.
-
So this is the final copy? Is anything else going to be added to this or should I just download it now?
-
got many errors extracting with WinRAR.. is it normal?
-
StExo,
Again - Thank you for selflessly backing the forums up and making them available.
good luck for the future
-
Here we are guys, FINALLY got it uploaded:
https://anonfiles.com/file/2cb4eefb1492f43b9510a6186bc66259
Take a copy and pass it along peeps :)
Many thanks, brother!
+1
-
Again if somebody can provide me with an easier way of providing this file for people to download, I will happily distribute it! But uploading 300mb over Tor is an extremely slow process.
Would this not be great use of usenet?? Encrypt the file with a easy password and post on usenet with innocent subject.
-
StExo - thanks so much for posting that! I know it was probably a huge pain in the ass, but you've done everyone a great service by posting it.