Silk Road forums
Support => Feature requests => Topic started by: Tessellated on December 29, 2012, 07:56 pm
-
Any chance of a simple API? I would love to automate my vending but your captcha makes it impractical.
I would like the following features:
- Download orders
- Change status of orders
That is it really. I can however appreciate that working on security and uptime is a higher priority.
-
That damn spam bot has pushed everything out of this section so I am bumping this. I don't normally bump, but in this case it is just spam pushing it down the list.
A simply API for handling orders is crucial for any business that wants to scale. If we secure the supply we want then we can set prices that we would be dealing with hundreds of orders per day, this needs to be automated.
If there was even just a way for a program to bypass the captcha with some sort of key or alternate login portal then the rest could be done with screen scraping.
Something as simple as an XML version of the orders page would be 80% of what we need.
-
I've actually been playing around by writing a library that scrapes the site to provide an API; originally I only wanted an accessible API as an interface like you're talking about, but my toying has turned it into a tray icon that monitors for new orders or messages and notifies you whenever any come in. It has a lot of really, really unpolished functionality, but nothing else is actually superior to a web browser... I mean it can send messages, navigate the site without proper HTML display (as in it shows you the text dumps), that kind of thing. But what you're asking for it doesn't provide an interface for, and unless somebody gives me the sanitized source for a vendor's orders page, I can't do anything about it either (I don't sell).
I'm actually finding it more convenient than I expected, since it effectively simulates push notification of messages and orders; and since it retries any network activity that failed (unlike the damn browser...), I'm actually getting a lot of use out of it. The site's stable all the time with it. Slow as fuck because it times out half the time during peak hours, but at least you don't have to even pay attention to it until it pops up and flashes at you...
But honestly I wouldn't say it's got much else going for it really. Just for my own fun, I'd like to finish the interface for order manipulation if you give me the source of your orders page (after being scrubbed clean of course). I mean seriously, if you don't mind sparing the time it'd be very helpful for me.
Frankly, writing this has made me notice just what a mess the site is... I mean I love SR as much as the next guy, but the HTML is so poorly organized it's virtually impossible to reliably scrape some of the data you see in the browser.
For example, this is a (sanitized) snippet of the HTML on the messages page:
<div id="content">
<div style="text-align: right"><form action="http://silkroadvb5piz3r.onion/messages/mass_delete/XXXXXX" method="post" accept-charset="utf-8"><div class="dropdown_wrapper" id="cat_dropdown_wrapper" style="display: inline-block; position: relative; top: 5px; margin-right: 3px"><select name="delete">
<option value="read">all read messages</option>
<option value="sent">all sent messages</option>
</select></div><input type="submit" name="submit" value="delete" class="submit" /></form></div><div class="container" style="float: left; margin-right: 20px"><span style="font-style: italic; font-weight: bold; color:black;">inbox</span><br /><a href="http://silkroadvb5piz3r.onion/messages/unread">unread</a><br /><a href="http://silkroadvb5piz3r.onion/messages/sent_folder">sent</a><br /><div class="hr"></div><a href="http://silkroadvb5piz3r.onion/messages/bulk_reply">bulk reply</a><br /></div><div style="display: inline-block"><table class="zebra"><tr><th>from</th><th>subject</th><th>freshness</th><th> </th></tr><tr>
<td style="white-space: nowrap"><a href="http://silkroadvb5piz3r.onion/messages/send_message/XXXXX">SomeUsername</a></td>
<td><span style="">1 minute</span></td>
<td style="white-space: nowrap">5 hours</td>
<td style="white-space: nowrap"><a href="http://silkroadvb5piz3r.onion/messages/message/XXXXXX">read</a> | <a href="http://silkroadvb5piz3r.onion/messages/delete_message/XXXXXXX">delete</a></td>
</tr><tr>
<td style="white-space: nowrap"><a href="http://silkroadvb5piz3r.onion/messages/send_message/XXXXXX">XXXXXX</a></td>
<td><span style="">subject</span></td>
<td style="white-space: nowrap">5 hours</td>
<td style="white-space: nowrap"><a href="http://silkroadvb5piz3r.onion/messages/message/XXXXXXX">read</a> | <a href="http://silkroadvb5piz3r.onion/messages/delete_message/XXXXXXX">delete</a></td>
</tr><tr>
<td style="white-space: nowrap"><a href="http://silkroadvb5piz3r.onion/messages/send_message/XXXXXX">XXXXXXX</a></td>
<td><span style="">subject</span></td>
<td style="white-space: nowrap">6 hours</td>
<td style="white-space: nowrap"><a href="http://silkroadvb5piz3r.onion/messages/message/XXXXXXX">read</a> | <a href="http://silkroadvb5piz3r.onion/messages/delete_message/XXXXXX">delete</a></td>
</tr><tr>
... that's after conversion of HTML whitespace for display (<b> to a newline, etc.). Notice how nothing's really labeled beyond being aligned into columns, but otherwise not at all organized and totally lacking what HyperTextMarkupLanguage does so well -- adding metadata to text, I mean. It's the flimsiest most fragile scraping you can have: if all of a sudden those table data elements got reversed, well... so much for scraping *that* page until it gets fixed. I really hope it's more organized before it's turned into HTML, at least...
-
Adding ids and classes would help screen scraping a lot, I agree. There are ways around it but if the structure of the page changes they break. If everything was labelled with ids and classes then they could re-arrange the page all they want it could still be found automatically.
How does your taskbar app get past the login capture? Do you login manually and then feed it the cookie?
Any code I create to make vending more automatic I will share open source once it is up to snuff.
-
Er, no -- since the idea is to bypass the silly browser and compensate for the instability of the whole Tor hidden service thing, you just login through a little popup window in the top-level window. It's setup like Pidgin at the moment -- click on the tray icon and it reappears, click it again and it vanishes again until there's activity to report.
When you close your browser, you don't actually log out. The site has no way of knowing you're done, so it keeps your authentication token around incase it sees you request another page. All my app does is not logout either -- except unlike a browser, I wrote in optional saving & loading of the session cookie to/from a file. It just uses the same session if you enable the option. I haven't logged in again for days.
-
So am I to take your statement of "whatever I write will be FOSS," to mean that you're not going to show me the source of the page so that you can be the first one to write it, lol? To be clear, I didn't mean to imply I'm hoarding anything for myself -- I don't see why anybody would trust a program that wasn't open source, for one thing, and for another if it ended up useful to people I don't really mind trusting them to toss a few bitcoins my way. I mean if somebody else wrote the thing and I used it all day long the way I have been the past few days, they'd get a couple from me.
Not really worried about that. I mean it's literally partially finished and has several half-done features right now, but if I misunderstood you & you meant you'd like a copy, that's fine. Just don't blame me for its lack of polish. I wasn't kidding though, I do this sort of thing for fun pretty often and want to finish the order API -- I wasn't joking about the source of the page.
-
So am I to take your statement of "whatever I write will be FOSS," to mean that you're not going to show me the source of the page so that you can be the first one to write it, lol?
ERROR: Does not compute!
I have read that sentence several times and I cannot figure out what you mean.
I just mean when I am finished writing software that will download orders and change the state of orders that I will publish the source code into the public domain. That way people use it, and they can read it so they don't have to trust it.
My tools will be all command line perl scripts, no GUI as we don't need one.
-
It was my way of politely saying, "I'm not going to spend $500 just for the amusement of tinkering further, and would appreciate it if you (or anybody) could clean the HTML of identities and give me a copy of it so I can write that in." I also wouldn't keep it to myself or anything.
I guess I explained myself really poorly: the only reason this program I mentioned functions is because it uses the API I initially started out to write. Anything else could do the same -- whether in a terminal or a GUI doesn't matter, it's just a matter of using a language that can call the methods or otherwise trigger the code. But since I can't even see what the page looks like in order to write the fake-API (which amounts to pretending to be a person with a browser clicking buttons), I guess that part won't be working for awhile longer...