Rachel Nabors ([info]crowhen) wrote,
@ 2009-04-06 22:37:00
Previous Entry  Add to memories!  Tell a Friend  Next Entry
Rubifruit may have to close.
Edit (or, before you use the word "bandwidth")
I've said this before, but the bandwidth is not a problem. It's the RAM usage and resources that cost money. Every time we display a banner hosted via Rubifruit, our server has to make a calculation. That calculation takes up processing power, usually in the form of RAM.

Bandwidth is how much information you can transfer from the host across the Internet, like images, files, etc.

But in this case we're talking about core processes that happen before the images ever even hit your screen, before they leave the server. And RAM isn't cheap. Most personal sites won't have to use much of it, but a calculation-heavy site like Rubifruit gobbles it up in order to serve your banners.

So don't even mention bandwidth in your reply, or you will make me very sad and grumpy. Your reply will be of no help and will only give the impression that Rubifruit is receiving sound advice and support, which she is not. I appreciate your effort and understand that you wish to help, though.

End edit.

Rubifruit is using so much RAM on its server from hosting and processing all them impressions and clicks that I'm allocating something like 600 megabytes of RAM to it on the server and it is still popping out a 503 error. That costs me $60 every month. I'd love to bump it up to a number that will stop the errors, but I can't afford it. Yesterday I released the pay-for-clicks system I proposed, but so far no one is using it.

I don't know what to do. It's such a good exchange and so many people have benefited from it. But I just can't afford it these days. If I can't find a less expensive host by the end of the month, I'll have to seriously consider letting it go.

BUT I DON'T WANT TO!

(22 comments) - (Post a new comment)


[info]grimepoch
2009-04-07 03:27 am UTC (link)
Wow, $60 is quite a bit. Have you tried like DreamHost? I've been using them for about 5 years now and it's very inexpensive for how much bandwidth you get. My site was down for one day in five years. I know others experience has been different, but at least this would be an option of not having to let it go.


(Reply to this) (Thread)

[info]crowhen
2009-04-07 03:52 am UTC (link)
I am in fact with Dreamhost.

I've said this before, but the bandwidth is not a problem. It's the RAM usage and resources that cost money. Every time we display a banner hosted via Rubifruit, our server has to make a calculation. That calculation takes up processing power, usually in the form of RAM.

Bandwidth is how much information you can transfer from the host across the Internet, like images, files, etc. But we're talking about core processes that happen before the images ever even hit your screen, before they leave the server. And RAM isn't cheap. Most personal sites won't have to use much of it, but a calculation-heavy site like Rubifruit apparently gorges on it.

I currently work for a company that hosts many calculation-intensive processes, and they face many of the same difficulties I am encountering now.

(Reply to this) (Parent)

[info]grimepoch
2009-04-07 04:35 am UTC (link)
What kind of calculations are you doing? In fact, every file, image, text, etc goes through the processor on its way from the hard drive to the network connection. This is why many websites, like eBay and the like, put a lot of the information into memory (cache) so the images stay persistent and don't require access to the hard drive which DOES block CPU performance and degrade the operation of a machine (IO blocking).

So I say, what exactly are you doing at rubifruit that is so calculation intensive? Are you keeping a record of how many times a banner is hit? dereferencing a banner click to a re-direct to another website? Are you taking a hit on database access? What kind of CGI infrastructure are you using? Perl, Python, and are you getting bottlenecks for the performance of the CGI? For instance, we switched from Perl to modPerl on a site and had a MASSIVE improvement in performance. Even database access when your database gets bigger or you are getting lots of hits can really degrade performance.

The other thing is this, if your CGI code loads for one user and required 200k to process an images, or load a database, if 10 people are doing this simultaneously, thats 2Meg. It's a function of how long your code has to run for, and what the peak is of people using the code. First thing I would look at is figuring out if there is any way to speed up or enhance the CGI portion of your site first. Simple things like a lint checker or profiler can really find bottlenecks in your code. And worse, for databases, when they need to be locked, when you have lots of users this can be HORRIBLE.

(Reply to this) (Thread)

[info]crowhen
2009-04-07 02:22 pm UTC (link)
We're PHP-based, not CGI-based. The code keeps track of every impression and click on every banner. Beyond that, it's beyond me. I'm not a programmer. I use Easy Banner Pro, though, I can tell you that.

I've spoken several time with the host and the person who made the script, and we've modified it a few times to improve performance. But then there are 20 people reading through the Penny and Aggie archives and another 40 over at Yume, indeed it does add up fast!

(Reply to this) (Parent)(Thread)

[info]grimepoch
2009-04-07 04:23 pm UTC (link)
My understanding of PHP is that it depends if the interpreter is compiled into the web server (I am assuming Apache) or not. CGI just indicates when the webserver calls a program to generate output, and you can in fact call PHP, Perl, native code, anything you want! (provided the webserver allows it to run). So you may or may not be running all inside of apache. I am also not aware which one has better performance or even if there is an option to run either with DreamHost, but I am sure there is a lot of discussion about this on the web.

But that aside, another huge problem is if each time the php script is called, it has to lock and open a file to store statistics about the operation. If 40 people need to access that file, each one is going to have to wait while the others are waiting, and each of those has to open the file, read the contents, change the contents, write out the file. While that is expensive enough as it is, again, the whole process blocks.

One thing you could try doing, if it is free (your site), and you are not wanting to get rid of it, you could try and not keep statistics of impressions for awhile and see how the performance is.

Unfortunately, since you are probably selecting a random banner all the time, you can NEVER get the advantage of caching on the server OR the client web browser.

The other thing I would look into, and I don't know the code, but to make sure the image doesn't actually process through your script itself, but instead, the randomness generates HTML code that points to the image with a unique name so that if the same banner gets called on a computer more than once, it will be cached locally and you wont take additional hits. That of course is probably a question for the code maintainer :)

(Reply to this) (Parent)(Thread)

[info]crowhen
2009-04-07 04:36 pm UTC (link)
Yes, we're using Apache, I'm pretty sure that's what Dreamhost uses.

Well, we kind of have to keep the statistics of the impressions. That's how we know how many impressions to hand out. See, users only get impressions if the banners on their sites generate clicks. 1 click gives them 200 impressions. This encourages people to put banners where they will be seen as well as make their banners look interesting. So we have to keep track of every impression AND every click.

(Reply to this) (Parent)(Thread)

[info]grimepoch
2009-04-07 05:10 pm UTC (link)
That certainly makes sense with the need then to have that ability. Unfortunately, only the programmer is going to be able to tell you were the bottle necks are, and, trying to set-up a test environment to reproduce the problem might be a pain.

You're just too popular of a site! :)

(Reply to this) (Parent)

[info]joamette
2009-04-07 06:24 am UTC (link)
That would be horrible. o_o

I actually just bought some impressions. I hope it helps.

(Reply to this) (Thread)

[info]crowhen
2009-04-07 04:37 pm UTC (link)
Thanks, Joey :)

(Reply to this) (Parent)

[info]taiki
2009-04-07 08:35 am UTC (link)
argh. Today our Joomla based CMS at work is doing something wonky where MySQL takes forever to respond when going through Joomla's mysql API but just fine from raw php mysql transactions.

This makes page requests go very slow.

yet I keep getting emails, "Is the bandwidth fine to the webserver? Are you guys dropping packets?"

Even worse, i work for an ISP's techsupport department, the group that should be vaguely aware of how to troubleshoot speed issues.

Can I cry now?

(Reply to this)


[info]kimonos_house
2009-04-07 04:56 pm UTC (link)
I intended to buy some impressions, but I haven't transferred any Project Wonderful earnings to my Paypal account yet. My Paypal is completely empty.

I'm wondering you might end up having to change the Rubifruit model altogether. You might just have to settle for a purely random banner exchange for the moment, something that just serves up banners instead of tracking clicks and impressions. I know it's not exciting, and it does nothing to reward those who place banners where they get clicks. But it would keep the exchange alive until you're able to work out issues with RAM usage and money.

(Reply to this) (Thread)

[info]crowhen
2009-04-07 05:06 pm UTC (link)
I don't think it has a purely random setting. The closest I could get would be a 2:1 exchange and that would mean that sites like panney and aggie would be perpetually advertised throughout the network while kicking back very little traffic due to the nature of their readers' browsing habits.

(Reply to this) (Parent)(Thread)

[info]kimonos_house
2009-04-07 06:08 pm UTC (link)
Maybe switch to another code for the time being?

(Reply to this) (Parent)

[info]kgammons
2009-04-07 08:40 pm UTC (link)
{htdiwdnab}

(Reply to this) (Thread)

[info]crowhen
2009-04-08 12:45 am UTC (link)
??

(Reply to this) (Parent)(Thread)

[info]kimonos_house
2009-04-08 01:09 am UTC (link)
Bandwidth fail.

(Reply to this) (Parent)

[info]goodkingnerdnor
2009-04-07 09:07 pm UTC (link)
What kind of memory are you using that isn't cheap? The vast majority of memory in use today can be purchased at $50 for 4gb, often even less. Do you absolutely have to have ECC memory? If you're using standard DDR2 memory, you can probably get donations that would cover as much as 16gb pretty easily. Heck, even a 4gb single stick of ECC DDR2 is only $100 from Crucial, one of the more reputable memory makers.

Also, not to be nitpicky or anything, but bandwidth isn't solely an internet term. The amount of data you can transfer from your memory to your processor and back in a certain amount of time *is* bandwidth, it's just the bandwidth of your memory controller, not the bandwidth of your cable line or NIC.

(Reply to this) (Thread)

[info]goodkingnerdnor
2009-04-07 09:09 pm UTC (link)
I suppose an equally relevant question is, "Do you own the hardware the site is stored on?" If the answer to that is "no", I can see why you might have memory problems. If you don't own the physical server itself, it may behoove you to make the long term investment in purchasing the server hardware and just leasing the rack space and bandwidth. I understand these may not be instantaneously economically feasible, but just something to think about for the future.

(Reply to this) (Parent)(Thread)

[info]crowhen
2009-04-08 12:43 am UTC (link)
The answer is in fact no. Elsewhere I have stated that we're on Dreamhost.

I can't think about leasing rackspace unless the exchange can earn something of its keep. Alas.

(Reply to this) (Parent)(Thread)

[info]goodkingnerdnor
2009-04-08 01:49 am UTC (link)
I know you stated you were on Dreamhost, but I didn't know if you were renting storage on their server or using them solely as a bandwidth provider (most hosting companies offer both services). Depending on who you rent the rack space from, it can be significantly less expensive than renting hosting space (discounting the initial outlay for the hardware which would be recouped over time). It can also be more burdensome if your server is taxed enough that you have to do hardware swaps frequently. I'm sure you've already thought of all of this though. My apologies for the annoyance.

(Reply to this) (Parent)(Thread)

[info]crowhen
2009-04-08 02:20 am UTC (link)
That's interesting information. I use Dreamhost to host and to provide bandwidth. Keep in mind I know nothing about upkeeping a server, or I would probably rather do it myself, heh. Alas, so much is involved with serving your own sites, it's just beyond me to do it myself. It's a whole 'nother field.

(Reply to this) (Parent)
Alternate idea.
[info]kahm007
2009-04-17 04:50 pm UTC (link)
I know this is on a serious budget here so don't shoot me but about the only way to get around this is possibly Co-locate a machine. This is by no means Cost effective but I figured that if we were throwing ideas out there it would take care of the provider having to crank up the power on their end. Of course then you have to figure out how much CPU and Ram you are using and expect it to grow in next 36 months an project for that and so on and so forth.

The feeling I am getting is that the system you are hosted on is a Virtual Server which is why or how they can crank up the CPU / memory as necessary.

(Reply to this)

(22 comments) - (Post a new comment)

Create an Account
Forgot your login or password?
Login w/ OpenID
English • Español • Deutsch • Русский…