Usually, out of memory errors (and other errors that are similar) is due to your server basically running out of memory also known as RAM. The cause though, is a bit more difficult to determine. Note that every Minecraft package we sell has a set amount of RAM. The use of plugins will use up this RAM in addition to each user in your server taking up a chunk of that RAM themselves. Thus, the more plugins you have, the fewer players you can have online at the same time - so you will need to lower the max player slots in this case. Using multi-world plugins is notorious for causing out of memory crashes because each world you add uses up quite a lot of memory even if no players are on that world.
Other causes of out of memory errors can be due to out of date plugins or incompatible plugins with the version of CraftBukkit your server uses. It is always best to ensure that everything on your server is up to date to reduce the chances of this occurring (note that a 100% stable server is nearly impossible to have due to the nature of Minecraft/CraftBukkit servers and plugins). It can also be caused by things in your world such as mobs, lava, water, redstone, pistons, entities etc, as these can cause memory leaks and have to be removed manually.