admin
HBRstudios 2:36am, 20 June 2008
OK, I am looking to the not too distant future and seeing my storage space sucked down to nothing even with 2TB currently working.

What is the best way to make an "expandable storage" up and above things like the Buffalo RAID tower (which I believe max's out at 2TB) or similar system, and has no way of adding more, to my knowlege.

Ideally, it would be a system that I could always add more drives (presumably 2 at a time, one for the backup) without the need for additional ethernet/firewire/usb ports in the CPU.

Can external drives be "chained" without any issues and at a speed rivaling an internal drive? If so, what brands are made to do this and come in the 1TB size?

Any ideas, all you geeks out there?
admin
Hambone Lewinski 10 years ago
Brian, what are you using 2 TB of space for? Please don't say that it's anything you care about like all your photos because hard drives are volatile little creatures. It becomes pretty likely for one of them to eventually fail when you have a bunch of them in one machine, wiping out all your data if they're in a non-mirrored, non-parity.

Our big deep dish boxes at work usually only have 6 disks in them and then we add a rack of disks if we want more space.

What I'm saying is, at the same time as you do this, you might want to start looking at a good tape-based backup system.

As for RAID, many motherboards have SATA RAID controllers built-in, but I don't think they can do RAID with parity -- i.e. RAID 5. For that you may have to go with (expensive) SCSI disks.

Jason
NorthernMinnesotaPhoto 10 years ago
There is something called a "drobo"???. I got the name off a podcast that I listen to. It is a device that does what your asking about. I'll check my podcasts, and get back to you.
admin
K. Praslowicz 10 years ago
Skip backups. Its fun to be forced to start fresh every so often. ;)
admin
It isn't fun if you are making money off of your images.
NorthernMinnesotaPhoto 10 years ago
Check out www.drobo.com personally we burn all our portraits to cdr disks by event.
admin
HBRstudios Posted 10 years ago. Edited by HBRstudios (admin) 10 years ago
Thanx, guys!

1TB is for storage, and another for backup. Tape drives are great for backup, but that does not solve my impending storage issue.

I will check out drobo and keep looking..............

And, yes, Jason, I have considered tape drives for backup, but as of now, my backup is strictly external HD.
admin
HBRstudios 10 years ago
OK, so I have indeed seen the Drobo advertised and did look at it before some time ago (and forgot about it) and didn't think it was the way to go because of its "redundancy" features -- essentially cutting your amount of space in half as it copies all information twice as it makes backups of the files spread across the usable drives. Yes, this is what happens, anyway, when you are backing up your info, but I don't see myself having this as the storage device and the back up all in one.

Wait, can that feature be turned off??? Have to check.

I like the idea of just being able to pop in a new SATA drive whenever needed, however, --- can RAID arrays be set up to be handled this way? If so, what kind of hardware would be needed?
admin
Brian, how about something like this? www.amazon.com/LaCie-301248U-Biggest-Quadra-Drive/dp/B000...
admin
Hambone Lewinski Posted 10 years ago. Edited by Hambone Lewinski (admin) 10 years ago
"didn't think it was the way to go because of its "redundancy" features"

That's what RAID is -- a "Redundant Array Of Independent Disks". The idea is, the more disks that you have, the more likely it is to for one to fail and cause data loss. If you arrange data in a clever way across multiple disks, you can make it so that if any one hard drive fails, the data can be recovered from another disk.

RAID comes in a few flavors. There's strict mirroring; that's where you have two copies of each logical disk. Then there is striping, which splits single blocks of data across multiple disks. Finally there is parity -- which uses algebraic error correction codes to make data on the lost disk recoverable.

Most modern RAID implementations using striping with a parity check -- RAID 5. I'm not mistaken, but I believe that if you have a 3 disk RAID 5, you will get 2 disks worth of space on it.

Jason

EDIT: I should mention, another big reason why RAID is popular is that it increases drive throughput. If you have three drives each of which have a copy of a file that you want, you can access the file faster because you are reducing seek time (whichever drive gets there first, gets there first) and if one drive wasn't enough to fill the data channel, three will be.

Jason
admin
HBRstudios 10 years ago
There is a 4TB equivalent of this as well that I have seen, but it is essentially a DROBO with the redundancy storage --- it does have a plus with firewire connections, however.

Here is the ideal situation: A "rack" of internal HD's that I can add 2 at a time and set one of those 2 to be a mirror as a backup. Much like the LaCie and Drobo, but without the redundancy storage and having it indefinitely expandable (or nearly, just adding another drive letter to the lot M:, N:, O: P:, etc.) That way, whenever one is full (and the backup to it), I can plug in another to the system without filling up all of my USB and firewire ports in the process, or having USB or firewire hubs chained.

Or would that be somewhat of a solution is a big-ass firewire or USB hub with lots of ports to add externals whenever I need???? I would think that there would be issues with this kind of a set up so I didn't really consider it.
admin
HBRstudios 10 years ago
Jason -- OK, if I were to go to a RAID system of drives, what kind of hardware is needed that can it be expandable beyond the 4 drives of the DROBO or similar units? What kind of interface to my CPU will it need?

Thanx to all and keep the info coming because, as you have figured out, I am no expert on how these systems work -- that is one of the reasons why I am starting this discussion.
admin
HBRstudios Posted 10 years ago. Edited by HBRstudios (admin) 10 years ago
OK, so please take a look at this and tell me if I am understanding this correctly:

www.pcconnection.com/IPA/Shop/Product/Detail.htm?sku=8075613

1) There will essentially be 2TB of useable space
2) There will not be the need to back up the information due to the redundancy storage (if set up that way)
3) I could purchase more of these arrays when needed to plug into the other 3 ports of the PCI-X card.

Is this about right?
admin
Hambone Lewinski 10 years ago
Brian,

There are costs and benefits to going with an internal vs external enclosure. An external enclosure will be more expensive, external, but probably more pluggable and easy for you to configure.

If you configure the disks internally, you'll have to buy a card and disks separately and build a RAID volume manually. It might not get along with your current machine -- you'd have to be sure that the RAID card works with your computer, be sure that it dissipates enough heat for four new hard drives, etc.

So an internal solution would require more fiddling to achieve RAID 5, but would only be a few bucks more than buying an additional hard disk controller (which you would need to add four disks anyway) and four big disks.

FYI, with RAID 5, if you have n disks, 1/n of the space is parity data. So, for example, if you have 4 disks, 25% of the space is lost to parity data and the rest is available for storage.

Jason
admin
Hambone Lewinski Posted 10 years ago. Edited by Hambone Lewinski (admin) 10 years ago
...That would work, but it would cost you a shit-ton of money.

Jason

EDIT: At least it would to me. Of course, no matter what it you do, it will be expensive to buy four 1 TB disks. At least $700.
admin
HBRstudios Posted 10 years ago. Edited by HBRstudios (admin) 10 years ago
Well, it will have to be external since I don't have the space needed to keep adding drives, and, yes, it is an expensive venture, but one that will be necessary in the not so distant future.

I do need a bit of clarification, however:
1) Can new arrays be added to the PCI-X card whenever needed?
2) Is there a reason why you would need to backup the RAID array?
3) What kind of configuration issues would I have with the MicroNet system linked above?
admin
Hambone Lewinski 10 years ago
HBR,

Ideally, you want to select a disk array that's large enough to accommodate your needs -- you want to avoid using multiple arrays.

It looks like this device that you've picked out does pretty much the same as the drobo, but with an eSATA interface (much faster) and without fancy branding.

Looks like your device can do RAID 5, so you'll get closer to 2.7 TB of space. This is significant -- .7 TB is 700 GB. That's more than the amount of non-volatile disk space that I have in all my computers combined.

It's a NewEgg, too. www.newegg.com/Product/Product.aspx?Item=N82E16822204074

There are a few similar options (some of which have reviews) at NewEgg, too: www.newegg.com/Product/ProductList.aspx?Submit=ENE&N=...

Jason
admin
duluthiscool 10 years ago
HBR

1) RAID is a HIGH AVAILABILITY system. it is NOT a backup solution. (yes yes... you can have a raided backup solution... but that's not my point)

2) We can discuss the merits of hardware over brew tomorrow night, Luce', 10PM. Jerree Small is playing in the bar. Good good good!!

What is RAID? Basically, depending upon how it is configured, you can lose one or more RAID members, and have the storage device continue to operate.

So... to answer your general questions above... in an equally general fashion...

Can new arrays be added to the pci-x card whenever needed. In general, NO. There will always be physical limit imposed by the underlying hardware. Also in general, maybe one or two more raid sets... again... depending upon the underlying hardware.

Is there a reason to backup the RAID? As noted... RAID is high-availability, NOT disaster proof. If you have a two disk RAID 1 mirror, and you lose both disks... (theft, fire, massive hardware failure due to electrical spike) (or any other RAID configuration for that matter)... you are left with nothing. Actually... maybe even LESS THAN NOTHING... as recovering a failed hardware RAID set is much much more difficult than recovering data from a failed single drive. Short answer? YES. Backup all of your data. Always. Twice. Once to a "hot-backup" such as inexpensive disk. The other to another off-site location.

MicroNet? I didn't follow the link... but will try to do so tonight.
admin
Shawn Thompson - Lake Superior Photographer Posted 10 years ago. Edited by Shawn Thompson - Lake Superior Photographer (admin) 10 years ago
I am sure you are looking for something different, but I have all of my images on my laptop HD, then I back them up to an external HD. I am also picking up another external HD that I will swap every week or two and take off site. This way I have 3 copies, 2 at home and one off site in the case of fire or theft.

With the prices of 500gb-1TB external HD's it is hard for me to buy into a huge, expensive system that would be hosed in the case of a fire.

Also, by using Microsoft's Briefcase, you can set it to automatically synch the changes when new files and folders are added or changed.

shutterbug.com/equipmentreviews/software_computers/0408ba...
admin
Hambone Lewinski 10 years ago
What Rich said. :-)

Jason
admin
HBRstudios 10 years ago
Thanx, guys

@Rich -- I might up for a brew tomorrow night, not sure as of now. Maybe we can talk then.

@ Shawn & Rich -- That is basically what I have been doing is backing up to externals, but now my internal HD space is full and files need to be copied and the drive needs to be wiped. So now I save all of my images on al larger external (500GB) which fills as well and the issue continues. I know I should be backing up off site. Just haven't done it yet.

Here is the kicker with this system: Lightroom is my organizing tool and when my internal HDs are full, the files need to be put on an external drive (unless I have 100 DVD's available and a whole lot of time to copy them.) This needs to be copied then Lightroom needs to be set to find the files at the new location (and you can't just point it toward the folder and let it find all of them, you have to do it folder by folder!) This is a very time consuming process so I am saving all of the images on 500GB external with a 1TB backup. Well, on the 2nd 500GB external, my 1TB back up is also full and I am in need of even more space which would normally involve pluggin in a new external.

Well, Lightroom doesn't seem to like that because it is trying to find the images on the other drive that was unplugged to fit the empty one. The next time I need to do this, I will give the new drive a new letter designation before using Lightroom at all to import and that may solve this issue, but it still is a lousy system.

Instead of getting rid of lightroom, I want a central location with enough space so I don't have to deal with this in the future (at least for a long time!)

If I can work out the 4TB with 2.7TB of "secure" space, that will last me a while, but it will fill up and the issue starts all over again. If I cannot add another one of these arrays at that point, I will have to unplug it, mark it with the dates, and plug in a new one making sure that Lightroom recognizes it as a completely different location, which I am not confident it will right now.

So there it is! Much more than you ever wanted to know!
admin
duluthiscool Posted 10 years ago. Edited by duluthiscool (admin) 10 years ago
HBR

Regarding your directory debacle... you will be interested in the following topic:

"How to create and use NTFS mounted drives in Windows XP". Read the article... I believe that this has very good potential for forcing your lightroom to behave.

(This technology is available on Windows 2000 and later... a similar type of technology is available on Mac OS X and later... perhaps earlier as well... but certainly OS X)

EDIT: Really like the specs on the Drobo WITH Network (DroboShare)... which essentially makes it a home network attached storage NAS. I like the mostly-idiot proof approach to disk provisioning... but *really* *really* like the NAS component... ESPECIALLY for BACKUPS!! Disks and related fans are the noisiest part of the computer and I hate the noise...

The Lacie is good too... hardware raid (so it seems... although this could all be smoke and mirror driven software raid...) mostly idiot proof... although it appears to lack the networking support, it supports USB2, and more importantly, Firewire 400/800... if you plug multiple USB2 devices in, and are read/writing to both... um... contention... equals slow. The Lacie will win here, every time.

Micronet... Hardware raid, but not in the box... must have hardware raid controller... which is another potential pain point... and another potential failure point... if the controller blows, you lose access to your data until you get a new controller... whereas if you motherboard and USB and FireWire controller blows, you just plug in another network computer for the Drobo or plug the Lacie into a different computer with USB or Firewire, and you are back in biz.

So. What's the budget? You don't have to say here... however... I like a big-ass-disk attached locally for speed, and a NAS with DOUBLE (minimum) the local attached disk for backup. If you use the Windows or MAC mount points as noted above, you will likely fool your software so that it all runs slickery smooth.

And remember... if you don't have an AUTOMATED process that AUTOMATICALLY makes backups your data for you... you don't have backups.

...rtn
admin
HBRstudios 10 years ago
Plug and play of the Drobo is good, networking to get a total of 2 Drobos a a time is good, USB connection is not so good.

LaCie is wonderful, but don't know if the firewire is worth spending another $700.

Micronet seems to be the best of the lot (and does come with a controller card) as far as price and speed goes, but what about the configuration for a guy who knows nothing about such things?

On the Micronet system, why do you get a PCI-X card with 4 ports if you can't hook another RAID array to it?
admin
duluthiscool 10 years ago
well... i believe that you can hook another box to it... and you would not require an additional raid card... so you might save $50-100 if you can purchase it that way...

even better, would be if the box supported two connections, from two separate controllers (known as duplexing) in order to give you even better redundancy...

if you are leaning towards the micronet... ensure that you have a robust return policy... and just order it. i will volunteer time, along with jason and shawn to get it up and running, if required...
Groups Beta