Hi all Sorry for the delay in writing this, but here is the summary of responses regarding my SAN Implementation email, BIG thanks go to the following people for excellent information and taking the time out to answer my questions: Adam Levin Christopher Saul Debbie Bingham Gregory Shaw Jon Hudson Jeremy Loukinas Charlotte M Ratliff Thank you again to everyone who responded. The general concensus was that Hitachi was the way to go, but we will still take a look at what EMC, LSI/STK and SUN have to offer as other options. We are now in the process of meeting with HITACHI to see what they can offer us, particularly I am interested in there THUNDER 9570V - I am hoping that this is going to provide the performance/cost benefit that has been recommended, it certainly looks more than capable kit.. We will need to upgrade our switches (Brocade 2400 Silkworms 1GB) and core fabrics to 2GB, I do understand that 4GB is out at the end of the year though we will not be persuing that at this stage. I dont think that there is any real need for us. We want to change our backup strategy, allowing us to use BCV's to perform offline backups and also potentially reporting, TimeFinder looks very good for this if we went the EMC route, other vendors offer similar products. Remote replication was important to us, rather than doing this at Veritas Software level, most vendors support this (IPStor by FalconStor was also mentioned which looks great) - so its just about finding a configuration that fits in with what we want, this will also allow us to go HW RAID 10 as opposed to SW RAID 1 and HW RAID 5 which is crippling our write performance and bogging down our bandwidth. It was mentioned that 66mhz HBA's in our SF3800's is going to limit us in terms of hitting the new arrays hard enough, aswell as having only 5 hosts connected to it, though this is going to grow - it will give us alot of breathing space to be adding more systems into the mix. Replies to all questions/queries are included below, thanks to the list, a very valuable resource. ------------------------------------------------------------------------------------------------------------------------------------------------------------- Gregory Shaw wrote: fyi: I work for StorageTek. However, I'm in IT not sales or any other customer facing group. The current products by StorageTek will support what you want without issue. We've got LOTS of that type of connectivity. The concerns regarding multi-host attach to a single array: 1. Make sure that you enable all call-home functionality. All your data is in one place; you sure don't want to have a problem that could have been avoided. 2. Use zoning. The reason for zoning is this: If your switch is one big zone, and a single host has a problem (such as a continuous reboot or bad FC card), everybody will see impact. If you build zones that contain only the server and the array (one zone for each server, with the array existing in multiple zones), the servers will not impact each other. 3. Use LUN masking inside the array. This binds what the servers can see to a particular set of LUNs. That way, you don't have the 'browsing admin' problem of an admin finding 'spare disk' then nuking something important on another server. 4. If you can afford it, use multiple arrays and mirror between them. This is a business continuity issue rather than a real array issue. I'm not saying that anybody's arrays are unreliable, however, the probability of two arrays having a problem is far lower than a single array. It always makes me nervous to have all of my eggs in a single basket. 5. Make sure that you have enough spindles to spread out the i/o. Don't try to do too much with a small set of disk, as you'll find that the spindles will saturate long before the controllers will saturate. The arrays will talk to whatever switch you'd like to use. Of course, YMMV. ------------------------------------------------------------------------------------------------------------------------------------------------------------- Anshul Mathur wrote: Hi Martin, We also implemented SAN in our environment couple of months back.We went for HP (1024) as a centralized storage.We had initail issues with Sun Enterprise servers 3500 and 4500 giving problem in recognizing the disks , we were using JNIC cards for connectivity. AT a different location we used EMC which had absolutely no issues of such kind. This was one issue we faced and wanted to bring to your notice just in case but you can still go ahead and search for more plausable solutions. Regards ------------------------------------------------------------------------------------------------------------------------------------------------------------- Eugene Schmidt wrote: Hi A bit of a curved ball, but well worth looking at: What about NetApp. The fact is that of all NAS type devices, NetApp is the ONLY ONE Meta will recommend. Further, NFS, iSCSI etc are supported on them, As regards replication, this is probably the easiest system to replicated. The side that counts is that the cost is a fraction of what traditional SANs cost, whereas you get all the SAN type functionality. Veritas also have agents for them if required. Also look at SPEC on the benchmarking side. Best of luck (and happy research) Regards Eugene ------------------------------------------------------------------------------------------------------------------------------------------------------------- Christopher Saul wrote: Hi Martin The 6120s we sell are usually inside the 6320 SAN (they call it the 6020 when it's in the 6320 for some reason). The market here tends to be 3510s or 6320s - it's rare to quote for just a 6120. Glad you had a chance to look at IPStor. Zero Impact Backup allows you to back up the volumes that IPStor controls, via IPStor itself, so there's no impact on the 'client' servers - the data's being processed via the IPStor appliance. It's a neater way of doing 'server free' backup - the other way I've configured that had to involve Veritas Volume Manager and clever snapshotting, mirror breaking and various other things. Another advantage is that you only pay for the licence to backup the IPstor server - take the example of 10 servers getting their storage via IPStor - rather than pay for a backup agent for each server, you pay for the agent for the IPstor server only. It's not something I've had a chance to play with yet, but it's a key IPStor feature. Good luck with your investigations, I was interested to see you have some 3800s - you must be the second company I know of to have bought them! Chris ------------------------------------------------------------------------------------------------------------------------------------------------------------- Adam Levin wrote: Our guys are used to EMC, which has significantly more cache than 1GB. According to STK, cache isn't important to their systems because of the way they do things with their asics. However, every disk theory I've known makes cache extremely important for performance. I'm skeptical that 1GB cache is anywhere near good enough, even with asic magic. So, the cache issue is one issue. The other issue was that they were really pushing that FalconStor storage virtualization product, as if the D280 couldn't stand alone, but that we would really need the FalconStor stuff to get the most out of the product. That made us leary of it, especially since the hardware is just being resold by STK. In other words, STK was just reselling stuff that other companies designed and developed. All you're getting with STK as added value is their support, I guess. They're a great tape company, but I'm just not sure they know anything about disks. Sun isn't a storage company either. Their high-end stuff is Hitachi now. I don't know who makes their midrange, but it used to be LSI -- the same folks who make the STK. They sent us a comptetive bulletin between the D280 and the Hitachi 9580, and it was full of mistakes about the 9580. The Hitachi, even in STK's own bulletin, beat the heck out of the D280 on sequential reads and writes (can you say cache? -- 8GB vs. 2GB). The random reads were very close (Hitachi won on cache reads, and was almost the same on disk reads: 186k IOPS cache based reads, and 50k IOPS disk-based reads). On the sequential, the Hitachi had twice as many MB/s as the D280. I'm not a Hitachi dealer or salesman, but they've got some pretty impressive products. As for price, you're probably looking at around $200-$250k for 8TB raw. > We did a few tests on it but its difficult to simulate major load in a > lab, iometer and another for random io - ioslam, didnt stress it > particularly much. They have offered a trial before buy option though > which is pretty good. Honestly, you're dealing with enterprise-level disk stuff, so I think anything you look at will be pretty solid. I don't think you'd go wrong with a D280, but there just don't seem to be too many people out there with them, and STK isn't known as a big disk storage company, at least compared to the big guys like Hitachi and EMC. Personally, I'd stay far away from EMC. > I'll be getting in contact with Hitachi and Sun when I get back, > specifically Hitachi, hearing very good things about them on this list - > the 9580 is the kind of area we will be looking at. That's exactly what we were going to buy, and it was a pretty impressive piece of gear. > If you didnt mind, I'd like to email you when I get back in the office, > if you didnt mind answering some further questions? I'm glad to help -- email away. -Adam ------------------------------------------------------------------------------------------------------------------------------------------------------------- Christopher Saul wrote: Hi Martin Caveat - I work for Sun's Middle East distributor! I would encourage you, however, to check out Sun's offerings. They are not bad and they are well priced - particularly the 3510, which now scales up to 15TB.. On another note, I would take a look at a product called IPStor from Falconstor. It sits between the server and storage layer and lets you perform a tonne of different useful things - easy virutalisation, support for tonnes of platforms, disaster recovery, 'zero impact backup', etc, etc, whilst allowing a lot of flexibility at the storage layer. We've started working with it for customers in our region and it meets a lot of varied requirements. Installation and management is also dead easy. Well worth a peak, if only out of professional interest. ------------------------------------------------------------------------------------------------------------------------------------------------------------- Jeremy Loukinas describing TIMEFINDER (EMC Software): The way ours works is at 1:00 am Oracle goes into hot backup mode. The Primary disks syncs to the BCV. The BCV is then mounted ( via veritas vg import )up on the Netbackup server then written to tape. It's a really sharp process. You can have 3 BCV's per primary data set. So we BCV one copy to the backup server, one to dev and one to test. Then on Sunday the database gets shutdown and a cold copy gets BCV'd and written to tape. The entire process is scripted via scripts EMC provided to us. There are plenty of options out there but I think EMC takes the cake. I hear the Hitachi units are pretty sharp also. There is also the Magnitude which I think is a big piece of crap. It's a little different then a snapshot. We put our db in hotbackup mode m-s @ 1:00 am. Then we put it in cold mode on Sunday. The hot is bcv'd to our netbackup server and written to tape. Then there is a copy written to a dev server via bcv. So in total we have 3 bcv copies.. you can have a max of 4. We use our 3rd for a backup for a second database server. TF actually monitors how far out of sync the blocks are. So it only syncs what has changed.. that's the beauty. It takes about 1.5 hours to do a nightly copy of the database. For a complete sync to disks which are 100% different it takes us about 6 hours for a 400gb database. Let me know if I can explain or detail anything else. J ------------------------------------------------------------------------------------------------------------------------------------------------------------- JV wrote: Martin, read the Brocade "Building SANs with brocade switches" book from cover to cover if you have not already. #1) get any vendor claim in writing and in your SOW. We got screwed royally by IBM. Everything they say verbally over powerpoint presentations is poppycock. #2) don't mix windows and unix in the SAN, if you can help it. #3) have your admins meditate on the supported OS/firmware/driver/HBA matrix like Tibetian monks with their prayer books. good luck JV ------------------------------------------------------------------------------------------------------------------------------------------------------------- Jon Hudson wrote: Welcome to the dark side ;-) You are also going to want to look at HDS. Though I personally would not go through Sun unless you have to, it will often cost more. With only 5 boxes ( I assume 2hbas per box) you are quite unlikely to really push anything you get. The newer mid-range and high end boxex (cx700 and DMX from EMC, 9580 and 9980 from HDS) are all so over engineered it would be quite hard for 5 hosts to really push it. Assuming you set up the luns properly. Your two main limitations will be the 66MHz bus speed on those 3800s and disk speed. Nomatter how fast they are, more than 180 iop/s out of each disk is really rare. More like 120-150 I would say. Though I haven't tested the 15k FC drives. One really important thing. When it comes to price, don't let anyone (especially EMC) quote you only 3yrs for support. This is how they get you. EMC includeds the first 3yrs, but the 4th is a killer. I just went through this last month. I was pitting EMCs cx700 against HDS 9580 for a MES system (which I just set up yesterday in Malaysia of all places). If you just projected out 3yrs, the EMC was about 20k cheaper. If you went out 5yrs, the HDS was 25k cheaper :) so be careful. (side note I ended up getting box, a HDS 9580 for Malayisa and a EMC cx700 for Corp) I would base you descision on a couple things, in this order. 1.) Relationship with vendor. you will be working with these guys for a while. Make sure you like them, they are technically sharp, and that they will support you after the PO is cut. 2.) Uptime & DR. This will determine if you go with midrange or high-end. NOT performance. There is a really good reason why EMC doesn't ever realease numbers compairing the cx line and the sym line. The mid range storage is REALLY fast. Is some cases faster than the high-end. The only reason to buy the high end is uptime and DR. This is becoming less and less true as mid range get more redudant. But if you want 5 9s and you plan to do hardware based replication you want the high end. If you can handled one or two scheduled downtimes a year, and plan to do software replication, the mid-range will be fine. 3.) Support and expansion costs. Most vendors based support costs on the LIST price of the storage, not the price they give you. So if they sell you a 1 million dollar array for 100k, you may get excited, till you see the support bill 2-3yrs later that will be about 100k per year. Also find out what it will cost to add disk, shelves, software and support. They will try to hide it everywhere they can. 4.) Performance. As I said, untill sun moves to 133MHz pci and you add a lot more than 5 hosts, you are going to have trouble really pushing anything hard. To give you an idea. I have about 16 hosts (10 suns, v880s, 4500s, 6800) and 6 windows servers (mssql and exchange) on a 2Gb san using really nice silk 3900s (I love them). This runs all email, finance, and manufacturing (oracle 11i) for a 2500 person international company. I have this all on a last generation EMC Sym 8530. It never even breaks a sweat. It's a 3yr old box, and it's way over powered for what I do. The only thing I have problems with is overloading the 1Gb FA boards on the Sym. After you push a 1Gb link beyond 75-80MB/s performance goes to hell (time to first date, read/write end completion times). But you won't have this problem with 2Gb links. and who even it is, don't let them make you set sd_throttle_max in /etc/system on your solaris boxes. It will throttle swap. The hbas can manage queue-depths just fine, you don't need to set it at the solaris kernel. as a total side note, if you are really concerned about performance afterward and are willing to spend some money let me know. I can direct you to some tools to monitor really low level performance, but they aren't cheap. Feel free to email me directly any questions you run into, I'd be happy to help out. Oh, one last thing. No matter how cheap they get, they will aways go lower. To give a quick example. I pit EMC vs HDS on a first deal, EMC wouldn't go lower than about 120k down from about 350k. They lost the deal to HDS. When deal 2 came, I got EMC down to 69k =) best of luck. ------------------------------------------------------------------------------------------------------------------------------------------------------------- Adam Levin wrote: On Tue, 6 Apr 2004, Martin Thorpe wrote: > This is a little off topic, however my company are investing in a new > SAN and I am looking into the possibilities, so far I have looked at: > LSI (StorageTek), EMC and HP > What I would like to know is any tips for researching a solution from > people that have been through this, and also any pitfalls to avoid. Wow, good for you! There's a lot of neat stuff out there. My company just finished a similar task, and I was the main contact person for the vendors involved. We contacted STK, Hitachi and EMC, just like you. Before I continue, I should say that we needed both SAN and NAS, and we had no existing SAN infrastructure. We ended up going with Network Appliance, because of our mixed needs. However, before we knew what the NetApp could do, we were ready to go with a SAN/Veritas/Sun solution, and we had chosen Hitachi over the others. Specifically, we were looking at the STK D280, the Hitachi Thunder 9580V, and the EMC Clariion C600/DMX800. > accurate. Whats the best way to measure this per application (if the > Application vendor cannot give you a rough idea) to give a rough idea as > to what we need minimum? That's a really difficult question. Sadly, we didn't figure out a good way of testing this either. You can test plenty of activity at the OS level, but when it comes to how the applications actually deal with the disk, it's very difficult to simulate. Of course, the other issue is that it's not easy to get your hands on the hardware to actually confirm what the vendor claims is the performance. As for the hardware, what we found (the short version) is as follows: STK: As you know, their product is not their own. They're re-badging the LSI products. For our NAS needs, they were strongly pushing a "storage virtualization" product from yet another vendor, FalconStor. It concerned us that they needed to add all of this extraneous stuff to make their products competitive. Be prepared for them to push the FalconStor stuff to replace the Veritas. It might be good, it might not -- we had demos, and it was slick, but I don't know how field-tested it is, and it relies on a couple of linux servers as black boxes to do their magic. It also bothered us that the cache was so small compared to the other companies. They claim amazing performance because of dedicated asics in the controllers, but I'm not so sure. We actually saw comparative numbers from them, and not only were they incorrect about their competitors' numbers, but Hitachi stacked up quite favorably to their D280. Incidentally, if you're looking at the EMC DMX line or the Hitachi Lightning 9900 stuff, the STK isn't even in the ballpark. EMC: I don't know if you've ever dealt with EMC. The biggest problem is that they'll lowball the hardware, charge a lot for the software, and *kill*you on the maintenance moving forward. The Clariion stuff is old news -- there haven't been any significant advances in the technology; it's still the same old Data General architecture, just with EMC's software in front of it. In fact, EMC is really more of a software company. We got demos of the software, and it was *really complicated*. It's powerful, but it's a very complex design, and it does require a Windows PC or three to manage everything. The DMX line has better performance, naturally, but the DMX800 is somewhat crippled. To get the true EMC performance, you really need the 1000, and that gets to be a *lot* of money. Hitachi: Ultimately, we felt they had the best price/performance advantages with their Thunder 9580 line. The 9900 is even more impressive. The sales staff was very responsive, and the engineering support really seemed to know what they were talking about. They have some nice software control, but it's still somewhat minimal compared to the other vendors. Using Veritas in addition should give you a lot of power and performance. Hope this helps, -Adam ------------------------------------------------------------------------------------------------------------------------------------------------------------- Charlotte M Ratliff wrote: Hi! I'm the storage architect for Mens Wearhouse, a retail business. We currently have a Hitachi 9980V. We did extensive research before determining this to be the best. We currently run multiple peoplesoft w/oracle svrs, oracle svrs, informix, universe, and exchange servers. I feel we made an excellent choice. I'd recommend checking out gartner.com. I'd also recommend upgrading the brocades to the newer 2G swithches with zoning. I'm out of the office this week but if you'd like to call and ask any questions that would be fine. One last thing DO NOT get EMC I've ran SANs with EMC and they do not provide the level of service you'd expect. Also check what the maintenance cost of later years will be. If you do choose EMC buy it thru Dell, then you get Dells support. Good luck! ------------------------------------------------------------------------------------------------------------------------------------------------------------- Jeremy Loukinas wrote: More info on what I said. We use Timefinder to do Oracle database backups to another server at night. Its all fiber connected using Veritas to import / export the data groups between servers. Works out really well. For a 400gb database it takes about 1.5 hours to complete. EMC has a complete set of tools to monitor performance on the unit. ------------------------------------------------------------------------------------------------------------------------------------------------------------- Jeremy Loukinas wrote: Since you need to support multiple types of hosts and I am sure you possible want to use Timefinder I would suggest an EMC unit. ------------------------------------------------------------------------------------------------------------------------------------------------------------- Hike1272-sunhelp wrote: don't buy emc! if you must buy emc, purchase an emc san from dell. the service is much better the san and parts are cheaper hp has the reputation of being complicated. ---------------------------------------------------------------------------------------------------------------------- ORIGINAL MESSAGE: ---------------------------------------------------------------------------------------------------------------------- --- Martin Thorpe > Sun Managers> > This is a little off topic, however my company are investing in a new> > SAN and I am looking into the possibilities, so far I have looked at:> LSI (StorageTek), EMC and HP> > What I would like to know is any tips for researching a solution from> > people that have been through this, and also any pitfalls to avoid.> > We already have a SAN (I was not involved in the purchase of this), > however we are looking at purchasing a more powerful and scalable> system > that can provide us with the necessary building blocks for future > growth, big bandwidth, 24x7 always-on and flexibility in order to> house > multiple-operating systems as the business grows (any-to-any > connectivity), our data on the current SAN will be migrated to the> new, > we do not want to upgrade, so therefore I have to start from the> ground > up in my research.> > IOPs is a big figure in scaling a solution, however arriving at> somekind > of rough ballpark figure for this seems very difficult without > purchasing software, even with this, its not going to be perfectly > accurate. Whats the best way to measure this per application (if the > Application vendor cannot give you a rough idea) to give a rough idea> as > to what we need minimum?> > The new SAN will need to support 5 hosts in the early stages but this> > will develop as the company gains confidence in it:> > SUNFIRE 3800 Solaris 9 - RDBMS server (PICK).> SUNFIRE 3800 Solaris 9 - RDBMS server (PICK)> SUNFIRE V210 Mail System Server> IBM X345 SQL Cluster> IBM X345 SQL Cluster> > VxVM will be used to control volumes, this will enable us to > deport/import RDBMS volumes on two different servers from two> different > arrays, as true clustering is not supported by the RDBMS.> > We already have 2xBrocade 2400 silkworms, however these are end of > lifed, they are also 1GB per port, most new arrays are 2GB. We also> do > NOT use zoning on these Brocades, it is disabled, I have noticed many> > people talking about using this being critically important?> > Userbase is around 500 for the two SF3800's, 600 for the V210 and> less > than 100 per SQL Cluster, so not a very large userbase.> > Of course, I will summarise the replies back to the list.> > > Thanks> > Martin [demime 1.01b removed an attachment of type image/gif which had a name of wink_n.gif] [demime 1.01b removed an attachment of type image/gif which had a name of smile_n.gif] _______________________________________________ sunmanagers mailing list sunmanagers@sunmanagers.org http://www.sunmanagers.org/mailman/listinfo/sunmanagersReceived on Mon May 10 10:32:44 2004
This archive was generated by hypermail 2.1.8 : Thu Mar 03 2016 - 06:43:33 EST